Quantcast

Mono repo breaks on some azure VMs but not others - CDN issue?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Mono repo breaks on some azure VMs but not others - CDN issue?

Aleksey Kliger via Mono-devel-list

Hi everyone,

 

First, thank you for all your hard work on the Mono project. We use F# as our primary development language. Mono helps us rapidly develop for Linux as well as Windows while keeping the benefits of using F#.

 

Unfortunately we are seeing a problem with the CentOS repository at download.mono-project.com that we believe is related to a CDN issue. Please let me know if this is not the right place to ask about this, I would appreciate any pointers. The issue is breaking Linux deployments of our Azure hosted service by causing yum install failures of Mono on our Linux VMs. We would be grateful for a fix or workaround and we are happy to help with further troubleshooting.

 

Details:

 

We deploy Mono on Red Hat Enterprise Linux 7.2 VMs hosted on Microsoft Azure. We see yum fails to install package sqlite from the CentOS repo on download.mono-project.com from some VMs but not other VMs hosted in Azure. The issue is breaking our Linux deployments of our service because we have set yum to install Mono on startup for our Linux VMs.

 

We see that yum is attempting to download this URL

http://download.mono-project.com/repo/centos/repodata/d839a3db898c8bd7e9a08097357b3568ffc7b128e846b3943d1d61a94ecc2bc6-primary.sqlite.bz2

 

The URL is specified in http://download.mono-project.com/repo/centos/repodata/repomd.xml , which we download directly from the web site, has a last modified date of 12 April 2017. The file we have shows revision 1491476506 . Is this the most recent version of repomd.xml?

 

The URL gives a 404 on some of our Azure-hosted VMs and not others. We can reliably reproduce the 404 error using wget on failing machines 100% of the time.

 

When we examine http://download.mono-project.com/repo/centos/repodata/  from a web browser we see that the sqlite package is at this URL, updated 12 April 2017:

http://download.mono-project.com/repo/centos/repodata/4e94d255551bd1fa2ce69f946cf41519aed3b645206aa23e1aa2d06f349763ae-primary.sqlite.bz2

There is no “d839…” URL in the repodata directory.

 

On all our machines that successfully download the “d839…” URL, we observe from wget a reference to a CDN cache hit and a “last modified” date of Thursday 6 April 2017. Output from wget appended to this message.

 

Because of the CDN reference and the CDN hitting only on machines that succeed in downloading the “d839…” URL, we suspect that there is a CDN which is caching an old version of the primary.sqlite.bz2 package. Because the “d839…” URL shows upin repomd.xml, we also suspect that either repomd.xml is being cached with the previous URL or that repomd.xml has not been updated properly with a new URL for a new version of the package.

 

Does this error report make sense? Would it be possible to please check repomd.xml and the propagation of the repo data to relevant CDNs? If this issue is hitting us, I expect it may also break others using CentOS / RHEL as well.

 

Again, if this is not the best place to report, I would appreciate any pointers on where to follow up. Thanks again for all your hard work on Mono!

 

David Molnar

 

 

WGET output showing CDN hit:

 

wget http://download.mono-project.com/repo/centos/repodata/d839a3db898c8bd7e9a08097357b3568ffc7b128e846

b3943d1d61a94ecc2bc6-primary.sqlite.bz2

StatusCode        : 200

StatusDescription : OK

Content           : {66, 90, 104, 57...}

RawContent        : HTTP/1.1 200 OK

                    X-Cache: HIT

                    Accept-Ranges: bytes

                    Content-Length: 555139

                    Content-Type: application/x-bzip2

                    Date: Mon, 17 Apr 2017 06:11:49 GMT

                    ETag: "87883-54c7d707d2df2"

                    Last-Modified: Thu, 06...

Headers           : {[X-Cache, HIT], [Accept-Ranges, bytes], [Content-Length, 555139], [Content-Type,

                    application/x-bzip2]...}

RawContentLength  : 555139

 

 

 

 

 


_______________________________________________
Mono-devel-list mailing list
[hidden email]
http://lists.dot.net/mailman/listinfo/mono-devel-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mono repo breaks on some azure VMs but not others - CDN issue?

Aleksey Kliger via Mono-devel-list

Hi all,

 

I apologize for the self-follow-up, but we have found that http://download.mono-project.com/repo/centos/repodata/repomd.xml is also hitting the CDN cache with a last modified date of 6 April 2017.

 

This 6 April 2017 is earlier than the date of 12 April 2017 which is listed on http://download.mono-project.com/repo/centos/repodata/ .

 

We hit this apparently stale repomd.xml on all our VMs. The apparently stale repomd.xml contains a link to the “d839…” URL that is breaking on some VMs and not others.

 

Is there a way for us to bypass the CDN and access an updated repodata.xml? nslookup reports the IP address as 72.21.81.200, but http://72.21.81.200/repo/centos/repodata/ gives a 404 error.

 

If repodata.xml is in fact out of date, would it be possible to push an update to repodata.xml to the CDN?

 

Thank you,

David Molnar

 

 

Wget –d output:

GET /repo/centos/repodata/repomd.xml HTTP/1.1                                                                                                 

 

User-Agent: Wget/1.17.1 (linux-gnu)                                                                                                           

 

Accept: */*                                                                                                                                    

 

Accept-Encoding: identity                                                                                                                     

 

Host: download.mono-project.com                                                                                                               

 

Connection: Keep-Alive                                                                                                                         

 

                                                                                                                                              

 

---request end---                                                                                                                              

 

HTTP request sent, awaiting response...                                                                                                       

 

---response begin---                                                                                                                           

 

HTTP/1.1 200 OK                                                                                                                               

 

Accept-Ranges: bytes                                                                                                                          

 

Content-Type: application/xml                                                                                                                  

 

Date: Mon, 17 Apr 2017 18:42:49 GMT                                                                                                           

 

Etag: "bb3-54c7d707d5cd8"                                                                                                                      

 

Last-Modified: Thu, 06 Apr 2017 11:02:05 GMT                                                                                                  

 

Server: ECAcc (sjc/16E4)                                                                                                                       

 

Vary: Accept-Encoding                                                                                                                         

 

X-Cache: HIT                                                                                                                                   

 

Content-Length: 2995                                                                                                                          

 

                                                                                                                                               

 

---response end---                                                                                                                            

 

200 OK                                                                                                                                         

 

Registered socket 3 for persistent reuse.                                                                                                     

 

Length: 2995 (2.9K) [application/xml]                                                                                                          

 

Saving to: 'repomd.xml.1'

 

From: David Molnar
Sent: Monday, April 17, 2017 11:31 AM
To: '[hidden email]' <[hidden email]>
Cc: William Blum <[hidden email]>; Marina Polishchuk <[hidden email]>; Stas Tishkin <[hidden email]>
Subject: Mono repo breaks on some azure VMs but not others - CDN issue?

 

Hi everyone,

 

First, thank you for all your hard work on the Mono project. We use F# as our primary development language. Mono helps us rapidly develop for Linux as well as Windows while keeping the benefits of using F#.

 

Unfortunately we are seeing a problem with the CentOS repository at download.mono-project.com that we believe is related to a CDN issue. Please let me know if this is not the right place to ask about this, I would appreciate any pointers. The issue is breaking Linux deployments of our Azure hosted service by causing yum install failures of Mono on our Linux VMs. We would be grateful for a fix or workaround and we are happy to help with further troubleshooting.

 

Details:

 

We deploy Mono on Red Hat Enterprise Linux 7.2 VMs hosted on Microsoft Azure. We see yum fails to install package sqlite from the CentOS repo on download.mono-project.com from some VMs but not other VMs hosted in Azure. The issue is breaking our Linux deployments of our service because we have set yum to install Mono on startup for our Linux VMs.

 

We see that yum is attempting to download this URL

http://download.mono-project.com/repo/centos/repodata/d839a3db898c8bd7e9a08097357b3568ffc7b128e846b3943d1d61a94ecc2bc6-primary.sqlite.bz2

 

The URL is specified in http://download.mono-project.com/repo/centos/repodata/repomd.xml , which we download directly from the web site, has a last modified date of 12 April 2017. The file we have shows revision 1491476506 . Is this the most recent version of repomd.xml?

 

The URL gives a 404 on some of our Azure-hosted VMs and not others. We can reliably reproduce the 404 error using wget on failing machines 100% of the time.

 

When we examine http://download.mono-project.com/repo/centos/repodata/  from a web browser we see that the sqlite package is at this URL, updated 12 April 2017:

http://download.mono-project.com/repo/centos/repodata/4e94d255551bd1fa2ce69f946cf41519aed3b645206aa23e1aa2d06f349763ae-primary.sqlite.bz2

There is no “d839…” URL in the repodata directory.

 

On all our machines that successfully download the “d839…” URL, we observe from wget a reference to a CDN cache hit and a “last modified” date of Thursday 6 April 2017. Output from wget appended to this message.

 

Because of the CDN reference and the CDN hitting only on machines that succeed in downloading the “d839…” URL, we suspect that there is a CDN which is caching an old version of the primary.sqlite.bz2 package. Because the “d839…” URL shows upin repomd.xml, we also suspect that either repomd.xml is being cached with the previous URL or that repomd.xml has not been updated properly with a new URL for a new version of the package.

 

Does this error report make sense? Would it be possible to please check repomd.xml and the propagation of the repo data to relevant CDNs? If this issue is hitting us, I expect it may also break others using CentOS / RHEL as well.

 

Again, if this is not the best place to report, I would appreciate any pointers on where to follow up. Thanks again for all your hard work on Mono!

 

David Molnar

 

 

WGET output showing CDN hit:

 

wget http://download.mono-project.com/repo/centos/repodata/d839a3db898c8bd7e9a08097357b3568ffc7b128e846

b3943d1d61a94ecc2bc6-primary.sqlite.bz2

StatusCode        : 200

StatusDescription : OK

Content           : {66, 90, 104, 57...}

RawContent        : HTTP/1.1 200 OK

                    X-Cache: HIT

                    Accept-Ranges: bytes

                    Content-Length: 555139

                    Content-Type: application/x-bzip2

                    Date: Mon, 17 Apr 2017 06:11:49 GMT

                    ETag: "87883-54c7d707d2df2"

                    Last-Modified: Thu, 06...

Headers           : {[X-Cache, HIT], [Accept-Ranges, bytes], [Content-Length, 555139], [Content-Type,

                    application/x-bzip2]...}

RawContentLength  : 555139

 

 

 

 

 


_______________________________________________
Mono-devel-list mailing list
[hidden email]
http://lists.dot.net/mailman/listinfo/mono-devel-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mono repo breaks on some azure VMs but not others - CDN issue? (Mono-devel-list Digest, Vol 144, Issue 12)

Jo Shields


On 17/04/17 20:06, David Molnar wrote:

> Hi all,
>
> I apologize for the self-follow-up, but we have found that http://download.mono-project.com/repo/centos/repodata/repomd.xml is also hitting the CDN cache with a last modified date of 6 April 2017.
>
> This 6 April 2017 is earlier than the date of 12 April 2017 which is listed on http://download.mono-project.com/repo/centos/repodata/ .
>
> We hit this apparently stale repomd.xml on all our VMs. The apparently stale repomd.xml contains a link to the "d839..." URL that is breaking on some VMs and not others.
>
> Is there a way for us to bypass the CDN and access an updated repodata.xml? nslookup reports the IP address as 72.21.81.200, but http://72.21.81.200/repo/centos/repodata/ gives a 404 error.
>
> If repodata.xml is in fact out of date, would it be possible to push an update to repodata.xml to the CDN?
>

I've forced cache invalidation
_______________________________________________
Mono-devel-list mailing list
[hidden email]
http://lists.dot.net/mailman/listinfo/mono-devel-list
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Mono repo breaks on some azure VMs but not others - CDN issue? (Mono-devel-list Digest, Vol 144, Issue 12)

David Molnar


On Mon, Apr 17, 2017 at 12:24 PM, Jo Shields <[hidden email]> wrote:

> If repodata.xml is in fact out of date, would it be possible to push an update to repodata.xml to the CDN?
>

I've forced cache invalidation

Thank you, Jo!  I confirm that this fixes the issue for us. Our Red Hat Enterprise Linux VMs are now happily installing mono. Deeply appreciate the quick response.

David Molnar  


_______________________________________________
Mono-devel-list mailing list
[hidden email]
http://lists.dot.net/mailman/listinfo/mono-devel-list
Loading...