Skip to content

CDNs may still serve up the old version of a replaced crate #10887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LawnGnome opened this issue Mar 24, 2025 · 1 comment · Fixed by #10888
Closed

CDNs may still serve up the old version of a replaced crate #10887

LawnGnome opened this issue Mar 24, 2025 · 1 comment · Fixed by #10888
Assignees
Labels
A-backend ⚙️ C-bug 🐞 Category: unintended, undesired behavior

Comments

@LawnGnome
Copy link
Contributor

It looks like we have at least one case where a crate has been published, deleted, and then republished where at least one CDN node has held onto the old version of the crate file and is still serving it up. (The rough timeline in this specific case is: published on the 13th, deleted on the 17th, republished on the 19th.)

The index file was correctly invalidated, and is correct across the CDNs, as far as I can tell.

I thought we were invalidating on delete, but I'm not so sure after 30 seconds with rg, so I'm going to dig into this further. If we're not, I'll add a job to ensure that we do so.


(For those with Zendesk access, this is derived from #2140.)

@LawnGnome
Copy link
Contributor Author

This was deployed on Friday.

I've just finished purging the crate files and rendered READMEs for previously deleted versions that are still potentially within the TTL (one week for READMEs; one year for crate files) on Fastly. Looks like it went fine.

For now, I'm choosing not to invalidate the same paths on Cloudfront — we'd be charged to do so, since there would be more than 1000 paths to invalidate, and given the low percentage of traffic that goes through Cloudfront, the relatively low probability that any deleted crate files remain in a Cloudfront node, and there not being any non-public data there, I'm comfortable relying on cargo's checksumming to catch any (extremely unlikely) user-facing issues. If we do get reports, though, I'll go ahead and invalidate the paths there as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-backend ⚙️ C-bug 🐞 Category: unintended, undesired behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant