Nightly version mismatch

[SiegeLord]

Right now, the Windows nightly version is rustc 0.11.0-pre-nightly (6b1426b 2014-05-25 01:11:19 -0700) while the Linux version is rustc 0.11.0-pre-nightly (33c3edd 2014-05-23 00:31:27 -700). The Windows binary nightlies cannot compile the source nightlies, so there's another version mismatch there (there's no obvious way to tell what version the source nightlies are).

This should not happen, and all nightlies should be in sync. It becomes very difficult to port between platforms when the compiler versions are all different.

[huonw]

This is because the linux nightlies are failing (but not the windows ones).

[alexcrichton]

This was just a hiccup in our automation. Adding more automation to ensure that everything is in sync 100% of the time is probably overkill for now, so I'm going to close this.

[sfackler]

Reopening. This keeps cropping up and we should really just buckle down and fix it. https://botbot.me/mozilla/rust-internals/2014-07-09/?msg=17718208&page=8

[brson]

@alexcrichton Think I might tackle this problem before the full release channel automation. Plan:

• add dist-staging/nightly, dist-staging/beta, dist-staging/stable s3 folders
• build master uploads artifacts to staging folders
• after uploading the artifacts, for each artifact we also upload a ${name}.commit-id file, which contains the commit sha
• add doc-staging/nightly, doc-staging/beta, doc-staging/stable s3 folders
• build master uploads docs to staging folders
• after uploading the docs we upload doc-staging/${branch}/commit-id, which contains the commit sha
• every hour a cron job on the buildmaster runs rust-buildbot/dist-promote.py, which checks for a list of artifacts, and that all the commit id's agree, then moves them to the right place.

[brson]

Actually, after writing all that out, it's a lot of work. May not be right to do it first.

[brson]

cc #17398

[brson]

There's a race condition in this process between the time the script checks for the commit ids and moves everything. Without downloading the entire set there may not be a solution.

[alexcrichton]

I think it's also a little more racy in terms of uploading docs because they're entire directory trees as opposed to one file. Here's how I'd tweak it:

• In each commit-id file, store both the commit SHA and the checksum of the tarball. That way we can download all the commit-id files, make sure the commits all match, and then when we download the tarballs we're sure we're downloading the actual tarball
• In each tarball, include the docs. We should do this for distributions anyway and this way we can ensure that we're always uploading docs for the exact same version of the compiler that's being distributed.
• As part of the dist-promote.py script, unpack the tarball and sync docs to the relevant doc folder on S3.

[alexcrichton]

That would involve not having staging areas for docs on S3 as well, you just couldn't view staging docs.

[brson]

Servo wants this for cargo as well.