Skip to content

ansible: promote release assets from staging r2 bucket #3620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 3, 2024
Merged

ansible: promote release assets from staging r2 bucket #3620

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions ansible/www-standalone/tools/promote/_resha.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ site=$1
dstdir=$2
version=$3

if [ "X${site}" == "X" ]; then
echo "site argument not provided"
exit 1
fi

if [ "X${dstdir}" == "X" ]; then
echo "dstdir argument not provided"
exit 1
Expand All @@ -16,6 +21,26 @@ if [ "X${version}" == "X" ]; then
exit 1
fi

if [ -z ${dist_rootdir+x} ]; then
echo "\$dist_rootdir is not set"
exit 1
fi

if [ -z ${staging_bucket+x} ]; then
echo "\$staging_bucket is not set"
exit 1
fi

if [ -z ${cloudflare_endpoint+x} ]; then
echo "\$cloudflare_endpoint is not set"
exit 1
fi

if [ -z ${cloudflare_profile+x} ]; then
echo "\$cloudflare_profile is not set"
exit 1
fi

(cd "${dstdir}/${version}" && shasum -a256 $(ls node* openssl* iojs* win-*/* x64/* 2> /dev/null) > SHASUMS256.txt) || exit 1
if [[ $version =~ ^v[0] ]]; then
(cd "${dstdir}/${version}" && shasum $(ls node* openssl* x64/* 2> /dev/null) > SHASUMS.txt) || exit 1
Expand All @@ -25,3 +50,7 @@ nodejs-dist-indexer --dist $dstdir --indexjson ${dstdir}/index.json --indextab
find "${dstdir}/${version}" -type f -exec chmod 644 '{}' \;
find "${dstdir}/${version}" -type d -exec chmod 755 '{}' \;

relativedir=${dstdir/$dist_rootdir/"$site/"}
aws s3 cp ${dstdir}/index.json $staging_bucket/$relativedir/index.json --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile
aws s3 cp ${dstdir}/index.tab $staging_bucket/$relativedir/index.tab --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile
Comment on lines +53 to +55
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we also copy the two shasum files?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shasum files are already put into the dist-prod bucket by the release script, https://github.com/nodejs/node/blob/a923fed8743cf787add2a22def0fdbb88d736a42/tools/release.sh#L20C1-L20C18

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_resha.sh is not only called by releasers. Automation for nightly, test, rc, builds also calls it.

Copy link
Member Author

@flakey5 flakey5 May 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, should I remove the upload of the .asc and .sig files from the release script then?

Copy link
Member

@targos targos May 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe? I'm not entirely sure about the order in which things happen during a release.
I noticed yesterday when I released v22.2.0 that SHASUMS256.txt wasn't uploaded to R2 (only the .asc and .sig files were).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it so that it will copy the SHASUM256.txt into the staging bucket now. The later call that syncs the version in the staging bucket -> dist bucket recursively copies the entire directory, so no change should be needed there.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI doesn't upload Shasums. For releases it happens when the release script is executed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHASUM256.txt is generated on the www machine by the promotion script (specifically _resha.sh, which also regenerates index.json and index.tab). For releases promotion is manually triggered by the releaser. For nightly builds there's a 30 min cron job that runs promote_nightly.sh:

build/ansible/www-standalone/tasks/site-setup.yaml

Line 63 in 3d2cdcc

- "*/30 * * * * dist /home/staging/tools/promote/promote_nightly.sh nodejs"

The signed artifacts (SHASUM256.txt.asc and SHASUM256.txt.sig) are only generated for releases as part of the release script. These are generated on the releaser's machine and uploaded as part of that script.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flakey5 I assume you have updated the code based on the comments above?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding yes

aws s3 cp ${dstdir}/${version}/SHASUMS256.txt $staging_bucket/$relativedir/${version}/SHASUM256.txt --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile
3 changes: 2 additions & 1 deletion ansible/www-standalone/tools/promote/settings
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,4 +39,5 @@ chakracore_release_dirmatch=.*

cloudflare_endpoint=https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com
cloudflare_profile=worker
destination_bucket=s3://dist-prod
staging_bucket=s3://dist-staging
dist_bucket=s3://dist-prod
14 changes: 9 additions & 5 deletions ansible/www-standalone/tools/promote/upload_to_cloudflare.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,12 @@ if [ -z ${dist_rootdir+x} ]; then
echo "\$dist_rootdir is not set"
exit 1
fi
if [ -z ${destination_bucket+x} ]; then
echo "\$destination_bucket is not set"
if [ -z ${staging_bucket+x} ]; then
echo "\$staging_bucket is not set"
exit 1
fi
if [ -z ${dist_bucket+x} ]; then
echo "\$dist_bucket is not set"
exit 1
fi
if [ -z ${cloudflare_endpoint+x} ]; then
Expand All @@ -38,6 +42,6 @@ fi
relativedir=${dstdir/$dist_rootdir/"$site/"}
tmpversion=$2

aws s3 cp $dstdir/$tmpversion/ $destination_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --recursive --no-follow-symlinks
aws s3 cp $dstdir/index.json $destination_bucket/$relativedir/index.json --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile
aws s3 cp $dstdir/index.tab $destination_bucket/$relativedir/index.tab --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile
aws s3 cp $staging_bucket/$relativedir/$tmpversion/ $dist_bucket/$relativedir/$tmpversion/ --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile --recursive --no-follow-symlinks
aws s3 cp $staging_bucket/$relativedir/index.json $dist_bucket/$relativedir/index.json --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile
aws s3 cp $staging_bucket/$relativedir/index.tab $dist_bucket/$relativedir/index.tab --endpoint-url=$cloudflare_endpoint --profile $cloudflare_profile