CI/CD status

[prabhu]

Discussion to track our build status, downtime etc.

self-hosted agents

We have two Mac Minis (64GB RAM each) and one Mini PC (32 GB RAM) powering cdxgen.

Todo

• Move to BDC1 (Upcoming Bude data centre) and invest in a physical lock.
• Publish OBOMs for the compute units.

[prabhu]

Thursday 3 Apr BST 23:00 (UTC+1)

Noticed errors with our container builds due to installation problems. Identified issues with the pip install blint command. Created and released blint 2.4.1 to fix. Re-ran the container builds to make the master branch green. Total DevOps time spent: 4 hours. This is, however, non-chargeable since I am also responsible for the bug originating from the blint 2.4.0 release.

[prabhu]

Tuesday 8 Apr BST 11:00 (UTC+1)

Noticed errors with our container builds due to installation problems. Identified issues with the pip install blint command. Have created this upstream ticket to investigate the problem.

[prabhu]

pip install has started working by simply re-running the workflows a few times. If I come across any RCA from the PyPI team, I will link it here.

[prabhu]

Thursday 10 Apr BST 10:20 (UTC+1)

Noticed timeouts in the upload base images workflow with the macos-hosted agent. Suspected the issue to be disk space, so re-ran the workflow after doing docker system prune -a.

[prabhu]

docker system prune did the trick. A couple of failures with sle .NET builds, which is like the one we have seen before.

#5 29.73 To install missing framework, download:
#5 29.73 https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.15&arch=x64&rid=linux-x64&os=sles.15.6
#5 ERROR: process "/bin/sh -c zypper refresh && zypper --non-interactive update && zypper --non-interactive install -l --no-recommends git-core nodejs20 npm20 python311 python311-pip wget zip unzip make gawk java-21-openjdk-devel     && dotnet --list-sdks     && dotnet workload install android wasm-tools wasm-tools-net6 wasm-tools-net7     && dotnet workload list     && npm install -g corepack     && zypper clean -a" did not complete successfully: exit code: 150

[prabhu]

My attempts to build sle dotnet images without performing zypper refresh and update did not work.

148.8 Running post-transaction scripts [....done]
150.4 8.0.407 [/usr/share/dotnet/sdk]
151.7
156.1 Failed to update the advertising manifest microsoft.net.workload.emscripten.net6: Value cannot be null or an empty string. (Parameter 'value').
156.1 Failed to update the advertising manifest microsoft.net.sdk.maccatalyst: Value cannot be null or an empty string. (Parameter 'value').
156.1 Advertising manifest not updated. Manifest package for microsoft.net.sdk.maccatalyst doesn't exist.
156.1 Advertising manifest not updated. Manifest package for microsoft.net.workload.emscripten.net6 doesn't exist.
156.2 Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.current: Value cannot be null or an empty string. (Parameter 'value').
156.2 Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.current doesn't exist.
156.2 Failed to update the advertising manifest microsoft.net.sdk.tvos: Value cannot be null or an empty string. (Parameter 'value').
156.2 Advertising manifest not updated. Manifest package for microsoft.net.sdk.tvos doesn't exist.
156.3 Failed to update the advertising manifest microsoft.net.workload.emscripten.net7: Value cannot be null or an empty string. (Parameter 'value').
156.3 Advertising manifest not updated. Manifest package for microsoft.net.workload.emscripten.net7 doesn't exist.
156.3 Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net7: Value cannot be null or an empty string. (Parameter 'value').
156.3 Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.net7 doesn't exist.
156.4 Failed to update the advertising manifest microsoft.net.sdk.maui: Value cannot be null or an empty string. (Parameter 'value').
156.4 Advertising manifest not updated. Manifest package for microsoft.net.sdk.maui doesn't exist.
156.5 Failed to update the advertising manifest microsoft.net.workload.emscripten.current: Value cannot be null or an empty string. (Parameter 'value').
156.5 Advertising manifest not updated. Manifest package for microsoft.net.workload.emscripten.current doesn't exist.
156.5 Failed to update the advertising manifest microsoft.net.sdk.android: Value cannot be null or an empty string. (Parameter 'value').
156.5 Advertising manifest not updated. Manifest package for microsoft.net.sdk.android doesn't exist.
156.6 Failed to update the advertising manifest microsoft.net.sdk.macos: Value cannot be null or an empty string. (Parameter 'value').
156.6 Advertising manifest not updated. Manifest package for microsoft.net.sdk.macos doesn't exist.
156.6 Failed to update the advertising manifest microsoft.net.sdk.aspire: Value cannot be null or an empty string. (Parameter 'value').
156.6 Advertising manifest not updated. Manifest package for microsoft.net.sdk.aspire doesn't exist.
159.3 Failed to update the advertising manifest microsoft.net.sdk.ios: Unable to get repository signature information for source https://api.nuget.org/v3-index/repository-signatures/5.0.0/index.json..
159.3 Advertising manifest not updated. Manifest package for microsoft.net.sdk.ios doesn't exist.
159.3 Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net6: Unable to get repository signature information for source https://api.nuget.org/v3-index/repository-signatures/5.0.0/index.json..
159.3 Advertising manifest not updated. Manifest package for microsoft.net.workload.mono.toolchain.net6 doesn't exist.
159.3 Installing pack Microsoft.Android.Sdk.Linux version 34.0.43...
159.5 Workload installation failed. Rolling back installed packs...
159.5 Rolling back pack Microsoft.Android.Sdk.Linux installation...
159.5 Workload installation failed: Value cannot be null or an empty string. (Parameter 'value')

SLE issues usually resolve themselves in a day or two. Let's check again tomorrow without wasting any more DevOps time.

[prabhu]

SLE dotnet 8 and 9 images are still broken. The error message lists two conflicting versions, 8.0.14 and 8.0.15, indicating upstream issues. Raised a query here.

#5 35.12 8.0.408 [/usr/share/dotnet/sdk]
#5 35.12 You must install or update .NET to run this application.
#5 35.12 
#5 35.12 App: /usr/share/dotnet/sdk/8.0.408/dotnet.dll
#5 35.12 Architecture: x64
#5 35.12 Framework: 'Microsoft.NETCore.App', version '8.0.15' (x64)
#5 35.12 .NET location: /usr/share/dotnet/
#5 35.12 
#5 35.12 The following frameworks were found:
#5 35.12   8.0.14 at [/usr/share/dotnet/shared/Microsoft.NETCore.App]
#5 35.12 
#5 35.12 Learn more:
#5 35.12 https://aka.ms/dotnet/app-launch-failed
#5 35.12 
#5 35.12 To install missing framework, download:
#5 35.12 https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=8.0.15&arch=x64&rid=linux-x64&os=sles.15.6
#5 ERROR: process "/bin/sh -c zypper refresh && zypper --non-interactive update && zypper --non-interactive install -l --no-recommends git-core nodejs20 npm20 python311 python311-pip wget zip unzip make gawk java-21-openjdk-devel     && dotnet --list-sdks     && dotnet workload install android wasm-tools wasm-tools-net6 wasm-tools-net7     && dotnet workload list     && npm install -g corepack     && zypper clean -a" did not complete successfully: exit code: 150

[prabhu]

Have removed the dotnet workload install commands with this commit, since it's been days. Recommendation to deprecate the SLE dotnet images in favour of the Debian ones, which have been stable as well as support multiple architectures.

[prabhu]

To vent my frustration with the reliability of SLE Docker builds, below is an error from failed Python 3.11 builds. I have to now wait a while for all other builds to finish before restarting the failed ones.

16.22 Retrieving: bzip2-1.0.8-150400.1.122.aarch64 (SLE_BCI) (1/46),  45.8 KiB    
16.27 Retrieving: bzip2-1.0.8-150400.1.122.aarch64.rpm [....error]
17.50 Download (curl) error for 'https://public-dl.suse.com/SUSE/Products/SLE-BCI/15-SP6/aarch64/product/aarch64/bzip2-1.0.8-150400.1.122.aarch64.rpm':
17.50 Error code: Curl error 35
17.50 Error message: OpenSSL/3.1.4: error:0A000438:SSL routines::tlsv1 alert internal error
17.50 
17.50 Abort, retry, ignore? [a/r/i/...? shows all options] (a): a
17.52 Problem occurred during or after installation or removal of packages:
17.52 Installation has been aborted as directed.
17.52 Please see the above error message for a hint.
------
Dockerfile.python311:21

[prabhu]

Wednesday 16 Apr BST 11:00 (UTC+1)

Power is down, so the self-hosted agents are unavailable.

(Power must be back in six hours. The good news is that we have an EcoFlow 1KWh battery backup. The bad news is that I am travelling in the opposite direction, so I couldn't physically deliver the battery to the co-location centre. Note to self: we need to distribute compute units across locations.)

[prabhu]

Power is back at 4 p.m. Have to revisit my plans to scale entirely with in-house computes.

[prabhu]

Thursday 17 Apr BST 17:00 (UTC+1)

Noticed that the "Upload base images" had errors. Apparently, this is a known issue with colima which requires a workaround after every macOS update.

brew uninstall colima qemu lima
rm -rf ~/.colima
brew install colima
brew services restart colima

[prabhu]

For reminder, we cannot use Docker Desktop for Mac since it requires access to the keychain and doesn't really work well in a headless environment. Things like Rancher Desktop and nerdctl require rework in the workflow to replace Docker-based steps with custom scripts.

[prabhu]

Could have tried this simpler workaround too.

rm -rf ~/.colima/_lima/_networks/user-v2

[malice00]

FYI: keychain can be interacted with from CLI. Not sure what you need, but I use it in CI all the time!

…

On Thu, Apr 17, 2025, 18:41 prabhu ***@***.***> wrote: For reminder, we cannot use Docker Desktop for Mac since it requires access to the keychain and doesn't really work well in a headless environment. Things like Rancher Desktop and nerdctl require rework in the workflow to replace Docker-based steps with custom scripts. — Reply to this email directly, view it on GitHub <#1720 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADC7EIPPXZPRAUMWKWXEBMT2Z7KUZAVCNFSM6AAAAAB2NKT6X2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBWHE4DQOA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[prabhu]

Interesting. I don't have the exact error with me, but it only worked when run from a screensharing session, since macOS wanted to show a password prompt. Upon restart, nothing docker related will work unless it was unlocked. In contrast, colima works in clear text password mode, but does have other issues.

[malice00]

I believe the correct command would be `keychain unlock`. It needs credentials, but these can be given on the cli.

…

On Thu, Apr 17, 2025, 18:59 prabhu ***@***.***> wrote: Interesting. I don't have the exact error with me, but it only worked when run from a screensharing session, since macOS wanted to show a password prompt. Upon restart, nothing docker related will work unless it was unlocked. In contrast, colima works in clear text password mode, but does have other issues. — Reply to this email directly, view it on GitHub <#1720 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADC7EIJKXQIQR45ZMJ3ELWL2Z7MV3AVCNFSM6AAAAAB2NKT6X2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEOBXGAYDKMI> . You are receiving this because you commented.Message ID: ***@***.***>

[prabhu]

Will keep in mind. So far we are ok with colima.