feature - Build alternate ubuntu images #7691

raymyers · 2025-04-03T17:14:53Z

This change is worth documenting at https://docs.all-hands.dev/
Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

End-user friendly description of the problem this fixes or functionality that this introduces.

Add a new docker image option based on Ubuntu

Give a summary of what the PR does, explaining any non-trivial design decisions.

We are trying to reduce the vulnerabilities found in image scans using SCA tools like trivy. Unfortunately, the current Debian Stable (Bookworm) which our nikolaik bases image use has vulnerabilities with no patches available showing up.

This PR introduces Ubuntu 24.04 to address this but retains the existing nikolaik build as the default.

For reference, here is the nikolaik dockerfile template:
https://github.com/nikolaik/docker-python-nodejs/blob/main/templates/debian.Dockerfile

Other changes

Removing the apt-get upgrade (which is done by upstream images regularly)

Adding another cache dir to to the ci config.

Removing several extensions dirs from the openvscode-server install because they show up as false positives (extension version is confused for npm package of the same name. These are handlebars, pug, json, diff, grunt, ini, npm.

Docker template logic

Since the Ubuntu base image does not have python and node installed by default, there are commands added to install them. Right now these activate when base_image contains "ubuntu", but perhaps people might try to configure a custom image with that in the name. Should we introduce another flag to toggle this? I'm not sure what the interface should be, maybe this is ok for now.

Link of any specific issues this addresses.

To run this PR locally, use the following command:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:6a55b88-nikolaik   --name openhands-app-6a55b88   docker.all-hands.dev/all-hands-ai/openhands:6a55b88

xingyaoww

Overall LGTM

xingyaoww · 2025-04-03T23:05:06Z

.github/workflows/ghcr-build.yml

@@ -208,7 +214,7 @@ jobs:
    strategy:
      fail-fast: false
      matrix:
-        base_image: ['nikolaik']
+        base_image: ['nikolaik', 'ubuntu']


Should we just remove nikolaik - or do you think we can keep both and using it for a while and if everything goes well in ubuntu, we do the switch then?

Good question. So the concern with changing over completely is that it might have unforeseen impact, either on the runtime image behavior itself or for people using the build process with other custom base images.

Personally long term I could see making ubuntu the default and phasing the other out as you suggest, this PR is just avoiding committing to that.

Would python:3.12 work? It's also debian so it's presumably more compatible with the nikolaik one.

@li-boxuan Actually, it wouldn't work exactly because it's Debian-based, since the goal is to fix the security findings. If that were possible, we could just stay with nikolaik base and run upgrades as was tried first.

To show what I mean, for instance looking at a run of trivy image --severity HIGH,CRITICAL --scanners vuln python:3.12

python:3.12 (debian 12.10) ========================== Total: 101 (HIGH: 96, CRITICAL: 5)

Zooming in on one of these criticals, the Debian page for CVE-2023-52425 shows that in the stable distro (Bookworm) there is no upgrade available. There happens to be a patch in the older distro Bullseye, but that's not true in general, for instance the CVE's associated with the package linux-libc-dev.

Ah I see, thanks

BTW its worth noting that Debian Trixie will freeze in a few months and move from the current Testing to Stable, so possibly there will be a Debian with newer packages at that point. If we're lucky we'll be able to consider the option of just using the upgraded Debian to meet the need instead of deciding if we want to make Ubuntu the default.

https://wiki.debian.org/DebianTrixie

raymyers · 2025-04-03T23:50:55Z

I'll have to troubleshoot the runtime test failures, looks like

tofarr · 2025-04-04T14:06:36Z

How does the ubuntu image compare in size with the nikolaik one?

@tofarr It does slightly better, probably because Bookworm's stock image starts out larger than Ubuntu's, but most of the space we add ourselves.

runtime:78509e5-nikolaik - 10.5 gb
runtime:78509e5-ubuntu - 9.86 gb

raymyers · 2025-04-04T19:56:45Z

.github/workflows/ghcr-build.yml


          TEST_RUNTIME=docker \
-          SANDBOX_USER_ID=$(id -u) \


Edit: This user id is for file permissions when mounting a volume and needs to stay that way

For some reason, a lot of our docs suggest using id -u for the user_id. When running in Docker mode, that makes no sense to me.

But the effect when running from Ubuntu using an Ubuntu image is: It tries to create an openhands user with uid 1000, the same id as the ubuntu user. This fails. Therefore, to use the Ubuntu runtime, we should set SANDBOX_USER_ID to some other number.

It's confusing for our users to have to know that, and we should revisit it - maybe see if we can eliminate the need to specify a uid at all.

As a workaround I tried to just remove the ubuntu user in the image build or change its ID, but wasn't able to get that working.

… user

raymyers · 2025-04-07T22:44:03Z

openhands/runtime/impl/docker/docker_runtime.py

            'VSCODE_PORT': str(self._vscode_port),
+            'PIP_BREAK_SYSTEM_PACKAGES': '1',


This gets around this issue where we can't pip install in Ubuntu. There may be better ways we can look at.

$ pip install requests error: externally-managed-environment × This environment is externally managed ╰─> To install Python packages system-wide, try apt install python3-xyz, where xyz is the package you are trying to install. If you wish to install a non-Debian-packaged Python package, create a virtual environment using python3 -m venv path/to/venv. Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make sure you have python3-full installed. If you wish to install a non-Debian packaged Python application, it may be easiest to use pipx install xyz, which will manage a virtual environment for you. Make sure you have pipx installed. See /usr/share/doc/python3.12/README.venv for more information. note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification.

Curious, why was this necessary when it wasn't in Debian?

@enyst Good question, I think it's because the official Python docker image (used by Nikolaik) installs python from source rather than with the APT package.

https://github.com/docker-library/python/blob/master/Dockerfile-linux.template

I think if we followed this pattern in Ubuntu we could also avoid it, that's an option. Generally it's probably not great to assemble the Python installation within the OpenHands build process, we could build an image with node+python in a separate repo maybe.

raymyers · 2025-04-07T22:45:01Z

openhands/runtime/impl/docker/docker_runtime.py

@@ -293,6 +294,8 @@ def _init_container(self):
            self.container = self.docker_client.containers.run(
                self.runtime_container_image,
                command=command,
+                # Override the default 'bash' entrypoint because the command is a binary.
+                entrypoint=[],


Oddly this specifically is needed in Ubuntu when the default ubuntu user has been removed - but the updated config seems more correct all around.

raymyers mentioned this pull request Apr 3, 2025

chore - Switch to slim base image for improved security #7680

Closed

2 tasks

raymyers force-pushed the ray/base-ubuntu branch 2 times, most recently from 10462fd to 78509e5 Compare April 3, 2025 22:39

xingyaoww approved these changes Apr 3, 2025

View reviewed changes

raymyers commented Apr 4, 2025

View reviewed changes

mamoodi assigned raymyers Apr 7, 2025

raymyers added 13 commits April 7, 2025 15:06

feature - Build alternate ubuntu images

b1dff8a

Remove comment from dockerfile because of newlines

9d8243f

Add playwright to cache

6e4ac37

Add more packages to ubuntu build

916d671

Fix jinja spacing in dockerfile

76f6f29

Put corepack on path

c55be3c

Update ubuntu docker commands

ad28795

Install corepack using npm

f013ebd

Use ensurepip for ubuntu dockerfile

c2b78f4

Install more python packages in ubuntu

5d4a858

Update test and remove some vscode extension dirs

c01f402

Avoid 1000 user id for running tests in ubuntu

f1c3685

Change entrypoint config so docker can run binary with deleted ubuntu…

9d48b70

… user

raymyers force-pushed the ray/base-ubuntu branch from 4c5b9e7 to 9d48b70 Compare April 7, 2025 20:11

raymyers added 3 commits April 7, 2025 15:33

Merge branch 'main' into ray/base-ubuntu

bf209b1

Cleanup ci

de1a92a

Allow pip to install in ubuntu

6a55b88

raymyers commented Apr 7, 2025

View reviewed changes

raymyers merged commit 99e493b into main Apr 8, 2025
20 checks passed

raymyers deleted the ray/base-ubuntu branch April 8, 2025 00:04

raymyers mentioned this pull request Apr 8, 2025

chore - Skip building Ubuntu runtime image on PR #7765

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature - Build alternate ubuntu images #7691

feature - Build alternate ubuntu images #7691

raymyers commented Apr 3, 2025 •

edited by github-actions bot

Loading

xingyaoww left a comment

xingyaoww Apr 3, 2025

raymyers Apr 3, 2025 •

edited

Loading

li-boxuan Apr 4, 2025 •

edited

Loading

raymyers Apr 4, 2025 •

edited

Loading

li-boxuan Apr 4, 2025

raymyers Apr 4, 2025

raymyers commented Apr 3, 2025

tofarr commented Apr 4, 2025 •

edited by raymyers

Loading

raymyers Apr 4, 2025 •

edited

Loading

raymyers Apr 7, 2025

enyst Apr 8, 2025

raymyers Apr 8, 2025 •

edited

Loading

raymyers Apr 7, 2025

		'VSCODE_PORT': str(self._vscode_port),
		'PIP_BREAK_SYSTEM_PACKAGES': '1',

feature - Build alternate ubuntu images #7691

feature - Build alternate ubuntu images #7691

Conversation

raymyers commented Apr 3, 2025 • edited by github-actions bot Loading

xingyaoww left a comment

Choose a reason for hiding this comment

xingyaoww Apr 3, 2025

Choose a reason for hiding this comment

raymyers Apr 3, 2025 • edited Loading

Choose a reason for hiding this comment

li-boxuan Apr 4, 2025 • edited Loading

Choose a reason for hiding this comment

raymyers Apr 4, 2025 • edited Loading

Choose a reason for hiding this comment

li-boxuan Apr 4, 2025

Choose a reason for hiding this comment

raymyers Apr 4, 2025

Choose a reason for hiding this comment

raymyers commented Apr 3, 2025

tofarr commented Apr 4, 2025 • edited by raymyers Loading

raymyers Apr 4, 2025 • edited Loading

Choose a reason for hiding this comment

raymyers Apr 7, 2025

Choose a reason for hiding this comment

enyst Apr 8, 2025

Choose a reason for hiding this comment

raymyers Apr 8, 2025 • edited Loading

Choose a reason for hiding this comment

raymyers Apr 7, 2025

Choose a reason for hiding this comment

raymyers commented Apr 3, 2025 •

edited by github-actions bot

Loading

raymyers Apr 3, 2025 •

edited

Loading

li-boxuan Apr 4, 2025 •

edited

Loading

raymyers Apr 4, 2025 •

edited

Loading

tofarr commented Apr 4, 2025 •

edited by raymyers

Loading

raymyers Apr 4, 2025 •

edited

Loading

raymyers Apr 8, 2025 •

edited

Loading