Functionality for additional control over the chroot's mounts

[rrbutani]

As described in #74, this PR adds functionality to nix-user-chroot that allows for:

• mounting in extra absolute paths into the chroot
  • for example, mounting / to /mount/extra-filesystem-root)
• mounting in extra paths relative to the current user's latest nix profile into the chroot
  • for example, mounting /bin/bash from the user's nix-profile to /bin/bash
  • as implemented, we search for the user's latest nix profile at <path to nix dir>/var/nix/profiles/per-user/$USER/profile
    • if this dir doesn't exist and profile relative mounts are provided we emit a warning (instead of exiting)
• choosing paths on the root filesystem to not mount into the chroot
  • for example, skipping mounting /var/run into the chroot

Currently, the config file is read from <path to nix dir>/etc/nix-user-chroot/path-config.toml. Here's an example of the format:

[excludes]
paths = [
    "/var/run/nscd",
]

[profile]
"/bin/sh"               = "/bin/sh"
"/bin/bash"             = "/bin/bash"
"/bin/env"              = "/usr/bin/env"
"/etc/profile.d/nix.sh" = "/etc/profile"

[absolute]
"/some/disk/config/group"  = "/etc/group"
"/some/disk/config/passwd" = "/etc/passwd"

---

#74 details some of the motivating use cases (overriding /etc/group, not mounting in socket files for daemons running outside the chroot, etc.).

This PR isn't quite ready for merge yet. I think the path resolution logic for nix profile relative paths is pretty questionable and should be reviewed, run_chroot should be split up a bit, and some tests (particularly for nix profile relative symlink resolution) would be good to have too.

I'm opening this PR as is anyways to solicit feedback and to get a feel for whether this is the right direction for this kind of feature/something that nix-user-chroot would be interested in adding.

If this is deemed out of scope for this project, please feel free to close this PR.

[Mic92]

@DavHau what is your opinion on this, since you already maintain https://github.com/DavHau/nix-portable
I am missing a bit of a usecase for nix-user-chroot, which is why it does not has a high priority for me personally.
But I also see how it can be useful for others so I am not so sure how to move forward with the project.

@rrbutani the functionality that you are describing make sense to have but I feel like it's turning a bit into what bubblewrap already provide to us. However I also see that value in just having one binary that is easy to install. So I am a bit unsure.

[Mic92]

So I thought about this a bit longer and I feel that simply exposing these mounts to users would make the tool harder to use because they generally would not know what to fill in for /etc/passwd to fix bugs like: #74 (comment)

That's why I see two options here:

1. Just hardcode sane bind mounts and override configuration that works for most users
2. Integrate this with some system that already provides working defaults but allow for customization. For example there could be a home-manager module that provides profiles/configuration to be consumed by nix-user-chroot.

[rrbutani]

@Mic92 Thanks for the quick response.

1. Just hardcode sane bind mounts and override configuration that works for most users

For the exact issue in #74 I think this is very doable; adding in an /etc/group that remaps the tty group seems pretty uncontroversial and leaving out /var/run/nscd/daemon is probably a defensible default too.

However, there are also other reasons why I wanted this kind of functionality. On the servers I use nix-user-chroot on, the default $HOME directories have fairly anemic disk quotas; I use a modified /etc/passwd to remap my home dir. I also map in the nixpkgs version of env to /usr/bin/env so that tools that futz with ld.so and libc (like bear) don't break when running scripts with #!/usr/bin/env ....

Use cases like these hard to anticipate and harder still to settle on good one-size-fits-all behavior for.

However, I think it's perfectly reasonable to say that such use cases are out of scope for nix-user-chroot.

2. Integrate this with some system that already provides working defaults but allow for customization. For example there could be a home-manager module that provides profiles/configuration to be consumed by nix-user-chroot.

I am very much willing to implement this but I'm confused what something like this would look like in practice. Would the home-manager module generate a config file for nix-user-chroot?

I think I get the appeal of doing something like this but I'm struggling to conceive of an implementation where the first step isn't adding some kind of config file like thing to this project.

Edit: sorry, I think I maybe misunderstood; are you just saying that the defaults would be encoded in the home-manager module?

[Mic92]

Edit: sorry, I think I maybe misunderstood; are you just saying that the defaults would be encoded in the home-manager module?

Yes there could be a configuration format that is easy to generate by home-manager. I just don't know how one would bootstrap nix-user-chroot in this case.

[rrbutani]

I just don't know how one would bootstrap nix-user-chroot in this case.

Hmm. It seems like the obvious way would be to have users manually grab nix-user-chroot, install nix, and set up their home-manager profile. On subsequent launches of nix-user-chroot, it can look for the config file home-manager emits in the profile.

But I get that this is not exactly optimal.

---

The other setup I can think of is one where nix-user-chroot takes over the nix install process and has you give it a home-manager config to get set up (and maybe also goes and adds something to your ~/.profile to enter the chroot using a nixpkgs supplied version of nix-user-chroot). The bootstrap process would then be grabbing a nix-user-chroot binary that gets run once to set everything up; after that nix-user-chroot from the nix store is used, etc.

This'd let users skip a few steps but it also seems complicated and kind of ties us to home-manager and the use cases it fits for.

[rrbutani]

In any case, in general is this a path you feel comfortable taking for nix-user-chroot?

If so, I think it might make sense to start by nailing down what the configuration file format should look like and what the defaults should be.

[Mic92]

I sent you an invite to nix-community and nix-user-chroot. I think as a first step we can have the more manual route and than later converge to auto install nix + home-manager.

[Mic92]

In any case, in general is this a path you feel comfortable taking for nix-user-chroot?

If so, I think it might make sense to start by nailing down what the configuration file format should look like and what the defaults should be.

From a nix perspective probably json would work best but is than less editable by users...
But I think home-manager now generates all sorts of formats anyway, so maybe readability and lightweight dependencies are favoured. It's probably best if more defaults are set in the default configuration than in nix-user-chroot itself. Maybe we need enough defaults to bootstrap the second phase that runs home-manager.

[Mic92]

I think we are both on the same page regarding the project. I would give you a free hand about the details. Let me know when you want to have a code review.

[rrbutani]

I sent you an invite to nix-community and nix-user-chroot.

Thanks!

From a nix perspective probably json would work best but is than less editable by users... But I think home-manager now generates all sorts of formats anyway, so maybe readability and lightweight dependencies are favoured.

Oh hmm, good point. There isn't a builtins.toTOML but there is a toml.generate in nixpkgs that gets used a bunch in home-manager.

I'm happy to switch to or add a JSON config file format thing – it's just using serde so it'd be a quick and painless change.

It's probably best if more defaults are set in the default configuration than in nix-user-chroot itself. Maybe we need enough defaults to bootstrap the second phase that runs home-manager.

I think this makes perfect sense! I can't think of any settings we'd need to bootstrap; keeping the current defaults (except maybe moving the thing about mounting in /run/opengl-driver/lib into the config, maybe?) seems like it would work fine.

I think we are both on the same page regarding the project. I would give you a free hand about the details. Let me know when you want to have a code review.

Thank you, this is much appreciated. I think the PR is ready for review now; I stared at the path resolving logic a bit more and, while it's definitely not pretty and it still feels like there's a more elegant way to do it, I think it handles the cases it claims to.

I think it still could use some tests though; I'll work on adding some, hopefully in the next few days.

[DavHau]

@DavHau what is your opinion on this, since you already maintain https://github.com/DavHau/nix-portable I am missing a bit of a usecase for nix-user-chroot, which is why it does not has a high priority for me personally. But I also see how it can be useful for others so I am not so sure how to move forward with the project.

@rrbutani the functionality that you are describing make sense to have but I feel like it's turning a bit into what bubblewrap already provide to us. However I also see that value in just having one binary that is easy to install. So I am a bit unsure.

@Mic92, I'm in a similar situation like you, where I do not really need nix-portable myself. Maintenance is done purely according to feedback from the community.
My main motivation creating it was to have a zero-configuration nix which also works without unprivileged user namespaces, so I can easily distribute other nix based projects without requiring the user to install nix first.

What benefits does nix-user-chroot provide over a static bubblewrap binary?

If nix-user-chroot has benefits over just using bubblewrap with the right parameters, then I should probably add nix-user-chroot as a possible backend for nix-portable. If it doesn't, then the question arises why nix-user-chroot should be maintained any longer.

[rrbutani]

@DavHau I hadn't taken a serious look at nix-portable before. Thanks for writing it; I think everything about the way it bootstraps itself (and especially the zip file header trickery) is extremely clever 😄.

---

In general I think nix-portable is indisputably much better at supporting use cases where the goal is to run a project or tool that uses nix than nix-user-chroot is.

However, to me, nix-user-chroot still feels a little better suited for use cases where you want to stand up a nix environment (you get to manage the nix installation yourself instead of having it be tied to nix-portable; the install process is the same as that of a typical nix installation, just with a couple of extra steps at the start). But this is maybe a tenuous assertion; it's definitely possible to use nix-portable in a way that mirrors a more typical installation (just use it to start a shell, install nix there and use that, etc.).

What benefits does nix-user-chroot provide over a static bubblewrap binary?

I think on master there isn't anything nix-user-chroot has to offer over bwrap. Special logic for var/nix/opengl-driver/lib is the only thing I can come up with that nix-user-chroot adds over nix-portable with bwrap and that's entirely trivial to add.

However, with this PR nix-user-chroot offers a way to exclude paths from being bound (it effectively goes and adjusts the other binds so that a particular path isn't mapped in) which I don't think (?) is functionality bwrap provides.

This PR also adds a way for users to map in paths from their nix profile into the chroot which isn't functionality nix-portable provides as far as I know (though I see no technical reason why it couldn't grow this functionality if desired).

If nix-user-chroot has benefits over just using bubblewrap with the right parameters, then I should probably add nix-user-chroot as a possible backend for nix-portable. If it doesn't, then the question arises why nix-user-chroot should be maintained any longer.

Unless nix-portable is interested in the functionality described above I don't think there's anything to be gained from adding nix-user-chroot as a backend.

Personally I think there's enough of a difference in the use cases to warrant nix-user-chroot's existence (things like home-manager integration don't really seem to fit nix-portable well, I think).

But I also think the distinction I'm trying to make is genuinely very narrow. I'm not sure there are other users that are looking to effectively run all their shell sessions inside a chroot with a /nix bind mount on a machine where they don't have privileges. Given that, something like "for your use case you should just set up and use bwrap yourself; everyone else should use nix-portable" might be a reasonable stance.