[Pre-RFC] Enable .cargo/root, CARGO_ROOTS + --root to limit config + manifest searches

[rib]

This is a draft proposal that I think could address a number of issues in Cargo (e.g. #5418, #7871) and could be an alternative (or complementary) approach to addressing the safety issues discussed in rust-lang/rfcs#3279

The intention here is to prototype a practical solution that can then hopefully inform any follow up discussions.

My current motivation for looking at this comes from using Cargo as part of a rustdoc-markdown toolchain that should be able to build third-party crates without worrying about side effects from unrelated configs or workspace manifests in parent directories.

--

This adds the notion of root directories to Cargo, which act as limits to how far Cargo will search ancestors looking for .cargo config files or Cargo.toml manifests (instead of walking to the root of the filesystem).

This has some similarity to Git's GIT_CEILING_DIRECTORIES mechanism.

There are three notable searches that root directories affect:

1. How far should Cargo search when loading .cargo/Config.toml files
2. How far should Cargo search from when loading a Cargo.toml package manifest
3. How far should Cargo search from a package manifest directory looking for a Cargo.toml workspace manifest.

Roots can be specified in three ways:

1. By setting the CARGO_ROOTS environment variable to a colon (semicolon on Windows) separated list of directories
2. By passing --root on the command line (can be passed multiple times)
   i. Suited to tool configurations where appending to environment variables may be limited (e.g. limited environment variable interpolation in vscode settings)
3. The existence of a .cargo/root file (discovered by checking parents up towards the root of the filesystem)
   i. Suited to sandboxing directories when the absolute path might vary, such in within monorepos that could be cloned to different locations or directories that may get mounted in Docker containers.

By default, if CARGO_ROOTS has not been set then the user's home directory is implicitly added as a root directory.

• This is for safer default behaviour that stops Cargo from loading configs from shared directories like /tmp/.cargo/, or C:/.cargo or searching for workspace manifests.

The default home-directory root also mitigates issues with triggering automounters on some systems.

Whenever Cargo needs to search ancestors for files it will resolve the closest root directory that is an ancestor of the starting directory and that root is the last directory Cargo will search.

If the starting directory is not within any root then Cargo will only search the starting directory. Cargo will also print a warning to indicate that it's loading of configs + workspace manifests is limited.

As a safety / ergonomics (+ backwards compatibility) trade-off, there is a special case that allows Cargo to search up to the root of the filesystem when searching for the first Cargo.toml package manifest, IFF the starting directory is outside of all roots. This enables a package to be unpacked under a shared directory like /tmp/my-package and start a build from /tmp/my-package/sub/dir which will load /tmp/my-package/Cargo.toml but will not then allow Cargo to attempt loading a workspace at /tmp/Cargo.toml.

What does this PR try to resolve?

Fixes: #5418

• For use cases where the workaround described in Add more suggestions on how to deal with excluding a package from a workspace. #6805 isn't appropriate (to add an empty workspace to nested packages)

Fixes: #7871

• Addresses safety issues with building under shared directories like /tmp
• Mitigates triggering home directory automounters

Addresses: rust-lang/rfcs#3279

This gives more flexibility for tools that want to use cargo as an implementation detail. In particular this allows you to sandbox the build of nested third-party workspaces that may be (unknowingly) dynamically unpacked within an outer workspace, in situations where neither the workspace being built and the outer workspace are owned by the tool that is managing the build.

This does not attempt to support nesting workspaces with inheritence (as described in #5042).

For example a tool based on rustdoc-json should be able to fetch and build documentation for third-party crates under any user-specified build/target directory without having to worry about spooky action at a distance due to config files and workspaces in ancestor directories.

In my case, I have an runtime for coding with LLMs that is given a repo to work on and is expected to keep its artifacts contained to a .ai/ directory. This runtime supports building markdown documentation for Rust crates, which involves using cargo to generate rustdoc-json data. That tool is expected to keep its artifacts contained within .ai/docs/rust/build/ within the project being worked on. It's possible that the project itself is Rust based and could define a workspace or .cargo/config.toml but from the pov of this toolchain those have nothing to do with the crate whose documentation is being generated (which are packages downloaded from crates.io).

How to test and review this PR?

TODO: write tests

Root directories can be smoke tested by attempting to build any package that doesn't define a workspace, nested within a repo that does, E.g.:

mkdir cargo-root-test
cd cargo-root-test

# Clone anything that defines an outer workspace...
git clone https://github.com/serde-rs/serde.git

# Then clone a nested repo that doesn't have a workspace
cd serde
git clone https://github.com/meta-rust/rust-hello-world

# Attempting to build will lead to an error like "error: current package believes it's in a workspace when it's not"
cargo build

# But if we know that the outer workspaces is unrelated we can specify a root directory and isolate the build...
cargo --root . build
# Or
env CARGO_ROOTS="$PWD" cargo build
# Or
mkdir .cargo
touch .cargo/root
cargo build

Known Issues

The current implementation makes heavy use of std::fs::canonicalize while the original expectation had been that the UNC prefixes for Windows wouldn't matter when just internally checking paths for equivalence.

As things have evolved I think the use of canonicalize should be replaced with a normalization that just maps root directories to absolute paths and uses Cargo's normalize_path utility. This way we would avoid resolving symlinks and rely on lexical path comparisons.

Ideally the normalized paths can even be kept as a hidden implementation detail when resolving a SearchRoute that would allow the following use of path::ancestors to iterate in accordance with any non-normalized starting directory.

[epage]

I guess something like this might need a [Pre-]RFC?, but as I'm #5418 (comment) with nested workspaces while using Cargo as part of a rustdoc-markdown toolchain, I figured I'd go ahead and take a stab at drafting a solution that could at least work for my use case, to get some practical insight into the problem.

Personally, I do not look at PRs until a design is agreed on. Having design discussions here splits the conversation and makes it harder to follow. This is also feature with a lot of potential workflow considerations and I would personally want to see a more deliberate approach, like an RFC,. to make sure we understand what problems we are trying to solve, how this has been solved before, and how this will impact people.

FYI there is also discussion at #7871 and this is somewhat related to rust-lang/rfcs#3279

[rib]

I guess something like this might need a [Pre-]RFC?, but as I'm #5418 (comment) with nested workspaces while using Cargo as part of a rustdoc-markdown toolchain, I figured I'd go ahead and take a stab at drafting a solution that could at least work for my use case, to get some practical insight into the problem.

Personally, I do not look at PRs until a design is agreed on. Having design discussions here splits the conversation and makes it harder to follow. This is also feature with a lot of potential workflow considerations and I would personally want to see a more deliberate approach, like an RFC,. to make sure we understand what problems we are trying to solve, how this has been solved before, and how this will impact people.

FYI there is also discussion at #7871 and this is somewhat related to rust-lang/rfcs#3279

Right, I at least like to write some code, where possible, as a proof of concept / point of reference before getting into design discussions to try and avoid getting too academic, but it makes sense to avoid fragmented design discussions here, beyond linking to existing discussions or perhaps giving pointers for framing a new RFC.

Thanks for the links - I hadn't come across those discussions from searching myself.

[rib]

While I've been looking at some of the other related issues in #7871 (including issues with /tmp/Config.toml or /tmp/.cargo/Config.toml files or triggering home directory automounters) and the safety issues discussed in rust-lang/rfcs#3279 I've taken another pass at this prototype and updated the description accordingly.

I'll try and find some time to follow up on the discussions for those separately, but I'm currently optimistic that something close to this draft could be a viable solution for several of these issues.