Tracking Issue for garbage collection

[ehuss]

Summary

Original proposal: https://hackmd.io/@rust-cargo-team/SJT-p_rL2
Implementation: #12634
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#gc
Issues: Z-gc Nightly: garbage collection

The -Zgc flag enable garbage collection for deleting old, unused files in cargo's cache.

Status

• Tracking of last-use data started in Rust 1.78 via Stabilize global cache data tracking. #13492.
• Automatic cleaning of unused data started in Rust 1.88 via Stabilize automatic garbage collection. #14287.
• Manual cleaning is tracked in gc: Determine CLI design for manual cleaning #13060.
• What kind of config options to expose is tracked in gc: Determine which config options should be exposed and their defaults #13061.
• Build artifact tracking and cleaning is tracked in Garbage collect whole target/ #13136.

Unresolved Issues

• Finalize the CLI
  • Enumerate prior art from other tools with cache management (poetry, docker, git, nix, zig, dune, etc)
    • Apparently buck2 and Turborepo grow unbounded

Future Extensions

No response

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

[epage]

For when we get to target/:

From @bjorn3 at https://hachyderm.io/@bjorn3/111047792430714997

The incr comp cache is GCed on every compilation by only copying artifacts that were used for the current compilation session to the new incr comp cache dir and then removing the old incr comp cache dir entirely. In effect it is a maximally eager semi-space garbage collector. This is not suitable for cargo at all because alternating between cargo build -p foo and cargo build -p bar is guaranteed to rebuild things every time as each build would delete the artifacts of the other build.

[epage]

Quick scan of brew

• autoremove
• cleanup
  • "Removes all downloads more than 120 days old. This can be adjusted with HOMEBREW_CLEANUP_MAX_AGE_DAYS."
• HOMEBREW_CLEANUP_PERIODIC_FULL_DAYS (default 30 days)

One complaint that came up was " "brew cleanup has not been run in 30 days, running now" ... and then proceeds to run an interminable process in the middle of you attempting to do something else." (mastadon)

[epage]

From https://hachyderm.io/@[email protected]/111048319933010803

Also: global vs. local? Just keep all things local. Errant cache hits pulling from a non-scoped global cache are a disaster. Errant cache hits from the same local cache are a "whoops." For shared artifacts (e.g. a singular tar downloaded from the registry), global is fine, and should just grow unbounded. (Seriously: nobody cares.)

pnpm, bun, npm, yarn: package cache is intended to be shared global immutable, unbounded growth.

I'm assuming our global package cache is to help with CI but we should probably explicitly document out priority use cases.

[ehuss]

In https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Stabilizing.20global.20cache.20tracking/near/422500781 I am proposing to stabilizing just the recording of the cache data as a first step. This doesn't enable automatic or manual gc.

[airstrike]

I would wager most rust users associate the words "garbage collection" with memory rather than cached files that have gone stale. It's unfortunate that the term is being overloaded here

[ssokolow]

I would wager most rust users associate the words "garbage collection" with memory rather than cached files that have gone stale. It's unfortunate that the term is being overloaded here

It depends. Some of us are familiar with the git gc command.

[epage]

The feature is in development and how we present it to the user is not yet decided. In #13060, we are exploring how to present it in the CLI, including looking at prior art from other tools.

[cessen]

Thanks for redirecting me to the naming discussion here @epage

I think(?) one of the distinguishing characteristics of garbage collection implementations (whether they be for memory, git, nix, etc.) is that they remove things that are "unreachable" in some sense, and thus can be confidently disposed of as not used. That particular characteristic is specifically not true of this feature, as discussed in #13176.

Having said that, in practice I'm skeptical if calling this feature "garbage collection" is actually going to confuse people. Nevertheless, it does seem like one of those "might as well be more accurate" kind of situations. So calling it "cache cleaning" or similar.

[ehuss]

I have proposed to stabilize the automatic side of this feature in #14287.

[tgrushka]

New (1.3 year or so) Rust developer here. I see this issue is still open. There was a merge of #14287. Hope I can still post a question on this.

I have no knowledge of internals, so please bear with my question if it's completely naive.

Why is this garbage collection by date/timestamp complexity necessary for cargo to clean artifacts automatically? Am I missing something?

In my mind:

• the Rust compilation unit is the crate
• Cargo knows what version of a crate is being built (doesn't it?) and what platform it is building it for
  • and possibly its source repo -- whether crates.io, git, path, etc. -- but maybe this part doesn't matter, because it's the crate name, version, and platform that really "counts" for the project, right? e.g. if I override a dep with a particular version, that "replaces" that version in my project as far as Cargo is concerned?
• Each time Cargo builds an artifact, it is put in a hashed structure under target/** -- so Cargo obviously hashes the artifact somehow, but I have no idea what info this hash represents (the contents of the binary I assume?):

├── deps
│   ├── adler2-e884df5f7f1a4eb1.d
│   ├── ahash-323a9839de8403ad.d
│   ├── aho_corasick-015c2a7e4d760308.d
...
│   ├── libserde_derive-70b20053052b9820.dylib
│   ├── libserde_json-04aeac63bb2c54fb.rlib
│   ├── libserde_json-04aeac63bb2c54fb.rmeta
│   ├── libserde_json-ee256ea891812ead.rmeta
│   ├── libserde_path_to_error-fcc0061e09c623b4.rmeta
...

• see the 3 libserde_json- files with hashes for example -- leading me to believe the hashes are for the contents of those binaries, and a cargo tree | grep serde_json confirms to me that there's only one version of serde_json in my project, but 3 artifacts.

Maybe it would be a lot of additional work/extra feature, but couldn't Cargo somehow keep track of what crate + version + platform each of these artifacts was compiled for?

Then, the next time Cargo builds that crate and version(s) + platform(s) for that project (maybe this wouldn't work for a shared Cargo directory), knowing its dependency tree, Cargo would automatically delete the old hashed artifact(s) when it builds the new one?

• EDIT: Maybe it could work for a shared dir, if Cargo "hashed" the source repo, version, and platform of each crate it builds. The worst case scenario in my mind would be two projects depending on the same crate and version + platform but from a git repo main branch, but then you'd want it to rebuild anyway. If in different repos, or different git refs, the "hash" Cargo generates for that combo of crate + version + platform + source would be different.

I guess this would require a file to store the hashes and version combos, but then there's a .fingerprint directory with all these other metadata files:

dep-lib-serde_json	lib-serde_json
invoked.timestamp	lib-serde_json.json

So maybe the necessary info is already there? Of course, unlinking old files would add more build time, but how much? A whole day of cargo build on my project takes about 30 seconds to a minute to clean, with several GB of files. I wonder what's creating so very much cruft?

What am I missing? Isn't trying to determine what should be deleted by access time a brittle and possibly over-complicated way of achieving something that Cargo could itself already be smart enough to solve without this extra guesswork? Couldn't Cargo just sweep itself during compile -- with an optional flag or perhaps per-project or global setting?

EDIT: Oh yeah, if I bump versions, the old artifcats for the old versions would no longer get swept.

So yes, I see the usefulness of timestamp garbage collection as well.

But I don't bump my versions much in one day working a project, if at all, and yet so very much cruft -- GB and GB -- builds up -- maybe from crates that generate a lot of code or bindings or something. No idea. But I would think there could be some way for Cargo to track the crate + version + platform + source hash of each artifact, and if those match, just delete the old files?

[weihanglo]

There are different ways build cache directory can be filled. #5026 (comment) is about toolchain upgrades. The original hackmd note of this feature also called out some unresolved questions (search for “deps” in it). The #7150 mega tracking issue has more details. One of the current plan is splitting intermediate build artifact directory out from final artifact directory, and then we can have a better space to restructure the layout to be more reasonable and easier to track.

(the contents of the binary I assume?):

No. The hash is determined before any compilation. For what hashes trakc, see https://doc.rust-lang.org/nightly/nightly-rustc/cargo/core/compiler/fingerprint/index.html.

…but couldn't Cargo somehow keep track of what crate + version + platform each of these artifacts was compiled for?

Theoretically yes, with the aid from rustc. The reality is rustc could emit more files than Cargo knows.

and if those match, just delete the old files?

Even if Cargo already tracks everything, Cargo still can't simply by default do that because the build caches directory could be shared between different workspaces. Also switching between different version of toolchain isn't an uncommon workflow. It needs to be considered.

[epage]

New (1.3 year or so) Rust developer here. I see this issue is still open. There was a merge of #14287. Hope I can still post a question on this.

Why is this garbage collection by date/timestamp complexity necessary for cargo to clean artifacts automatically? Am I missing something?

For context, there are multiple resources to garbage collect. What has been stabilized is GC of user-wide resources like the Index, .crate files that were downloaded, and the source extracted from .crate files. We have not done any work on GCing the contents of target/ yet. @weihanglo pointed out different degrees of doing this.

Then, the next time Cargo builds that crate and version(s) + platform(s) for that project (maybe this wouldn't work for a shared Cargo directory), knowing its dependency tree, Cargo would automatically delete the old hashed artifact(s) when it builds the new one?

The compile clears out anything from the incremental compilation cache that wasn't used in the current compilation. This doesn't work well for Cargo because users can switch between different feature combinations that would affect compilation. Maybe crate source + name + version could be used to clean up but that can also make it harder to jump back and forth between versions. There are times I wish we had different caches of local code so I could more easily jump back and forth between commits.