'Gracefully' fallback from failing substituters

[philipwilk]

Motivation

Nix currently hard fails if a substituter is inaccessible, even when they are other substituters available, unless fallback = true.
This breaks nix build, run, shell et al entirely.
This would modify the default behaviour so that nix would actually use the other available substituters and not hard error.

Here is an example before vs after when using dotenv where I have manually stopped my own cache to trigger this issue, before and after the patch. The initial error is really frustrating because there is other caches available.

Context

#3514 (comment) is the earliest issue I could find, but there are many duplicates.

There is an initial PR at #7188, but this appears to have been abandoned - over 2 years with no activity, then a no comment review in jan. There was a subsequent PR at #8983 but this was closed without merge - over a year without activity.

I have visualised the current and proposed flows. I believe my logic flows line up with what is suggested in #7188 (comment) but correct me if I am wrong.
Current behaviour:

Proposed behaviour:

Charts in lucid

Possible issues to think about:

• I could not figure out where the curl error is created... I can't figure out how to swallow it and turn it into a warn or better yet, a debug log.
• Unfortunately, in contrast with the previous point, I'm not sure how verbose we want to warns/traces to be - personally I think that the warn that a substituter has been disabled (when it happens) is sufficient, and that the next one is being used, but this is personal preference.

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[philipwilk]

Further consideration:
Potentially a configuration option to mark a substituter as silently fail-able/optional - this would solve the issue of whether we want to log fails (some if not all) or not - such as in the use case of a LAN only cache... would still need to figure out how to silence the curl fails though.

[philipwilk]

Further consideration: Potentially a configuration option to mark a substituter as silently fail-able/optional - this would solve the issue of whether we want to log fails (some if not all) or not - such as in the use case of a LAN only cache... would still need to figure out how to silence the curl fails though.

Without a json config #8373, I think per-store configuration is nasty to implement (builders/machines config is just list in list) if not impossible, which would also make this currently infeasible...

[philipwilk]

I think there's a flake (literally) edge case where it still hard errors out in a flake - not sure what is causing this, as I only just tested it working fine... and will need to taken another look at it.