[RFC 0164] Bootspec v2

This introduces the second revision of RFC-0125, Bootspec,
addressing the feedback we received in NixOS#125
and building on our experience of Bootspec v1.

Author: RaitoBezarius

rfcs/0164-bootspecv2.md

@@ -0,0 +1,300 @@
+---
+feature: bootspec_v2
+start-date: 2022-11-01
+author: Ryan Lahfa
+co-authors:
+shepherd-team: 
+shepherd-leader:
+---
+
+# Summary
+[summary]: #summary
+
+Bootspec v2 is the second revision of the Bootspec document, introduced in RFC-0125.
+
+These facts are used as the primary input for bootloader backends like systemd-boot and grub, for creating files in `/boot/loader/entries/` and `grub.cfg`.
+
+In this proposal, we aim to tackle known weaknesses of Bootspec v1, namely:
+
+- Lack of devicetree
+- Multiple initrds support
+- Rework of initrd secrets mechanism
+- Extra metadata that was discovered to be useful during the year of usage of Bootspec v1
+
+This document describes **Bootspec v2**.
+
+# Motivation
+[motivation]: #motivation
+
+The motivation of Bootspec v1 remains but we address the v1 weaknesses and
+includes our experience of running with it for a while.
+
+## Multiple initrds
+
+The Boot Loader Specification, developed by the Userspace API group, allows the
+specification of multiple initrd entries [1], with the purpose of merging them into
+CPIO archives. This functionality is valuable in the systemd-stub ecosystem,
+where various types of initrds are combined, including specific credential
+initrds, global credential initrds, system extension initrds, PCR signature
+initrds, and PCR public key initrds. Some initrds, such as credential initrds,
+are dynamically generated from an EFI System Partition (ESP) location. [2]
+
+Moreover, an additional initrd is frequently used to store CPU microcode. To
+ensure compatibility and flexibility, it is essential to rework the initrd
+support in the Boot Loader Specification. The proposed changes aim to treat
+initrds as a set, allowing bootloaders to handle a list of initrds or a single
+initrd in cases where multiple initrds are not supported.
+
+[1]: https://uapi-group.org/specifications/specs/boot_loader_specification/#type-1-boot-loader-entry-keys
+[2]: https://github.com/systemd/systemd/blob/main/src/boot/efi/stub.c#L778-L793
+
+## Initrd secrets
+
+Initrd secrets play a crucial role in the NixOS ecosystem, but they have raised
+concerns, including security vulnerabilities (e.g., [GHSA-3rvf-24q2-24ww](https://github.com/NixOS/calamares-nixos-extensions/security/advisories/GHSA-3rvf-24q2-24ww
+)) and
+issues with booting in the Heads platform (e.g., [Issue #1348](https://github.com/linuxboot/heads/issues/1348.
+)).
+
+Initrd secrets serve to protect boot-time secrets from exposure within the Nix
+store. This is achieved by using runtime scripts that append CPIO archives to
+the generation's initrd during bootloader installation. However, the term
+"initrd secrets" can be misleading, as the secrets are plaintext and can be
+accessed from the ESP partition, offering limited confidentiality and no
+integrity.
+
+In practice, initrd secrets are often employed to establish stable fingerprints
+for SSH servers within the initrd, aiding in remote disk decryption on servers.
+To address these issues, this RFC proposes moving away from the "appender
+script" model in Bootspec v1 and instead adopting a hash map format to
+represent secrets and their corresponding values.
+
+This change allows for greater flexibility and enhanced security. For users in
+the NixOS ecosystem relying on systemd and its semantics, the proposal suggests
+offering a systemd-credentials approach for handling initrd secrets,
+potentially enabling encryption using TPM2 if available, or a straightforward
+key if TPM2 is not present. Additionally, this approach provides the means to
+build using the `LoadEncryptedCredential` abstraction.
+
+For those not using systemd, the hash map format offers full flexibility to
+bootloader implementations to determine how to add the secrets, such as
+appending to the initrd or other methods.
+
+**Key Takeaway**: This RFC recommends changing the format of Bootspec v2 to support
+`initrdSecrets` as a hashmap with string keys and values. The key represents a
+name used to organize secrets, and the value denotes an accessible path during
+bootloader installation.
+
+_Example_
+
+```
+"initrdSecrets": {
+   "my-private-key": "/etc/nixos/secrets/wireguard-key"
+}
+```
+
+## Device Tree and Device Tree Overlays
+
+Non-x86 systems often rely on device trees to inform firmware or bootloader about the available devices and hardware support. 
+
+There are two distinct requirements for device trees: 
+
+- generic hardware support, where firmware or bootloader selects the appropriate device tree
+- device-specific support, where a hardcoded device tree is required. 
+
+The latter indicates a potentially problematic, non-upstreamed, or in-development platform.
+
+The current Bootspec v1 does not formally encode information about hardcoded device trees or the folder containing available device trees. As a result, unformalized extensions are needed to address these fundamental use cases.
+
+Regarding overlays in NixOS, any expressed overlay is incorporated into the final device tree. This eliminates the necessity to formalize an additional overlays field list, as overlays can be transformed into device trees as needed.
+
+# Goals
+[goals]: #goals
+
+- Improve non-x86 support in Bootspec, emphasizing the importance of
+  devicetrees.
+- Address and reduce the risks associated with initrd secrets, providing a
+  default "secure" implementation within the systemd ecosystem and offering
+  flexibility for other ecosystems.
+- Enhance initrd flexibility, allowing developers to optimize their systems by
+  supporting multiple initrds, including one for microcode and other specific
+  purposes.
+
+### Non-Goals
+[non-goals]: #non-goals
+
+- Store TPM2-related information (hashes)
+  We believe that Bootspec is still too immature for this and pcrlock
+  (https://github.com/systemd/systemd/pull/28891) offers a more reliable and
+  robust solution for generating signed PCR policies.
+- Supporting SecureBoot.
+  Secure Boot has one maintained implementation that is being upstreamed: https://github.com/nix-community/lanzaboote
+  which was enabled by Bootspec v1.
+- Specifying how to discover generations. This is desirable, but should not be tied to bootspec directly since bootspec may be useful with diverse discovery mechanisms.
+
+# Proposed Solution
+[proposed-solution]: #proposed-solution
+
+- `initrd` will be removed from the v2
+- `initrds` will be introduced as a list of initrd (compressed or uncompressed
+  CPIO archives), this list can be empty, but the field is **required**
+  nonetheless.
+- `initrdSecrets` will now be a hashmap of strings (key) and strings (value)
+  where the key is the "name" of a secret and the value is the "path" towards a
+  secret.
+- `fdtdir` will be introduced as an opaque string to a directory in a shape as
+  expected by U-Boot extlinux `FDTDIR` directive — it is **optional**
+- `devicetree` will be introduced as a path to a single devicetree that will be
+  hardcoded — it is **optional**
+
+All the Bootspec ecosystem will be updated as part of this specification.
+
+### Bootspec Format v2
+[format-v2]: #format-v2
+
+
+Using the following JSON:
+
+```json5
+{
+  // Toplevel key describing the version of the specification used in the document
+  "org.nixos.bootspec.v2": {
+    // (Required) System type the bootspec is intended for (e.g. `x86_64-linux`, `aarch64-linux`)
+    "system": "x86_64-linux",
+
+    // (Required) Path to the stage-2 init, executed by the initrd (if present)
+    "init": "/nix/store/xxx-nixos-system-xxx/init",
+
+    // (Required) List of paths to the initrd, can be empty
+    "initrds": [ "/nix/store/xxx-initrd-linux/initrd" ],
+
+    // (Required) Hash map of desired secrets for that generation inside of the initrd.
+    // Implementors of a bootloader installation procedure should examine their options
+    // to securely make available the secret inside the initrd phase.
+    // This may involve leveraging TPM2 via systemd-credentials or any measure you deem
+    // to be reasonable in the context.
+    // The legacy behavior is to prepare a CPIO archive for each file and
+    // extend the `initrds` fields with those CPIO archives.
+    // Make sure the location where the secrets are dropped in the initrd are visible
+    // for the user.
+    "initrdSecrets": {
+      "my-private-key": "/etc/nixos/secrets/wireguard-private-key",
+    }
+
+    // (Required) Path to the kernel image
+    "kernel": "/nix/store/xxx-linux/bzImage",
+
+    // (Required) Kernel commandline options
+    "kernelParams": [
+      "amd_iommu=on",
+      "amd_iommu=pt",
+      "iommu=pt",
+      "kvm.ignore_msrs=1",
+      "kvm.report_ignored_msrs=0",
+      "udev.log_priority=3",
+      "systemd.unified_cgroup_hierarchy=1",
+      "loglevel=4"
+    ],
+
+    // (Required) The label of the system. It should contain the operating system, kernel version,
+    // and other user-relevant information to identify the system. This corresponds
+    // loosely to `config.system.nixos.label`.
+    "label": "NixOS 21.11.20210810.dirty (Linux 5.15.30)",
+
+    // (Required) Top level path of the closure, in case some spelunking is required
+    "toplevel": "/nix/store/xxx-nixos-system-xxx",
+    
+    // (Optional) FDTDIR is assumed to be a path to a directory in the shape
+    // of what `FDTDIR` in U-Boot extlinux would expect.
+    // At the time of writing, it is assumed to follow the kernel output shape.
+    "fdtdir": "/nix/store/xxx-uboot-fdtdir",
+    
+    // (Optional) devicetree is assumed to be path to a single devicetree file
+    // which will be hardcoded for that generation.
+    "devicetree": "/nix/store/xxx-arm64-machine/my-device.dtb"
+  },
+  // The top-level object may contain arbitrary further keys ("extensions"), whose semantics may be defined by third parties.
+  // The use of reverse-domain-name namespacing is recommended in order to avoid name collisions.
+
+  // (Optional) Specialisations are an extension to the specification which allows bundling multiple variants of a NixOS configuration with a single parent.
+  // These are shaped like the top level; to be precise:
+  //  - Each entry in the toplevel "org.nixos.specialisation.v2" object represents a specialisation.
+  //  - In order for the top-level document to be a valid v2 bootspec, each specialisation must have a valid "org.nixos.bootspec.v2" key whose value conforms to the same schema as the toplevel "org.nixos.bootspec.v2" object.
+  //  - The behaviour of nested specialisations (i.e. entries in "org.nixos.specialisation.v2" which themselves contain the "org.nixos.specialisation.v2" key) is not defined.
+  //  - In particular, there is no expectation that such nested specialisations will be handled by consumers of bootspec documents.
+  //  - Each specialisation document may contain arbitrary further keys (extensions), like the top-level document.
+  //  - The semantics of these should be the same as when these keys are used at the top level, but only apply for the given specialisation.
+  "org.nixos.specialisation.v2": {
+    // Each key in this object corresponds to a specialisation as defined by the `specialisation.<name>` NixOS option.
+    "<name>": {
+      "org.nixos.bootspec.v2": {
+        // See above
+      }
+    }
+  },
+}
+```
+
+An *optional* field means: a field that is either missing or present, but **never `null`**.
+
+### Risks
+[risks]: #risks
+
+- Some of the bootloader backends are quite complicated, and in many cases have
+  inadequate tests. We could accidentally break corner cases.
+- The bootloader backends are inherently a weak point for NixOS, as it is our
+  last option for rolling back. We cannot roll back a broken bootloader. This
+  and the previous point are risks, but also help demonstrate the value of
+  reducing the amount of code and complexity in the generator.
+
+### Milestones
+[milestones]: #milestones
+
+- Update Bootspec with the version 2 of that specification.
+- Implement changes inside at least one bootloader backend.
+
+# FAQ
+[faq]: #faq
+
+Familiarize yourself with Bootspec v1 which may already contain answers.
+
+## How will I be able to express complicated logic for initrd secrets, e.g. dynamic secrets?
+
+Some users may have used the appender script to provide dynamic logic that
+provides a secret at activation time rather than storing it on the long run.
+
+While this usecase is interesting, it is very advanced and the footgun that
+initrd secrets represent cannot be made up by our only goal of supporting that
+usecase.
+
+Authors may propose to think about how a filesystem could implement the dynamic
+fetching at activation time, e.g. a FUSE secretfs that will dynamically query
+the secret engine for a secret and make it available for short time.
+
+If any logic requires user interaction, it is preferable to invest in a custom
+bootloader installer logic and use the static fields to refer to secrets that
+will be requested. We never defined "path" on purpose as it could be a network
+path or an application-specific path.
+
+It could even be a UEFI device path!
+
+# Open Questions
+[open-questions]: #open-questions
+
+- Should the `initrdSecrets` work itself with systemd-credentials to load
+  further credential and have a closed loop of credentials, this would require
+  the activation to run inside systemd:
+  https://github.com/NixOS/nixpkgs/pull/258571 !
+- What are lessons we can learn from
+  https://github.com/aarch64-laptops/edk2/tree/dtbloader-app#dtbloader for
+  devicetrees manipulation?
+
+# Future Work
+[future]: #future-work
+
+- Continue the migration from filesystem-spelunking into using the bootspec
+  data.
+- Implement a systemd-credentials based of `initrdSecrets`
+- Implement an [Verified Boot for
+  Embedded](https://u-boot.readthedocs.io/en/latest/develop/vbe.html)
+  bootloader installation script using bootspec data