diff --git a/changes/2025-01-17_key-store-storage-mutations/background.md b/changes/2025-01-17_key-store-storage-mutations/background.md new file mode 100644 index 00000000..d0d89ccc --- /dev/null +++ b/changes/2025-01-17_key-store-storage-mutations/background.md @@ -0,0 +1,157 @@ +[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." +[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" + +# Branch Key Storage Changes for Mutating Branch Keys + +# Definitions + +## All items of a Branch Key + +"All items of a Branch Key" is all the items +persisted in the Key Store with the same Branch Key ID; +these will be one or more DECRYPT_ONLY versions, +the ACTIVE version, and the Beacon Key. + +In this document, +the result of the kms:GenerateDataKeyWithoutPlaintext of a Branch Key Version +is the "Branch Key Version's Cryptographic Material". + +All the results of the kms:GenerateDataKeyWithoutPlaintext +of all items of a Branch Key +are the "Branch Key's Cryptographic Materials". + +## MPL Consumer + +An "MPL Consumer" is a library, service, or other application +that uses the AWS Cryptographic Material Providers Library (MPL) +to manage cryptographic materials. +An "MPL Consumer" may or may not be using an AWS Crypto Tools product, +such as the AWS Encryption SDK or the AWS Database Encryption SDK. + +## Conventions used in this document + +The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", +"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be +interpreted as described in [RFC 2119](https://tools.ietf.org/html/rfc2119). + +# Background + +The [branch key store](../../framework/branch-key-store.md) +persists Branch Keys and their metadata; +the [Key Storage Interface](../../framework/key-store/key-storage.md) +is the interface for persisting this data. + +The Interface does yet specify how to safely change +properties of these Branch Key items. + +This change adds new Operations to the interface, +which facilitate changing the +properties of all items of a Branch Key. + +These new operations are constructed with _Optimistic Locks_, +ensuring that modifications do not accidentally erase data. + +Additionally, +they facilitate the creation, maintenance, and deletion +of items that are used to ensure a change to a Branch Key +is consistently applied to all items of that Branch Key. + +# Design Questions + +These are not answered, +as the Mutations Background document addresses +these questions. + +# Changes + +## New Structures + +The following structures are introduced to the Key Storage Interface: + +### EncryptedHierarchicalKeys + +A List of +[EncryptedHierarchicalKey](../../framework/key-store/key-storage.md#encryptedhierarchicalkey). + +### MutationCommitment + +A structure holding information on +an in-flight Mutation of a Branch Key +that is never updated in-flight. + +The members are: + +- `Identifier`: a string, The Branch Key (ID) under Mutation +- `CreateTime`: a string, The ISO 8061 UTC Timestamp of when the Mutation was started +- `UUID`: a string, A unique identifier for the Mutation +- `Original`: a binary blob, A commitment of the original mutable properties of the Branch Key +- `Terminal`: a binary blob, A commitment of the terminal mutable properties of the Branch Key +- `Input`: a binary blob, A Description of the input to initialize a Mutation +- `CiphertextBlob`: a binary blob, which MAY be used to protect the Mutation Commitment + +### MutationIndex + +A structure holding information on +an in-flight Mutation of a Branch Key +that is regularly changed in-flight. + +The members are: + +- `Identifier`: a string, The Branch Key (ID) under Mutation +- `CreateTime`: a string, The ISO 8061 UTC Timestamp of when the Mutation was started +- `UUID`: a string, A unique identifier for the Mutation +- `PageIndex`: a binary blob, Indirectly describes which items of the Branch Key have already been Mutated +- `CiphertextBlob`: a binary blob, which MAY be used to protect the Mutation Index + +### OverWriteMutationIndex + +A structure, much like +[OverWriteEncryptedHierarchicalKey](../../framework/key-store/key-storage.md#overwriteencryptedhierarchicalkey), +that affords a conditioned update to the Mutation Index. + +The members are: + +- `Index`: The new Mutation Index +- `Old`: The last read Mutation Index + +## New Operations + +The following new Operations are added: + +### GetItemsForInitializeMutation + +Retrieves the items necessary to initialize a Mutation, +while checking for any in-flight Mutations. + +### WriteInitializeMutation + +Atomically updates the Active Items +of a Branch Key along with +the Mutation Commitment and Mutation Index. + +### WriteAtomicMutation + +Atomically updates all the Items of a Branch Key. + +#### QueryForVersions + +Query Storage for a page of version (decrypt only) items of a Branch Key. + +### WriteMutatedVersions + +Atomically updates a page of version (decrypt only) +items while maintaining the Mutation Commitment and Mutation Index. + +### GetMutation + +Check for Mutation Commitment on a Branch Key ID. +If one exists, returns the Mutation Commitment. +Otherwise, returns nothing. + +### DeleteMutation + +Deletes an existing Mutation Commitment & Index. + +#### WriteMutationIndex + +Creates a Mutation Index, conditioned on the Mutation Commitment. diff --git a/framework/key-store/dynamodb-key-storage.md b/framework/key-store/dynamodb-key-storage.md index a29bd2aa..f4014ee1 100644 --- a/framework/key-store/dynamodb-key-storage.md +++ b/framework/key-store/dynamodb-key-storage.md @@ -156,20 +156,63 @@ If the record does not contain the defined fields, this operation MUST fail. A branch key record MUST include the following key-value pairs: 1. `branch-key-id` : Unique identifier for a branch key; represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) -1. `type` : One of the following; represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +2. `type` : One of the following; represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) - The string literal `"beacon:ACTIVE"`. Then `enc` is the wrapped beacon key. - The string `"branch:version:"` + `version`, where `version` is the Branch Key Version. Then `enc` is the wrapped branch key. - The string literal `"branch:ACTIVE"`. Then `enc` is the wrapped beacon key of the active version. Then -1. `version` : Only exists if `type` is the string literal `"branch:ACTIVE"`. +3. `version` : Only exists if `type` is the string literal `"branch:ACTIVE"`. Then it is the Branch Key Version. represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) -1. `enc` : Encrypted version of the key; +4. `enc` : Encrypted version of the key; represented as [AWS DDB Binary](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) -1. `kms-arn`: The AWS KMS Key ARN used to generate the `enc` value. +5. `kms-arn`: The AWS KMS Key ARN used to generate the `enc` value. represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) -1. `create-time`: Timestamp in ISO 8601 format in UTC, to microsecond precision. +6. `create-time`: Timestamp in ISO 8601 format in UTC, to microsecond precision. Represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) -1. `hierarchy-version`: Version of the hierarchical keyring; +7. `hierarchy-version`: Version of the hierarchical keyring; represented as [AWS DDB Number](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) A branch key record MAY include [custom encryption context](../branch-key-store.md#custom-encryption-context) key-value pairs. These attributes should be prefixed with `aws-crypto-ec:` the same way they are for [AWS KMS encryption context](../branch-key-store.md#encryption-context). + +## Mutation Commitment DDB Item Format + +A Mutation Commitment DDB Item MUST include ONLY the following key-value pairs: + +1. `branch-key-id` : Unique identifier for a branch key; + represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +2. `type` : the constant `branch:MUTATION_COMMITMENT` + represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +3. `create-time` : Timestamp in ISO 8601 format in UTC, to microsecond precision + represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +4. `hierarchy-version` : Version of the hierarchical keyring; + represented as [AWS DDB Number](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +5. `input` : A Description of the input to initialize a Mutation; + represented as [AWS DDB Binary](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +6. `original` : A commitment of the original mutable properties of the Branch Key; + represented as [AWS DDB Binary](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +7. `terminal` : A commitment of the terminal mutable properties of the Branch Key; + represented as [AWS DDB Binary](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +8. `uuid` : A unique identifier for the Mutation + represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +9. `enc` : `enc` : The output of the Key Store Admin's System Key, + which MAY protect the record (depending on the System Key configuration); + represented as [AWS DDB Binary](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) + +## Mutation Index DDB Item Format + +A Mutation Index DDB Item MUST include ONLY the following key-value pairs: + +1. `branch-key-id` : Unique identifier for a branch key; + represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +2. `type` : the constant `branch:MUTATION_INDEX` + represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +3. `create-time` : Timestamp in ISO 8601 format in UTC, to microsecond precision + represented as [AWS DDB String](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +4. `hierarchy-version` : Version of the hierarchical keyring; + represented as [AWS DDB Number](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +5. `pageIndex` : The `type` of the latest Branch Key Item modified, + encoded by the Key Store Admin into a binary; + represented as [AWS DDB Binary](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) +6. `enc` : The output of the Key Store Admin's System Key, + which MAY protect the record (depending on the System Key configuration); + represented as [AWS DDB Binary](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes) diff --git a/framework/key-store/key-storage.md b/framework/key-store/key-storage.md index 5640c877..d48678e9 100644 --- a/framework/key-store/key-storage.md +++ b/framework/key-store/key-storage.md @@ -5,10 +5,12 @@ ## Version -0.2.0 +0.3.0 ### Changelog +- 0.3.0 + - [Branch Key Storage Changes for Mutating Branch Keys](../../changes/2025-01-17_key-store-storage-mutations/background.md) - 0.2.0 - [Mitigate Update Race in the Branch Key Store](../../changes/2025-01-16_key-store-mitigate-update-race/background.md) - 0.1.0 @@ -71,8 +73,74 @@ the UTF8 Encoded value of the version of the branch key. A structure that holds two related [EncryptedHierarchicalKeys](#encryptedhierarchicalkey): -- Item: the [EncryptedHierarchicalKey](#encryptedhierarchicalkey) that will be written -- Old: the [EncryptedHierarchicalKey](#encryptedhierarchicalkey) that was read and is presumed to be the currently persisted item that will be replaced by `Item`. +- `Item`: the [EncryptedHierarchicalKey](#encryptedhierarchicalkey) that will be written +- `Old`: the [EncryptedHierarchicalKey](#encryptedhierarchicalkey) that was read and is presumed to be the currently persisted item that will be replaced by `Item`. + +Both `Item` and `Old` MUST have the same Branch Key ID and Type. + +### EncryptedHierarchicalKeys + +A List of +[EncryptedHierarchicalKey](../../framework/key-store/key-storage.md#encryptedhierarchicalkey). + +### MutationCommitment + +A structure holding information on +an in-flight Mutation of a Branch Key +that is never updated in-flight. + +The members are: + +- `Identifier`: a string, The Branch Key (ID) under Mutation +- `CreateTime`: a string, The ISO 8061 UTC Timestamp of when the Mutation was started +- `UUID`: a string, A unique identifier for the Mutation +- `Original`: a binary blob, A commitment of the original mutable properties of the Branch Key +- `Terminal`: a binary blob, A commitment of the terminal mutable properties of the Branch Key +- `Input`: a binary blob, A Description of the input to initialize a Mutation +- `CiphertextBlob`: a binary blob, which MAY be used to protect the Mutation Commitment + +See [Dynamodb Key Storage's Mutation Commitment DDB Item Format](./dynamodb-key-storage.md#mutation-commitment-ddb-item-format) +for the DynamoDB Item representation. + +### MutationIndex + +A structure holding information on +an in-flight Mutation of a Branch Key +that is regularly changed in-flight. + +The members are: + +- `Identifier`: a string, The Branch Key (ID) under Mutation +- `CreateTime`: a string, The ISO 8061 UTC Timestamp of when the Mutation was started +- `UUID`: a string, A unique identifier for the Mutation +- `PageIndex`: a binary blob, Indirectly describes which items of the Branch Key have already been Mutated +- `CiphertextBlob`: a binary blob, which MAY be used to protect the Mutation Index + +See [Dynamodb Key Storage's Mutation Index DDB Item Format](./dynamodb-key-storage.md#mutation-index-ddb-item-format) +for the DynamoDB Item representation. + +### OverWriteMutationIndex + +A structure, much like +[OverWriteEncryptedHierarchicalKey](../../framework/key-store/key-storage.md#overwriteencryptedhierarchicalkey), +that affords a conditioned update to the Mutation Index. + +The members are: + +- `Index`: The new Mutation Index +- `Old`: The last read Mutation Index + +### WriteInitializeMutationVersion + +A structure that allows for a Mutation to either +create a new Version (DECRYPT*ONLY) of a Branch Key +or update the Active's Version (DECRYPT_ONLy) of a Branch Key, +via an \_Optimistic Lock*. + +The members are: + +- `rotate`: An [EncryptedHierarchicalKey](#encryptedhierarchicalkey) with a [type](#type) of HierarchicalSymmetricVersion +- `mutate`: An [OverWriteEncryptedHierarchicalKey](#overwriteencryptedhierarchicalkey) with both `Item` and `Old` with [type](#type) of HierarchicalSymmetricVersion ## Interface @@ -84,6 +152,14 @@ The KeyStorageInterface MUST support the following operations: - [GetEncryptedBranchKeyVersion](#getencryptedbranchkeyversion) - [GetEncryptedBeaconKey](#getencryptedbeaconkey) - [GetKeyStorageInfo](#getkeystorageinfo) +- [GetItemsForInitializeMutation](#getitemsforinitializemutation) +- [WriteInitializeMutation](#writeinitializemutation) +- [WriteAtomicMutation] +- [QueryForVersions] +- [WriteMutatedVersions] +- [GetMutation] +- [DeleteMutation] +- [WriteMutationIndex] ### WriteNewEncryptedBranchKey @@ -94,7 +170,7 @@ The WriteNewEncryptedBranchKey caller MUST provide: - An [EncryptedHierarchicalKey](#encryptedhierarchicalkey) with a [type](#type) of ActiveHierarchicalSymmetricBeacon All three keys need to be written together with an atomic transactional write. -See the [default key stores's write new key to store specification](./default-key-storage-interface.md#writenewencryptedbranchkey) for more details about what storage properties are expected. +See the [default key store's write new key to store specification](./default-key-storage-interface.md#writenewencryptedbranchkey) for more details about what storage properties are expected. ### WriteNewEncryptedBranchKeyVersion @@ -104,7 +180,7 @@ The WriteNewEncryptedBranchKeyVersion caller MUST provide: - An [EncryptedHierarchicalKey](#encryptedhierarchicalkey) with a [type](#type) of HierarchicalSymmetricVersion Both keys need to be written together with a consistent transactional write. -See [default key stores's write new branch key version to store specification](./default-key-storage-interface.md#writenewencryptedbranchkeyversion) for more details about what storage properties are expected. +See [default key store's write new branch key version to store specification](./default-key-storage-interface.md#writenewencryptedbranchkeyversion) for more details about what storage properties are expected. ### GetEncryptedActiveBranchKey @@ -124,3 +200,120 @@ It MUST return an [EncryptedHierarchicalKey](#encryptedhierarchicalkey). ### GetKeyStorageInfo It MUST return the physical table name. + +### GetItemsForInitializeMutation + +The `GetItemsForInitializeMutation` caller MUST provide an `Identifer` that is the Branch Key ID. + +The operation MUST fetch the following from the storage, +if they exist: + +- [`MutationCommitment`](#mutationcommitment) for the `Identifier` +- [`MutationIndex`](#mutationindex) for the `Identifier` + +The operation MUST fetch the following from the storage: + +- The Active Branch Key for the `Identifier` +- The Active Beacon Key for the `Identifier` + +If the Active Branch Key or Active Beacon Key are not found, +an error MUST be thrown. + +If any of the items are malformed, +an error MUST be thrown. + +Otherwise, +at least the Active Branch and Active Beacon Keys MUST be +returned as [EncryptedHierarchicalKey](#encryptedhierarchicalkey), +along with the Mutation Commitment or Mutation Index IF they are present. + +### WriteInitializeMutation + +The caller MUST provide: + +- `ActiveItem`: An [OverWriteEncryptedHierarchicalKey](#overwriteencryptedhierarchicalkey) with [type](#type) of ActiveHierarchicalSymmetricVersion +- `BeaconItem`: An [OverWriteEncryptedHierarchicalKey](#overwriteencryptedhierarchicalkey) with [type](#type) of ActiveHierarchicalSymmetricBeacon +- `Version`: An [WriteInitializeMutationVersion](#writeinitializemutationversion) +- `MutationCommitment`: A [MutationCommitment](#mutationcommitment) +- `MutationIndex`: A [MutationIndex](#mutationindex) + +All of these structures MUST have the same Branch Key ID/Identifier. + +The operation MUST atomically persist all 5 items +in one write. + +[//]: # "- the CiphertextBlob of the Branch Key ID's Active is equal to `ActiveItem`'s `old` CiphertextBlob." +[//]: # "- the CiphertextBlob of the Branch Key ID's Beacon is equal to `BeaconItem`'s `old` CiphertextBlob." +[//]: # "- If the `Version` is `mutate`, the CiphertextBlob of the Version is equal to `mutate`'s `old` CiphertextBlob." + +The write MUST only succeed if: + +- there is no existing Mutation Commitment for the Branch Key ID +- the ACTIVE value in Storage is the `ActiveItem`'s `old`. +- the Beacon value in Storage is the `BeaconItem`'s `old`. +- If the `Version` is `mutate`, the value of the Version Item in Storage is the `mutate`'s `old`. +- If the `Version` is `rotate`, there is no Branch Key Item with the same `type` value as `rotate`. + +If the write does not succeed because of the above constraints, +the operation MUST yield error; +this error SHOULD suggest a race is occurring. + +The write MUST persist: + +- the `ActiveItem`'s `Item` +- the `BeaconItem`'s `Item` +- if the `Version` is `mutate`, the [EncryptedHierarchicalKey](#encryptedhierarchicalkey), + else the `rotate`'s `Item` +- the `MutationCommitment` +- the `MutationIndex` + +If the write fails for reasons other than the conditions, +the operation MUST yield error; +this error SHOULD suggest why the write failed. + +Otherwise, the operation MUST return a success. + +### WriteAtomicMutation + +The caller MUST provide: + +- `ActiveItem`: An [OverWriteEncryptedHierarchicalKey](#overwriteencryptedhierarchicalkey) with [type](#type) of ActiveHierarchicalSymmetricVersion +- `BeaconItem`: An [OverWriteEncryptedHierarchicalKey](#overwriteencryptedhierarchicalkey) with [type](#type) of ActiveHierarchicalSymmetricBeacon +- `Version`: An [WriteInitializeMutationVersion](#writeinitializemutationversion) +- `Items`: A list of [OverWriteEncryptedHierarchicalKey](#overwriteencryptedhierarchicalkey) with [type](#type) of HierarchicalSymmetricVersion + +All of these structures MUST have the same Branch Key ID/Identifier. + +If the Storage implementation cannot write +the size of `Items` + 3 in an atomic write, +the operation MUST yield an error. + +The operation MUST issue an atomic write that ONLY succeeds if: + +- There is no existing Mutation Commitment for the Branch Key ID. +- There is no existing Mutation Index for the Branch Key ID. +- The ACTIVE value in Storage is the `ActiveItem`'s `old`. +- The Beacon value in Storage is the `BeaconItem`'s `old`. +- If the `Version` is `mutate`, the value of the Version Item in Storage is the `mutate`'s `old`. +- If the `Version` is `rotate`, there is no Branch Key Item with the same `type` value as `rotate`. +- For every [OverWriteEncryptedHierarchicalKey](#overwriteencryptedhierarchicalkey) in `Items`, + the [EncryptedHierarchicalKey](#encryptedhierarchicalkey)'s value in Storage is the `old`. + +If the write does not succeed because of the above constraints, +the operation MUST yield error; +this error SHOULD suggest a race is occurring. + +The atomic write MUST persist: + +- the `ActiveItem`'s `Item` +- the `BeaconItem`'s `Item` +- if the `Version` is `mutate`, the [EncryptedHierarchicalKey](#encryptedhierarchicalkey), + else the `rotate`'s `Item` +- for every [OverWriteEncryptedHierarchicalKey](#overwriteencryptedhierarchicalkey) in `Items`, + the [OverWriteEncryptedHierarchicalKey](#overwriteencryptedhierarchicalkey)'s `Item` + +If the write fails for other reasons, +the operation MUST yield error; +this error SHOULD suggest why the write failed. + +Otherwise, the operation MUST return a success.