Skip to content

Commit 08f063a

Browse files
mratsimmratsim
authored
[RFC - Doc] State of Nimbus block & attestation flows (#2351)
* Expand documentation on block flow [skip ci] * address review comments [skip ci] * Update with GossipFlow out [skip ci] * LocalBlockProposer -> LocalValidatorDuties + WeakSubjectivitySync * First outline of attestation flow * finish up prose
1 parent 9c241f8 commit 08f063a

8 files changed

+401
-62
lines changed

‎beacon_chain/rpc/beacon_api.nim

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -392,7 +392,7 @@ proc installBeaconApiHandlers*(rpcServer: RpcServer, node: BeaconNode) =
392392
if head.slot >= blck.message.slot:
393393
node.network.broadcast(getBeaconBlocksTopic(node.forkDigest), blck)
394394
# The block failed validation, but was successfully broadcast anyway.
395-
# It was not integrated into the beacon node''s database.
395+
# It was not integrated into the beacon node's database.
396396
return 202
397397
else:
398398
let res = proposeSignedBlock(node, head, AttachedValidator(), blck)

‎docs/attestation_flow.dot

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
digraph architecture{
2+
node [shape = invhouse]; Eth2RPC GossipSub;
3+
node [shape = octagon]; AttestationPool Eth2Processor DoppelgangerDetection;
4+
node [shape = doubleoctagon]; AsyncQueueAttestationEntry AsyncQueueAggregateEntry;
5+
node [shape = octagon]; ForkChoice Quarantine;
6+
{rank = same; Eth2RPC GossipSub;}
7+
8+
AsyncQueueAttestationEntry [label="AsyncQueue[AttestationEntry]"];
9+
AsyncQueueAggregateEntry [label="AsyncQueue[AggregateEntry]"];
10+
11+
GossipSub -> Eth2Processor [label="getAttestationTopic: attestationValidator->attestationPool.validateAttestation()"];
12+
GossipSub -> Eth2Processor [label="node.topicAggregateAndProofs: aggregateValidator->attestationPool.validateAggregate()"];
13+
14+
Eth2Processor -> AttestationAggregation [label="validateAttestation() validateAggregate()"];
15+
Eth2Processor -> AttestationPool [label="attestation_pool.selectHead()"];
16+
Eth2Processor -> DoppelgangerDetection
17+
18+
Eth2Processor -> AsyncQueueAttestationEntry [label="Move to queue after P2P validation via validateAttestation()"];
19+
Eth2Processor -> AsyncQueueAggregateEntry [label="Move to queue after P2P validation via validateAggregate()"];
20+
AsyncQueueAttestationEntry -> AttestationPool [label="runQueueProcessingLoop() -> processAttestation()\n->addAttestation (to fork choice)"];
21+
AsyncQueueAggregateEntry -> AttestationPool [label="runQueueProcessingLoop() -> processAggregate()\n->addAttestation (to fork choice)"];
22+
23+
AttestationPool -> Quarantine [label="Missing block targets"];
24+
AttestationPool -> ForkChoice [label="addAttestation(), prune(), selectHead()"];
25+
Eth2RPC -> AttestationPool [dir="back" label="get_v1_beacon_pool_attestations() -> attestations iterator"]
26+
27+
LocalValidatorDuties -> AttestationPool [label="Vote for head\nMake a block (with local validators attestations)"];
28+
LocalValidatorDuties -> AttestationAggregation [label="aggregate_attestations()"];
29+
AttestationAggregation -> AttestationPool
30+
}

‎docs/attestation_flow.md

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Attestation Flow
2+
3+
This is a WIP document to explain the attestation flows.
4+
5+
## Validation & Verification flow
6+
7+
Important distinction:
8+
- We distinguish attestation `validation` which is defined in the P2P specs:
9+
- single: https://github.com/ethereum/eth2.0-specs/blob/v1.0.0/specs/phase0/p2p-interface.md#beacon_attestation_subnet_id
10+
- aggregated: https://github.com/ethereum/eth2.0-specs/blob/v1.0.0/specs/phase0/p2p-interface.md#beacon_aggregate_and_proof
11+
A validated attestation or aggregate can be forwarded on gossipsub.
12+
- and we distinguish `verification` which is defined in consensus specs:
13+
https://github.com/ethereum/eth2.0-specs/blob/v1.0.0/specs/phase0/beacon-chain.md#attestations
14+
An attestation needs to be verified to enter fork choice, or be included in a block.
15+
16+
To be clarified: from the specs it seems like gossip attestation validation is a superset of consensus attestation verification.
17+
18+
### Inputs
19+
20+
Attestations can be received from the following sources:
21+
- Gossipsub
22+
- Aggregate: `/eth2/{$forkDigest}/beacon_aggregate_and_proof/ssz` stored `topicAggregateAndProofs` field of the beacon node
23+
- Unaggregated `/eth2/{$forkDigest}/beacon_attestation_{subnetIndex}/ssz`
24+
- Included in blocks received
25+
- the NBC database (within a block)
26+
- a local validator vote
27+
- Devtools: test suite, ncli, fuzzing
28+
29+
The related base types are
30+
- Attestation
31+
- IndexedAttestation
32+
33+
The base types are defined in the Eth2 specs.
34+
On top, Nimbus builds new types to represent the level of trust and validation we have with regards to each BeaconBlock.
35+
Those types allow the Nim compiler to help us ensure proper usage at compile-time and zero runtime cost.
36+
37+
#### TrustedAttestation & TrustedIndexedAttestation
38+
39+
An attestation or indexed_attestation that was verified as per the consensus spec or that was retrieved from the database or any source of trusted blocks is considered trusted. In practice we assume that its signature was already verified.
40+
41+
_TODO Note: it seems like P2P validation is a superset of consensus verification in terms of check and that we might use TrustedAttestation earlier in the attestation flow._
42+
43+
## Attestation processing architecture
44+
45+
How the various modules interact with block is described in a diagram:
46+
47+
![./attestation_flow.png](./attestation_flow.png)
48+
49+
Note: The Eth2Processor as 2 queues for attestations (before and after P2P validation) and 2 queues for aggregates. The diagram highlights that with separated `AsyncQueue[AttestationEntry]` and `AsyncQueue[AggregateEntry]`
50+
51+
### Gossip flow in
52+
53+
Attestatios are listened to via the gossipsub topics
54+
- Aggregate: `/eth2/{$forkDigest}/beacon_aggregate_and_proof/ssz` stored `topicAggregateAndProofs` field of the beacon node
55+
- Unaggregated `/eth2/{$forkDigest}/beacon_attestation_{subnetIndex}/ssz`
56+
57+
They are then
58+
- validated by `validateAttestation()` or `validateAggregate()` in either `attestationValidator()` or `aggregateValidator()`
59+
according to spec
60+
- https://github.com/ethereum/eth2.0-specs/blob/v1.0.0/specs/phase0/p2p-interface.md#beacon_aggregate_and_proof
61+
- https://github.com/ethereum/eth2.0-specs/blob/v1.0.0/specs/phase0/p2p-interface.md#attestation-subnets
62+
- It seems like P2P validation is a superset of consensus verification as in `process_attestation()`:
63+
- https://github.com/ethereum/eth2.0-specs/blob/v1.0.0/specs/phase0/beacon-chain.md#attestations
64+
- Enqueued in
65+
- `attestationsQueue: AsyncQueue[AttestationEntry]`
66+
- `aggregatesQueue: AsyncQueue[AggregateEntry]`
67+
- dropped in case of error
68+
69+
### Gossip flow out
70+
71+
- After validation in `attestationValidator()` or `aggregateValidator()` in the Eth2Processor
72+
- Important: P2P validation is different from verification at the consensus level.
73+
- We jump into libp2p/protocols/pubsub/pubsub.nim in the method `validate(PubSub, message)`
74+
- which was called by `rpcHandler(GossipSub, PubSubPeer, RPCMsg)`
75+
76+
### Eth2 RPC in
77+
78+
There is no RPC for attestations but attestations might be included in synced blocks.
79+
### Comments
80+
81+
### Sync vs Steady State
82+
83+
During sync the only attestation we receive are within synced blocks.
84+
Afterwards attestations come from GossipSub
85+
86+
#### Bottlenecks during sync
87+
88+
During sync, attestations are not a bottleneck, they are a small part of the large block processing.
89+
90+
##### Backpressure
91+
92+
The `attestationsQueue` to store P2P validated attestations has a max size of `TARGET_COMMITTEE_SIZE * MAX_COMMITTEES_PER_SLOT` (128 * 64 = 8192)
93+
94+
The `aggregatesQueue` has a max size of `TARGET_AGGREGATORS_PER_COMMITTEE * MAX_COMMITTEES_PER_SLOT` (16 * 64 = 1024)
95+
96+
##### Latency & Throughput sensitiveness
97+
98+
We distinguish an aggregated attestation:
99+
- aggregation is done according to eth2-spec rules, usually implies diffusion of aggregate signature and the aggregate public key is computed on-the-fly.
100+
- and a batched validation (batching done client side), implies aggregation of signatures and public keys are done on-the-fly
101+
102+
During sync attestations are included in blocks and so do not require gossip validation,
103+
they are also aggregated per validators and can be batched with other signatures within a block.
104+
105+
After sync, attestations need to be rebroadcasted fast to:
106+
- maintain the quality of the GossipSub mesh
107+
- not be booted by peers
108+
109+
Attestations need to be validated or aggregated fast to avoid delaying other networking operations, in particular they are bottleneck by cryptography.
110+
111+
The number of attestation to process grows with the number of peers. An aggregated attestation is as cheap to process as a non-aggregated one. Batching is worth it even with only 2 attestations to batch.

‎docs/attestation_flow.png

148 KB
Loading

‎docs/block_flow.dot

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
digraph architecture{
2+
node [shape = invhouse]; Eth2RPC GossipSub WeakSubjectivitySync;
3+
node [shape = octagon]; SyncManager SyncProtocol RequestManager;
4+
SyncManager [label="SyncManager (Sync in)"];
5+
node [shape = doubleoctagon] SharedBlockQueue;
6+
{rank = same; SyncManager RequestManager;}
7+
{rank = same; Eth2RPC GossipSub WeakSubjectivitySync;}
8+
9+
WeakSubjectivitySync [label="WeakSubjectivitySync (impl TBD)"];
10+
11+
node [shape = octagon]; Eth2Processor RequestManager;
12+
node [shape = octagon]; ChainDAG Quarantine Clearance;
13+
14+
Eth2RPC -> SyncProtocol [dir=both]
15+
SyncProtocol -> SyncManager [dir=both, label="beaconBlocksByRange() (mixin)"]
16+
17+
GossipSub -> Eth2Processor [label="node.topicBeaconBlocks: blockValidator->isValidBeaconBlock (no transition or signature check yet)\nthen enqueued in blocksQueue"];
18+
GossipSub -> Eth2Processor [dir=back, label="node.topicBeaconBlocks: blockValidator()->ValidationResult.Accept->libp2p/gossipsub.nim\nvalidate() in rpcHandler()"];
19+
20+
Eth2Processor -> Clearance [label="storeBlock(): enqueue in clearance/quarantine and callback to fork choice"];
21+
SyncProtocol -> RequestManager [dir=both, label="fetchAncestorBlocksFromNetwork()"];
22+
23+
SyncManager -> SharedBlockQueue [dir=both, label="Eth2Processor.blocksQueue\n== SyncManager.outQueue (shared state!)"];
24+
Eth2Processor -> SharedBlockQueue [dir=both, label="Eth2Processor.blocksQueue\n== RequestManager.outQueue (shared state!)"];
25+
SharedBlockQueue -> RequestManager [dir=both, label="SyncManager.outQueue\n== RequestManager.outQueue (shared state!)"];
26+
27+
LocalValidatorDuties -> Clearance
28+
29+
RequestManager -> Quarantine [dir=back, label="Retrieve missing ancestors"]
30+
Clearance -> Quarantine
31+
Clearance -> ChainDAG
32+
33+
Eth2Processor -> ForkChoice
34+
LocalValidatorDuties -> ForkChoice
35+
36+
node [shape = cylinder]; BeaconChainDB;
37+
ChainDAG -> BeaconChainDB [dir=both]
38+
39+
SyncProtocol -> ChainDAG [dir=back, label="Sync out: getBlockRange()\nbeaconBlocksByRoot()\n"]
40+
}

0 commit comments

Comments
 (0)