Merge pull request #51 from NilFoundation/nil-core-concepts

Nil core concepts

Author: khannanov-nil

docusaurus.config.js

@@ -6,6 +6,9 @@
 
 import { themes as prismThemes } from 'prism-react-renderer';
 
+import remarkMath from 'remark-math';
+import rehypeKatex from 'rehype-katex';
+
 /** @type {import('@docusaurus/types').Config} */
 const config = {
   title: '=nil; Foundation Documentation Portal',
@@ -82,13 +85,24 @@ const config = {
     [
       '@docusaurus/plugin-content-docs',
       {
-        id: 'zksharding',
-        path: 'zksharding',
-        routeBasePath: 'zksharding',
-        sidebarPath: './sidebars.js'
+        id: 'nil',
+        path: 'nil',
+        routeBasePath: 'nil',
+        sidebarPath: './sidebar-nil.js',
+        remarkPlugins: [remarkMath],
+        rehypePlugins: [rehypeKatex],
       }
     ],
   ],
+  stylesheets: [
+    {
+      href: 'https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.css',
+      type: 'text/css',
+      integrity:
+        'sha384-odtC+0UGzzFL/6PNoE8rX/SPcQDXBJ+uRepguP4QkPCm2LBxH3FA3y+fKSiJ+AmM',
+      crossorigin: 'anonymous',
+    },
+  ],
   themeConfig:
     /** @type {import('@docusaurus/preset-classic').ThemeConfig} */
     (
@@ -113,6 +127,11 @@ const config = {
             src: 'img/nil-logo.png',
           },
           items: [
+            {
+              position: 'left',
+              label: '=nil;',
+              to: '/nil/intro'
+            },
             {
               position: 'left',
               label: 'zkLLVM',
@@ -124,11 +143,6 @@ const config = {
               label: 'Proof Market',
               to: '/proof-market/intro'
             },
-            {
-              position: 'left',
-              label: 'zkSharding',
-              to: 'https://nil.foundation/blog/post/nil_zkSharding'
-            },
             {
               position: 'left',
               label: 'Crypto3',

nil/core-concepts/accounts.mdx

@@ -0,0 +1,36 @@
+# Accounts
+
+## Definition
+
+In =nil;, an account is the minimal unit of data for sharding. An account consists of an address, its balance, its storage root and the hash of its source code.
+
+Each execution shard handles only a dedicated part of all accounts in the global state of the cluster while the consensus shard stores the mapping of accounts and their corresponding shards.
+
+:::info[Execution shards]
+
+Each execution shard acts as a separate blockchain that starts from a genesis block and continues until a possible last block. When an account is placed in a shard, its location persists unless [**contract co-location**](./contract-co-location) is employed.
+
+:::
+
+## Security and rollbacks
+
+In the case of malicious nodes taking over a shard (and forcing commitment to corrupted data), validators for all other execution shards must initiate a rollback of all affected accounts. 
+
+The rollback must revert accounts to their last verified state, meaning the state before the corrupted shard had started sending messages to other shards.
+
+There exist two possible methods for rolling back accounts.
+
+* The 'carpet' method where the entire state is reset to the last previous valid state
+* The 'surgical' method where only the corrupted accounts are rolled back
+
+=nil; employs the 'carpet' method: during a rollback, all accounts are reset to their previous valid states regardless of whether they were affected by the attack.
+
+:::tip[Justification]
+
+Rolling back all accounts might seem excessive.
+
+However, attacks propagate faster than state transition proofs are generated. While the cluster is figuring out what accounts to roll back, the attack might grow in size, making the 'surgical' method costly and risky. 
+
+For now, this makes the 'carpet' method preferable despite its limitations. 
+
+:::

nil/core-concepts/contract-co-location.mdx

@@ -0,0 +1,33 @@
+# Contract co-location
+
+## Definition
+
+[**Cross-shard communications**](./shards-parallel-execution#message-passing-checks) are a built-in feature of the =nil; protocol ensuring horizontal scalability with no fragmentation.
+
+However, =nil; also supports contract co-location, which is a technique for ensuring that two accounts are located on the same shard. In this case, interaction between these accounts does not require cross-shard communications.
+
+In plain terms, co-location is a request made to change the value stored under the account's address in [**the mapping of accounts and shards**](./accounts) stored in the consensus shard. 
+
+:::note[Explanation]
+
+When two accounts (Account 1 and Account 2) are co-located, communications between them can occur synchronously similarly to regular Ethereum transactions. 
+
+However, this is not the case if Account 1 and Account 2 are placed in different shards and Account 1 calls a function in Account 2. In this situation, the transaction from Account 1 must create another transaction that subsequently performs an async call to the function in Account 2. This async transaction could then be executed in a future block in the shard where Account 2 is located.
+
+This ensures that sharding does not result in fragmentation.
+
+:::
+
+## Scalability considerations
+
+On the one hand, contract co-location presents a valuable opportunity for developers wanting to ensure the lowest possible transaction processing times for their users.
+
+On the other hand, co-location encourages having as many accounts as possible on one shard, undermining the benefits of horizontal scaling.
+
+This concern, however, is mitigated by basic market mechanics: 
+
+* The more accounts exist on the same shard (and the more transactions are submitted by them), the higher the gas price 
+* The higher the gas price, the higher the cost of each transaction
+* The higher the cost of each transaction, the lower the benefits-to-costs ratio of co-locating additional accounts
+
+It is up to the market participants to determine whether co-location is worth incurring high transaction costs.

nil/core-concepts/shards-parallel-execution.mdx

@@ -0,0 +1,49 @@
+# Shards and parallelized execution
+
+## Definition
+
+Shards are smaller chains that are usually responsible for managing a portion of the global state of the main chain. In =nil;, there exist two types of shards.
+
+* Execution shards process transactions of the accounts allocated to these shards
+* The consensus shard coordinates execution shards and submits ZKPs of the global state to Ethereum
+
+## Execution shards
+
+All work related to processing transactions, storing accounts, and managing the local state occurs at the execution shard level. Execution shards are also tasked with passing message passing checks.
+
+:::info[Mempools]
+
+Although each shard has its own mempool of transactions, there are no restrictions for the participants on what shard they can cooperate with.
+
+:::
+
+### Message passing checks
+
+To support [**cross-shard communications**](../principles#cross-shard-communication), execution shards must record all necessary outgoing messages posted in other execution shards.
+
+Each message contains information about its destination account. If a shard sees that a message targets one of the accounts in the shard, the shard must include this message in its next block. Such outgoing messages are called necessary messages and new blocks are considered invalid unless they include all necessary messages.
+
+:::info
+
+As the number of execution shards grows, this process may be changed so that shards are only tasked with tracking messages from their neighbors. The closeness of shards can be determined by their Hamming distance.
+
+:::
+
+
+## Consensus shard
+
+In contrast to execution shards, [**the consensus shard**](../principles#consensus-shard) is responsible for the following functions:
+
+* Managing global consensus
+* Producing 'master' ZKPs for verifying global state changes
+* Setting the protocol rules and parameters
+
+:::note[Security guarantees]
+
+Although =nil; relies on ZKPs to verify state transitions statelessly, ZKPs can be costly to generate. As a result, a standard security protocol is used to provide security guarantees while ZKPs are generated.
+
+The committee of an execution shard runs a [**Multi-Threshold BFT security protocol**](https://dl.acm.org/doi/10.1145/3460120.3484554) which is a varioation of Sync HotStuff. This is done so that the safety threshold at the local level is not tied to the safety threshold of the cluster.
+
+After running the local consensus protocol, leaders of all committees transmit block digests and quorum certificates to the leader of the consensus shard. The consensus shard leader then proposes a new block to the consensus shard committee, which includes the entire set of validators. These validators run a HotStuff-2 consensus protocol to achieve global consensus.
+
+:::

nil/core-concepts/transaction-lifecycle.mdx

@@ -0,0 +1,54 @@
+# Transaction lifecycle
+
+## Definition
+
+In =nil;, a transaction passes the following stages before reaching finality.
+
+1. A transaction is submitted by an account
+2. Execution shards survey other execution shards to determine the shard where the transaction should be included in a block
+3. The transaction goes to the mempool of an execution shard
+4. The transaction is picked up by a collator that bundles it with other transactions and sends the resulting batch for execution
+5. The transaction is executed and is sent into a block
+6. The block passes local consensus while the shard generates a ZKP
+7. The block with the transaction is sent into storage and the transaction reaches soft finality
+8. The ZKP and the block are sent to the consensus shard that produces a 'master' ZKP
+
+Additional remarks about each of the above stages are given below.
+
+## Stages
+
+### Transaction submission and processing
+
+Execution shards poll other execution shards about outgoing messages. After a new transaction is submitted, its destination shard retrieves it from its shard of origin and places it in its mempool for subsequent execution.
+
+To learn more about cross-shard communication, [**click here**](./shards-parallel-execution#message-passing-checks).
+
+### Passing the collator
+
+The collator then retrieves the transaction from the mempool and bundles it together with other transactions. The resulting bundle is executed, and the transaction is included in the new block produced by the execution shard.
+
+At the end of this stage, the block with the transaction is sent for verification via local consensus. [**Click here**](./shards-parallel-execution#consensus-shard) to learn more about local consensus.
+
+### Achieving soft finality
+
+After the block containing the transaction is built, execution shards produce a ZKP and send it to the consensus shard along with block hashes. In the meantime, the block with the transaction reaches storage, and the transaction achieves soft finality.
+
+:::info
+
+When =nil; supports communications with L1, a special application running on top of the consensus shard will submit [**data availability transactions**](../specification/data-availability) to Ethereum. 
+
+In this case, transactions will reach soft finality when the corresponding data availability transactions are verified in Ethereum slots.
+
+:::
+
+:::note[ZKPs]
+
+ZKPs allow for trustless verification of state transitions inside execution shards and the consensus shard. ZKP generation is handled by dedicated participants (proof producers). To learn more about this process, [**click here**](../specification/finality).
+
+:::
+
+### Achieving hard finality
+
+The consensus shard also verifies the proof from the shard with the transaction and creates a 'master' ZKP. 
+
+When this feature is supported, the 'master' ZKP will be sent to the =nil; smart contract on Ethereum where it will be verified. At this stage, the transaction will achieve absolute finality.

nil/intro.mdx

@@ -0,0 +1,62 @@
+import Thesis from '@site/static/img/nil/nil-icons/thesis.png'
+import Principles from '@site/static/img/nil/nil-icons/principles.png'
+import {Card, CardSection} from '@site/src/components/CardSection'
+import Accounts from '@site/static/img/nil/nil-icons/accounts.png'
+import Transaction from '@site/static/img/nil/nil-icons/transaction.png'
+import Shards from '@site/static/img/nil/nil-icons/shards.png'
+import Finality from '@site/static/img/nil/nil-icons/finality.png'
+import Data from '@site/static/img/nil/nil-icons/data.png'
+import Contract from '@site/static/img/nil/nil-icons/contract.png'
+import Sequencing from '@site/static/img/nil/nil-icons/sequencing.png'
+
+# Overview
+
+## Definition
+
+=nil; is an Ethereum layer-2 blockchain powered by zkSharding. The design of =nil; embodies three core qualities:
+
+* **Horizontal scalability**. =nil; unlocks the power of many machines by partitioning state across shards with cross-shard communications being a built-in feature of the protocol.
+* **Modularity**. =nil; acts as an execution layer while using Ethereum for data availability and consensus.
+* **Verifiable security**. The entire =nil; network can be verified with a single zero-knowledge proof attesting to the correctness of all shards.
+
+The design of =nil; transcends the typical monolithic vs. modular discourse as it emphasizes horizontal scalability above all else. =nil; scales out beyond the constraints of a single machine's processing power, by leveraging zkSharding, a new type of sharding architecture. In zkSharding, execution shards produce zero-knowledge proofs (ZKPs) verifying intra-shard state transitions while the state of the cluster is managed by a designated consensus shard. The consensus shard is responsible for synchronizing execution shards and producing a 'master' ZKP that is sent to Ethereum for global state verification.
+
+=nil; offers an effective strategy for scaling Ethereum while avoiding the limitations typically associated with the modular approach such as state and liquidity fragmentation.
+
+## Structure of the documentation
+
+### Introduction
+
+Learn the basics of what =nil; is and how it works.
+
+<CardSection>
+    <Card icon={<img src={Thesis}/>} id='thesis' title='Thesis' description='Read about the "how" and "why" of =nil;' to='./thesis'/>
+    <Card icon={<img src={Principles}/>} id='principles' title='Principles' description='Grasp the fundamentals of the architecture of =nil;' to='./principles'/>
+</CardSection>
+
+### Core concepts
+
+Familiarize yourself with the key components of the architecture of =nil;.
+
+<CardSection>
+    <Card icon={<img src={Accounts}/>} id='accounts' title='Accounts' description='Learn about how =nil; handles accounts' to='./core-concepts/accounts'/>
+    <Card icon={<img src={Transaction}/>} id='lifecycle' title='Transaction lifecycle' description='Follow a transaction to finality' to='./core-concepts/transaction-lifecycle'/>
+    <Card icon={<img src={Shards}/>} id='shards' title='Shards and parallelized execution' description='See how =nil; achieves scalability' to='./core-concepts/shards-parallel-execution'/>
+    <Card icon={<img src={Contract}/>} id='contract' title='Contract co-location' description='Ensure account placement on the same shard' to='./core-concepts/contract-co-location'/>
+</CardSection>
+
+### Specification
+
+Learn more about how =nil; plans to resolve issues central to any blockchain.
+
+:::info[WIP provisions]
+
+The materials in this section may describe features and components that are not yet part of =nil;.
+
+:::
+
+<CardSection>
+    <Card icon={<img src={Sequencing}/>} id='sequencing' title='Sequencing' description='Read about how =nil; adopts the PBS model' to='./specification/sequencing'/>
+    <Card icon={<img src={Finality}/>} id='finality' title='Finality' description='Learn about the strong security guarantees of =nil;' to='./specification/finality'/>
+    <Card icon={<img src={Data}/>} id='data' title='Data availability' description='See how =nil; achieves DA at all levels' to='./specification/data-availability'/>
+</CardSection>

nil/principles.mdx

@@ -0,0 +1,56 @@
+import Flows from '@site/static/img/nil/flows.png'
+
+# Principles
+
+This page offers a high-level explanation of the architecture of =nil;.
+
+## Flows
+
+The following diagram shows how =nil; organizes execution shards, the main shard, the transfer of ZKPs, and communications with Ethereum.
+
+<img src={Flows}/ >
+
+Each execution shard is responsible for generating a ZKP for their respective state change. While ZKPs are generated, execution shards employ a protocol based on [**Sync HotStuff**](https://eprint.iacr.org/2019/270.pdf) to reach consensus. Afterward, the proofs are verified via a [**HotStuff2-based**](https://eprint.iacr.org/2023/397.pdf) global consensus protocol. 
+
+The main shard uses all received ZKPs to create a 'master' ZKP that is then transmitted to Ethereum. The global state of =nil; is verified and transactions reach finality.
+
+## Consensus shard
+
+The consensus shard performs two essential functions:
+
+* The shard stores and sets consensus rules and parameters
+* The shard stores hashes of the most recent blocks from execution shards and ensures synchronization between execution shards
+
+:::info
+
+There is only one consensus shard in the cluster.
+
+:::
+
+## Execution shards
+
+As implied by their name, execution shards process transactions. 
+
+Each shard is allocated with a subset of accounts from the global state. The shard is responsible for executing transactions made to and from these accounts. Consensus is handled by a committee of randomly selected validators.
+
+To learn more about execution shards, [**click here**](./core-concepts/shards-parallel-execution#execution-shards).
+
+:::info[Validator rotation]
+
+At the end of each epoch, a new set of validators is generated for each execution shard based on the Verifiable Secret Sharing (VSS) scheme.
+
+:::
+
+## Cross-shard communication
+
+Instead of handling cross-shard communication via bridges (which are often insecure and unstable), =nil; integrates such communications in its protocol.
+
+Each execution shard is tasked with surveying other execution shards and retrieving necessary transactions. A transaction is necessary for a shard if the shard holds the account that the transaction calls. Blocks within a shard are considered invalid unless they include all necessary messages. 
+
+To learn more about cross-shard communication, [**click here**](./core-concepts/shards-parallel-execution#message-passing-checks).
+
+## Contract co-location
+
+=nil; also offers participants the option of contract co-location, which is a request for two accounts to be placed in the same shard.
+
+Co-location ensures that transaction execution speeds remain high. To learn more about contract co-location and its benefits, [**click here**](./core-concepts/contract-co-location).