jupyter: add base and tutorial

Author: willcl-ark

01_overview-and-development.adoc

@@ -776,33 +776,81 @@ Subsequent sections will contain various exercises related to their subject area
 To prepare for this we will begin with the following exercises which will ensure that our environment is ready:
 
 . Build Bitcoin Core from source
-** Clone Bitcoin Core repository from GitHub
-** Check out the latest release tag (e.g. `v24.0.1`)
-** Install any dependencies required for your system
-** Follow the build instructions to compile the program
-** Run `make check` to run the unit tests
-** Follow the documentation to install dependencies required to run the functional tests
-*** Run the functional tests
-. Run a `bitcoind` node in regtest mode and control it using the `cli` tool
-** `./src/bitcoind -regtest` will start bitcoind in regtest mode. You can then control it using `./src/bitcoin-cli -regtest -getinfo`
+- [ ] Clone Bitcoin Core repository from GitHub
+- [ ] Check out the latest release tag (e.g. `v24.0.1`)
+- [ ] Install any dependencies required for your system
+- [ ] Follow the build instructions to compile the program
+- [ ] Run `make check` to run the unit tests
+- [ ] Follow the documentation to install dependencies required to run the functional tests
+- [ ] Run the functional tests
+. Run a `bitcoind` node in regtest mode and control it using the `cli` tool +
++
+TIP: `./src/bitcoind -regtest` will start bitcoind in regtest mode. You can then control it using `./src/bitcoin-cli -regtest -getinfo`
 . Run and control a Bitcoin Core node using the `TestShell` python class from the test framework in a Jupyter notebook
 ** See <<test_shell_nodes, Running nodes via Test Framework>> for more information on how to do this
+. Review a Pull Request from the repo
+- [ ] Find a PR (which can be open or closed) on GitHub which looks interesting and/or accessible
+- [ ] Checkout the PR locally
+- [ ] Review the changes
+- [ ] Record any questions that arise during code review
+- [ ] Build the PR
+- [ ] Test the PR
+- [ ] Break a test / add a new test
+- [ ] Leave review feedback on GitHub, possibly including:
++
+ACK/NACK
++
+Approach
++
+How you reviewed it
++
+Your system specifications if relevant
++
+Any suggested nits
 
 [#test_shell_nodes]
 .Running nodes via Test Framework
 ****
+[discrete]
+== Why
+
 Using Bitcoin Core's Test Framework means that nodes can be started, controlled and stopped using a python control class.
-Additionally, they are run in a temporary directory which is automatically removed by the system later, if not done
+Additionally, they are run in a temporary directory which is automatically removed by the operating system, if not done
 manually.
 
-In addition to this, the `TestShell` class has an extremely similar interface to `bitcoin-cli`, where `bitcoin-cli` commands are mapped to `TestShell` methods, and arguments can be supplied positionally or as named values.
+In addition to this, the `TestShell` class has an extremely similar interface to `bitcoin-cli`, where most `bitcoin-cli` commands are mapped to `TestShell` methods, and arguments can be supplied positionally or as named values.
+Specifically, all RPCs are available to `TestShell`.
+However, certain `bitcoin-cli` commands, for example `-getinfo` require `bitcoin-cli` to call multiple RPCs and combine the results into something more user-friendly.
+These commands are not available to `TestShell`, but you can re-create them yourself using the source code!
+
+When `TestShell` is combined with a jupyter notebook the result is easy-to-setup ephemeral nodes where iteration on complex commands is more pleasant than in the shell, and complex sequences of commands can be reproduced without having to write bash scripts or use shell history.
+
+Once a complex command or sequence of commands is established, they can generally be translated to `bitcoin-cli` commands without much difficulty.
+
+[discrete]
+== How
+
+You **MUST** have a compiled `bitcoind` binary in the Bitcoin Core source directory.
+You can use any recent supported version of Bitcoin Core.
 
-When combined with a jupyter notebook, the result is easy-to-setup ephemeral nodes where iteration on complex commands is more pleasant, and complex sequences of commands can be reproduced without having to write bash scripts or use shell history.
+In order to add startup (`bitcoind` program) options to our node(s) we need https://github.com/bitcoin/bitcoin/pull/26617/commits/989a52e0a50c0ae30a5c2bd3c08bb3ad1363a250[this^] commit.
+We can include this two ways:
 
-To begin it is recommended to create a python virtual environment which can be done as follows when in the onboarding to bitcoin core top level directory:
+. Use the master branch of Bitcoin Core and running `git pull`, which will include the change.
+. Use any recent tag (e.g. v24.0.1) and running `git cherry-pick 989a52e0` to pull that change into the Test Framework code.
+
+You **MUST** have a copy of the jupyter notebook, either manually downloaded from https://github.com/chaincodelabs/onboarding-to-bitcoin-core or by cloning the onboarding-to-bitcoin-core repo (recommended) with:
+
+[source, bash]
+----
+git clone https://github.com/chaincodelabs/onboarding-to-bitcoin-core.git
+----
+
+You **MAY** want to use a python virtual environment (recommended) which can be done as follows when in the onboarding to bitcoin core top level directory:
 
 [source, bash]
 ----
+cd /path/to/source/onboarding-to-bitcoin-core
 python3 -m venv "obc-venv"
 source obc-venv/bin/activate
 ----
@@ -824,7 +872,7 @@ jupyter notebook
 ----
 
 This will open a list of all the files in this directory.
-Opening the file named `exercises.ipynb` will start the empty notebook containing instructions on how to use `TestShell` from the test Framework.
+Opening the file named `exercise_tutorial.ipynb` will start the notebook containing instructions on how to use `TestShell` from the test Framework.
 
 When you are finished you can deactivate the venv using
 
@@ -834,6 +882,16 @@ deactivate
 ----
 
 TIP: Don't forget to re-activate your venv each time you want to start the Jupyter notebook after deactivating the venv!
+
+[discrete]
+== Quick use
+
+Once you have familiarized yourself with the `TestShell` method using `exercise_tutorial.ipynb`, you can instead start new notebooks for exercises based on the `exercise_base.ipynb` notebook, which has much of the instruction removed and will let you get started faster.
+
+If you correct the import path for your system in this file and save it, you can then easily make copies of it to use as start points for different exercises:
+
+image::jupyter_duplicate.png[width=300]
+
 ****
 
 ////

exercise_base.ipynb

@@ -0,0 +1,46 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "import sys, os\n",
+    "from test_framework.test_shell import TestShell\n",
+    "\n",
+    "sys.path.insert(0, os.path.expanduser(\"~/bitcoin/test/functional\"))\n",
+    "test = TestShell().setup(\n",
+    "    num_nodes=2,\n",
+    "    setup_clean_chain=True,\n",
+    "    extra_args=[[], ['-fallbackfee=0.0002']],\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.7"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

exercises.ipynb renamed to exercise_tutorial.ipynb

@@ -10,21 +10,21 @@
    "source": [
     "## Extending python path\n",
     "\n",
-    "First we must add the path to Bitcoin Core's test_framework directory to the python system path\n",
+    "First we must add Bitcoin Core's test_framework directory path to Python's system path.\n",
     "\n",
-    "The test_framework is found within the Bitcoin Core source directory's `/test/functional` directory, e.g. as shown below\n",
-    "`/home/will/src/bitcoin/test/functional`"
+    "`test_framework` is found within Bitcoin Core's source directory, under `/test/functional`.\n",
+    "Modify the value shown below to reflect the correct path on your system."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Add the functional test framework to our PATH\n",
-    "import sys\n",
-    "sys.path.insert(0, \"/home/will/src/bitcoin/test/functional\")"
+    "# Add the functional test framework to PATH\n",
+    "import sys, os\n",
+    "sys.path.insert(0, os.path.expanduser(\"~/bitcoin/test/functional\"))"
    ]
   },
   {
@@ -43,7 +43,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 4,
    "metadata": {
     "pycharm": {
      "name": "#%%\n"
@@ -65,12 +65,16 @@
    "source": [
     "## Node setup\n",
     "\n",
-    "Finally, we must set up one or more nodes using the `setup()` method."
+    "Finally, we must set up one or more nodes using the `setup()` method.\n",
+    "\n",
+    "We can pass `bitcoind` options to each node using the `extra_args` argument.\n",
+    "This is the equivalent of setting options in the config file for `bitcoind` on the command line.\n",
+    "Arguments are passed in as a list of lists, one list of arguments for each node."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 5,
    "metadata": {
     "pycharm": {
      "name": "#%%\n"
@@ -81,7 +85,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "2022-12-13T22:18:13.318000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_wbfjhnzg\n"
+      "2022-12-14T13:28:53.132000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_r8jqr6ge\n"
      ]
     }
    ],
@@ -90,6 +94,7 @@
     "test = TestShell().setup(\n",
     "    num_nodes=2,\n",
     "    setup_clean_chain=True,\n",
+    "    extra_args=[[], ['-fallbackfee=0.0002']],\n",
     ")"
    ]
   },
@@ -105,22 +110,47 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 6,
    "metadata": {
     "pycharm": {
      "name": "#%%\n"
     }
    },
    "outputs": [],
    "source": [
-    "node1 = test.nodes[0]\n",
-    "node2 = test.nodes[1]\n"
+    "test.nodes[0].getblockchaininfo()\n",
+    "\n",
+    "# or\n",
+    "\n",
+    "node2 = test.nodes[1]\n",
+    "node2.getmemoryinfo()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": []
+   "source": [
+    "## Shutdown\n",
+    "\n",
+    "When you are finished, you can shut down the nodes just you would via `bitcoin-cli` by using the `shutdown()` method as shown below.\n",
+    "\n",
+    "If you kill the jupyter notebook server without shutting down the nodes then they are usually torn down automatically, but there is a slim chance they remain operational.\n",
+    "You can check for running `bitcoind` processes using `htop` or `sudo pidof bitcoind`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "pycharm": {
+     "name": "#%%\n"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "test.nodes[0].stop()\n",
+    "test.nodes[1].stop()"
+   ]
   }
  ],
  "metadata": {
@@ -144,4 +174,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 1
-}
+}