feat: cargo CLI update target directory (#1662)

Resolves INT-3882, and records changes in the book as per INT-4038. Most
importantly, reworks command outputs to primarily use the `target/`
directory that Cargo uses and assumes full control over.

- User can specify `target_dir`; if unspecified, `cargo openvm` will use
the Cargo default
- Keys are stored in `${target_dir}/openvm/`, while all other artifacts
are stored in `${target_dir}/openvm/${profile}`
- User can copy outputs to `${output_dir}`
- **NOTE** - generated proofs are stored at `.`, as this file should
generally be the final output

Other changes include:
- `build` no longer generates app commits
- `run` will call `build` before attempting to execute
- `prove` will call `build` before attempting to generate a proof
- Replaced all references to `evm-proving-setup` with references to
`setup`
- `openvm.toml`'s default location is in the working directory instead
of `.`

The basic workflow should now be `cargo openvm prove ...` into `cargo
openvm verify ...`.

Author: stephenh-axiom-xyz

.github/workflows/cli.yml

@@ -49,9 +49,9 @@ jobs:
         working-directory: examples/ecc
         run: |
           cargo openvm build
-          mv openvm/app.vmexe app1.vmexe
+          mv target/openvm/release/ecc-example.vmexe app1.vmexe
           cargo openvm build
-          cmp app1.vmexe openvm/app.vmexe || (echo "Build is not deterministic!" && exit 1)
+          cmp app1.vmexe target/openvm/release/ecc-example.vmexe || (echo "Build is not deterministic!" && exit 1)
 
       - name: Build and run book examples
         working-directory: examples
@@ -72,7 +72,7 @@ jobs:
         run: |
           export RUST_BACKTRACE=1
           cargo build
-          cargo run --bin cargo-openvm -- openvm keygen --config ./example/app_config.toml --output app.pk --vk-output app.vk
+          cargo run --bin cargo-openvm -- openvm keygen --config ./example/app_config.toml --output-dir .
 
       - name: Run CLI tests
         working-directory: crates/cli

Cargo.lock

@@ -1,4 +1,4 @@
-use cargo_openvm::{default::DEFAULT_APP_CONFIG_PATH, util::read_config_toml_or_default};
+use cargo_openvm::util::read_config_toml_or_default;
 use clap::{Parser, ValueEnum};
 use eyre::Result;
 use openvm_benchmarks_utils::{get_elf_path, get_programs_dir, read_elf_file};
@@ -106,7 +106,7 @@ fn main() -> Result<()> {
             let elf_path = get_elf_path(&program_dir);
             let elf = read_elf_file(&elf_path)?;
 
-            let config_path = program_dir.join(DEFAULT_APP_CONFIG_PATH);
+            let config_path = program_dir.join("openvm.toml");
             let vm_config = read_config_toml_or_default(&config_path)?.app_vm_config;
 
             let exe = VmExe::from_elf(elf, vm_config.transpiler())?;

benchmarks/execute/src/main.rs

@@ -12,6 +12,7 @@
 - [Overview](./writing-apps/overview.md)
 - [Writing a Program](./writing-apps/write-program.md)
 - [Compiling](./writing-apps/build.md)
+- [Running a Program](./writing-apps/run.md)
 - [Generating Proofs](./writing-apps/prove.md)
 - [Verifying Proofs](./writing-apps/verify.md)
 - [Solidity SDK](./writing-apps/solidity.md)

book/src/SUMMARY.md

@@ -9,135 +9,142 @@ The command `cargo openvm build` compiles the program on host to an executable f
 It first compiles the program normally on your _host_ platform with RISC-V and then transpiles it to a different target. See here for some explanation of [cross-compilation](https://rust-lang.github.io/rustup/cross-compilation.html).
 Right now we use `riscv32im-risc0-zkvm-elf` target which is available in the [Rust toolchain](https://doc.rust-lang.org/rustc/platform-support/riscv32im-risc0-zkvm-elf.html), but we will contribute an OpenVM target to Rust in the future.
 
-## Build flags
+## Build Flags
 
-The following flags are available for the `cargo openvm build` command:
+The following flags are available for the `cargo openvm build` command. You can run `cargo openvm build --help` for this list within the command line.
 
-- `--manifest-dir <MANIFEST_DIR>`
+Generally, outputs will always be built to the **target directory**, which will either be determined by the manifest path or explicitly set using the `--target-dir` option. By default Cargo sets this to be `<workspace_or_package_root>/target/`. 
 
-  **Description**: Specifies the directory containing the `Cargo.toml` file for the guest code.
+OpenVM-specific artifacts will be placed in `${target_dir}/openvm/`, but if `--output-dir` is specified they will be copied to `${output-dir}/` as well.
 
-  **Default**: The current directory (`.`).
+### OpenVM Options
 
-  **Usage Example**: If your `Cargo.toml` is located in `my_project/`, you can run:
+- `--no-transpile`
 
-  ```bash
-  cargo openvm build --manifest-dir my_project
-  ```
+  **Description**: Skips transpilation into an OpenVM-compatible `.vmexe` executable when set.
 
-  This ensures the build command is executed in that directory.
+- `--config <CONFIG>`
 
-- `--target-dir <TARGET_DIR>`
+  **Description**: Path to the OpenVM config `.toml` file that specifies the VM extensions. By default will search the manifest directory for `openvm.toml`. If no file is found, OpenVM will use a default configuration. Currently the CLI only supports known extensions listed in the [Using Existing Extensions](../custom-extensions/overview.md) section. To use other extensions, use the [SDK](../advanced-usage/sdk.md).
 
-  **Description**: Specifies the directory where the guest binary will be built. If not specified, the default target directory is used.
+- `--output_dir <OUTPUT_DIR>`
 
-  **Default**: The `target` directory in the package root directory.
+  **Description**: Output directory for OpenVM artifacts to be copied to. Keys will be placed in `${output-dir}/`, while all other artifacts will be in `${output-dir}/${profile}`.
 
-  **Usage Example**: To build the guest binary in the `my_target` directory:
+- `--init-file-name <INIT_FILE_NAME>`
 
-  ```bash
-  cargo openvm build --target-dir my_target
-  ```
+  **Description**: Name of the generated initialization file, which will be written into the manifest directory.
 
-- `--features <FEATURES>`
+  **Default**: `openvm_init.rs`
 
-  **Description**: Passes a list of feature flags to the Cargo build process. These flags enable or disable conditional compilation features defined in your `Cargo.toml`.
+### Package Selection
 
-  **Usage Example**: To enable the `my_feature` feature:
+As with `cargo build`, default package selection depends on the working directory. If the working directory is a subdirectory of a specific package, then only that package will be built. Else, all packages in the workspace will be built by default.
 
-  ```bash
-  cargo openvm build --features my_feature
-  ```
+- `--package <PACKAGES>`
 
-- `--bin <NAME>`
+  **Description**: Builds only the specified packages. This flag may be specified multiple times or as a comma-separated list.
 
-  **Description**: Restricts the build to the binary target with the given name, similar to `cargo build --bin <NAME>`. If your project has multiple target types (binaries, libraries, examples, etc.), using `--bin <NAME>` narrows down the build to the binary target with the given name.
+- `--workspace`
 
-  **Usage Example**:
+  **Description**: Builds all members of the workspace (alias `--all`).
 
-  ```bash
-  cargo openvm build --bin my_bin
-  ```
+- `--exclude <PACKAGES>`
 
-- `--example <NAME>`
+  **Description**: Excludes the specified packages. Must be used in conjunction with `--workspace`. This flag may be specified multiple times or as a comma-separated list.
 
-  **Description**: Restricts the build to the example target with the given name, similar to `cargo build --example <NAME>`. Projects often include code samples or demos under the examples directory, and this flag focuses on compiling a specific example.
+### Target Selection
 
-  **Usage Example**:
+By default all package libraries and binaries will be built. To build samples or demos under the `examples` directory, use either the `--example` or `--examples` option.
 
-  ```bash
-  cargo openvm build --example my_example
-  ```
+- `--lib`
 
-- `--no-transpile`
+  **Description**: Builds the package's library.
 
-  **Description**: After building the guest code, doesn't transpile the target ELF into an OpenVM-compatible executable (by default it does).
+- `--bin <BIN>`
 
-  **Usage Example**:
+  **Description**: Builds the specified binary. This flag may be specified multiple times or as a comma-separated list.
 
-  ```bash
-  cargo openvm build --no-transpile
-  ```
+- `--bins`
 
-- `--config <CONFIG>`
+  **Description**: Builds all binary targets.
+
+- `--example <EXAMPLE>`
+
+  **Description**: Builds the specified example. This flag may be specified multiple times or as a comma-separated list.
+
+- `--examples`
+
+  **Description**: Builds all example targets.
+
+- `--all-targets`
+
+  **Description**: Builds all package targets. Equivalent to specifying `--lib` `--bins` `--examples`.
+
+### Feature Selection
+
+The following options enable or disable conditional compilation features defined in your `Cargo.toml`.
+
+- `-F`, `--features <FEATURES>`
+
+  **Description**: Space or comma separated list of features to activate. Features of workspace members may be enabled with `package-name/feature-name` syntax. This flag may also be specified multiple times.
+
+- `--all-features`
+
+  **Description**: Activates all available features of all selected packages.
+
+- `--no-default-features`
+
+  **Description**: Do not activate the `default` feature of the selected packages.
+
+### Compilation Options
+
+- `--profile <NAME>`
 
-  **Description**: Specifies the path to a .toml configuration file that defines which VM extensions to use.
+  **Description**: Builds with the given profile. Common profiles are `dev` (faster builds, less optimization) and `release` (slower builds, more optimization). For more information on profiles, see [Cargo's reference page](https://doc.rust-lang.org/cargo/reference/profiles.html).
 
-  **Default**: `./openvm.toml` if `--config` flag is not provided.
+  **Default**: `release`
 
-  **Usage Example**:
+### Output Options
 
-  ```bash
-  cargo openvm build --config path/to/openvm.toml
-  ```
+- `--target_dir <TARGET_DIR>`
 
-  This allows you to customize the extensions. Currently the CLI only supports known extensions listed in the [Using Existing Extensions](../custom-extensions/overview.md) section. To use other extensions, use the [SDK](../advanced-usage/sdk.md).
+  **Description**: Directory for all generated artifacts and intermediate files. Defaults to directory `target/` at the root of the workspace.
 
-- `--exe-output <EXE_OUTPUT>`
+### Display Options
 
-  **Description**: Sets the output path for the transpiled program.
+- `-v`, `--verbose`
 
-  **Default**: `./openvm/app.vmexe` if `--exe-output` flag is not provided.
+  **Description**: Use verbose output.
 
-  **Usage Example**: To specify a custom output filename:
+- `-q`, `--quiet`
 
-  ```bash
-  cargo openvm build --exe-output ./output/custom_name.vmexe
-  ```
+  **Description**: Do not print Cargo log messages.
 
-- `--profile <PROFILE>`
+- `--color <WHEN>`
 
-  **Description**: Determines the build profile used by Cargo. Common profiles are dev (faster builds, less optimization) and release (slower builds, more optimization).
+  **Description**: Controls when colored output is used.
 
-  **Default**: release
+  **Default**: `always`
 
-  **Usage Example**:
+### Manifest Options
 
-  ```bash
-  cargo openvm build --profile dev
-  ```
+- `--manifest-path <PATH>`
 
-- `--help`
+  **Description**: Path to the guest code Cargo.toml file. By default, `build` searches for the file in the current or any parent directory. The `build` command will be executed in that directory.
 
-  **Description**: Prints a help message describing the available options and their usage.
+- `--ignore-rust-version`
 
-  **Usage Example**:
+  **Description**: Ignores rust-version specification in packages.
 
-  ```bash
-  cargo openvm build --help
-  ```
+- `--locked`
 
-## Running a Program
+  **Description**: Asserts the same dependencies and versions are used as when the existing Cargo.lock file was originally generated.
 
-After building and transpiling a program, you can execute it using the `run` command. The `run` command has the following arguments:
+- `--offline`
 
-```bash
-cargo openvm run
-    --exe <path_to_transpiled_program>
-    --config <path_to_app_config>
-    --input <path_to_input>
-```
+  **Description**: Prevents Cargo from accessing the network for any reason.
 
-If `--exe` and/or `--config` are not provided, the command will search for these files in `./openvm/app.vmexe` and `./openvm.toml` respectively. If `./openvm.toml` is not present, a default configuration will be used.
+- `--frozen`
 
-If your program doesn't require inputs, you can (and should) omit the `--input` flag.
+  **Description**: Equivalent to specifying both `--locked` and `--offline`.

book/src/writing-apps/build.md

@@ -4,41 +4,46 @@ Generating a proof using the CLI is simple - first generate a key, then generate
 
 ```bash
 cargo openvm keygen
-cargo openvm prove [app | evm]
+cargo openvm prove [app | stark | evm]
 ```
 
 ## Key Generation
 
-The `keygen` CLI command has the following optional arguments:
+The `keygen` command generates both an application proving and verification key.
 
 ```bash
 cargo openvm keygen
     --config <path_to_app_config>
-    --output <path_to_app_pk>
-    --vk_output <path_to_app_vk>
 ```
 
-If `--config` is not provided, the command will search for `./openvm.toml` and use that as the application configuration if present. If it is not present, a default configuration will be used.
+Similarly to `build`, `run`, and `prove`, options `--manifest-path`, `--target-dir`, and `--output-dir` are provided.
 
-If `--output` and/or `--vk_output` are not provided, the keys will be written to default locations `./openvm/app.pk` and/or `./openvm/app.vk` respectively.
+If `--config` is not specified, the command will search for `openvm.toml` in the manifest directory. If the file isn't found, a default configuration will be used.
+
+The proving and verification key will be written to `${target_dir}/openvm/` (and `--output-dir` if specified).
 
 ## Proof Generation
 
-The `prove` CLI command has the following optional arguments:
+The `prove` CLI command, at its core, uses the options below. `prove` gets access to all of the options that `run` has (see [Running a Program](../writing-apps/run.md) for more information).
 
 ```bash
-cargo openvm prove [app | evm]
-    --app_pk <path_to_app_pk>
+cargo openvm prove [app | stark | evm]
+    --app-pk <path_to_app_pk>
     --exe <path_to_transpiled_program>
     --input <path_to_input>
-    --output <path_to_output>
+    --proof <path_to_proof_output>
 ```
 
+If `--app-pk` is not provided, the command will search for a proving key at `${target_dir}/openvm/app.pk`.
+
+If `--exe` is not provided, the command will call `build` before generating a proof.
+
 If your program doesn't require inputs, you can (and should) omit the `--input` flag.
 
-If `--app_pk` and/or `--exe` are not provided, the command will search for these files in `./openvm/app.pk` and `./openvm/app.vmexe` respectively. Similarly, if `--output` is not provided then the command will write the proof to `./openvm/[app | evm].proof` by default.
+If `--proof` is not provided then the command will write the proof to `./[app | stark | evm].proof` by default.
+
 
-The `app` subcommand is used to generate an application-level proof, while the `evm` command generates an end-to-end EVM proof.
+The `app` subcommand generates an application-level proof, the `stark` command generates an aggregated root-level proof, while the `evm` command generates an end-to-end EVM proof. For more information on aggregation, see [this specification](https://github.com/openvm-org/openvm/blob/bf8df90b13f4e80bb76dbb71f255a12154c84838/docs/specs/continuations.md).
 
 > ⚠️ **WARNING**
 > In order to run the `evm` subcommand, you must have previously called the costly `cargo openvm setup`, which requires very large amounts of computation and memory (~200 GB).