Skip to content

Commit 86d4fc6

Browse files
djdongjindjdongjin
committed
Add BUILD and INSTALL docs
Signed-off-by: Jin Dong <[email protected]>
1 parent 6939e94 commit 86d4fc6

9 files changed

+363
-58
lines changed

‎Makefile

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,8 +70,7 @@ vendor:
7070

7171
test:
7272
@echo "$@"
73-
@GO111MODULE=$(GO111MODULE_VALUE) go test -race ./...
74-
@cd ./cmd/soci ; GO111MODULE=$(GO111MODULE_VALUE) go test -timeout 20m -race ./...
73+
@GO111MODULE=$(GO111MODULE_VALUE) go test $(GO_TEST_FLAGS) -race ./...
7574

7675
integration: build
7776
@echo "$@"

‎README.md

Lines changed: 9 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ the container, and to instead lazily load data on demand, and also prefetch data
1818
## Design considerations
1919

2020
### No image conversion
21+
2122
Existing lazy loading snapshotters rely on a build-time conversion step, to produce a new image artifact.
2223
This is problematic for container developers who won't or can't modify their CI/CD pipeline, or don't
2324
want to manage the cost and complexity of keeping copies of images in two formats. It also creates
@@ -32,34 +33,16 @@ developed by the [OCI Reference Types working group](https://github.com/opencont
3233

3334
### Workload-specific load order optimization
3435

35-
Some lazy loading snapshotters support load order optimization, where some files are
36-
prioritized for prefetching. Typically, there is a one-to-one relationship between
37-
the list of to-be-prefetched files and the image or layer artifact.
38-
39-
For SOCI, we wanted a bit more flexibility. Often, which files to prefetch is highly
40-
dependent on the specific workload, not the image or base layer. For example, a customer
41-
may have a Python3 base layer that is shared by thousands of applications. To optimize
42-
the launch time of those applications using the traditional approach, the base
43-
layer can no longer be shared, because each application’s load order for that layer will be
44-
different. Registry storage costs will increase dramatically, and cache hit rates will plummet.
45-
And when it comes time to update that base layer, each and every copy will have to be reoptimized.
46-
47-
Secondly, there are some workloads that need to be able to prefetch at the subfile level. For example,
48-
we have observed machine learning workloads that launch and then immediately read a small header
49-
from a very large number of very large files.
50-
51-
To meet these use-cases, SOCI will implement a separate load order document (LOD), that can specify
52-
which files or file-segments to load. Because it is a separate artifact, a single image can have
53-
many LODs. At container launch time, the appropriate LOD can be retrieved using business logic
54-
specified by the administrator.
36+
Another big consideration that we haven't implmented/integrated
37+
into SOCI is to image load order based on your specific workload. See [design README](./docs/design-docs/README.md#workload-specific-load-order-optimization)
38+
for more details.
5539

56-
***Note:*** **SOCI Load order optimization is not yet implemented in SOCI.**
40+
## Documentation
5741

58-
## Learning More
59-
Check out our docs area:
60-
- [Getting
61-
Started](docs/GETTING_STARTED.md)
62-
- [Glossary](docs/GLOSSARY.md)
42+
- [Getting Started](docs/getting-started.md): walk through SOCI setups and features.
43+
- [Build](docs/build.md): how to build SOCI from source, test SOCI (and contribute).
44+
- [Install](docs/install.md): how to install SOCI as a systemd unit.
45+
- [Glossary](docs/glossary.md): glossary we use in the project.
6346

6447
## Project Origin
6548

‎docs/build.md

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Build soci-snapshotter from source
2+
3+
This document is helpful if you plan to contribute to the project (thanks!) or
4+
want to use the latest soci snapshotter/cli in the main branch.
5+
6+
This document includes the following sections:
7+
8+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
9+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
10+
11+
- [Dependencies](#dependencies)
12+
- [Build soci-snapshotter](#build-soci-snapshotter)
13+
- [Test soci-snapshotter](#test-soci-snapshotter)
14+
- [(Optional) Contribute your change](#optional-contribute-your-change)
15+
16+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
17+
18+
## Dependencies
19+
20+
soci-snapshotter has the following dependencies. Please follow the links or commands
21+
to install them on your machine:
22+
23+
> We only mention the direct dependencies of the project. Some dependencies may
24+
> have their own dependencies (e.g., containerd depends on runc/cni). Please refer
25+
> to their doc for a complete installation guide (mainly containerd).
26+
27+
- **[go](https://go.dev/doc/install) >= 1.18** - required to build the project;
28+
to confirm please check with `go version`.
29+
- **[containerd](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) >= 1.4** -
30+
required to run soci-snapshotter; to confirm please check with `sudo containerd --version`.
31+
- **fuse** - used for mounting without root access (`sudo yum install fuse`).
32+
- **zlib** - used for decompression and ztoc creation; soci builds zlib statically into its binaries
33+
(`sudo yum install zlib-devel zlib-static`).
34+
- **gcc** - used for compiling SOCI's c code, gzip's zinfo implementation (`sudo yum install gcc`).
35+
- **[flatc](https://github.com/google/flatbuffers)** - used for compiling ztoc
36+
flatbuffer file and generating corresponding go code.
37+
38+
For fuse/zlib/gcc, they can be installed by your Linux package manager (e.g., `yum` or `apt-get`).
39+
40+
For flatc, you can download and install a [release](https://github.com/google/flatbuffers/releases)
41+
into your `/usr/local` (or other `$PATH`) directory. For example:
42+
43+
```shell
44+
wget -c https://github.com/google/flatbuffers/releases/download/v23.3.3/Linux.flatc.binary.g++-10.zip
45+
sudo unzip Linux.flatc.binary.g++-10.zip -d /usr/local
46+
rm Linux.flatc.binary.g++-10.zip
47+
```
48+
49+
## Build soci-snapshotter
50+
51+
First you need `git` to clone the repository (if you intend to contribute, you
52+
can fork the repository and clone your own fork):
53+
54+
```shell
55+
git clone https://github.com/awslabs/soci-snapshotter.git
56+
cd soci-snapshotter
57+
```
58+
59+
soci-snapshotter uses `make` as the build tool. Assuming you're in the root directory
60+
of the repository, you can build soci-snapshotter by:
61+
62+
```shell
63+
make
64+
```
65+
66+
This builds the project binaries into the `./out` directory. You can install them
67+
to a `PATH` directory (`/usr/local/bin`) with:
68+
69+
```shell
70+
sudo make install
71+
# check soci can be found in PATH
72+
sudo soci --help
73+
```
74+
75+
When changing the ztoc's flatbuffer definition, you need to regenerate the generated
76+
code package with:
77+
78+
> It's rare to make such a change, especially delete a field which is a breaking
79+
> change and discouraged by flatbuffers.
80+
81+
```shell
82+
make flatc
83+
```
84+
85+
## Test soci-snapshotter
86+
87+
We have unit tests and integration tests as part of our automated CI, as well as
88+
benchmark tests that can be used to test the performance of soci-snapshotter. You
89+
can run these tests using the following `Makefile` targets:
90+
91+
- `make test`: run all unit tests.
92+
- `make integration`: run all integration tests.
93+
- `make benchmarks` (experimental): run all benchmark tests. This requires some
94+
setup and preparation for public images with SOCI index which are used for benchmarking.
95+
It is tracked in [#245](https://github.com/awslabs/soci-snapshotter/issues/245).
96+
97+
To speed up develop-test cycle, you can run individual test(s) by utilizing `go test`'s
98+
`-run` flag. For example, suppose you only want to run a test named `TestFooBar`, you can:
99+
100+
```shell
101+
# 1. if TestFooBar is a unit test
102+
GO_TEST_FLAGS="-run TestFooBar" make test
103+
104+
# 2. if TestFooBar is an integration test
105+
GO_TEST_FLAGS="-run TestFooBar" make integration
106+
```
107+
108+
## (Optional) Contribute your change
109+
110+
If you intend to contribute your change, you need to validate your changes pass
111+
all unit/integration tests. (i.e., `make test` and `make integration` pass).
112+
113+
Meanwhile, there are a few steps you should follow to ensure your change is ready
114+
for review:
115+
116+
1. If you added any new files, make sure they contain the SOCI license header. We
117+
provide a script (`./scripts/add-ltag.sh`) that can do this.
118+
119+
2. Make sure your change is well-formatted and you've run `gofmt`.
120+
121+
3. Make sure your commit is signed (`git commit -s`).
122+
123+
4. As a final step, run `make check` to verify your change passes these checks.
124+
125+
> `make check` requires some checking tools (`golangci`, `ltag`,
126+
> `git-validation`). We provide a script (`./scripts/install-check-tools.sh`) to
127+
> help install all these checking tools.
128+
129+
Once you pass all the tests and checks. You're ready to make your PR!

‎docs/design-docs/README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Design docs
2+
3+
We use this folder to track design docs for the project and their status (proposed
4+
, accepted, implemented, etc).
5+
6+
We also keep some features/ideas that we think will improve soci-snapshotter but
7+
haven't been converted into concrete design docs.
8+
9+
## Workload-specific load order optimization
10+
11+
Some lazy loading snapshotters support load order optimization, where some files are
12+
prioritized for prefetching. Typically, there is a one-to-one relationship between
13+
the list of to-be-prefetched files and the image or layer artifact.
14+
15+
For SOCI, we wanted a bit more flexibility. Often, which files to prefetch is highly
16+
dependent on the specific workload, not the image or base layer. For example, a customer
17+
may have a Python3 base layer that is shared by thousands of applications. To optimize
18+
the launch time of those applications using the traditional approach, the base
19+
layer can no longer be shared, because each application’s load order for that layer will be
20+
different. Registry storage costs will increase dramatically, and cache hit rates will plummet.
21+
And when it comes time to update that base layer, each and every copy will have to be reoptimized.
22+
23+
Secondly, there are some workloads that need to be able to prefetch at the subfile level. For example,
24+
we have observed machine learning workloads that launch and then immediately read a small header
25+
from a very large number of very large files.
26+
27+
To meet these use-cases, SOCI will implement a separate load order document (LOD), that can specify
28+
which files or file-segments to load. Because it is a separate artifact, a single image can have
29+
many LODs. At container launch time, the appropriate LOD can be retrieved using business logic
30+
specified by the administrator.

‎docs/GETTING_STARTED.md renamed to ‎docs/getting-started.md

Lines changed: 26 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ with soci-snapshotter.
77
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
88
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
99

10-
- [Prerequisites](#prerequisites)
10+
- [Dependencies](#dependencies)
1111
- [Install soci-snapshotter](#install-soci-snapshotter)
1212
- [Push an image to your registry](#push-an-image-to-your-registry)
1313
- [Create and push SOCI index](#create-and-push-soci-index)
@@ -22,46 +22,52 @@ with soci-snapshotter.
2222

2323
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
2424

25-
## Prerequisites
25+
## Dependencies
2626

27-
- **[go](https://go.dev/doc/install) >= 1.18** - to confirm please check with `go version`.
28-
- **[containerd](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) >= 1.4** - to confirm that you have containerd working please check with
29-
`sudo ctr version`.
30-
- **[ctr](https://github.com/containerd/containerd/blob/main/docs/getting-started.md)/[nerdctl](https://github.com/containerd/nerdctl#install)** - you need one of the containerd clients to interact with containerd/registry.
31-
- **fuse** - used for mounting without root access (`sudo yum install fuse`).
32-
- **zlib** - used for decompression and ztoc creation (`sudo yum install zlib-devel zlib-static`).
33-
- **gcc** - used for compiling SOCI's c code, gzip's zinfo implementation (`sudo yum install gcc`).
27+
soci-snapshotter has the following runtime dependencies. Please follow the links or commands
28+
to install them on your machine:
29+
30+
> We only mention the direct dependencies of the project. Some dependencies may
31+
> have their own dependencies (e.g., containerd depends on runc/cni). Please refer
32+
> to their doc for a complete installation guide (mainly containerd).
33+
34+
- **[containerd](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) >= 1.4** -
35+
required to run soci-snapshotter; to confirm please check with `sudo ctr version`.
36+
- **[ctr](https://github.com/containerd/containerd/blob/main/docs/getting-started.md)** -
37+
required for this doc to interact with containerd/registry.
38+
- **fuse** - used for mounting without root access (`sudo yum install fuse` or
39+
other Linux package manager like `apt-get`, dpending on your Linux distro).
3440

3541
## Install soci-snapshotter
3642

37-
The soci-snapshotter project consists of 2 main components:
43+
The soci-snapshotter project produces 2 binaries:
3844

3945
- `soci`: the CLI tool used to build/manage SOCI indices.
4046
- `soci-snapshotter-grpc`: the daemon (a containerd snapshotter plugin) used for lazy loading.
4147

42-
Currently to get the binaries, we need to build the project from source after cloing the repo:
48+
You can download prebuilt binaries from our [release page](https://github.com/awslabs/soci-snapshotter/releases)
49+
or [build them from source](./build.md).
50+
51+
In this doc, let's just download the release binaries and move them to a `PATH`
52+
directory (`/usr/local/bin`):
53+
54+
> You can find other download link in the release page that matches your machine.
4355
4456
```shell
45-
git clone https://github.com/awslabs/soci-snapshotter.git
46-
cd soci-snapshotter
47-
make
57+
wget -O https://github.com/awslabs/soci-snapshotter/releases/download/v0.1.0/soci-snapshotter-0.1.0-linux-amd64.tar.gz
58+
sudo tar -C /usr/local/bin -xvf soci-snapshotter-0.1.0-linux-amd64.tar.gz soci soci-snapshotter-grpc
4859
```
4960

50-
This builds the project binaries into the `./out` directory. You can install them
51-
to a `PATH` directory (`/usr/local/bin`) with:
61+
Now you should be able to use the `soci` CLI (and `soci-snapshotter-grpc` containerd plugin shortly):
5262

5363
```shell
54-
sudo make install
5564
# check soci can be found in PATH
5665
sudo soci --help
5766
```
5867

5968
Many `soci` CLI commands need to be run as `sudo`, because the metadata is saved
6069
in directories that a non-root user often does not have access to.
6170

62-
> This doc assumes SOCI binaries are installed into `PATH`. If not, please use
63-
> the full path of the binaries (e.g. `./out/soci`).
64-
6571
## Push an image to your registry
6672

6773
In this document we will use `rabbitmq` from DockerHub `docker.io/library/rabbitmq:latest`.

‎docs/GLOSSARY.md renamed to ‎docs/glossary.md

File renamed without changes.

‎docs/install.md

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# Install soci-snapshotter
2+
3+
This doc walks through how to install soci-snapshotter as a component managed by systemd.
4+
5+
The soci-snapshotter project produces 2 binaries:
6+
7+
- `soci`: the CLI tool used to build/manage SOCI indices.
8+
- `soci-snapshotter-grpc`: the daemon (a containerd snapshotter plugin) used for lazy loading.
9+
10+
You can get the prebuilt binaries from our [release page](https://github.com/awslabs/soci-snapshotter/releases)
11+
or [build them from source](./build.md).
12+
13+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
14+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
15+
16+
- [Dependencies](#dependencies)
17+
- [Configure soci-snapshotter (optional)](#configure-soci-snapshotter-optional)
18+
- [Install soci-snapshotter for containerd with systemd](#install-soci-snapshotter-for-containerd-with-systemd)
19+
- [Config containerd](#config-containerd)
20+
21+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
22+
23+
## Dependencies
24+
25+
soci-snapshotter has the following dependencies. Please follow the links or commands
26+
to install them on your machine:
27+
28+
> We only mention the direct dependencies of the project. Some dependencies may
29+
> have their own dependencies (e.g., containerd depends on runc/cni). Please refer
30+
> to their doc for a complete installation guide (mainly containerd).
31+
32+
- **[containerd](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) >= 1.4** -
33+
required to run soci-snapshotter; to confirm please check with `sudo containerd --version`.
34+
- **fuse** - used for mounting without root access (`sudo yum install fuse`).
35+
36+
For fuse/zlib, they can be installed by your Linux package manager (e.g., `yum` or `apt-get`).
37+
38+
## Configure soci-snapshotter (optional)
39+
40+
Similar to containerd, soci-snapshotter has a toml config file which is located at
41+
`/etc/soci-snapshotter-grpc/config.toml` by default. If such a file doesn't exist,
42+
soci-snapshotter will use default values for all configurations.
43+
44+
> Whenever you make changes to the config file, you need to stop the snapshotter
45+
> first before making changes, and restart the snapshotter after the changes.
46+
47+
## Install soci-snapshotter for containerd with systemd
48+
49+
If you plan to use systemd to manage your soci-snapshotter process, you can download
50+
the [`soci-snapshotter.service` unit file](../soci-snapshotter.service) in the
51+
repository root directory into `/usr/local/lib/systemd/system/soci-snapshotter.service`,
52+
and run the following commands:
53+
54+
```shell
55+
sudo systemctl daemon-reload
56+
sudo systemctl enable --now soci-snapshotter
57+
```
58+
59+
To validate soci-snapshotter is running, let's check the snapshotter's version.
60+
The output should show the version that you installed.
61+
62+
```shell
63+
$ sudo soci-snapshotter-grpc --version
64+
soci-snapshotter-grpc version f855ff1.m f855ff1bcf7e161cf0e8d3282dc3d797e733ada0.m
65+
```
66+
67+
## Config containerd
68+
69+
We need to configure and restart containerd to enable soci-snapshotter (this
70+
section assume your containerd is also managed by `systemd`):
71+
72+
- Stop containerd: `sudo systemctl stop containerd`;
73+
- Update containerd config to include soci-snapshotter plugin. The config file
74+
is usually in `/etc/containerd/config.toml`, and you need to add the following:
75+
76+
```toml
77+
[proxy_plugins]
78+
[proxy_plugins.soci]
79+
type = "snapshot"
80+
address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"
81+
```
82+
83+
- Restart containerd: `sudo systemctl restart containerd`;
84+
- (Optional) Check soci-snapshotter is recognized by containerd: `sudo ctr plugin ls id==soci`.
85+
You will see output like below. If not, consult containerd logs to determine the cause
86+
or reach out on [our discussion](https://github.com/awslabs/soci-snapshotter/discussions).
87+
88+
```shell
89+
TYPE ID PLATFORMS STATUS
90+
io.containerd.snapshotter.v1 soci - ok
91+
```

0 commit comments

Comments
 (0)