Improve Docker builds

[vgwidt]

Docker build speeds can be sped up significantly.

• Cache dependencies
• Eliminate cargo cache clearing commands
• Use pre-compiled binaries (i.e. diesel
• Move binary to a slim Debian image (depends on ability to build diesel binaries, until then use rust image)

wget -qO- https://github.com/thedodd/trunk/releases/download/v0.16.0/trunk-x86_64-unknown-linux-gnu.tar.gz | tar -xzf-
      chmod +x trunk
      cp trunk ~/.cargo/bin/

Separate Dockerfiles are not necessary either, especially once caching is properly utilized.

[vgwidt]

If using pre-compiled binaries don't forget to install wget:

RUN apt-get update && apt-get install -y wget && rm -rf /var/lib/apt/lists/*

[vgwidt]

Threw together a sample that doesn't cache cargo dependencies yet.

FROM rust:latest as base
RUN apt-get update && apt-get install -y wget && rm -rf /var/lib/apt/lists/*
RUN wget -qO- https://github.com/thedodd/trunk/releases/download/v0.16.0/trunk-x86_64-unknown-linux-gnu.tar.gz | tar -xzf- && chmod +x trunk && cp trunk /usr/local/cargo/bin/
RUN rustup target add wasm32-unknown-unknown
RUN wget -qO- https://github.com/rustwasm/wasm-bindgen/releases/download/0.2.84/wasm-bindgen-0.2.84-aarch64-unknown-linux-gnu.tar.gz | tar -xzf- && cp ./wasm-bindgen-0.2.84-aarch64-unknown-linux-gnu/wasm-bindgen /usr/local/cargo/bin/ && chmod +x /usr/local/cargo/bin/wasm-bindgen

FROM base as builder
WORKDIR /usr/src/sumi
COPY . .
RUN trunk build -d dist ./frontend/index.html --release
RUN cargo build --manifest-path ./backend/Cargo.toml --release

FROM rust:latest
WORKDIR /usr/src/sumi
RUN apt-get install -y libpq-dev
RUN cargo install diesel_cli --no-default-features --features postgres
COPY --from=builder /usr/src/sumi/target/release/sumi-backend ./sumi-backend

I'm not sure it is actually worth avoiding using cargo to install trunk and wasm-bindgen as that bit should be cached.

[vgwidt]

Sample with cargo-chef for caching project (so far only backend) dependencies:

FROM rust:latest as base
RUN cargo install cargo-chef
RUN cargo install trunk
RUN rustup target add wasm32-unknown-unknown

FROM base as planner
WORKDIR /usr/src/sumi
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM base as cacher
WORKDIR /usr/src/sumi
COPY --from=planner /usr/src/sumi/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

FROM base as builder
WORKDIR /usr/src/sumi
COPY . .
COPY --from=cacher /usr/src/sumi/target target
RUN trunk build -d dist ./frontend/index.html --release
RUN cargo build --manifest-path ./backend/Cargo.toml --release

FROM rust:latest
WORKDIR /usr/src/sumi
RUN apt-get install -y libpq-dev
RUN cargo install diesel_cli --no-default-features --features postgres
COPY ./backend/diesel.toml ./backend/diesel.toml
COPY ./certificates ./certificates
COPY ./dist ./dist
COPY ./backend/migrations ./backend/migrations
COPY --from=builder /usr/src/sumi/target/release/sumi-backend ./sumi

This builds trunk and wasm-bindgen as that stage will be cached anyways. Only the backend dependencies are cached; not sure how to handle frontend yet. Using the Rust image and building Diesel results in a 1.5GB image as there are no precompiled binaries readily available (diesel 2379).

[vgwidt]

COPY --from=builder /usr/src/sumi/target/release/sumi-backend ./sumi doesn't appear to be working.

[vgwidt]

COPY --from=builder /usr/src/sumi/target/release/sumi-backend ./sumi doesn't appear to be working.

Dist folder wasn't being copied from the builder, need to change COPY ./dist ./dist to COPY --from=builder /usr/src/sumi/dist ./dist.

Need to fix up .dockerignore as well, and can just copy . . on final stage. It'll copy unused source files but they use minimal space.

[vgwidt]

A related issue is that some .env variables are still required for building the frontend.

[vgwidt]

Merged 7ac1b16 which includes the new Dockerfile. It currently only caches the backend.

Environment variables are now written to the index.html file when the backend is run. I'll need to figure out a solution that allows de-coupling of the API and web server, though that is currently not possible at this point anyways.

[vgwidt]

The issue with caching the frontend has to do with Cargo workspace limitations.
We can tell the frontend to target wasm32-unknown-unknown by adding the following in .cargo/config.toml:

[build]
target = "wasm32-unknown-unknown"

However, when building from the root workspace folder, it does not look at this config file for members. Alternatively, Cargo nightly allows us to force the package target in the member Cargo.toml: rust-lang/cargo#9406. I tested it and it works great, but I don't want to depend on nightly.

It appears that the only other option is to move frontend out of the workspace (would need to do an exclude or eliminate workspace altogether).

[vgwidt]

It looks like I might be able to build frontend deps using config.toml while building in the frontend folder. Sample (not working) Dockerfile that might demonstrate how it could be done when building with Docker:

FROM rust:latest as base
RUN cargo install cargo-chef
RUN cargo install trunk
RUN rustup target add wasm32-unknown-unknown

FROM base as cacher
WORKDIR /usr/src/sumi
COPY ./Cargo.lock ./Cargo.lock
COPY ./frontend/Cargo.toml ./frontend/Cargo.toml
COPY ./backend/Cargo.toml ./backend/Cargo.toml
COPY ./frontend/.cargo ./frontend/.cargo
RUN mkdir ./backend/src && mkdir ./frontend/src && echo 'fn main() { println!("Dummy"); }' > ./backend/src/main.rs && echo 'fn main() { println!("Dummy"); } ' > ./frontend/src/lib.rs
RUN cargo build --release --manifest-path ./backend/Cargo.toml
WORKDIR /usr/src/sumi/frontend
RUN cargo build --release
WORKDIR /usr/src/sumi
RUN rm -rf ./backend/src && rm -rf ./frontend/src

FROM base as builder
WORKDIR /usr/src/sumi
COPY . .
COPY --from=cacher /usr/src/sumi/target target
RUN touch -a -m ./backend/src/main.rs && touch -a -m ./frontend/src/lib.rs
RUN trunk build -d dist ./frontend/index.html --release
RUN cargo build --manifest-path ./backend/Cargo.toml --release

FROM rust:latest
WORKDIR /usr/src/sumi
RUN apt-get install -y libpq-dev
RUN cargo install diesel_cli --no-default-features --features postgres
COPY . .
COPY --from=builder /usr/src/sumi/dist ./dist
COPY --from=builder /usr/src/sumi/target/release/sumi-backend ./sumi

[vgwidt]

Missed root Cargo.file in last comment. The following works and built in less than 2 minutes with everything cached:

FROM rust:latest as base
RUN cargo install trunk
RUN rustup target add wasm32-unknown-unknown

FROM base as cacher
WORKDIR /usr/src/sumi
COPY ./Cargo.lock ./Cargo.lock
COPY ./Cargo.toml ./Cargo.toml
COPY ./frontend/Cargo.toml ./frontend/Cargo.toml
COPY ./backend/Cargo.toml ./backend/Cargo.toml
COPY ./frontend/.cargo ./frontend/.cargo
RUN mkdir ./backend/src && mkdir ./frontend/src && echo 'fn main() { println!("Dummy"); }' > ./backend/src/main.rs && echo 'fn main() { println!("Dummy"); } ' > ./frontend/src/lib.rs
RUN cargo build --release --manifest-path ./backend/Cargo.toml
WORKDIR /usr/src/sumi/frontend
RUN cargo build --release
WORKDIR /usr/src/sumi
RUN rm -rf ./backend/src && rm -rf ./frontend/src

FROM base as builder
WORKDIR /usr/src/sumi
COPY --from=cacher /usr/src/sumi/target target
COPY . .
RUN touch -a -m ./backend/src/main.rs && touch -a -m ./frontend/src/lib.rs
RUN trunk build -d dist ./frontend/index.html --release
RUN cargo build --manifest-path ./backend/Cargo.toml --release

FROM rust:latest
WORKDIR /usr/src/sumi
RUN apt-get install -y libpq-dev
RUN cargo install diesel_cli --no-default-features --features postgres
COPY . .
COPY --from=builder /usr/src/sumi/dist ./dist
COPY --from=builder /usr/src/sumi/target/release/sumi-backend ./sumi

I attempted to get it working the same way with cargo-chef without success. I may try it again as it is a pretty clean solution.

It can be cleaned up a bit (i.e. remove RUN rm).

[vgwidt]

Slightly cleaned up version without touch, as I don't think we need it:

FROM rust:latest as base
RUN cargo install cargo-chef
RUN cargo install trunk
RUN rustup target add wasm32-unknown-unknown

FROM base as cacher
WORKDIR /usr/src/sumi
COPY ./Cargo.lock ./
COPY ./Cargo.toml ./
COPY ./frontend/Cargo.toml ./frontend/Cargo.toml
COPY ./backend/Cargo.toml ./backend/Cargo.toml
COPY ./frontend/.cargo ./frontend/.cargo
RUN mkdir ./backend/src && mkdir ./frontend/src && echo 'fn main() { println!("Dummy"); }' > ./backend/src/main.rs && echo 'fn main() { println!("Dummy"); } ' > ./frontend/src/lib.rs
RUN cargo build --release --manifest-path ./backend/Cargo.toml
WORKDIR /usr/src/sumi/frontend
RUN cargo build --release

FROM base as builder
WORKDIR /usr/src/sumi
COPY --from=cacher /usr/src/sumi/target target
COPY . .
RUN trunk build -d dist ./frontend/index.html --release
RUN cargo build --manifest-path ./backend/Cargo.toml --release

FROM rust:latest
WORKDIR /usr/src/sumi
RUN apt-get install -y libpq-dev
RUN cargo install diesel_cli --no-default-features --features postgres
COPY . .
COPY --from=builder /usr/src/sumi/dist ./dist
COPY --from=builder /usr/src/sumi/target/release/sumi-backend ./sumi

It would be nice to not have to rebuild the both if only one was changed. Is this because Docker ignores cache after change was detected at COPY . .?

[vgwidt]

Touch is needed otherwise it may use the builds made in the cacher stage. Removed cargo-chef.

I'm pretty satisfied with the current state, but I still would like to get a smaller final image. That would require building diesel_cli by ourselves.

FROM rust:latest as base
RUN cargo install trunk
RUN rustup target add wasm32-unknown-unknown

FROM base as cacher
WORKDIR /usr/src/sumi
COPY ./Cargo.lock ./
COPY ./Cargo.toml ./
COPY ./frontend/Cargo.toml ./frontend/Cargo.toml
COPY ./backend/Cargo.toml ./backend/Cargo.toml
COPY ./frontend/.cargo ./frontend/.cargo
RUN mkdir ./backend/src && mkdir ./frontend/src && echo 'fn main() { println!("Dummy"); }' > ./backend/src/main.rs && echo 'fn main() { println!("Dummy"); } ' > ./frontend/src/lib.rs
RUN cargo build --release --manifest-path ./backend/Cargo.toml
WORKDIR /usr/src/sumi/frontend
RUN cargo build --release

FROM base as builder
WORKDIR /usr/src/sumi
COPY . .
COPY --from=cacher /usr/src/sumi/target target
RUN touch -a -m ./backend/src/main.rs && touch -a -m ./frontend/src/lib.rs
RUN trunk build -d dist ./frontend/index.html --release
RUN cargo build --manifest-path ./backend/Cargo.toml --release

FROM rust:latest
WORKDIR /usr/src/sumi
RUN apt-get install -y libpq-dev
RUN cargo install diesel_cli --no-default-features --features postgres
COPY . .
COPY --from=builder /usr/src/sumi/dist ./dist
COPY --from=builder /usr/src/sumi/target/release/sumi-backend ./sumi

[vgwidt]

Addition of shared broke the Dockerbuild, fixed in 5d7778c

[vgwidt]

https://docs.rs/diesel_migrations/latest/diesel_migrations/macro.embed_migrations.html can be used in lieu of diesel_cli. It reads the migrations at compile time.

[vgwidt]

Dependency caching is not working very well. I find myself rebuilding them most of the time without any changes to Cargo.toml or Cargo.lock.

[vgwidt]

In case I want to continue to use diesel_cli but build in builder stage:

In builder:

RUN cargo install diesel_cli --no-default-features --features postgres

In final image:

RUN apt-get -y update
RUN apt-get install -y libpq-dev
COPY --from=builder /usr/local/cargo/bin/diesel /usr/bin/diesel

libssl.so.1.1 doesn't exist in the bookworm image, but it does in the Rust, so we need to either move it or build it. It might look like this:

RUN apt-get install -y wget
RUN wget https://www.openssl.org/source/openssl-1.1.1o.tar.gz && tar -zxvf openssl-1.1.1o.tar.gz
WORKDIR ./openssl-1.1.1o
RUN ./config && make && make test
RUN mv libcrypto.so.1.1 /usr/bin/ && mv libssl.so.1.1 /usr/bin/