With Moore’s law coming to an end, optimizing code to avoid performance pitfalls is becoming more and more useful. To this end, programming languages like Rust are designed to produce fast and memory-efficient programs out-of-the-box. When that is not sufficient, profilers like perf are useful to measure where the code is slow and therefore which algorithms and data structures should be optimized.

Now, you may wonder why profiling Rust code within Docker, an engine to run containerized applications. The main advantage of containers is that when configured properly, they provide isolation from the rest of your system, allowing to restrict access to resources (files, networking, memory, etc.).

Isolation is useful when developing code, because you soon end up running untrusted code, especially with modern languages that facilitate adding dependencies, that themselves collect more and more dependencies. You probably don’t want to run with malicious code that sneaked into a dependency (as happened last year in the event-stream npm package) with root privileges, nor even as your user.

Even if you trust the code, a container allows to very easily clean up any artifacts (e.g. configuration files) added by your code (or dependencies), intentionally or not, and to restart from a clean state for reproducibility. It also allows to test if your code depends on any resources and if your code reports useful errors when these resources are unavailable.

The drawback of isolation is that you need quite a bit of configuration to restrict privileges while giving access to the needed resources. In this context, I want to share my journey in profiling Rust code within Docker containers, and to answer the following questions.

  • What is the minimal setup to install a Rust toolchain in a Docker container?
  • What privileges are necessary to compile and profile Rust code?
  • How to tell the Rust compiler to create a profiling-friendly binary?

The goal of this blog post is also to explain why I’ve chosen each parameter, instead of just dumping a configuration that happens to work for me.

Installation: Docker image

The first step is to create a Docker image which contains a Rust compiler and the perf tool. We’ll then run this image to build our Rust application and profile it.

In this section we’ll walk through the Dockerfile (Docker’s instructions to build an image).

Docker base image

First of all, I suggest to start with a Debian testing base image. This is a reasonably trusted base as it is maintained by the Docker project and used by many other images. Debian is a well-maintained distribution, and the testing version will give us access to recent updates of all packages, while being more stable than – well – Debian unstable (a.k.a. sid). We’ll also pick the slim version, which strips away a bunch of unneeded files, such as man pages.

FROM debian:bullseye-slim

Within the image, the next step I suggest is to create an unprivileged (non-root) user; let’s name is dev. We’ll associate it with a unique user ID such as 5000 so that it doesn’t clash with any other user on your (host) system, which provides an additional layer of isolation. Let’s also create a home directory, as the Rust compiler will expect it to store packages downloaded by Cargo.

RUN useradd --uid 5000 --create-home --shell /bin/bash dev

Perf

Perf is a Linux profiling system that can collect performance events from the CPU, but also many kinds of resources in the Linux kernel, including filesystems, networking, memory access, the scheduler, etc. Brendan Gregg’s website contains many examples of how to use perf. Chandler Carruth also gave a talk with a live demo walking through using perf to profile programs.

perf is tightly integrated in the Linux kernel, which exposes these performance counters via a custom syscall called perf_event_open. However, using this syscall directly would be quite cumbersome, so there is a program named perf that exposes a user-friendly interface to run and profile other programs.

To install it, you need the linux-perf and linux-base Debian packages.

RUN apt-get install -y linux-perf linux-base

Rust toolchain

In order to compile Rust programs, you need the Rust toolchain, which consists of rustc – the Rust compiler – and cargo – Rust’s package manager. There are multiple ways to install it.

  • If you want to use stable Rust, and have a Debian testing image (like we did), you can simply install the cargo package (which depends on the rustc package). At the time of writing, the version of rustc available on testing is very close to the latest stable Rust. 
    RUN apt-get install -y cargo
  • If you need Rust nightly, there is unfortunately no Debian package for it, as it changes too often and has no stability guarantees. In that case, you can use rustup (Rust’s own toolchain management tool). I suggest to use the following steps.
    • Download the rustup script with something like wget, and compare it with a known checksum to verify integrity. This requires the ca-certificates package, so that an HTTPS connection can be established with rustup’s website. You can do the comparison with sha256sum -c. You’ll have to update this checksum every now and then when rustup changes.
    • Install the nightly toolchain and remove the rustup script.
    • Move the $HOME/.cargo folder to somewhere else (e.g. $HOME/cargo), so that we can later overlay a tmpfs on $HOME/.cargo (see the next section).
    • Add $HOME/cargo/bin to the $PATH, so that the Rust toolchain binaries are found by the shell.
    • Additionally, you will likely need the build-essential package, as some Rust crates need a C++ compiler to build.
    RUN apt-get install -y ca-certificates wget build-essential
RUN wget https://sh.rustup.rs -O rustup.sh \ 
    && sha256sum rustup.sh \
    && echo "<checksum>  rustup.sh" | sha256sum -c - \
    && sh rustup.sh --default-toolchain none -y \
    && rm rustup.sh \
    && /home/dev/.cargo/bin/rustup default nightly \
    && mv /home/dev/.cargo /home/dev/cargo
ENV PATH=${PATH}:/home/dev/cargo/bin

Misc packages

Additionally, it can be useful to install the following Debian packages: git to download code, and ca-certificates to connect with HTTPS (e.g. for a git repository with an https:// URL).

RUN apt-get install -y git ca-certificates

Recap

Here are Dockerfiles summarizing the configuration.

FROM debian:bullseye-slim
RUN useradd --uid 5000 --create-home --shell /bin/bash dev

RUN apt-get update \
    && apt-get install -y \
        linux-perf \
        linux-base \
        cargo \
        ca-certificates \
        git \
        --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

USER dev
WORKDIR "/home/dev"

ENTRYPOINT [ "/bin/bash" ]

FROM debian:bullseye-slim
RUN useradd --uid 5000 --create-home --shell /bin/bash dev

RUN apt-get update \
    && apt-get install -y \
        linux-perf \
        linux-base \
        ca-certificates \
        wget \
        build-essential \
        git \
        --no-install-recommends \
    && rm -rf /var/lib/apt/lists/*

USER dev
WORKDIR "/home/dev"

RUN wget https://sh.rustup.rs -O rustup.sh \
    && sha256sum rustup.sh \
    && echo "9a165af7c7446f07a978c6ab618dc5080bfe918b2bc88e9ad0f6df985b4e4dcb  rustup.sh" | sha256sum -c - \
    && sh rustup.sh --default-toolchain none -y \
    && rm rustup.sh \
    && /home/dev/.cargo/bin/rustup default nightly \
    && mv /home/dev/.cargo /home/dev/cargo

ENV PATH=${PATH}:/home/dev/cargo/bin

ENTRYPOINT [ "/bin/bash" ]

Running the Docker image with limited privileges

One advantage of Docker is that besides packaging your code and dependencies in a hermetic/reproducible environment, running code in a container is more isolated than a process running directly as your user. However, you need to be careful about what privileges you give to the container: running code as root (or even worse, in a privileged container) can be less secure than not using a container at all.

Unfortunately, the defaults of docker run still give quite a few unnecessary privileges, so I’ll quickly go through flags to tighten your container privileges. I’d also recommend to check the guidelines for Docker security published by OWASP.

We’ll also need to add one permission so that perf can do its profiling job.

Before we start, I strongly recommend NOT to add your user to the docker group. This may avoid you typing a few sudo, but as explained here, this is equivalent to granting root privileges to your user, due to the attack surface exposed by the Docker daemon. I don’t think that the reduced sudo-typing friction is worth the loss in security.

Running a temporary container in the foreground

Before looking at the security configuration, I just want to mention two useful flags when spawning a new container with docker run. These allow to start a temporary container with a shell in the foreground.

  • --rm, so that the container is automatically removed upon exiting.
  • -it, to run the container in foreground mode. This is actually a combination of two flags, -t (a.k.a. --tty) to allocate a pseudo-TTY, and -i (a.k.a. --interactive) to keep stdin open. This allows to run an interactive shell in the container (in our case, the Dockerfile’s ENTRYPOINT is /bin/bash).

Locking down the container

To reduce the attack surface when running Docker containers, I suggest to use the following flags when running docker run.

  • --cap-drop=all, to remove all privileged capabilities. You can then manually whitelist some capabilities, but we won’t need any.
  • --security-opt no-new-privileges, to prevent privilege escalation via setuid/setgid binaries.
  • --read-only, to mount the image’s file system as read-only. We’ll then add tmpfs mounts to have writable folders to build the code.
  • -u dev, to use our unprivileged dev user instead of root. Note that we’ve also specified it in the Dockerfile, but better be safe than sorry.
  • If you don’t need any networking, use --network=none. Unfortunately, Cargo needs networking to download package dependencies, so we’ll just leave the default, a bridge network distinct from the host network. See also Docker documentation on networking.

I also suggest the following flags to limit the resources available within the container.

  • --cpus 4, to only allow the container to use 4 CPUs.
  • --memory=1024m, to limit the RAM used by the container to 1 GB. The Rust compiler can be quite memory-hungry, so it’s hard to go below this limit for non-trivial cases.
  • --memory-swap=1024m, to limit the RAM + swap to 1 GB. This effectively prevents the container from using the swap (which could slow down your program and/or wear out your non-volatile storage).
  • --memory-swappiness=0, to turn off anonymous page swapping. This also prevents the container from using swap.

Last, here are a few security recommendations beyond docker run.

  • Keep your kernel up-to-date! Given that the kernel is shared between the host and the container, a kernel with public vulnerabilities undermines the isolation provided by Docker.
  • Given that we don’t need this feature, you should disable inter-container communication at the Docker daemon level. On Linux, this is done by adding an "icc": false entry in /etc/docker/daemon.json (you can create this JSON file if it doesn’t exist). After that, run sudo service docker restart to apply the changes (if using systemd to manage the Docker daemon). You can check that ICC is disabled by running sudo iptables -L -n -v and looking for the following entry in the FORWARD rule. 
    Chain FORWARD
pkts bytes target prot opt in      out     source   destination
...
   0     0 DROP   all  --  docker0 docker0 anywhere anywhere

Custom tmpfs mounts to build Rust code

In order to build code, the Rust toolchain needs to write some files. Because we passed the --read-only flag to run our container, we’ll need to configure some folders as writable. One way to do it is with a tmpfs, a temporary file system backed by volatile memory (RAM).

This has several advantages:

  • Memory is fast, because it’s backed by RAM.
  • It doesn’t wear your non-volatile storage (hard drive, SSD), because you don’t write on it.
  • It doesn’t “pollute” the host’s file system, as everything is temporary.

There are a few drawbacks as well, but the advantages usually outweigh them:

  • RAM capacity is more limited, but 1 GB is usually enough to compile an average Rust crate.
  • It’s not persistent, so the Rust compiler will have to download all dependencies every time you start a container.

Setting up a tmpfs with Docker is quite simple. You need to pass the --tmpfs flag along with a directory where to mount the tmpfs. You can also set a size limit, and configure options such as whether executables are allowed to run in the tmpfs.

In our case, we’ll need three folders:

  • $HOME/rust, where we’ll build and run our code,
  • $HOME/.cargo, where the Cargo package manager will download dependencies,
  • /tmp, so that the Rust compiler can write some temporary files.

Given that we’ll run our compiled code (to profile it), we need to set the exec bit in the first folder. The following size parameters are usually enough for common cases, but don’t hesitate to increase them if you get No space left on device (a.k.a. ENOSPC) errors.

sudo docker run \
    --tmpfs /home/dev/rust:exec,size=256m \
    --tmpfs /home/dev/.cargo:size=256m \
    --tmpfs /tmp:size=64m \
    ...

Custom seccomp-bpf profile for perf

By default, Docker uses seccomp-pbf (a secure computing mode feature of the Linux kernel) to ban some system calls from ever being run inside the container. The Docker documentation on seccomp explains why a bunch of system calls are not allowed. You can also learn about it in jessfraz’s blog post.

The main thing to know is that perf needs to use the perf_event_open system call to collect profiling information about your program, and that this system call is not allowed by Docker’s default profile (the rationale is that is is a « tracing/profiling syscall, which could leak a lot of information on the host. »).

To generate a custom profile, you can download the current default profile on Docker’s GitHub repository. Then make a copy of this JSON file (let’s call it seccomp-perf.json) and add perf_event_open to the long list of system calls around the beginning of the file. You can then use this custom profile with the following flag.

sudo docker run \
    --security-opt seccomp=seccomp-perf.json \
    ...

Don’t forget to re-download the default profile regularly to update it, as it changes quite frequently! Unfortunately, I don’t know of a way to tell Docker to use the default profile with a custom diff on top of it.

Recap

Here is a summary to build the Docker image and start a Docker container with suitable flags for profiling Rust code.

# Create a folder containing your Dockerfile.
mkdir rust-profiling
cp Dockerfile rust-profiling-folder/Dockerfile

# Build the Docker image.
sudo docker build -t rust-profiling rust-profiling-folder

# Launch a Docker container based on this image.
sudo docker run \
    --rm \
    -it \
    --cap-drop=all \
    --security-opt no-new-privileges \
    --read-only \
    -u dev \
    --cpus 4 \
    --memory=1024m \
    --memory-swap=1024m \
    --memory-swappiness=0 \
    --tmpfs /home/dev/rust:exec,size=256m \
    --tmpfs /home/dev/.cargo:size=256m \
    --tmpfs /tmp:size=64m \
    --security-opt seccomp=seccomp-perf.json \
    rust-profiling

Profiling Rust code

We now have a Docker image with a Rust toolchain and perf installed, and we know how to launch it in a container with suitable flags. Let’s compile and profile your Rust code!

Compiler flags to build a profiling-friendly binary

In order to take the most out of our profiling, we’ll need to adapt how we compile Rust code.

The first thing to do is to compile in release mode with cargo build --release, to obtain an optimized binary representative of what you run in production.

Frame pointers

However, one of the optimizations applied by the compiler with the --release flag is to omit the so-called frame pointers. In compiled languages like Rust or C, each function has a frame pointer which indicates the starting address of the function’s stack frame and is commonly stored in a specific CPU register (ebp on x86). You can learn more about frame pointers on Wikipedia or in these lecture notes.

However, the fact that ebp contains the frame pointer is mostly a convention (the CPU itself doesn’t have a concept of frames) and a compiler can omit updating the frame pointer for every function call, to save a few instructions and maybe use epb for other computation.

Back to profiling, the important part is that the frame pointer allows perf to retrieve the call stack when the program is interrupted by a perf event. This allows to retrieve which functions are run the most often (the goal of a profiler). Without these call traces, the output is not very useful.

So the second thing to do is to tell Cargo to keep the frame pointers. This will modify a little your binary compared to production, but in practice the difference is often negligible, and the benefit is that you get actionable profiling output. The easiest way is to use the RUSTFLAGS environment variable when running Cargo. You’ll probably want to clean your build first, so that Cargo recompiles all your code and dependencies with frame pointers.

cargo clean
RUSTFLAGS='-C force-frame-pointers=y' cargo build --release

Running with perf

Now that we have some binary, let’s profile it! This is actually the easy part, you simply have to run the following commands in the container.

perf record -g <binary>
perf report -g graph,0.5,caller

First perf record will call your binary and record perf events. The -g argument tells perf to record the call graph.

Then, perf report will open an interface in the terminal where you can navigate functions and assembly to figure out which code is run the most often.

asciicast Interactive demo of perf, recorded with asciinema (click to replay the recording).

On some systems, you may encounter an error saying that you don’t have permissions to collect stats. This is because the dev user running in the container doesn’t have CAP_SYS_ADMIN privileges (which we certainly don’t want to grant!), and perf event collection is restricted to these privileges by default (a good secure-by-default setting!).

To enable the relevant profiling, run the following command on the host (with root privileges).

echo 2 > /proc/sys/kernel/perf_event_paranoid

To disable profiling, toggle the setting back to 3.

echo 3 > /proc/sys/kernel/perf_event_paranoid

To learn more about how perf works and how to get the most of its user interface, I recommend watching Chandler Carruth’s talk.

If however you aren’t a big fan of terminal-based interfaces or are afraid of reading assembly, there’s another way of looking at profiling data.

Flame graphs

As we have seen, perf allows you to record events and walk through the call graph down to the assembly level to see which functions and instructions are executed most often. While this low-level command-line interface is powerful, it’s not necessarily the most intuitive way to get an overview of your profiled code.

Flame graph Flame graph profile of lzma-rs (click to open an interactive view).

Flame graphs provide a complementary visualization:

  • Each function is represented by a bar proportional to the time spent in that function. This allows to quickly see which functions are in the hot path.
  • Functions are stacked on top of each other according to function calls: if a function f calls g and h, then the bars for g and h will be stacked on top of f’s bar. This allows to understand how much of the time spent on a function is due to recursive calls to other functions.

Brendan Gregg’s published useful tools on GitHub to create flame graphs from perf output. Let’s use them to create a nice flame graph in SVG format in our container!

# Let's just clone the FlameGraph repository from within our container.
git clone https://github.com/brendangregg/FlameGraph
# Record with perf as usual (including call graph).
perf record -g <binary>
# Generate a flame graph.
perf script | ./FlameGraph/stackcollapse-perf.pl > out.perf-folded
./FlameGraph/flamegraph.pl out.perf-folded > rust-perf.svg

That’s it! We now have an SVG image containing our interactive flame graph.

Exfiltrating a file from our container

However, we don’t have anything to display SVG files in our Docker image…

We could add a program to visualize images to our Dockerfile, but our docker run setup doesn’t give any permissions to interact with the windowing system, so programs within the container cannot show anything on screen (apart from writing to the console). We could also configure that, but doing it securely would be a long blog post in and of itself1.

Instead, we can copy the SVG file out of the container, and then display it on the host. There is a command to do that – docker cp – but unfortunately it doesn’t work on tmpfs file systems.

A workaround is to use docker exec to exfiltrate the file contents via the terminal. docker exec is a useful tool allowing to run an additional command in an existing container, e.g. to inspect the contents of the container. To use it, you need to know the ID of the container, which you can obtain by running docker ps and look at the hexadecimal CONTAINER ID (the mnemonic NAME is also a valid identifier).

Back to exfiltrating the file, Docker’s documentation gives an example with tar, but we can simply use cat.

sudo docker exec <container_id> cat /path/to/container/file.svg > /path/to/host/file.svg

If you also find Bash syntax confusing, this command should be understood as follows.

(sudo (docker exec <container_id> (cat /path/to/container/file.svg))) > /path/to/host/file.svg

When you run it on the host shell, Docker will run cat /path/to/container/file.svg inside the container, sending the contents to stdout. The piping > /path/to/host/file.svg then happens on the host shell, taking the data passing through the container’s stdout and writing it to a file on the host.

Rust library in a C++ application

Another use case you might have is to compile a Rust library, and call it from another language like C++. For example, you had a C++ code base and migrated part of it to Rust. The binary that you are profiling is then a hybrid of Rust and C++. It turns out that perf can perfectly handle this case!

As before, you can compile your Rust library with RUSTFLAGS='-C force-frame-pointers=y' cargo build --release. The easiest is to create a static library (.a on Linux).

On the C++ side, the relevant compiler flag to keep frame pointers is -fno-omit-frame-pointer. You can then link your Rust library with linker flags such as -L/path/to/your/rust/library -lfoo (for a library at /path/to/your/rust/library/libfoo.a).

With that, the usual perf and flame graph commands should work on the resulting binary.

Conclusion

You should now be able to compile Rust programs and profile them inside Docker containers! As we have seen, although we ended up doing quite a bit of configuration, this doesn’t require any unstable or unsafe feature of Rust.

If you want to analyze the performance of Rust programs from a different angle, Cargo has a built-in feature: the cargo bench command. Although it requires the nightly compiler, this is a great integrated way of running micro-benchmarks. All you need is to tag a function to benchmark with the #[bench] attribute, and enable #![feature(test)]. You can find some examples in my lzma-rs crate or my Rust implementation of Gravity-SPHINCS cryptographic signatures.

asciicast Cargo bench of gravity-rs, recorded with asciinema (click to replay the recording).

This post was edited to clarify the risk of adding your user to the docker group, and to recommend that the Docker daemon be invoked with sudo. It was also edited to add the --default-toolchain none parameter to rustup.sh when installing the nightly toolchain.

  1. The /tmp/.X11-unix path could be mounted to the container, but this gives broad access to anything happening on your display, including keyboard/mouse inputs (no need to be the window on focus to collect them), clipboard contents, etc. See The Linux Security Circus: On GUI isolation by Joanna Rutkowska. ↩

Comments

To react to this blog post please check the Twitter thread.

RSS | Mastodon | GitHub

You may also like

Lessons learned from stracing a password manager in Docker
Optimization adventures: making a parallel Rust workload 10x faster with (or without) Rayon
Detecting SIMD support on ARM with Android (and patching the Rust compiler for it)
Making my website 10x smaller in 2024, with a dark mode
And 32 more posts on this blog!