rebuilderd 0.15.0 very recently released, this is a short intro into what it is, how it works and how to build our own integrations!

rebuilderd monitors an index of artifacts and parses it into a datastructure that looks like this. In the most basic case, based on the distro field it’s going to pick the right build script and attempt to generate an artifact identical to the file linked to in url.

We’re starting with a script that generates a json. In our case we’ll simply hard-code all values for demonstration purpose. Most of these values can be arbitrary strings and are simply used to sort packages into different buckets.

Let’s run the script and pipe it through jq for formatting and pretty colors. This describes one build that produces a single output-artifact.

Next up we’re going to import it into rebuilderd! Unless you’re on a distro that ships official rebuilderd packages, the easiest way to get started is cloning the repo and using docker-compose up daemon.

Test everything is working correctly with REBUILDERD_COOKIE_PATH=secret/auth cargo run --bin rebuildctl -- status you might have to set the permissions of ./secret/auth to 644 because the default permissions are fairly strict. If you don’t get errors everything is good to go.

Next up we’re going to import our hello-world package with the script we wrote in the first tweets. The arguments for this command need to match with the json.

If everything has worked correctly rebuildctl pkgs ls should now print that there’s one known package that’s in “unknown” state. This means the rebuild did neither succeed nor fail yet.

This has also been scheduled in the build queue with rebuildctl queue ls. Next up we need a worker that’s going to pick up this job, attempt the build and then report back the result. This is split into a different process so can run the builds on different servers.

These are usually wrapper shell scripts that wrap around the heavy lifting that’s implemented in rebuilder backends like archlinux-repro or debrebuild, but can be anything you want. This script does some basic math and then creates a file with hello world in the output folder.

Testing the script is working correctly…

Next edit the example config file at contrib/confs/rebuilderd-worker.conf, set enabled = true in the diffoscope section (or leave at false if you don’t want to install diffoscope). We also need to configure our script as rebuilder backend for our example “foo” distro.

Usually you’d use absolute paths but because we’re running the worker from our shell it should be fine this time. Remember that we’ve provided a url for our hello.txt artifact? We need an http server that hosts this for us. We’re going to let this fail intentionally first.

Open another terminal and change the directory into the rebuilderd checkout again. Then run the rebuilderd-worker and instruct it to connect to the local instance. If something gues wrong you can requeue the package with rebuildctl pkgs requeue.

The rebuild has been marked as BAD, as expected. Let’s have a look at the diffoscope. With REBUILDERD_COOKIE_PATH=secret/auth cargo run --bin rebuildctl -- pkgs diffoscope --name hello-world we see both the line from our http server and the line from our script.

Usually we wouldn’t use rebuilderd with basic text files and diffoscope is useful to diff complex nested packages. Let’s put the correct file on our local http server and try again.

It worked! The artifact that was generated by our build script was bit-for-bit identical with the reference file we downloaded.

Wrapping up for today, this work is currently funded by: myself and github sponsors! I’ve also setup a patreon account. Thanks! <3