This is the monthly report of what I’ve been up to in August 2021. 🙌

Reproducible Builds

There are many different reasons to be interested in Reproducible Builds. When I originally got involved in the project I wasn’t a maintainer in any Linux distribution yet, instead I was wondering if there’s a way to distribute pre-compiled artifacts as an independent open source dev without carrying all the responsibility alone.

A few years later I’ve now published a manual called i-probably-didnt-backdoor-this. It contains a hello world program and instructions on how to reproduce the various pre-compiled artifacts, explains all build instructions and why these controls are effective.

A similar project has been published in response by Michael Lieberman. This project also distributes binaries that can be rebuilt with very simple commands using NixOS and Bazel.

The project also got a shout-out at debconf21 in a talk about Reproducible Builds by Holger Levsen (around 11:23).

Reproducible Arch Linux

Quite a few (but not all) of the remaining 13% unreproducible packages on Arch Linux are unreproducible due to python bytecode files (.pyc). Debian doesn’t distribute .pyc files but because both Arch Linux and Alpine do I’ve published a blog post about Reproducible Python Bytecode. Levente ‘anthraxx’ Polyak of the Arch Linux Security Team was also involved in the investigation and a patch for pacman-6.0.1 was suggested by Allan McRae on the pacman-dev email list. The blog post has also been featured in Python Weekly!

The rebuilder setup of Reproducible Arch Linux has been mentioned in Looking Forward to Reproducible Builds (around 3:19) by Vagrant Cascadian. I’d recommend watching the whole video if you’re interested in rebuilders, regardless of the distribution you’re using!

I found two bugs in archlinux-repro and submitted patches #101, #102.

The following packages have been fixed:

• v2ray-domain-list-community (upstream) - Sort list before marshaling into dlc.dat
• gitlab-workhorse (upstream) - Add SOURCE_DATE_EPOCH support for build-time
• podman (upstream) - Fix embedded build-paths with -trimpath
• ktoblzcheck - Don’t download new data at build time
• calibre - PYTHONHASHSEED=0
• notion - Disable embedded build time
• gauche (upstream) - Disable embedded timestamps in info page compression
• swift.im (upstream) - Sort directory contents at build time

rebuilderd

rebuilderd is one of the components powering reproducible.archlinux.org. It monitors release artifacts (like the packages of a Linux distribution) and schedules rebuilds using supported rebuild backends.

Joy Liu worked on in-toto for Google Summer of Code 2021 and developed code for both in-toto-rs and rebuilderd to add in-toto attestation capabilities. This allows cryptographically verifying rebuild attestation in-toto/rebuilderd#1, #65.

Santiago Torres has contributed major fixes for the work-in-progress debian integration with debrebuild #59.

Aditya Sirish who’s operating the NYU rebuilder has discovered that rebuilderd can’t handle some compression formats for pacman database files, this has been fixed in #62.

Rebuilderd is now able to track which rebuilder-backend a worker needs to rebuild the package. A worker that is setup to rebuild an Arch Linux package might not have the setup to rebuild Debian packages too #64. The systemd unit for the rebuilder worker has been updated to use idle cpu and io priority #60. The debrebuild flag to specify the output directory is now passed correctly #63.

Binary Transparency

This isn’t directly related to Reproducible Builds but an important component for supply chain security as well. Binary Transparency helps if an attacker has gained control over an update signing key. Since a compromised update signing key is a very valuable asset for an attacker, they might hesitate to officially upload a malicious update to the distro’s archive. Instead they could use the key to target specific high-profile individuals directly to avoid burning the key. This kind of attack is very likely to go unnoticed.

Because in this scenario the attacker controls a signing key we can’t trust the signature alone, but if the update system requires Binary Transparency proofs the attacker is forced to log their malicious update to the transparency log. This doesn’t necessarily prevent the install of the malicious package, but it would likely get flagged as an incident by a transparency log monitor, making it a rather unappealing approach for the attacker.

I’ve released pacman-bintrans which acts as an experimental download plugin for pacman that performs additional security checks. I had an old codebase from a failed attempt to implement this with certificate transparency and SCTs, I’ve removed this part and replaced it with the rekor transparency log from the sigstore project.

You can already use this on your Arch Linux system today to ensure all updates you’re downloading have been properly logged to the transparency log, but keep in mind there’s no monitor auditing the log yet (although all data to implement this is public). The transparency signatures are hosted on https://pacman-bintrans.vulns.xyz/ and http://2iz5fzvuwjapcv5v2msvhlr5oqbwriznwu7hnsccrsho47ljqynrgryd.onion/. The transparency log used for this is the public rekor instance at https://rekor.sigstore.dev.

Luke Hinds from Red Hat sent a shout-out in his talk OCB: sigstore, Software Signing for All (around the 34:40 mark).

The rekor package in Arch Linux has been updated to include shell completions. I found a very minor bug that is now fixed upstream as well! I’ve also started working on an alpine package for rekor.

Acknowledgments

This project was funded by Google, The Linux Foundation, and people like you and me through GitHub Sponsors. Without this support I wouldn’t be able to do all of this, thanks! ♥️♥️♥️