During summer 2017 I picked up Rust as one of my programming languages. Since this was my first compiled programming language (ignoring some of my early C antics), I was faced with one inherent problem: “How do I distribute pre-compiled binaries without getting SolarWinded?”. Granted, those might not have been my exact words back in Fall 2017, but I was intrigued by this problem and got involved in the reproducible builds project as a volunteer contributor.

About 4 years later this is now changing to part-time open source security research thanks to sponsoring by Google & The Linux Foundation!

I’m very excited about this and would like to send a shout-out to the many people I’ve met that have been part of my journey from “I’m going to sign up on GitHub and maybe submit a patch or 3” to “being able to work on open source security in a sustainable way”. ♥️

Reproducible Alpine Linux

If you’ve followed my work you might be aware that I’m currently trying to build Reproducible Alpine Linux images for the Raspberry Pi. There have been two walkthrough-style posts so far (part 1 and part 2), originally published on twitter on an account operated by me and my cat.

The patch for a reproducible apk index has been rewritten to implement SOURCE_DATE_EPOCH instead of hard-coding 1970-01-01. The patch was merged and released with the help of Ariadne and Timo Teräs, thanks a lot! (pull request)

Next up are patches for mkinitfs (using the --renumber-inodes feature implemented by Ariadne) and abuild-sign. I’m also excited about the upcoming buildinfo work in this space!

Reproducible Arch Linux

Arch Linux already has multiple independent groups running rebuilders but we’re always looking for more people interested in this! Please reach out in #archlinux-reproducible on libera if you want to run one!

Two rebuilders disagreed on the status of the cross package, a popular Rust cross-compile tool, so I’ve published a detailed writeup on “What if two rebuilders disagree and what that means” and sent a patch upstream (pull request). This has sparked some discussion in the Rust community since “reproducible builds and custom build.rs files” is a fairly novel topic.

On the triage side of reproducible Arch Linux there’s now a git repository at reproducible-archlinux-notes to track unreproducible packages, along with tooling to work on this. The current stats are:

[+] 896 (54.37%) unreproducible packages are classified

When grouped by root cause there are currently two major categories:

  • haskell related - This is mostly due to an issue in ghc, the haskell compiler, when used with concurrency (which is necessary when operating on Arch Linux-scale).
  • python bytecode - We ship .pyc files in Arch Linux, those are only reproducible if you run export PYTHONHASHSEED=0 before compiling the byte code, SOURCE_DATE_EPOCH is not enough unfortunately.

I’ve also submitted a patch to archlinux-repro (developed by Morten Linderud) to display an error message if no matching PKGBUILD could be found in asp (pull request). This may happen if the rebuilder picked up the package before the svn2git sync finished.

I’ve also sent a patch to archweb to fix a link in the rebuilderd integration (pull request). This integration was recently developed by Jelle van der Waa and made triaging reproducible-builds issues significantly easier.

The following packages have been fixed:

  • hwinfo (upstream) - fix timezone issue in SOURCE_DATE_EPOCH code
  • perl - partial fix, resolves embedded build times, hostname and kernel version
  • mitmproxy - PYTHONHASHSEED=0
  • wakatime - PYTHONHASHSEED=0
  • flake8 - PYTHONHASHSEED=0
  • gnome-pie - rebuild with reproducible devtools
  • wallutils - -trimpath should be passed instead of -gcflags and -asmflags
  • go-ipfs - -trimpath should be passed instead of -gcflags and -asmflags
  • choria-io - fix embedded build date
  • curlie - fix embedded build date
  • man-pages-l10n - fix embedded build date
  • phpvirtualbox - remove .git folder from package
  • autopep8 - PYTHONHASHSEED=0
  • blender - PYTHONHASHSEED=0
  • cdrtools - embedded uname
  • woob - PYTHONHASHSEED=0
  • svgcleaner - missing Cargo.lock file

The issues in the Arch Linux grub package have been triaged but there’s currently no fix yet.

rebuilderd

Rebuilderd (pronounced “rebuilder dee”) is a scheduler that automatically attempts to reproduce the packages distributed by Linux distributions - this is referred to as “rebuilding”. Rebuilderd is currently used by reproducible.archlinux.org and others to verify Arch Linux packages.

There has been a new v0.13.0 release, not much has changed, but a has_diffoscope: bool field was introduced to the api, so frontends can hide the diffoscope link if there’s none available (pull request).

I’ve also fixed an issue related to blocking reqwest clients in async code (pull request). This issue was found and reported by Joy Liu, a GSoC intern working on in-toto integration for rebuilderd. Thanks!

I’ve also started a rebuilderd channel in the CNCF slack to collaborate with the Secure Systems Lab Team at NYU. You’re very welcome to join if you’re interested in this!

Acknowledgments

This project was funded by Google, The Linux Foundation, and people like you and me through GitHub Sponsors. Without this support I wouldn’t be able to do all of this, thanks! ♥️♥️♥️