Monthly Report (September 2021)

This is the monthly report of what I’ve been up to in September 2021. 🙌

Reproducible Builds

There have been 3 releases of rebuilderd this month, 0.14.0, and two minor bugfix releases, 0.14.1 and 0.14.2.

The 0.14.0 release introduced experimental support to rebuild Tails images in #66. Tails is a portable operating system that’s known for it’s strong focus on privacy and security, and commonly used by activists, journalists and various human-rights NGOs. It already had reproducible images for a long time (since around 2017), but you had to reproduce the images manually. Starting with this release you can setup rebuilderd to monitor Tails for new releases and automatically attempt to recreate the release from source, on your own independent build system. If this succeeds, the result looks like this:

Since the Tails build uses virtual machines it has special requirements for the build system, specifically /dev/kvm needs to be available. The VPS hoster I usually use didn’t offer nested-kvm and I instead used a bare-metal server to develop this.

Special requirements for rebuilders are common, for example archlinux-repro uses systemd-nspawn to build in a clean environment and debrebuild uses mount(2) to setup a build container. mount(2) is considered a very privileged action in Linux and requires the CAP_SYS_ADMIN kernel capability, which is effectively equivalent to root access. Requirements like this are now documented in the README, along with a link to the rebuilder-backend that’s being used:

Because the rebuilderd and rebuildctl binaries have previously only been packaged in Arch Linux I’ve also created a package for Alpine Linux that’s now available in the [testing] repository (aports!25695).

Cryptographic rebuild attestations with in-toto

One of the most notable changes in rebuilderd 0.14.0 is support for in-toto attestations, contributed by Joy Liu during GSoC 2021 (#65, #67). Attestations are human readable and can be fetched like this:

% rebuildctl -H https://wolfpit.net/rebuild/ pkgs attestation --name zstd | jq .
{
  "signatures": [
    {
      "keyid": "585a2a5c5efec5fc22d84f7fa7a4a22cc1c62507cf97b6d8e7df7aaea8e5f659",
      "sig": "0b22ed56c69d362a66bd247a206e7f97e0208e45f2d24bf16bf2e7a3ba69bc73ddd1bf4a457bb57039dc570dc3827ea6350bc4972f899cc9c78f43b41b0d960c"
    }
  ],
  "signed": {
    "_type": "link",
    "byproducts": {},
    "env": {},
    "materials": {
      "/tmp/rebuilderdyeQ5vC/inputs/zstd-1.5.0-1-x86_64.pkg.tar.zst": {
        "sha256": "0414f4baa43aef409b235184e6079ee0dac8e692912e1da61a8ae264c7606670",
        "sha512": "21a56b70b072673159277a9049ac61831ada6712d00ea1f45b4ca1207196d8005332e9c1b6371b6c8615df7dcfa878a8d440c1d85985b83d63eb4e9f970f9e4a"
      }
    },
    "name": "rebuild zstd-1.5.0-1-x86_64.pkg.tar.zst",
    "products": {
      "/tmp/rebuilderdyeQ5vC/out/zstd-1.5.0-1-x86_64.pkg.tar.zst": {
        "sha256": "0414f4baa43aef409b235184e6079ee0dac8e692912e1da61a8ae264c7606670",
        "sha512": "21a56b70b072673159277a9049ac61831ada6712d00ea1f45b4ca1207196d8005332e9c1b6371b6c8615df7dcfa878a8d440c1d85985b83d63eb4e9f970f9e4a"
      }
    }
  }
}

An in-toto attestation is a signed data structure that (in this case) cryptographically documents a successful rebuild. From a reproducible builds point of view, the essentials you need to know about in-toto:

    "materials": {
      "/tmp/rebuilderdyeQ5vC/inputs/zstd-1.5.0-1-x86_64.pkg.tar.zst": {
        "sha256": "0414f4baa43aef409b235184e6079ee0dac8e692912e1da61a8ae264c7606670",
        "sha512": "21a56b70b072673159277a9049ac61831ada6712d00ea1f45b4ca1207196d8005332e9c1b6371b6c8615df7dcfa878a8d440c1d85985b83d63eb4e9f970f9e4a"
      }
    },

materials are the input files that have been used. Since this is an attestation for an Arch Linux package we’re going to need the buildinfo file that describes the build environment that has been used (which compiler version, which library versions, etc). Arch Linux embeds this info inside the package, that’s why the pre-compiled package that’s used by all Arch Linux users is also listed as a build input. The content of this file is identified using two cryptographic checksums, sha256 and sha512. This is mostly for documentation (similar to some of the unused sections like env and byproducts).

    "products": {
      "/tmp/rebuilderdyeQ5vC/out/zstd-1.5.0-1-x86_64.pkg.tar.zst": {
        "sha256": "0414f4baa43aef409b235184e6079ee0dac8e692912e1da61a8ae264c7606670",
        "sha512": "21a56b70b072673159277a9049ac61831ada6712d00ea1f45b4ca1207196d8005332e9c1b6371b6c8615df7dcfa878a8d440c1d85985b83d63eb4e9f970f9e4a"
      }
    }

products are the most important part of the attestation, it records the artifacts that have been built from source by the rebuilder. In this case there’s only one and since both the sha256 and sha512 checksums are identical, this indicates that this rebuilder has successfully reproduced the binary package from source.

    "name": "rebuild zstd-1.5.0-1-x86_64.pkg.tar.zst",

name describes what this signature is actually about. In this case it indicates that this is a rebuild attestation and which package has been rebuilt. This helps avoid signature-reuse attacks that are common in more traditional systems like OpenPGP where a) signing-intent is implied b) keys are tied to identity instead of purpose.

For example, my pgp key is present in both the Arch Linux and Debian keyring, from a cryptographic point of view my signed Arch Linux packages are also valid signed Debian uploads. The only defense against signature-reuse attacks with systems like this are the file format parsers (Debian wouldn’t accept a tar file as a valid upload, and pacman wouldn’t accept the Debian upload text-format as a valid package). Fingers crossed I never sign a polyglot. 🤞

Recording the purpose in the signature (like in-toto does) prevents attacks like this.

  "signatures": [
    {
      "keyid": "585a2a5c5efec5fc22d84f7fa7a4a22cc1c62507cf97b6d8e7df7aaea8e5f659",
      "sig": "0b22ed56c69d362a66bd247a206e7f97e0208e45f2d24bf16bf2e7a3ba69bc73ddd1bf4a457bb57039dc570dc3827ea6350bc4972f899cc9c78f43b41b0d960c"
    }
  ],

signatures is the cryptographic portion of the attestation. It documents which key created the signature, and the signature itself. Because signatures work on binary blobs instead of data structures, the in-toto spec defines how to serialize the data structure into a canonical binary representation. Serializing the same data structure is always going to give you the same serialized binary representation.

There are currently slightly over 1.000 attestations that have been issued so far across all rebuilders at the time of writing:

for x in 'https://reproducible.archlinux.org' 'https://r-b.engineering.nyu.edu' 'https://wolfpit.net/rebuild'; do
    echo -n "$x "; curl -sSfm30 "$x/api/v0/pkgs/list" | jq '.[] | select(.has_attestation) | .name' | wc -l
done

Having cryptographic attestations like this is useful to design more secure package authentication systems for package updates that rely on confirmations from multiple rebuilders. I’m looking forward to the first package-manager integrations that fetch in-toto attestations from rebuilderd!

Acknowledgments

This project was funded by Google, The Linux Foundation, and people like you and me through GitHub Sponsors. Without this support I wouldn’t be able to do all of this, thanks! ♥️♥️♥️