Reproducible Builds: Debian and the case of the missing version string

If you’ve been following my twitter recently you probably noticed there’s now a rebuilderd based Debian rebuilder run by the Purdue Trustworthy Software Ecosystems Lab. The rebuilder backend - the code that’s actually re-creating the build environment and running the build - is debrebuild.py, written by Frédéric Pierret from the QubesOS project. The setup as a whole automatically monitors packages in Debian unstable, then downloads the source code, build-dependencies and attempts to compile a bit-for-bit identical binary package. If this succeeds, the package is marked as “reproducible”.

The 62.89% reproducible number is currently significantly lower than the 94.6% reproducible number¹ reported at tests.reproducible-builds.org/debian/. This blogpost is diving into why that is and why there are different challenges in “rebuilding” done in this setup vs “build environment fuzzing”² done by tests.reproducible-builds.org.

This is partially due to complexity about versions, let’s look into how other distros are approaching reproducible builds first:

Arch Linux

pkg version -> [ svntogit-*.git ] +-> source code
   |                              `-> build instructions
   |
   |
   `--> [ archive.archlinux.org ] -> *.pkg.tar.zst -> .BUILDINFO -> build dependencies (exact versions)

To build a reproducible binary we need a canonical (source code, build instructions, build dependencies with exact versions) combination that both the original build and all verification rebuilds are using.

In Arch Linux this information is spread out over multiple places (which is fine). There’s a git repository that can be used to locate the PKGBUILD, it links to the source code (pinned by cryptographic checksums) and contains the build instructions, but the PKGBUILD only specifies the build dependencies by name, not by version. We can’t use this alone for reproducible builds because different compiler versions are almost guaranteed to result in different binaries (unreproducible packages caused by mirror drift over time).

The “resolved dependencies” (aka “the build dependencies with exact versions”) are effectively tracked on archive.archlinux.org inside the packages. The (name, version) tuple of a package is required to be unique/canonical within Arch Linux, so we can use the package archive to resolve a pkg version to a canonical binary package that contains the buildinfo file. This file, together with the source code and build instructions we already had, gets us our required (source code, build instructions, build dependencies with exact versions) combination.

Neat!

Debian

pkg version -> [ source packages ] +-> source code
   |                               `-> build instructions
   |
   |
   `--> [ buildinfos.debian.net ] -> build dependencies (exact versions)

This looks very similar to what Arch Linux does (slightly simpler, even). Every binary package is built out of something called a “source package”, which contains both the source code, the build instructions and the dependencies (also by name, with no version).

The resolved dependencies are tracked on buildinfos.debian.net³. Using the extra info in the buildinfo file we get our canonical (souce code, build instructions, build dependencies with exact versions) combination and are able to implement a rebuilder.

Debian version strings (the happy path)

We still need to locate the buildinfo file though. Let’s look at the happy-path first using a package called “sniffglue”:

The Packages.xz is the index that’s used by apt to determine which packages are available in Debian. It contains entries like this:

Package: sniffglue
Source: rust-sniffglue
Version: 0.14.0-2
Installed-Size: 2344
Maintainer: Debian Rust Maintainers <[email protected]>
Architecture: amd64
Depends: libc6 (>= 2.32), libgcc-s1 (>= 4.2), libpcap0.8 (>= 1.5.1), libseccomp2 (>= 0.0.0~20120605)
Description: Secure multithreaded packet sniffer
Multi-Arch: allowed
Built-Using: rust-nix (= 0.23.0-1), rust-pktparse (= 0.5.0-1), rust-seccomp-sys (= 0.1.3-1), rustc (= 1.56.0+dfsg1-2)
Description-md5: e7f1183e49341488d3bd8fbe63b63f37
X-Cargo-Built-Using: rust-aho-corasick (= 0.7.10-1), rust-ansi-term (= 0.12.1-1), rust-anyhow (= 1.0.44-2), rust-arrayvec (= 0.5.1-1), rust-atty (= 0.2.14-2), rust-base64 (= 0.13.0-1), rust-bitflags (= 1.2.1-1), rust-block-buffer (= 0.9.0-4), rust-block-padding (= 0.2.1-1), rust-bstr (= 0.2.17-1), rust-byteorder (= 1.4.3-2), rust-cfg-if-0.1 (= 0.1.10-2), rust-cfg-if (= 1.0.0-1), rust-clap (= 2.33.3-1), rust-cpuid-bool (= 0.1.2-4), rust-dhcp4r (= 0.2.0-1), rust-digest (= 0.9.0-1), rust-dirs-next (= 2.0.0-1), rust-dirs-sys-next (= 0.1.1-1), rust-dns-parser (= 0.8.0-1), rust-enum-primitive (= 0.1.1-1), rust-env-logger (= 0.9.0-1), rust-generic-array (= 0.14.4-1), rust-humantime (= 2.1.0-1), rust-itoa (= 0.4.3-1), rust-lazy-static (= 1.4.0-1), rust-lexical-core (= 0.4.8-3), rust-libc (= 0.2.103-1), rust-log (= 0.4.11-2), rust-memchr (= 2.4.1-1), rust-memoffset (= 0.6.4-1), rust-nix (= 0.23.0-1), rust-nom (= 5.0.1-4), rust-num-cpus (= 1.13.0-1), rust-num-traits (= 0.2.14-1), rust-opaque-debug (= 0.3.0-1), rust-pcap-sys (= 0.1.3-2), rust-phf (= 0.8.0-2), rust-phf-shared (= 0.8.0-1), rust-pktparse (= 0.5.0-1), rust-quick-error (= 1.2.3-1), rust-reduce (= 0.1.1-1), rust-regex-automata (= 0.1.8-2), rust-regex (= 1.5.4-1), rust-regex-syntax (= 0.6.25-1), rust-rusticata-macros (= 2.0.4-1), rust-ryu (= 1.0.2-1), rust-seccomp-sys (= 0.1.3-1), rust-serde (= 1.0.130-2), rust-serde-json (= 1.0.41-1), rust-sha2 (= 0.9.2-2), rust-siphasher (= 0.3.1-1), rust-static-assertions (= 1.1.0-1), rust-strsim (= 0.9.3-1), rust-structopt (= 0.3.20-1), rust-strum (= 0.19.2-1), rust-syscallz (= 0.15.0-1), rust-termcolor (= 1.1.0-1), rust-textwrap (= 0.11.0-1), rust-time (= 0.1.42-1), rust-tls-parser (= 0.9.2-3), rust-toml (= 0.5.8-1), rust-typenum (= 1.12.0-1), rust-unicode-width (= 0.1.8-1), rust-users (= 0.11.0-1), rust-vec-map (= 0.8.1-2), rustc (= 1.56.0+dfsg1-2)
Section: net
Priority: optional
Filename: pool/main/r/rust-sniffglue/sniffglue_0.14.0-2_amd64.deb
Size: 732980
MD5sum: 177f9229266ad5eef3fb42fff0c07345
SHA256: 448c781a9e594227bc9f0d6c65b8beba2b3add68d3583020de188d4cfa365b40

Package: - this is the name of the binary package as used by apt.
Source: - this is the name of the source package. There can be multiple source packages with the same name in Sources.xz at the same time, if that’s the case this line may contain a version string in parentheses.
Version: - this is the version of the binary package as used by apt.
Architecture: - this is the architecture this package was built for.
Filename: - this value is used by apt to generate a url in order to download the .deb

The Sources.xz is an index of all source packages, it’s only relevant if you want to compile Debian packages yourself. It contains entries like this:

Package: rust-sniffglue
Binary: librust-sniffglue-dev, sniffglue
Version: 0.14.0-2
Maintainer: Debian Rust Maintainers <[email protected]>
Uploaders: kpcyrd <[email protected]>
Build-Depends: debhelper (>= 12), dh-cargo (>= 25), cargo:native, rustc:native, libstd-rust-dev, librust-ansi-term-0.12+default-dev, librust-anyhow-1+default-dev, librust-atty-0.2+default-dev, librust-base64-0.13+default-dev, librust-bstr-0.2+default-dev (>= 0.2.12-~~), librust-dhcp4r-0.2+default-dev, librust-dirs-next-2+default-dev, librust-dns-parser-0.8+default-dev, librust-env-logger-0.9+default-dev, librust-libc-0.2+default-dev, librust-log-0.4+default-dev, librust-nix-0.23+default-dev, librust-nom-5+default-dev, librust-num-cpus-1+default-dev (>= 1.6-~~), librust-pcap-sys-0.1+default-dev (>= 0.1.3-~~), librust-pktparse-0.5+default-dev, librust-pktparse-0.5+serde-dev, librust-reduce-0.1+default-dev (>= 0.1.1-~~), librust-serde-1+default-dev, librust-serde-derive-1+default-dev, librust-serde-json-1+default-dev, librust-sha2-0.9+default-dev, librust-structopt-0.3+default-dev, librust-syscallz-0.15+default-dev, librust-tls-parser-0.9+default-dev, librust-toml-0.5+default-dev, librust-users-0.11+default-dev
Architecture: any
Standards-Version: 4.5.1
Format: 3.0 (quilt)
Files:
 cdb6bf3fb7a8725986b313a65dd3339b 3079 rust-sniffglue_0.14.0-2.dsc
 80a7d2ab6becacf69213d9ce57d29274 134805 rust-sniffglue_0.14.0.orig.tar.gz
 9902c121e4f5cf268b19ee8b88201979 7816 rust-sniffglue_0.14.0-2.debian.tar.xz
Vcs-Browser: https://salsa.debian.org/rust-team/debcargo-conf/tree/master/src/sniffglue
Vcs-Git: https://salsa.debian.org/rust-team/debcargo-conf.git [src/sniffglue]
Checksums-Sha256:
 b9a77f9f918769ecded338c07a344257b17d1112918b1597a8c939a719444ea4 3079 rust-sniffglue_0.14.0-2.dsc
 f056bfa09e8fae5f4cc0e1d4e8ae3619050644b321800d0d6a8cc778eb80aaf3 134805 rust-sniffglue_0.14.0.orig.tar.gz
 cb3498dd85e18e7b2c7ad5cbef2ac56e4c598df0a0ac5024aa480f97de79b096 7816 rust-sniffglue_0.14.0-2.debian.tar.xz
Package-List: 
 librust-sniffglue-dev deb net optional arch=any
 sniffglue deb net optional arch=any
Testsuite: autopkgtest
Testsuite-Triggers: dh-cargo
Directory: pool/main/r/rust-sniffglue
Priority: extra
Section: misc

Package: - this is the name of the source package (this is matched with Source: from the previous file).
Binary: - this is the list of binary packages that are built out of this source package, in this case our source package results in two binary packages when built.
Version:- this is the version of our source package, here it’s identical to the version of the binary package but this isn’t always the case, more on that later.

Ok now using all of these together we try to locate the buildinfo file like this:

"https://buildinfos.debian.net/buildinfo-${Source:Directory}/${Source:Package}_${Source:Version}_${Binary:Architecture}.buildinfo"

This results in⁴:

https://buildinfos.debian.net/buildinfo-pool/r/rust-sniffglue/rust-sniffglue_0.14.0-2_amd64.buildinfo

We try to open the link and get something that looks like this:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.0
Source: rust-sniffglue
Binary: librust-sniffglue-dev sniffglue sniffglue-dbgsym
Architecture: amd64
Version: 0.14.0-2
Checksums-Md5:
 6d57946f2f56b1b58d906eb29d6a021f 125404 librust-sniffglue-dev_0.14.0-2_amd64.deb
 20fd28a9824a2c485d2e9b155b0a073c 8995232 sniffglue-dbgsym_0.14.0-2_amd64.deb
 177f9229266ad5eef3fb42fff0c07345 732980 sniffglue_0.14.0-2_amd64.deb
Checksums-Sha1:
 9fdde2501245e7db1eb2eaa104922b1b2b05fe57 125404 librust-sniffglue-dev_0.14.0-2_amd64.deb
 0dc99414756c846aadee63484642958c1ca7ff60 8995232 sniffglue-dbgsym_0.14.0-2_amd64.deb
 d2e4f34a46527effd3375764463d2c1bbe3eeecc 732980 sniffglue_0.14.0-2_amd64.deb
Checksums-Sha256:
 c452054c216359ef44adc9a5d35870d707f47e503051dfcb736f47df17058961 125404 librust-sniffglue-dev_0.14.0-2_amd64.deb
 214817662f43ec4ae0766dd23700a694c45985cb03d28fe82a791a61202e0705 8995232 sniffglue-dbgsym_0.14.0-2_amd64.deb
 448c781a9e594227bc9f0d6c65b8beba2b3add68d3583020de188d4cfa365b40 732980 sniffglue_0.14.0-2_amd64.deb
Build-Origin: Debian
Build-Architecture: amd64
Build-Date: Mon, 06 Dec 2021 21:35:27 +0000
Build-Path: /build/rust-sniffglue-1KDXF6/rust-sniffglue-0.14.0
Installed-Build-Depends:
 autoconf (= 2.71-2),
 automake (= 1:1.16.5-1.1),
 autopoint (= 0.21-4),
 autotools-dev (= 20180224.1+nmu1),
 base-files (= 12),
 base-passwd (= 3.5.52),
 bash (= 5.1-5),
 binutils (= 2.37-10),
 binutils-common (= 2.37-10),
 binutils-x86-64-linux-gnu (= 2.37-10),
 bsdextrautils (= 2.37.2-4),
 bsdutils (= 1:2.37.2-4),
 build-essential (= 12.9),
 bzip2 (= 1.0.8-5),
 cargo (= 0.57.0-3),
 coreutils (= 8.32-4.1),
[...]

This is the full list of dependencies with exact versions we need!

Debian version strings (the not-so-happy path)

Let’s look into a different package next, one that’s slightly quirky. This package is going to be mariadb-server, which is currently on version 1:10.6.5-2.

First of all, there are multiple entries starting with Package: mariadb-server but when filtering by version we get the one we’re looking for:

Package: mariadb-server
Source: mariadb-10.6
Version: 1:10.6.5-2
Installed-Size: 66
Maintainer: Debian MySQL Maintainers <[email protected]>
Architecture: all
Depends: mariadb-server-10.6 (>= 1:10.6.5-2)
Description: MariaDB database server (metapackage depending on the latest version)
Homepage: https://mariadb.org/
Description-md5: 810926b6e0e35edd4901d8603857b214
Tag: devel::lang:c++, devel::lang:sql, devel::library, implemented-in::c++,
 interface::commandline, interface::daemon, network::server,
 protocol::db:mysql, role::devel-lib, role::metapackage, role::program,
 works-with::db
Section: database
Priority: optional
Filename: pool/main/m/mariadb-10.6/mariadb-server_10.6.5-2_all.deb
Size: 34360
MD5sum: e32900fc549e3be67c98578e185a2eb9
SHA256: 4a37a29f28b7e8e81f9d432acb96265258e9dd5e24af8026a4c8f234d1485334

This is referencing source package mariadb-10.6, let’s try to find that next:

Package: mariadb-10.6
Binary: libmariadb-dev, libmariadb-dev-compat, libmariadb3, libmariadbd19, libmariadbd-dev, mariadb-common, mariadb-client-core-10.6, mariadb-client-10.6, mariadb-server-core-10.6, mariadb-server-10.6, mariadb-server, mariadb-client, mariadb-backup, mariadb-plugin-connect, mariadb-plugin-s3, mariadb-plugin-rocksdb, mariadb-plugin-oqgraph, mariadb-plugin-mroonga, mariadb-plugin-spider, mariadb-plugin-gssapi-server, mariadb-plugin-gssapi-client, mariadb-plugin-cracklib-password-check, mariadb-test, mariadb-test-data
Version: 1:10.6.5-2
Maintainer: Debian MySQL Maintainers <[email protected]>
Uploaders: Otto Kekäläinen <[email protected]>
Build-Depends: bison, cmake, cracklib-runtime <!nocheck>, debhelper (>= 10), dh-exec, gdb <!nocheck>, libaio-dev [!linux-any], libboost-dev, libcrack2-dev (>= 2.9.0), libcurl4-openssl-dev | libcurl4-dev, libedit-dev, libedit-dev:native, libjemalloc-dev [linux-any], libjudy-dev, libkrb5-dev, liblz4-dev, libncurses5-dev (>= 5.0-6~), libncurses5-dev:native (>= 5.0-6~), libnuma-dev [linux-any], libpam0g-dev, libpcre2-dev, libpmem-dev [amd64 arm64 ppc64el], libsnappy-dev, libssl-dev, libssl-dev:native, libsystemd-dev [linux-any], liburing-dev [linux-any], libxml2-dev, libzstd-dev (>= 1.3.3), lsb-release, perl:any, po-debconf, psmisc, unixodbc-dev, uuid-dev, zlib1g-dev (>= 1:1.1.3-5~)
Architecture: any all
Standards-Version: 4.5.0
Format: 3.0 (quilt)
Files:
 6ee9e176b9b8b380b5de85af20720083 4573 mariadb-10.6_10.6.5-2.dsc
 3049b2c7f83f5e99eab9fb871c7743cc 81853489 mariadb-10.6_10.6.5.orig.tar.gz
 7b80c984b8ec0e1d6a7ea4791087bb59 221456 mariadb-10.6_10.6.5-2.debian.tar.xz
Vcs-Browser: https://salsa.debian.org/mariadb-team/mariadb-server
Vcs-Git: https://salsa.debian.org/mariadb-team/mariadb-server.git
Checksums-Sha256:
 114e04d6916437218fd995deb90af216b20c387c6e22136af16964c96c4e3ee0 4573 mariadb-10.6_10.6.5-2.dsc
 0831debda6ff6f2942d756732a9e5886ef2c5526ad360119502ae3e03b13e013 81853489 mariadb-10.6_10.6.5.orig.tar.gz
 6c33c0e43f6e24d07af754b960c8b0e7473bfa7c6bece7488125cd13eb260206 221456 mariadb-10.6_10.6.5-2.debian.tar.xz
Homepage: https://mariadb.org/
Package-List: 
 libmariadb-dev deb libdevel optional arch=any
 libmariadb-dev-compat deb libdevel optional arch=any
 libmariadb3 deb libs optional arch=any
 libmariadbd-dev deb libdevel optional arch=any
 libmariadbd19 deb libs optional arch=any
 mariadb-backup deb database optional arch=any
 mariadb-client deb database optional arch=all
 mariadb-client-10.6 deb database optional arch=any
 mariadb-client-core-10.6 deb database optional arch=any
 mariadb-common deb database optional arch=all
 mariadb-plugin-connect deb database optional arch=any
 mariadb-plugin-cracklib-password-check deb database optional arch=any
 mariadb-plugin-gssapi-client deb database optional arch=any
 mariadb-plugin-gssapi-server deb database optional arch=any
 mariadb-plugin-mroonga deb database optional arch=any-alpha,any-amd64,any-arm,any-arm64,any-i386,any-ia64,any-mips64el,any-mips64r6el,any-mipsel,any-mipsr6el,any-nios2,any-powerpcel,any-ppc64el,any-sh3,any-sh4,any-tilegx
 mariadb-plugin-oqgraph deb database optional arch=any
 mariadb-plugin-rocksdb deb database optional arch=amd64,arm64,mips64el,ppc64el
 mariadb-plugin-s3 deb database optional arch=any
 mariadb-plugin-spider deb database optional arch=any
 mariadb-server deb database optional arch=all
 mariadb-server-10.6 deb database optional arch=any
 mariadb-server-core-10.6 deb database optional arch=any
 mariadb-test deb database optional arch=any
 mariadb-test-data deb database optional arch=all
Testsuite: autopkgtest
Testsuite-Triggers: eatmydata
Directory: pool/main/m/mariadb-10.6
Priority: optional
Section: misc

The version is 1:10.6.5-2 in both cases, but this doesn’t check out with the .buildinfo file url. Using our previous scheme we’d get:

https://buildinfos.debian.net/buildinfo-pool/m/mariadb-10.6/mariadb-10.6_1:10.6.5-2_all.buildinfo

This link 404’s, the correct url is:

https://buildinfos.debian.net/buildinfo-pool/m/mariadb-10.6/mariadb-10.6_10.6.5-2_all.buildinfo

Note the missing 1: in the version. Unfortunately there’s no field containing just 10.6.5-2 so there’s no clean way to get this value. Whatever, we just cut off everything before the : and call it a day, the link is now working correctly:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.0
Source: mariadb-10.6
Binary: mariadb-client mariadb-common mariadb-server mariadb-test-data
Architecture: all
Version: 1:10.6.5-2
Checksums-Md5:
 f06a4f7d1a760131b2e94d88d011e4a4 34244 mariadb-client_10.6.5-2_all.deb
 5863a24052e165e68565b5e310e3051d 35768 mariadb-common_10.6.5-2_all.deb
 e32900fc549e3be67c98578e185a2eb9 34360 mariadb-server_10.6.5-2_all.deb
 f8ab5df076102d28fafb4b3c4c62396a 17958780 mariadb-test-data_10.6.5-2_all.deb
Checksums-Sha1:
 c472cef1dc83f77d5f267d3f48e34b6b7518f52e 34244 mariadb-client_10.6.5-2_all.deb
 338b14e8e96e2e261bf4d1f669b86a5321c5ea06 35768 mariadb-common_10.6.5-2_all.deb
 629f8c52395f3e59ebbfa095059277d06f541eaa 34360 mariadb-server_10.6.5-2_all.deb
 bacd0f09f0d465133b213bd992320b3af78fb5c4 17958780 mariadb-test-data_10.6.5-2_all.deb
Checksums-Sha256:
 6c198e06111c1de8389ca798bd24d712b8c2776ad8b7c97052cba6d2b8a8f675 34244 mariadb-client_10.6.5-2_all.deb
 d2f17ff78af73dea232a6890b12becaaf9292100a515f63935d370df0c2515ff 35768 mariadb-common_10.6.5-2_all.deb
 4a37a29f28b7e8e81f9d432acb96265258e9dd5e24af8026a4c8f234d1485334 34360 mariadb-server_10.6.5-2_all.deb
 f603d7f330f4e8988e899e5e0ab85ff7eb68caf1967b8a7d6d0a4c8c91c5a06b 17958780 mariadb-test-data_10.6.5-2_all.deb
Build-Origin: Debian
Build-Architecture: amd64
Build-Date: Wed, 15 Dec 2021 05:41:50 +0000
Build-Path: /build/mariadb-10.6-kGWtxs/mariadb-10.6-10.6.5
Installed-Build-Depends:
 autoconf (= 2.71-2),
 automake (= 1:1.16.5-1.1),
 autopoint (= 0.21-4),
 autotools-dev (= 20180224.1+nmu1),
[...]

The .deb file is also using this truncated version, but the Packages.xz has an entry that contains a full path for every .deb so we don’t really worry about that:

Filename: pool/main/m/mariadb-10.6/mariadb-server_10.6.5-2_all.deb

Debian version strings (the miserable path)

I didn’t really mention this yet, but there are essentially three version strings so far:

The “binary package version”
The “source package version”
The “source package version but without the epoch”

The first two are fully independent from one-another and can have completely different values (although they didn’t in the examples so far). But there’s a 4th one you possibly never heard of, even if you’re fairly familiar with Debian:

The binNMU version⁵.

But what’s a binNMU? It stands for “binary non-maintainer-upload” and it’s essentially recycling a past source upload and building new binary packages out of it in a more recent build environment. This is useful if you want to re-resolve the dependencies in the source package to more recent dependencies that have been uploaded in the meantime, for example if you want to compile again with a more recent compiler. In other distros like Arch Linux this would be done with a pkgrel bump, although a pkgrel bump also allows you to change the build instructions. I don’t know why it’s implemented like this in Debian, but it’s potentially been like that for a really long time⁶.

Let’s look into the courier-imap package next. It’s built out of the courier source package (the source package version is 1.0.16-3):

Package: courier
Binary: courier-base, courier-mlm, courier-mta, courier-faxmail, courier-webadmin, sqwebmail, courier-pcp, courier-pop, courier-imap, courier-ldap, courier-doc
Version: 1.0.16-3
Maintainer: Markus Wanner <[email protected]>
Build-Depends: automake, courier-authlib-dev (>= 0.66.4-5~), debhelper-compat (= 13), default-libmysqlclient-dev, dh-exec, dh-apache2, expect, ghostscript, gnupg2, gnutls-bin, groff-base, libcourier-unicode-dev (>= 2.1-3~), libgamin-dev, libgcrypt-dev, libgdbm-dev | libgdbmg1-dev, libgnutls28-dev, libidn11-dev, libldap2-dev, libpam0g-dev, libpcre3-dev, libperl-dev, libpq-dev, libsasl2-dev | libsasl-dev, libtool-bin | libtool, mgetty-fax, mime-support, netpbm, po-debconf, procps, wget, zlib1g-dev
Build-Conflicts: automake1.4
Architecture: any all
Standards-Version: 4.5.1
Format: 3.0 (quilt)
Files:
 dc395743184bd43c6bef647e29907439 3874 courier_1.0.16-3.dsc
 25f1c97a9ee74b7b264402b52d424ea9 7644196 courier_1.0.16.orig.tar.bz2
 f421e270aa2bb0eef69883b8dfc67661 866 courier_1.0.16.orig.tar.bz2.asc
 d82c935aa59897dc40adf85d8cc951b0 108396 courier_1.0.16-3.debian.tar.xz
Vcs-Browser: https://salsa.debian.org/debian/courier
Vcs-Git: https://salsa.debian.org/debian/courier.git
Checksums-Sha256:
 dd89bad1059adfba65b6f06be895b97c7d3d28d4f177d6a4f055407d374ef683 3874 courier_1.0.16-3.dsc
 87fc35ddff4f273aa04f43fdffc73f9236abf39bc3234a449eab88742d885ebb 7644196 courier_1.0.16.orig.tar.bz2
 e2d574353654d2a3e473d481a1354f2d8eb6412e77277a489d6545ef41e6122d 866 courier_1.0.16.orig.tar.bz2.asc
 565912449f530457892ccee787585184d644bac76f7055698d7f41600308519f 108396 courier_1.0.16-3.debian.tar.xz
Homepage: http://www.courier-mta.org/
Package-List: 
 courier-base deb mail optional arch=any
 courier-doc deb doc optional arch=all
 courier-faxmail deb mail optional arch=any
 courier-imap deb mail optional arch=any
 courier-ldap deb mail optional arch=any
 courier-mlm deb mail optional arch=any
 courier-mta deb mail optional arch=any
 courier-pcp deb mail optional arch=any
 courier-pop deb mail optional arch=any
 courier-webadmin deb mail optional arch=any
 sqwebmail deb mail optional arch=any
Testsuite: autopkgtest
Testsuite-Triggers: default-mta
Directory: pool/main/c/courier
Priority: source
Section: mail

But the version of the binary package is sorta wild (5.0.13+1.0.16-3+b1):

Package: courier-imap
Source: courier (1.0.16-3)
Version: 5.0.13+1.0.16-3+b1
Installed-Size: 593
Maintainer: Markus Wanner <[email protected]>
Architecture: amd64
Replaces: courier-imap-ssl (<< 4.16.2+0.75.0-1~), imap-server
Provides: imap-server
Depends: courier-base (= 1.0.16-3+b1), debconf | debconf-2.0, gamin, default-mta | mail-transport-agent, sysvinit-utils (>= 2.88dsf-50) | init-d-script, courier-authlib (>= 0.71), libc6 (>= 2.15), libcourier-unicode4 (>= 2.1.2), libgamin0 | libfam0, libgdbm6 (>= 1.16), libidn12 (>= 1.13)
Suggests: courier-doc, imap-client
Conflicts: imap-server
Breaks: courier-imap-ssl (<< 4.16.2+0.75.0-1~)
Description: Courier mail server - IMAP server
Homepage: http://www.courier-mta.org/
Description-md5: aedad44242f18297b70663ef077f0e63
Tag: interface::daemon, mail::imap, network::server, network::service,
 protocol::imap, role::program, works-with::mail
Section: mail
Priority: optional
Filename: pool/main/c/courier/courier-imap_5.0.13+1.0.16-3+b1_amd64.deb
Size: 277952
MD5sum: f8eb4b0c42f191581189b2aee4c31793
SHA256: 67acfd8593f6c0a12a2681906e43dfe2872a694bd74e5438726646ec0e2af0a6

Let’s naively use our scheme again, so the buildinfo link we get is:

https://buildinfos.debian.net/buildinfo-pool/c/courier/courier_1.0.16-3_amd64.buildinfo

This link works, but hold on:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.0
Source: courier
Binary: courier-base courier-base-dbgsym courier-faxmail courier-imap courier-imap-dbgsym courier-ldap courier-ldap-dbgsym courier-mlm courier-mlm-dbgsym courier-mta courier-mta-dbgsym courier-pcp courier-pcp-dbgsym courier-pop courier-pop-dbgsym courier-webadmin courier-webadmin-dbgsym sqwebmail sqwebmail-dbgsym
Architecture: amd64
Version: 1.0.16-3
Checksums-Md5:
 15a8c74a2a05ae71c433e40693c7fe6e 460316 courier-base-dbgsym_1.0.16-3_amd64.deb
 b0570504e8e76072260e4b954b0d2c1e 327972 courier-base_1.0.16-3_amd64.deb
 d0d6cedf35b6555cd3e22be5dc9845fa 137208 courier-faxmail_1.0.16-3_amd64.deb
 5712d505d26c9d121ab7f233df825af6 505112 courier-imap-dbgsym_5.0.13+1.0.16-3_amd64.deb
 a099c775cf8801b00106974307c9fb9d 277672 courier-imap_5.0.13+1.0.16-3_amd64.deb
 3df62f2c9c29843cbb22699316fac2d8 31956 courier-ldap-dbgsym_1.0.16-3_amd64.deb
 56de5a9a609e1c098abe717279fe81c2 142028 courier-ldap_1.0.16-3_amd64.deb
 e1f1e737ca49da98fe1121ccaf331d07 2975104 courier-mlm-dbgsym_1.0.16-3_amd64.deb
 a80662978d886ae2feee732e3c0f8abd 390368 courier-mlm_1.0.16-3_amd64.deb
 4f81470483ceafdeb7a9d06971baa923 3235056 courier-mta-dbgsym_1.0.16-3_amd64.deb
 76fe544615420d64092b335c9dc1a965 634624 courier-mta_1.0.16-3_amd64.deb
 f55197352abe8bd6aa39f1e8252e9159 148224 courier-pcp-dbgsym_1.0.16-3_amd64.deb
 f922ee7b466cdbf7c24e953b5ff44c2d 169028 courier-pcp_1.0.16-3_amd64.deb
 620698dda8139f0d8012e59f89d5e3d0 135076 courier-pop-dbgsym_1.0.16-3_amd64.deb
 701684c8508858dd26e6fce133737c54 179620 courier-pop_1.0.16-3_amd64.deb
 6d96808e88e071ce2da86185d1e87f5a 4076 courier-webadmin-dbgsym_1.0.16-3_amd64.deb
 5c45649f63422f17c5a19187c2bacbac 147616 courier-webadmin_1.0.16-3_amd64.deb
 4073e99b9e047d263256a1a07428c596 1009160 sqwebmail-dbgsym_6.0.5+1.0.16-3_amd64.deb
 55319a910101a4742bebe7476bb93258 496464 sqwebmail_6.0.5+1.0.16-3_amd64.deb
[...]

It’s for the wrong package! This one describes the build environment for courier-imap_5.0.13+1.0.16-3_amd64.deb. The correct buildinfo file is:

https://buildinfos.debian.net/buildinfo-pool/c/courier/courier_1.0.16-3+b1_amd64.buildinfo

This link shows the correct buildinfo file for courier-imap_5.0.13+1.0.16-3+b1_amd64.deb (note the +b1):

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Format: 1.0
Source: courier (1.0.16-3)
Binary: courier-base courier-base-dbgsym courier-faxmail courier-imap courier-imap-dbgsym courier-ldap courier-ldap-dbgsym courier-mlm courier-mlm-dbgsym courier-mta courier-mta-dbgsym courier-pcp courier-pcp-dbgsym courier-pop courier-pop-dbgsym courier-webadmin courier-webadmin-dbgsym sqwebmail sqwebmail-dbgsym
Architecture: amd64
Version: 1.0.16-3+b1
Binary-Only-Changes:
 courier (1.0.16-3+b1) sid; urgency=low, binary-only=yes
 .
   * Binary-only non-maintainer upload for amd64; no source changes.
   * Rebuild against libidn12
 .
  -- amd64 / i386 Build Daemon (x86-ubc-01) <[email protected]>  Sun, 22 Aug 2021 22:12:19 +0000
Checksums-Md5:
 224dacec94be1775639ef61ff01efea9 459456 courier-base-dbgsym_1.0.16-3+b1_amd64.deb
 06d0be1d5211f0d425803633806663a5 328148 courier-base_1.0.16-3+b1_amd64.deb
 24f9a41c6380ea2ca7559ce9e6f6f54a 137492 courier-faxmail_1.0.16-3+b1_amd64.deb
 56f61b3dd8a9b855a688bf10fe982e9a 504672 courier-imap-dbgsym_5.0.13+1.0.16-3+b1_amd64.deb
 f8eb4b0c42f191581189b2aee4c31793 277952 courier-imap_5.0.13+1.0.16-3+b1_amd64.deb
 56ab0d8f58360e7f9e9cdd1e550ef4aa 31896 courier-ldap-dbgsym_1.0.16-3+b1_amd64.deb
 fda5c8d7f98633778de1beb7197d257c 142284 courier-ldap_1.0.16-3+b1_amd64.deb
 786868f86bedc808f60f185b7e1ca2a1 2975168 courier-mlm-dbgsym_1.0.16-3+b1_amd64.deb
 4f8cc681fee4364d1a89e118c16bb5a7 390264 courier-mlm_1.0.16-3+b1_amd64.deb
 bab1bb694fe552a42472b2b0c9cf39bb 3233060 courier-mta-dbgsym_1.0.16-3+b1_amd64.deb
 788546e04c005d285cc084c1dea6b2ab 634364 courier-mta_1.0.16-3+b1_amd64.deb
 1e94cc602065e1df6b0a57212da60ffc 148160 courier-pcp-dbgsym_1.0.16-3+b1_amd64.deb
 16d6b5a9f33b6da3e8cbe0fb52105aa5 169328 courier-pcp_1.0.16-3+b1_amd64.deb
 97c64a4c770f9b613ac9eca381ddf162 134468 courier-pop-dbgsym_1.0.16-3+b1_amd64.deb
 97bbb2d0a27f34d1b48482cfd003b081 179872 courier-pop_1.0.16-3+b1_amd64.deb
 04031603ee4c00f5d972bd7b61c89bc3 3992 courier-webadmin-dbgsym_1.0.16-3+b1_amd64.deb
 146605f7e774c98d94da493cc3c19c4e 147916 courier-webadmin_1.0.16-3+b1_amd64.deb
 594fb250bfd24cd9b570cd8881ab88df 1008928 sqwebmail-dbgsym_6.0.5+1.0.16-3+b1_amd64.deb
 5fa070104394a53254b244b16685508e 496944 sqwebmail_6.0.5+1.0.16-3+b1_amd64.deb
Checksums-Sha1:
 5c9481e660beb92d12a315e1eee3174ed0024bcf 459456 courier-base-dbgsym_1.0.16-3+b1_amd64.deb
 47f73e25e38bcc966bd9db4f01b20a89cacdac38 328148 courier-base_1.0.16-3+b1_amd64.deb
 a7df6d382b73c34bb22d14cca2c6228d5a4ef0e0 137492 courier-faxmail_1.0.16-3+b1_amd64.deb
 da99a63aa0653506414a167b93cd55b801b1a45c 504672 courier-imap-dbgsym_5.0.13+1.0.16-3+b1_amd64.deb
 e9ac59293bcad55ae1a8e884510633f0b2387fc7 277952 courier-imap_5.0.13+1.0.16-3+b1_amd64.deb
 cae9c8e7ec94c448844a29531e5c6ad0a3dc0527 31896 courier-ldap-dbgsym_1.0.16-3+b1_amd64.deb
 1c4550c6dcca151a94b5bd23e859d5ec325d84f9 142284 courier-ldap_1.0.16-3+b1_amd64.deb
 5a0f79081975dc7fd91b0a84664b6a32639ba509 2975168 courier-mlm-dbgsym_1.0.16-3+b1_amd64.deb
 1faa6b2ba5e02e349cbb9bbb0f5c18bd196722da 390264 courier-mlm_1.0.16-3+b1_amd64.deb
 b9a1eb39e916317db01a9b5df760066736b8db51 3233060 courier-mta-dbgsym_1.0.16-3+b1_amd64.deb
 0e1b9802303b48bab4b2f0377c40b18b0464991a 634364 courier-mta_1.0.16-3+b1_amd64.deb
 38cfe238d668a31a24356af200380dd96160c38e 148160 courier-pcp-dbgsym_1.0.16-3+b1_amd64.deb
 31f33ecfd6b8f59a1826c9b695bdd2f3bf37bdd5 169328 courier-pcp_1.0.16-3+b1_amd64.deb
 f701fe7cf62ca401161a92353bebf9d3892611e7 134468 courier-pop-dbgsym_1.0.16-3+b1_amd64.deb
 2ab1cb53c878452b40d9b266bd41b5fbe5a23c87 179872 courier-pop_1.0.16-3+b1_amd64.deb
 f0a7436e84b642562f50659aa8c0b772a934077f 3992 courier-webadmin-dbgsym_1.0.16-3+b1_amd64.deb
 a00ddf85da6ad81ba1db25317779165f273bd6c4 147916 courier-webadmin_1.0.16-3+b1_amd64.deb
 14288aedb222350809795ca82502c9e36078b82b 1008928 sqwebmail-dbgsym_6.0.5+1.0.16-3+b1_amd64.deb
 34d298eaa1065ad5cff91bdec577cb7a91142654 496944 sqwebmail_6.0.5+1.0.16-3+b1_amd64.deb
Checksums-Sha256:
 786673efeab0460e94d1a1e60f36aa8732bb4535ae85746dee471a6db49fb676 459456 courier-base-dbgsym_1.0.16-3+b1_amd64.deb
 688b7c11b8ec92514929d37e207681e4b9ac754db9e8cf0ab0632374433eed7e 328148 courier-base_1.0.16-3+b1_amd64.deb
 6cb78e731f845dd98ab792c43fc01a6dc3416140b08d2db00e7415eff5973527 137492 courier-faxmail_1.0.16-3+b1_amd64.deb
 c90cc58b7c957b90120b606f39422315bca05293defacc1d4cf42ce4aa178128 504672 courier-imap-dbgsym_5.0.13+1.0.16-3+b1_amd64.deb
 67acfd8593f6c0a12a2681906e43dfe2872a694bd74e5438726646ec0e2af0a6 277952 courier-imap_5.0.13+1.0.16-3+b1_amd64.deb
 a146803c918d1160ca4681bd62b3bc61e14f15bed0cd483ae66916e7b279c57a 31896 courier-ldap-dbgsym_1.0.16-3+b1_amd64.deb
 733c1e1f620b416fb107e115fc8fa3cbe511d1e47ce1e171e1ffb8b5b1cecd05 142284 courier-ldap_1.0.16-3+b1_amd64.deb
 aaf96264630a4526c7aff2502a8015c0d409ca6e772213d7b9014cce3f1ecb63 2975168 courier-mlm-dbgsym_1.0.16-3+b1_amd64.deb
 209018400fd2dfa4caf4ede13a8865a110a9dec1c387e8eb9e45e2cd0550b771 390264 courier-mlm_1.0.16-3+b1_amd64.deb
 5a513509987478410e297f99be9d0d644f999928c7bc9ad1e854ad25473eeee0 3233060 courier-mta-dbgsym_1.0.16-3+b1_amd64.deb
 a05a38e8aa0986067b40d1f82dbf59732c6f4697ea91d8717ef0ea88b388ae6a 634364 courier-mta_1.0.16-3+b1_amd64.deb
 254defd40412e5f6a09d4ca4784b07255f617d2a1bb2ed1f7deac6d67a1b9771 148160 courier-pcp-dbgsym_1.0.16-3+b1_amd64.deb
 7b118060d17983ff0a070825819a5b6ac46bad0262288e5f63c7bd0baefcf72d 169328 courier-pcp_1.0.16-3+b1_amd64.deb
 655cd79a41636291ed2eb4be416828d14d7d44c55cebeb139ddce65d33390dcf 134468 courier-pop-dbgsym_1.0.16-3+b1_amd64.deb
 efc865b63f19efda4feb15917789b9390667d973ac18d7aa6e641290a7ba8461 179872 courier-pop_1.0.16-3+b1_amd64.deb
 e7f7abf143674c7ac1fe5a6c0c21ac2d9505c3ade37625e7623a6d13ccc6e45f 3992 courier-webadmin-dbgsym_1.0.16-3+b1_amd64.deb
 bc65ecb8eac668c0c5fe18c6784217dd67541d2347899e0096b4b2f9f2ab0059 147916 courier-webadmin_1.0.16-3+b1_amd64.deb
 e48e7fb6a38b6dd2057f61db875368cf5d175bc5b012c5b88eba8f4c15b4321e 1008928 sqwebmail-dbgsym_6.0.5+1.0.16-3+b1_amd64.deb
 51ebf109a5257b34a521a68967db5b328d2301ad9e7585ccf797dfb549ed5e6e 496944 sqwebmail_6.0.5+1.0.16-3+b1_amd64.deb
Build-Origin: Debian
Build-Architecture: amd64
Build-Date: Sun, 22 Aug 2021 22:24:21 +0000
Build-Path: /build/courier-DEOrho/courier-1.0.16
Installed-Build-Depends:
 adduser (= 3.118),
 apache2-dev (= 2.4.48-4),
 autoconf (= 2.69-14),
 automake (= 1:1.16.4-1),
 autopoint (= 0.21-4),
 autotools-dev (= 20180224.1+nmu1),
 base-files (= 11.1),
 base-passwd (= 3.5.51),
 bash (= 5.1-3+b1),
 binutils (= 2.37-4),
 binutils-common (= 2.37-4),
[...]

So the binnmu-version we’re looking for is 1.0.16-3+b1, which, again, is not easily available in any fields of either index.

Ok but why is this important?

Excellent question! Of course there’s the even better question “is anything important” that I’m occasionally asking myself, but for the version string there’s an easier answer:

It seems there are currently two ways to locate the correct 1.0.16-3+b1 version string:

Option #1: courier-imap_5.0.13+1.0.16-3+b1_amd64.deb contains a file called usr/share/doc/courier-imap/changelog.Debian.amd64.gz, this file contains a changelog entry that contains the binnmu-version:

courier (1.0.16-3+b1) sid; urgency=low, binary-only=yes

  * Binary-only non-maintainer upload for amd64; no source changes.
  * Rebuild against libidn12

 -- amd64 / i386 Build Daemon (x86-ubc-01) <[email protected]>  Sun, 22 Aug 2021 22:12:19 +0000

One could download all packages in Debian and parse the changelog file inside of them to get an accurate binnmu-version for every package.

Option #2: Crawl buildinfos.debian.net, download all buildinfo files and generate a lookup table. There’s an implementation of this at builtin-pho but unfortunately no publicly available database.

With reproducible builds taking a somewhat central role in supply-chain security there’s quite a bit of power around the answer to “What’s inside of a distro and how is it built”. The security of Packages.xz and Sources.xz is fairly well understood at this point, I’d like to keep the missing link as simple as possible and the barrier of running a rebuilder as low as possible.

There’s a related video by José Miguel Parrella called Traversing deb packages upstream provenance and downstream integrity that I highly recommend looking into, it goes into further detail how these trust-chains work.

Thanks

This work is currently crowd-funded on github sponsors. I’d like to thank @jvoisin, @SantiagoTorres, @repi and @rgacogne for their support in particular, this work wouldn’t be possible without them. ♥️

Footnotes

isdebianreproducibleyet.com, also, yey, I figured out how to do footnotes! ↩
build environment fuzzing is attempting to build something twice and varying things in the environment you don’t want to end up in the binary. For example the clock of the second build server could be off by multiple years or have a different username for the build server, if the binary of the second build server is bit-for-bit identical you’ve successfully confirmed there are no non-normalized values leaking into the binary. ↩
Historically there’s also buildinfo.debian.net (buildinfo without the s vs buildinfos) but at this point of the reproducible Debian project it’s fairly irrelevant. ↩
Actually, this results in https://buildinfos.debian.net/buildinfo-pool/main/r/rust-sniffglue/rust-sniffglue_0.14.0-2_amd64.buildinfo, but the main/ part needs to be removed from the Directory: field. This is an implementation quirk of buildinfos.debian.net. ↩
Or at least that’s the name somebody suggested when I asked in #debian-devel. ↩
Since at least 2006. ↩