Today we’ve noticed a disagreement between the Arch Linux rebuilders about the “cross” package, a popular @rustlang cross-compile tool. One rebuilder reported they’ve succesfully reproduced the package, while the other reported they couldn’t. Let’s have a look what that means.

The official rebuilder says “yup, we took the official package, took the source code, built it in the same environment and got a bit-for-bit identical package, there was probably no build-server compromise (or if there was, they at least didn’t mess with this build)”.

Another rebuilder also tried but had less luck. It periodically retries to get a match with the official package but no success yet. This is likely a bug somewhere instead of a supply-chain compromise, there’s diffoscope reporting enabled so let’s look at the most recent attempt!

Ok, there’s basically one long list of target triplets. The content of that list is likely the same, but the order seems to be different. Stuff like that usually doesn’t happen in rust, unless there’s a custom ./build.rs file in the project. Let’s check if that’s the case!

There’s indeed a ./build.rs file in the project. While regular rebuilds are a best-effort attempt to produce the same binary we’re going to run this build with reprotest first. This intentionally varies some things that we don’t want to have an impact on the build output.

This includes stuff like the system time, filesystem internals or user names/ids. Things like the source code or our compiler versions are not varied of course. Let’s run this and see what happens.

Oops, build failed, we quickly check line 66 in the ./build.rs file. It seems there’s a docker/ folder that we didn’t specify as source input in our reprotest command. We’re going to look closer into this function later.

There seems to be some kind of problem here. faketime is fairly rootkit-y and sometimes causes.. unusual problems like this segfault, we’re going to disable it with -time for now and try again, our problem is likely not related to the build time/date anyway.

It compiles successfully now. We built it twice and we could reproduce this differently ordered target-triplet list. Let’s have a look into that docker/ folder.

huh, this looks oddly familiar. Did you know, the order the kernel returns the file entries for fs::read_dir is filesystem dependent? reprotest uses disorderfs to shuffle the order because this is the kind of filesystem internals we don’t want in our binary. Let’s fix this!

We’re reading the content of the folder into a vector first, then sort it alphabetically. This means the order is always the same and isn’t dependent on the filesystem anymore. Let’s see how that impacts the build!

reprotest confirms both builds yield identical results now, even though disorderfs is still heavily randomizing filesystem internals, trying to introduce.. disorder. That’s probably good enough to prevent the in-the-wild problem we’ve observed on one rebuilder.

This problem isn’t Arch Linux specific, so let’s send our patch upstream and fix this for everybody! If this gets merged the next version is hopefully reproducible on first try, regardless of the filesystem used. 🎉

Our wrap-up is slightly different today, instead of just thanking our supporters over at github sponsors (who’ve covered my cat’s expenses for >1 year now! 😺♥️ ), we’d also like to thank Google and The Linux Foundation who’re now sponsoring all our repro work. Mad props for that!

This post was originally published on Twitter.