This weekend, I was excited to deploy my first Ryzen 3000-powered workstation in my home office. Unfortunately, a microcode bug—originally discovered in July, but still floating around in large numbers in the wild—wrecked my good time. I eventually got my Ryzen 3700X system working, and it’s definitely fast. But unfortunately, it’s still bugged, and there’s no easy way to fix it.
Not long after the product launch, AMD Ryzen 3000 customers started noticing problems with their shiny new CPUs. Windows users couldn’t successfully launch (due to a power-management bug, unrelated to the one sidelining my system), and Linux users in many cases couldn’t even get their system to boot. Jason Evangelho covered the initial discovery and report of the bug at Forbes back in July, and an AMD representative provided him a statement by email:
AMD has identified the root cause and implemented impacting the ability to run certain Linux on Ryzen 3000 processors.
This sounds happy and upbeat, but the reality isn’t quite so simple. When there’s a bug in the CPU microcode, you’re at the mercy of your motherboard vendor to release a new system BIOS that will update it for you—you can’t just go to some download link at AMD and apply a fix yourself.
AMD responded to the bug in July—although so far as I can tell, only by direct email response; there’s no press release about it—and AMD’s response sounded like everything would be fixed in a week or two.
AMD responded to the bug in July. As far as I can tell, AMD did so only by direct email response; there’s no press release about it—and the company’s response made it sound like everything would be fixed in a week or two.
I have the unfortunate duty of reporting to you, three months later, that it is not.
What’s an RDRAND?
The microcode bug in question is a faulty response to the
RDRAND instruction. Modern x86_64 CPUs—beginning with Intel’s Broadwell and AMD’s Zen architectures—are supposed to have high-quality onboard random number generators (RNGs), which use thermal “noise” to very rapidly offer high-entropy pseudorandom numbers to anybody with kernel-level access who wants it. RDRAND is, in turn, the instruction which provides these random numbers.
All of this is supposed to be fairly failsafe. There’s a CPUID function call that checks for the availability of
RDRAND, and there’s also a “carry bit” in the return value from a call to
RDRAND that’s supposed to let the calling application know if the CPU’s RNG was unable to generate a sufficiently random number due to lack of entropy. Unfortunately, unpatched Ryzen 3000 says “yes” to the CPUID 01H call, sets the carry bit indicating it’s successfully created the most artisanal, organic high-quality random number possible… and gives you a
0xFFFFFFFF for the “random” number, every single time.
Obvious RDRAND bug impacts
RDRAND bug in Ryzen 3000 first surfaced back in June, Linux users widely reported that their entire Ryzen 3000-powered systems wouldn’t boot. The failure to boot was due to systemd’s use of
RDRAND—and it wasn’t systemd’s first clash with AMD and a buggy random-number generator, unfortunately.
A much earlier bug in older CPUs caused some AMD systems to stop generating properly “random” numbers after resuming from suspend. The new bug caused Ryzen 3000 users to never get any proper random numbers at all. Both problems caused lockups in Linux operating systems using systemd, so in May systemd committed a patch that falls back to using alternate RNG sources if systemd receives the characteristic
0xFFFFFFFF back from the RNG. (This kinda sucks, because
0xFFFFFFFF is technically a perfectly valid random number—the implication here is that, after a sufficient length of time, systemd will decide system has a buggy RNG when it eventually receives the “bad” number, even if it’s never seen that number before.)
Systemd’s patch is ugly, but it certainly works well enough to allow systems to boot. Unfortunately, it doesn’t fix the actual , which is that the CPU’s random number generator is no more “random” than a two-headed penny. On my own system, I spent my entire weekend chasing phantom problems, first suspecting the system’s brand new RX 590 graphics card and (necessarily) updating distro and kernel versions before haring off from there.
Eventually, after many false trails and much swearing, coffee, and less-respectable beverages, I actually the call trace from my frequent CPU lockups—and “WireGuard” was right there, in every one of them. As it turns out, WireGuard relies on
RDRAND (when available) to generate new session IDs. The session IDs need to be unique, and WireGuard wants them not to be simple consecutive integers, so it pulls a pseudorandom value from
RDRAND, compares it against its existing session ID list to make sure there’s no collision, then assigns it to the session.
Read that last part again carefully—it first. If an existing session has the same ID as the new number, WireGuard asks RDRAND for another “random” number, checks it for uniqueness, and so on. Since
RDRAND on my system—and any non-microcode-updated Ryzen 3000 system—always returned
0xFFFFFFFF no matter what, that means infinite loop. Infinite loops in kernel code are ; they introduce you to the value of the hardware reset button in a hurry.
I want to be very clear here, this is not a WireGuard bug! WireGuard correctly checks to see if
RDRAND is available, fetches a value if it is, and correctly checks to see if the carry bit is set. Then it indicates that, not only is there a value, it’s a properly random one. Nevertheless, it’s one that will lock up affected systems hard. So after considerable discussion this morning, the project decided it will implement a simple detection-and-fallback routine.
The fallback routine will allow WireGuard—and systems with WireGuard installed—to work even in the presence of the bug. But it still doesn’t fix the . A modern system needs high-quality pseudo-random numbers for lots of tasks, and the security implications of “random” meaning “always return
0xFFFFFFFF” are difficult to predict. One obvious candidate is Address Space Layout Randomization (ASLR). Both Windows and Linux use
RDRAND as at least part of the randomness used to make sure code is never loaded in the same order twice, which mitigates against stack-smashing attacks.
Fixing the problem—or at least recognizing it
As AMD’s representatives told reporters back in July, the real fix comes from applying BIOS updates to your motherboard and hoping that the BIOS update also includes the microcode patch for the CPU itself. When I checked my own BIOS using the dmidecode utility, I saw a date of August 12, 2019. But when I looked at Asus’ download page for my motherboard, I saw downloads dated in September! Hurray! So I downloaded the BIOS update, saved it to a FAT32 thumb drive, rebooted my system, and went into setup.
Unfortunately, after successfully applying the update and rebooting again, I realized my error—yes, Asus showed a later for the BIOS, but the actual version was the same as the one I already had—3.2.0. My CPU still thought
0xFFFFFFFF was the randomest number ever, always, no matter what.
At this point, I began to get paranoid—systemd had already quietly worked around the bug, and WireGuard was (thankfully) about to do the same. But with most applications just quietly ignoring the problem, how would I know if it ever had been patched? What if two years later, I was still vulnerable to stack-smashing that I shouldn’t have been, due to ASLR that wasn’t actually randomizing?
I discovered that I could use the linux utility
hexdump against the kernel device
/dev/hwrng to demonstrate that I had the problem. Unfortunately, the WireGuard project’s Jason Donenfeld warned me that
/dev/hwrng could, on some systems, derive its randomness from other sources—so while seeing a bunch of
FF from it demonstrates that you the problem, seeing valid pseudorandom data doesn’t necessarily demonstrate that you don’t. So he generously whipped up a couple test utilities for the purpose, that safely access
If you’re a linux user, you can download rdrand-test.zip, unzip it, and run it directly in the folder that you unzipped it in.
./amd-rdrandbug will tell you in plain English whether you have this specific bug or not, and
./test-rdrand will output a 20 test
RDRAND fetches. So you can confirm for yourself that you’re not vulnerable to similar bugs either—if running
./test-rdrand produces the same set of values every time, it doesn’t really matter whether they “look random” or not, your RNG is broken!
If you’re a Windows user, you have a little more work ahead of you. First, download an Ubuntu desktop installer, then create an Ubuntu installer thumb drive. Then you can boot into the Ubuntu thumb drive’s live environment (click “Try Ubuntu”) and download and run the tests from there:
[email protected]:~$ wget https://cdn.arstechnica.net/wp-content/uploads/2019/10/rdrand-test.zip [email protected]:~$ unzip rdrand-test.zip [email protected]:~$ cd rdrand-test [email protected]:~$ ./amd-rdrand.bug
A broken random-number generator is a very serious bug, and it’s troubling that more hasn’t been said or done about this issue by AMD in the last three months. Ryzen 3000 is a great CPU platform in general, and I’ve been very impressed with the new system… for spending an entire frustrated weekend troubleshooting it, being uneasy about the impact this will have on my overall system security, and having no idea when I can expect to be able to actually fix it.
I reached out to AMD representatives earlier today, and they’ve responded with questions about my hardware but no solutions yet. I’ll update this article with any fixes or recommendations as they arrive.