Break It Till You Make It: Q&A with Hardware Hackers Colin O'Flynn and Jasper van Woudenberg
Submitted by No Starch Press on Tue, 2021-10-05 14:19
To kick off the Cybersecurity Awareness Month edition of our ongoing Author Spotlight series, we're joined by Colin O’Flynn and Jasper van Woudenberg, co-authors of The Hardware Hacking Handbook (available November, 2021). In the following Q&A, we talk with Colin (@colinoflynn) and Jasper (@jzvw) about the perils of proprietary protocols being replaced with network devices, the problem of having too many interesting targets to test your tools on, the beauty of AI-designed attack systems, the indisputable power of “hammock hacking,” and why nobody cares about fault injection until they get hacked with fault injection.
Colin runs NewAE Technology, Inc., a startup based on his ChipWhisperer project that designs tools to make hardware attacks more accessible, and teaches engineers about embedded security – a topic he frequently speaks about at conferences and on tech podcasts.
Jasper is CTO of Riscure North America, where he leads the company’s pentesting teams, and has a special interest in integrating AI with security. His research has been published in various academic journals, and he’s a regular speaker at educational and hacking conferences.
No Starch Press: I’ll start by saying that your book is timely! Hardware hacking, once a niche field of the exploit world, has become far more relevant amidst the proliferation of embedded devices all around us. What do you think accounts for this, and why are side-channel attacks in particular becoming increasingly common (and difficult to prevent)?
Colin O'Flynn: Hardware hacking has been a niche field, but one with an extensive and long history. Most of the powerful attacks we’re discussing today have been demonstrated for 20 years, so I’d say they should be “well-known.” But the truth seems to be that, until recently, advanced hardware attacks weren’t needed for most IoT devices. Default passwords and unlocked debug interfaces were the norm, so most hardware hackers never needed to dig deeper. Many people I’ve talked to at events have told me they were interested in side-channel and similar advanced attacks but never had time to actually learn them, as they were always able to break devices with easier and faster attacks!
The good news is that device manufacturers seem to be taking security more seriously these days, which means side-channel attacks have become a real threat. So I guess we’re seeing the industry fast-forwarding that 20-year lag of security research to catch up.
Jasper van Woudenberg: Hacking always moves with interesting targets. Once pinball machines started requiring money to play, people “hacked” them by just tilting the whole machine. Nowadays physical pinball machines have a tilt sensor – if you tilt the machine in order to affect the ball, it ceases operation. Of course, we’re talking about digital hardware in our book, but bypassing security systems is as old as security systems. So, the abundance of digital devices naturally increases the amount of hacking going on. Side-channel attacks are fascinating if you’re into the intersection between electronics, signal processing and cryptography. Beyond being fascinating though, they only become relevant when more straightforward attacks are mitigated.
NSP: Fault injection (FI) attacks – which inject a glitch into a target device that alters its behavior so you can bypass security mechanisms – used to be too “high end” for most hackers to bother with, often requiring expensive tools and intimate technical knowledge of the specific system under attack. But those days are over. Not only are low-cost FI toolkits readily available, the explosion of IoT has led to the rise of new defensive features, like Secure Boot, that can be easily subverted by a well-timed FI attack. What are the potential risks to a larger IoT network once a device is compromised this way?
CO: In the past we’ve seen end devices used as a pivot point into a more sensitive network. When it comes to commercial devices, we’re seeing many proprietary protocols replaced with network devices. For example, recent access-control readers are now simply PoE devices that talk back to a central server. With many of these devices, the original designers haven’t considered what happens if an end node becomes compromised. While the network may be correctly secured, you still see sensitive credentials stored in end devices become accessible to an attacker. And if an attacker is able to access these credentials, it means they may be able to pivot off the external network and into more sensitive internal networks.
JVW: I think the cost of the tools is a common misunderstanding – they can be really inexpensive. In our lab, we’ve done attacks literally by soldering a single wire to a bus, connecting it to a button, and when we pressed the button at the right time, the system boots our own code. The cost usually comes from the many days and weeks spent trying to figure out how to carry out the attack. And yes, some attacks do require high-end equipment, or at least equipment that can bring down the time used to figure out the attack.
One common stepping-stone attack we see is the firmware dump. Typically, embedded-device firmware does not receive a lot of scrutiny, and may have lingering vulnerabilities that can be exploited. This usually means gaining control over a single device, but there have been wormable firmware issues in the past.
NSP: What measures can be taken to harden embedded systems against FI attacks, and do you see this happening throughout the industry (why or why not)?
JVW: We always advise our customers to threat model and see if it makes sense to consider FI in scope. Usually that’s the case for embedded systems that are out in the field and have some sensitive assets to protect. Next is the question of whether faults can be mitigated in hardware and/or software. Both is ideal, but that’s not always feasible. Our book contains a chapter on countermeasures that also has a lab, so people can try out some ideas for FI countermeasures. Finally, verification of countermeasures early and often is critical. It’s virtually impossible, as a human, to predict all the ways a system can fault. Pre-silicon fault simulation and post-silicon fault injection, without exception, turn up surprises. Iteration and adaptation are key.
And then the million-dollar question: why is the hardening not happening throughout the industry? It’s a combination of cost and human nature. There is a real engineering cost to these countermeasures, so typically we only see customers that have had their devices compromised requiring FI resistance. If a compromise hasn’t happened, it’s very easy to write the attacks off as unrealistic or irrelevant. Nobody cares about fault injection until they get hacked with fault injection.
CO: Fault injection can be tricky to prevent, as we see countermeasures applied that aren’t effective. For instance, Jasper and I demonstrate a few examples in the book where compilers might remove the effect of your clever countermeasures. There seems to be a lot more interest in this now – for many companies, they just need some “end customer” to ask about it. I talked to silicon vendors a few years ago who were tracking countermeasure ideas, but basically none of their customers (people who actually build products) cared about FI attacks. So that meant they weren’t going to pay for engineering efforts to add those countermeasures. We seem to be seeing a very fast shift in the last couple of years though, so people who were tracking this early-on are in a good position to quickly offer solutions.
NSP: Speaking of low-cost fault-injection toolkits, Colin, you developed one of the most popular models out there, the ChipWhisperer, and built a company around it (NewAE Technology). Given that just about everything we use in our homes and offices has embedded computing systems and could be vulnerable to attack, how do you pick which devices to test your boards and analysis algorithms on? An example from your book would be smart toothbrushes – are you ever doing something like brushing your teeth when it suddenly occurs to you, “Wow, I could totally hack this thing”?
CO: This is actually a big problem! Unfortunately I tend to buy a lot of devices (microcontrollers, IoT products, industrial control systems, etc.) because I think they will be interesting to poke at! As a result, I’ve got a storage cabinet full of various devices along these lines… I’m slowly working through some of them, and when we get some time at the company, we’ll pick away at one or two of those devices as well.
But as more devices include embedded security, there are more “interesting" targets than there is hope of having time to deal with them. Part of why we design many different target-board devices (our “UFO targets” for ChipWhisperer) is actually to help out other researchers by giving them an easier platform to work with.
NSP: Once you successfully exploit a commonly used product, do you let the manufacturer know or is that generally considered an exercise in futility?
CO: If I plan on talking about the issue publicly I’ll reach out, even if I don’t think it’s a serious issue. Sometimes it takes a bit of time to reach the correct person (or team), but so far they have all generally spun it into positive experiences all around.
With one ongoing disclosure, for example, the engineering team had internally flagged that there could be some issues related to a relatively unsecure microcontroller that they were using in a product, and my report had validated their internal concerns. In this case they were already working on a new design, but I’m sure my report was a nice bonus for the people involved, as they can point to that as proof that the issue would be found eventually “in the wild.” In the meantime it gave them the opportunity to provide an interim fix via a firmware update for existing customers.
NSP: Jasper, one of your specialized areas of interest is combining AI with security research. Would you explain what this entails? And looking into the future, how could AI applications be leveraged to improve hardware and embedded security at the design level?
JVW: What I love about AI is also what I love about hacking: making a computer device do more than the original designer put in. With AI, this is tying a couple of artificial neurons together and getting a cat-and-dog image detector. With hacking, this is sending some weird input into a program and all of a sudden it executes arbitrary code.
The combination, I find fascinating. For instance, we’ve used neural networks to do side-channel analysis and outperform human-designed algorithms. We created an algorithm with colleagues that automatically optimizes fault injection parameters. I’ll work very hard to create some automation so I can be – paradoxically – lazy afterwards.
I firmly believe that most if not all cognitive activities, such as designing or attacking systems, will be better performed using AI rather than brains – the big question is when. I prefer to be on the side of making systems more secure through AI, so my research is going towards automating both the detection and mitigation of vulnerabilities, at scale. For instance, a big push we have currently is in pre-silicon security – detecting side-channel and fault issues before they make it into products. I wouldn’t say we’ve arrived at using AI yet, but the first steps are being made.
NSP: Both of you have advanced degrees, which makes sense given all of the academic knowledge involved with embedded security. Yet, The Hardware Hacking Handbook makes very little assumption about a reader’s background. What was your approach to making this challenging field accessible to novices and newcomers, and why was it important enough that you wrote an entire book on this premise?
CO: My career path on paper seems relatively full of academic love – I was an assistant professor for several years in the Electrical & Computer Engineering department at Dalhousie University. But back at the start, when I was considering applying to start my undergraduate degree in electrical engineering, I came relatively close to not attending university at all. I had self-taught myself a fair amount about electronics in high school, and managed to get a summer job that was effectively an electrical engineering internship, and was considering just continuing to grow with the “on the job” experience instead. In the end I fell onto the academic path, but I’ve always believed that it is not the only path, and part of this shapes my desire to make this as accessible as possible.
While many readers may be undergraduate or grad students, it’s clear that a classic academic textbook would cut out readers coming from other backgrounds (including everyone from high school students to professionals interested in looking at other careers). Practically, what we write down isn’t the only consideration – one of the great things about working with No Starch Press is that the pricing of the books makes them more accessible as well. From academic publishers, this book would have been $150+. And there would never be Humble Bumble sales that make it completely accessible on the level that NSP does!
JVW: I’ve taught courses on side-channel and fault injection for years, and it has taught me that the group of people that has to defend against these attacks is not necessarily interested in all the theory and all the research in this field. They want to focus on their goals of creating a system.
Then there’s the group of people like teenage me. I started hacking software before I had an internet connection, so I know the struggle of having to figure out everything by yourself. Looking around at the amazing blogs, videos, tutorials, etc. that exist for the software space currently, it really made me realize what a gap there is in the hardware space.
So, for both these groups, it’s really about breaking things down into practical tips and tricks, and then some of the unavoidable theoretical background. I really would like to show people that this space isn’t daunting, and that even someone like me – who came off a software background – can learn and enjoy it.
NSP: I’ll end with an easy one (I think) – what is your favorite hacking tool, and has that changed since you first got interested in hardware hacking when you were young?
CO: I should probably say my favourite tool is one of my own more-advanced products. But really, a good DMM is the most important tool! And in that regard, it hasn’t changed much over the years – one of my first “dream gifts” (back when Santa would be responsible for it) was a Fluke 12 multimeter, long before I knew about hardware hacking. I’ve since upgraded to a nicer meter (Fluke 179/EDA2 kit), but as we talk about in the book, there is so much you can do with this tool! Finding where pins go, checking the state of logic levels and voltages – it’s still my most used tool when I’m looking at a new device.
JVW: I started being “creative with technology” in the mid-’90s. What has changed is the amount of information available, and the fact that security is now an actual career – I still don’t always believe people are willing to pay me to do this. What hasn’t changed is my curiosity, and the rush that comes with solving a complex problem.
Favorite hacking tool? Hah. Although I use devices for a significant portion of the day, they are also a source of frustration. So, those are out. I’m going to say: my hammock. When I get stuck on a problem and I sense no more new ideas are being produced, or I get frustrated, I drop the problem for a few hours or days. Then I hop in my hammock for what I call “hammock hacking.” This is where I hang back and relax. I’ll almost always have a new view on the problem, or another way of connecting some dots that I hadn’t considered before. Or I fall asleep. But it’s a win in either case.
