This Author Spotlight is on Python programmer/educator extraordinaire Eric Matthes, author of the international bestseller (now in its third edition) Python Crash Course. In the following Q&A, we discuss how reader feedback shaped the latest edition of his book, why he chose teaching over a career in physics, the importance of applying new programming concepts to your own projects, and what Eric learned about life—and coding—from riding across the U.S. on a bicycle (five times).

Eric Matthes has been writing computer programs since childhood. For over a decade, he was a high school science and math teacher, as well as a Python programming educator. He now lives in Alaska as a full-time author and informal Python evangelist, making appearances on podcasts and at events like PyCon, where he hopes to meet you in person.

No Starch Press (NSP): Almost eight years ago, you wrote what has become the most popular Python programming book in the world. Even with well over a million readers of your work, you’re known for replying to pretty much every fan who emails you. How has such a high level of audience feedback informed the second and third editions of Python Crash Course? 

Eric Matthes (EM): When people decide to read a book like Python Crash Course, they’re making a pretty significant commitment. I want to know that when people start the book, they can make it through everything that interests them. So, when someone writes to me about something that’s not working for them, I want to help them if at all possible. The teacher in me can’t help but answer people’s questions.

When I respond to questions, I look for patterns in what people are asking about. When the book first came out and I noticed multiple people asking about the same parts of the book, I’d look at those sections to see how I might revise them to avoid common points of confusion. That feedback from readers helped shape the second edition in particular. Writing the second edition gave me the chance to bring all the code in the book up to date, but also allowed me to clarify numerous sections that weren’t written as well as they could have been.

I also post resources online to address the kinds of things people ask about. For example, I posted a full set of solutions early on, because having solutions available is a huge help to independent learners. I’ve posted recommendations about what resources might be most useful to people after finishing Python Crash Course, and some thoughts about how to find your first programming job. The cheat sheets that accompany the book have been downloaded millions of times.

Overall, I spend about 10-20 hours a week answering questions and paying attention to updates in Python, and all the libraries that the projects depend on. For example, over the last two weeks, I wrote a fully updated test suite for all the code in the book, that makes it easier to test any section of the book against new versions of Python and other dependencies. It surprises a lot of people to hear about all this ongoing work. I think most people assume that once a book is published, the work is done. But this is what keeps Python Crash Course relevant and effective through each new printing, and each new edition. It’s mostly joyful work that puts me in touch with people from all over the world on a regular basis.

NSP: Would you share a little about how you got to where you are today? Specifically, how did you go from studying chemical engineering in college (with plans of becoming a particle physicist) to becoming a public-school teacher and, later, a full-time author?

EM: I had a really good high school chemistry class, so I started college with a major in chemical engineering. But after a semester I changed my major to physics—I just loved the quest to understand the universe at the most fundamental levels.

I loved math and science, but I also noticed how many of my peers hated those two subjects. I joined a volunteer tutoring program and noticed that most of the people who came to us for help enjoyed the same subjects I did—it was just the way they’d been taught that wasn’t working for them. That tutoring work really opened my eyes to the impact of good (and bad) teaching.

I wanted to be a particle physicist, but I didn’t want to be a student forever. I tried teaching and found that the challenge of trying to reach every student in a class was as satisfying as hard science. I never really looked back from teaching. I taught middle school math and science for seven years in New York City and then transitioned to high school when I moved to Alaska.

My father was a software engineer in the ’70s and ’80s, so I learned the basics of programming as a kid. (That was a time when people were just starting to have computers in their homes.) I was a hobbyist programmer all my life, and I’d teach intro programming classes whenever I could. In the early 2010s, I was looking for a book I could give to my more motivated students so they could work at their own pace. I was frustrated to find, however, that every book available back then was either aimed at little kids or made too many assumptions about what people already knew.

I decided to write a book that reflected the way I teach: a focus on just enough fundamentals to start doing meaningful projects. I wrote Python Crash Course for “anyone old enough to not want a kids’ book.” It has met that goal beyond my wildest hopes. When the book first came out, I got a handwritten letter from a 10-year-old who said thank you for writing a book they could learn so easily from. I regularly get emails from people in their 70s and 80s who are finally learning programming to satisfy their curiosity or to keep their minds active. And I hear from everyone in between: high school students, undergrad and graduate students, and a whole bunch of people who are already working but are looking to make a career change. These emails, and conversations at conferences, are the highlight of my writing career.

NSP: Speaking of teaching, your early career in the field is what led to a chance encounter with No Starch Press's founder, Bill Pollock, following a lightning talk you gave at PyCon in 2013 about improving education by taking ideas from the programming world. (Tough crowd!) How did your appearance at the conference result in a publishing deal, and had you been thinking about writing a book at that point in your life?

EM: I first went to PyCon in 2012 because I wanted to do something meaningful with the programming skills I’d been informally building all my life. I was starting to build some tools that would help students navigate school better, and help teachers focus on teaching well. For example, I wrote a program that ingested students’ text-based transcripts, which were really hard to read, and generated a visual transcript for each student. You could look at one of these transcripts and in 30 seconds see what a student’s strengths and weaknesses were, and what they needed to focus on in order to graduate. That project didn’t just save time—it changed the nature of many conferences with students and their families. Instead of focusing on the frustrating process of trying to decipher a transcript, the focus was on celebrating successes and how to meet ongoing needs.

I gave a lightning talk in 2013 about how we could bring the well-established concept of openness in programming into the education world, in a way that goes beyond free software. Bill was in the audience, and he’d been having his own frustrations with the school system at that time. He said something along the lines of, “I like what you have to say about education. If you ever want to write a book, feel free to submit a proposal.” Some early No Starch books had been really influential in my life, so that invitation was compelling. I got my start on Linux by reading Ubuntu for Non-Geeks.

I didn’t really want to write a book, because I was focused on those educational infrastructure projects. But when I got back from the conference, I saw a handwritten poster I had put up in my classroom. It was titled “What’s the least you need to know about programming in order to work on your own projects?” I had listed out the most fundamental concepts of programming, along with three suggestions for projects: a video game, a data visualization project, and a web app. I realized that the poster was the table of contents of the book I wished I could teach from, and I basically turned that poster into a book proposal.

I naively thought I could write the book during the summer and edit it during the following school year. That turned into two and a half years of early mornings and late nights! But it was very much worthwhile; Python Crash Course has reached a global audience that I could hardly dream of back then. I kept teaching full-time until 2019. But I couldn’t continue to teach well and write well, so I left the classroom to focus full-time on writing and programming.

NSP: While we’re on the subject of conferences, you encourage your fans to go to events like PyCon or local coding meetups—that is, places where they can benefit from taking part in some sort of Pythonista community. Why do you believe it’s important for programmers (of every skill level) to step away from their monitor and go meet people?

EM: As a programmer, especially if you’re new to the field, it’s so easy to get stuck in your own head. It’s tempting to keep telling yourself you’re not good enough yet, that when you just learn a little more you’ll be ready to meet others and talk about programming. But when you step into a Python space—a meetup, a conference, an organized online event—you’ll almost always meet people you have something in common with.

You’ll definitely meet people who know more about Python than you do. But you’ll also probably meet people who are even newer to the field than you are. You’ll find people who are working on problems you’re interested in, and you’ll be inspired in ways you just can’t predict.

Some of what you hear will go over your head, but that’s quite fine. The concepts you hear about but don’t understand yet will be more familiar when you come across them later in your own work. And they won’t be dry concepts then—you’ll have a face you can connect to that topic. 

There’s also a fair chance you’ll make some lifelong friends, especially if you end up staying in the Python community for a while.

NSP: Given that PCC has been a bestseller for the better part of a decade, you probably come across people who read the book years ago and are now really advanced and/or professional Python programmers. Got any tips on how to continue making progress for those who only recently finished the book? Put another way: How does someone go from beginner to expert programmer, and is that even necessary in order to have success in the field of programming?

EM: That happened much faster than I thought it would! I remember talking to someone at a booth at the second conference after the book came out. They were explaining something technical about what their company does, and halfway through they realized who I was. They stopped what they were saying to share how much Python Crash Course had helped them get started in programming. Then they continued their explanation of an area of programming I hadn’t used much and didn’t fully understand. As an author, that was a very satisfying experience!

In the early days of Python, it was pretty reasonable to aim for mastery of the entire language. It wasn’t that big, and back then the community was a lot smaller as well. These days, Python is huge; it would be really hard to know every part of the language in depth. There are plenty of Python maintainers who know one part of the language well, but who wouldn’t consider themselves an expert in other areas. It’s just too big and touches too many specialized areas.

We all know what a beginner is because it’s relatively easy to define. But there’s no clear definition of “intermediate” or “advanced.” I would advise people to learn one language well enough that you can use it to solve meaningful, real-world problems without having to follow a step-by-step tutorial. That doesn’t mean you need to be able to write code from memory. It’s perfectly fine to look back at the tutorials and resources you’ve learned from as you’re working on your own projects.

The best advice I can give once you’ve learned the basics is to keep learning fundamental concepts, but make sure you’re applying those concepts in a variety of real-world projects. There are a lot of concepts in programming that you can’t fully make sense of until you’ve had the chance to apply them in a variety of contexts

NSP: Let’s talk about your passion for projects, then. Python Crash Course is peppered with a number of them—including a Space Invaders-–style video game—and these endeavors definitely contribute to the book’s effectiveness. Would you explain your philosophy that programmers should always have a project in mind? Bonus question: Care to share any cool projects that you are working on right now?

EM: Nobody really learns programming just to write lines of code. The code we write is only meaningful if it actually does something we (or someone else) cares about. If you have a project you’re yearning to build, you’re almost certainly going to be a more active, engaged learner. Every time a new concept comes up, you’ll ask yourself how that concept might help you implement your project. This pushes you to think more deeply about each new concept and try using it in different ways, until you start to internalize that concept.

At the same time, you probably shouldn’t just focus on one large project. It’s helpful to take on some smaller projects that you can complete, and practice the art of “finishing” a project. No codebase is perfect, so learning to recognize when a project is “good enough” is a tremendously valuable skill.

I always have a number of projects going on. I write a weekly newsletter called Mostly Python, and it consists of individual, one-off posts and series about specific topics. I’m working on a program that will convert these series into mini-ebooks. I’m working on django-simple-deploy, a project that automates configuration and deployment for Django projects. I’m slowly learning piano, and I had a hard time learning all the notes of the grand staff well enough to focus on actually playing music. I made an old-school JavaScript site that facilitates learning the notes, and after using it for two weeks, I’m now the fastest student at naming all the notes. (That really means I’m faster than a bunch of fifth graders, but it still feels good.) I’m going to polish that a bit more and then share it more widely.

NSP: Last question. A surprising detail about your life is that you’ve bicycled across the US [checks notes] five times—in fact, you quit your job at one point and lived on a bike for an entire year! Are there any lessons you learned during those hundreds (thousands?) of hours of pedaling that can be applied to the world of Python programming?

EM: Yes, it was thousands of hours of pedaling!

When I was in my 20s and had a few years of teaching behind me, everyone was focused on getting master’s degrees. I just wanted to live outside during the summer, so I biked across the US for two summers in a row. I rode across the northern states one summer and then took a southern route the next summer. Those were such good experiences, I quit my job so I could live outside for a full year. I rode from Seattle to Maine, down to Florida, over to California, and up to Alaska. I had a conviction that I’d learn more from that kind of journey than I would from getting a master’s degree, and looking back, it was definitely the right choice for me.

One thing I took away from that trip was a deep reserve of calm from having faced everything that a year of living outside brings: beautiful sunny days in wild places, sleety mountain passes, sleeping in a tent for weeks at a time with bears, and so much more. Another big life lesson was a better understanding of all the different ways people live their lives. When you travel alone for an extended period without an engine, people open up in a different way. This was before social media, smartphones, Google Maps, and photo-sharing sites. I stopped a lot to ask directions, and just talk to people on the side of the road. Mostly, I just listened—people want to tell you about the place they live, and many people want to share their life story when you’re outside and don’t have a schedule. People would say, “I wish I could just take off and live on a bike...” and then they’d open up and tell really honest stories about their lives. I think that happened a lot because people knew I was just passing through, and they’d never see me again.

I learned empathy from listening to so many different people. When I write code, I think about the people who will use what I’m making, and I consider who might be negatively impacted by whatever I contribute to. I enjoy time in front of a computer more because of the time I’ve spent far away from a computer. When I deal with stressful bugs and crashes, I remind myself, At least I’m not facing a bear alone on the side of a gravel road in the far north of British Columbia right now.

I believe most programmers benefit from having nontechnical hobbies. Anything you find meaning in away from a computer helps keep your technical work in perspective. Keeping things in perspective is really helpful in a yearslong technical career.

Finishing that trip made it easier to write a book because I had a sense of how to bring a long-term project to completion. In some sense, a 14,000-mile bike trip is nothing more than a whole bunch of individual pedal strokes. When I feel like I’ll never get to the end of a long project, I often think back to hard days of riding and remind myself that every step forward is meaningful. That’s more than just a passing thought, though. It brings back the feeling of living outside on a daily basis. I miss those days, but I’ll carry them with me for the rest of my life.

For anyone interested in hearing more about those adventures, I wrote a book about the trip a while back. It’s kind of funny to be the author of one book that’s sold over a million copies, and another that’s sold about a hundred copies. But I wrote that book more to solidify the memories than to reach a wide audience. 

New year, new spotlight—and this one shines on civil engineer and YouTube star Grady Hillhouse. His first book, Engineering in Plain Sight, was released this past Fall to critical acclaim. In the following Q&A, we talk with Grady about how he went from civil engineer to full-time video producer with over 3 million subscribers (hint: woodworking), why all he needed to know about science communication he learned in kindergarten, the importance of average citizens understanding how things work, and the joy of "infrastructure spotting" on the road.

Grady Hillhouse is a civil engineer and science communicator widely known for his educational video series "Practical Engineering," currently one of the largest engineering channels on YouTube, with millions of views each month. His videos, which focus on infrastructure and the human-made environment, have garnered media attention from around the world and been featured on the Science Channel, Discovery Channel, and in many publications. Before producing videos full- time, Grady worked as an engineering consultant, focusing primarily on dams and hydraulic structures. He holds degrees from Texas State University and Texas A&M University.

No Starch Press: You got your bachelor’s degree in geography, then later earned a master’s degree in civil engineering, and spent nearly a decade working in the field on infrastructure projects. How did you go from that path to becoming a full-time YouTube sensation?

Aglow in our Author Spotlight series this month are the daring duo behind Open Circuits: The Inner Beauty of Electronic Components—Windell Oskay and Eric Schlaepfer. Their book is a truly unique photographic exploration of the astonishing design hiding in everyday electronics, and it's as awesome as it sounds. In the following Q&A, we talk with Eric and Windell about how this project came about, the ins and outs of the hardware disassembly and macro-photography feats it took to make the book, the surprises—both good and bad—that they encountered along the way, and the many challenges of cutting a cathode ray tube in two.

Eric is a hardware engineer at Google, and runs the popular Twitter account @TubeTimeUS, where he posts cross-section photos, discusses retrocomputing and reverse engineering, and investigates engineering accidents. His better-known projects are the MOnSter 6502 (the world’s largest 6502 microprocessor, made out of individual transistors) and the Snark Barker (a retro recreation of the famous Sound Blaster sound card).

Windell is the co-founder of Evil Mad Scientist Laboratories, where he designs robots and produces DIY and OS hardware "for art, education, and world domination.” A longtime photographer, he holds a B.A. in Physics and Mathematics from Lake Forest College and a Ph.D. in Physics from the University of Texas at Austin. Besides Open Circuits, he's author of The Annotated Build-It-Yourself Science Laboratory (Maker Media, 2015).

No Starch Press: First of all, congratulations on all the hard work paying off. The response to your book has been incredible. Considering how popular cross-section pictures were when I was a kid, I guess it’s not too surprising. People still love peeking into things full of hidden complexities! But the books I remember were mostly just intricate drawings—for Open Circuits, you actually photographed real stuff that you cut in half. What inspired this project? Did you intend to add a whole new dimension to the “cutaway” genre?

Eric Schlaepfer: I grew up fascinated by cross sections and cutaways. I’m sure that influenced this book, but it’s not exactly what inspired me. It was a broken piece of equipment, and the problem was one of the electrical components (a tantalum capacitor similar to the one on page 40). I sanded it in half to see if I could figure out how it failed, tweeted a photo, and folks really enjoyed it. So I started cutting other parts in half, and that led to the book.

Windell Oskay: I’ve always been interested in how things are made, in addition to how they work and what’s inside them. One of the really remarkable things about physically cutting things is that you get to see so many features that are maybe incidental to the function of the device, but are signatures of the processes that went into making it. Each part tells a story. And, often, we’re not even saying anything about them. Those little stories are left for the readers to discover.

NSP: The two of you have professionally intersected over the years in Silicon Valley, and have worked together on some design projects for Evil Mad Scientist Laboratories. But who roped whom into this book idea? What compelled you to collaborate at such a level?

ES: We’ve worked together on a number of other projects, such as the Three Fives discrete 555 timer kit, as well as the world’s largest 6502 microprocessor—the MOnSter 6502. Windell had seen my photos on Twitter, and we started talking about how to turn it into a book. I don’t remember all the details but it was a very natural thing.

WO: We’ve had a number of fruitful collaborations. In addition to those that Eric mentioned, we also designed an educational project, “Uncovering the Silicon,” that we presented at Maker Faire (along with Ken Shirriff, our technical reviewer for Open Circuits, and John McMaster, who prepared some subjects for photography). In that project, we placed very simple integrated circuits under a microscope and showed how they worked by tracing their individual parts. There’s a sense in which our book is a successor to that project—we’re letting people look at things up close, and then talking through how they work. But, I think that there was actually a moment when I roped Eric into the book idea after seeing his early cross-section photos.

NSP: What was the most challenging aspect of putting this book together?

ES: There were many challenges. For me, the most difficult challenge was preparing the samples—it took a really long time to prepare each one, taking care to create a polished section with no scratches or blemishes, and being careful to remove every speck of dust.

WO: At one point we realized that we would have to cull the weakest subjects from our draft. We ended up deleting about a dozen—some quite interesting and beautiful—along with their descriptive text. The book is stronger as a whole because we did so, but it really stung at the time. Fine-tuning our text was also difficult in places. For a number of subjects, we only had a few sentences in which to flesh out subtle concepts clearly, to an audience composed of both laypeople and engineers.

NSP: How did you divide up all of the labor that the book entailed? I mean, you had to find hundreds of tiny electronic components, carefully cut them in half, photograph them, write the accompanying text for each page—the list goes on and on!

ES: Windell took on the photography and some of the sample preparation, introducing me to some more professional tools that I hadn’t used before. We spent time searching the local electronics surplus store for potential subjects, and I made a lot of exploratory cuts to see if a particular component would be good enough for the book. I’d say the writing was a 50/50 split—we spent so much time writing and editing over video-chat that I wouldn’t be able to point to any sentence and definitively say 'I wrote this.'

WO: In addition to the bulk of the cutting and sample preparation, Eric also drew the rough drafts of all of the illustrations and wrote the initial drafts of some of the most challenging subjects to describe. I took the photographs, fine-tuned the illustrations, and designed the initial page layout so that we could understand how much text could be paired with each photographic subject. And as Eric said, we worked together closely through all of the writing and editorial choices.

NSP: Eric—what was the hardest thing to cross-section, and how did you eventually make it work?

ES: The most challenging was the cathode ray tube (page 186). Windell had the idea to cut it on the slow-speed saw so we could remove the electron gun. I sectioned the glass envelope and the electron gun separately—each of those took several hours to wet sand. The parts were simply too fragile to section any other way. Cleaning the sectioned electron gun was difficult because of the small magnet inside, which vacuumed up the debris created during sanding.

NSP: Windell—unlike a cross-section illustration, capturing everything inside an object with a single photograph in a single frame had to be difficult at times. Can you give us some examples where you had to get creative to get the shot?

WO: One of the basic limitations that you can run into with macro photography is the limited depth of field—only a very narrow slice of the view is in focus at any given time. We used focus-stacking software to digitally combine pictures taken at different camera positions, stitching them together like a panorama where the entire subject is sharp and in focus. The circuit-board photograph on the front cover of the book was taken this way. Other times, the subject itself can just be plain hard to photograph.

For some of the LEDs, like the surface-mount LED on page 90, we took photos at different exposure levels and composited them (in a basic HDR—high dynamic range—process) so that you can see detail even in the brightly lit LED. For the color sensor on page 81, the photos came out drab until we added an additional light source at just the right position and brightness so that you could see the additional reflection.

NSP: How did you decide what samples and images ultimately made it into the book?

ES: During endless hours of video chat we discussed every potential sample and made a highly detailed spreadsheet. We’re both very organized.

WO: Some part of it was determined by which things we could get our hands on—there are probably 50 other things in the spreadsheet that we might have included if we had an example to disassemble. We did skip a number of potential subjects that were too similar to others, too difficult to section, too difficult to photograph, or that were less likely to be of general interest.

NSP: Anyone who’s into photography knows that what’s pleasing to the eye is not always pleasing to the lens. Were there any samples that you successfully cross-sectioned but just could not get a good photo of—things you left on the cutting-room floor, as it were?

WO: Yes, there were quite a few actually, including some that we put a lot of time into preparing. A good example is a reed relay, where we just couldn’t get a photo that clearly showed the features that we wanted to highlight.

NSP: Given that you both have backgrounds in hardware engineering—and professional tinkering in general—did you know in advance which electronic components would look cool from a cross-cutting perspective, or was there a lot of trial and error? Any surprises along the way, good or bad?

ES: I’ve seen a few component cross sections created for failure-analysis purposes, so I knew about certain components that would look good, but there were definitely a few surprises. We thought an RGB LED would look cool, but after cutting into one, it just didn’t really seem interesting. We took apart a boring-looking gray electronics module that turned out to be a fabulously complex jewel—the isolation amplifier (page 266).

WO: One of my favorites that took some experimentation was the multilayer ceramic capacitor (page 36). There’s never been any mystery about what is in one—stacked layers of metal electrodes—but it took us a lot of experimentation and cutting into different capacitors to find one where you could literally see and count the individual layers. There were definitely real surprises along the way. The way that the rocker DIP switch (page 110) works inside is just stunning elegance.

NSP: You include a “Retro Tech” section in the book for your vintage finds, like Nixie tubes, a mercury tilt switch, and even a magnetic tape head. From a purely aesthetic standpoint, which era wins (Old vs. Modern) as far as microscale interior design goes?

ES: They both fascinate me. Vintage components seem warm and natural to me, being made of less processed materials like brass, rubber, mica, and glass. Modern parts have a sort of cold Cartesian precision and a microscopic intricacy.

WO: Modern electronics has so much more to offer in interior design—there’s just so much more inside. If we were talking about exterior design, I’ll pick the vintage. I love all the brass and Bakelite.

NSP: Windell, your company’s motto is “Making the World a Better Place, One Evil Mad Scientist at a Time.” If you had to come up with a similar motto for your book, what would it be? I’ll go first: “Making the garage a messier place, one experiment at a time!” . . . I guess what I’m getting at is, what effect do you hope your work in this book has on people? Eric, same question for you.

WO: If the book needed a motto, other than our existing subtitle, I’d pick “Showing you the Hidden Wonders Inside Electronics.” I hope that it inspires people to open up their electronics and look inside. To look at the parts for the little clues about how they’re made, what they’re for, and how they work. To appreciate elegant design, where they weren’t looking for it before.

ES: I want to inflame curiosity. Earlier today my very young nephew was totally absorbed in a copy of the book, asking his mother afterwards if they had any circuits he could play with. The world is a better place with curious people living in it.

To kick off the Cybersecurity Awareness Month edition of our ongoing Author Spotlight series, we're joined by Colin O’Flynn and Jasper van Woudenberg, co-authors of The Hardware Hacking Handbook (available November, 2021). In the following Q&A, we talk with Colin (@colinoflynn) and Jasper (@jzvw) about the perils of proprietary protocols being replaced with network devices, the problem of having too many interesting targets to test your tools on, the beauty of AI-designed attack systems, the indisputable power of “hammock hacking,” and why nobody cares about fault injection until they get hacked with fault injection.

Colin runs NewAE Technology, Inc., a startup based on his ChipWhisperer project that designs tools to make hardware attacks more accessible, and teaches engineers about embedded security – a topic he frequently speaks about at conferences and on tech podcasts.

Jasper is CTO of Riscure North America, where he leads the company’s pentesting teams, and has a special interest in integrating AI with security. His research has been published in various academic journals, and he’s a regular speaker at educational and hacking conferences.

No Starch Press: I’ll start by saying that your book is timely! Hardware hacking, once a niche field of the exploit world, has become far more relevant amidst the proliferation of embedded devices all around us. What do you think accounts for this, and why are side-channel attacks in particular becoming increasingly common (and difficult to prevent)?

Colin O'Flynn: Hardware hacking has been a niche field, but one with an extensive and long history. Most of the powerful attacks we’re discussing today have been demonstrated for 20 years, so I’d say they should be “well-known.” But the truth seems to be that, until recently, advanced hardware attacks weren’t needed for most IoT devices. Default passwords and unlocked debug interfaces were the norm, so most hardware hackers never needed to dig deeper. Many people I’ve talked to at events have told me they were interested in side-channel and similar advanced attacks but never had time to actually learn them, as they were always able to break devices with easier and faster attacks!

The good news is that device manufacturers seem to be taking security more seriously these days, which means side-channel attacks have become a real threat. So I guess we’re seeing the industry fast-forwarding that 20-year lag of security research to catch up.

Jasper van Woudenberg: Hacking always moves with interesting targets. Once pinball machines started requiring money to play, people “hacked” them by just tilting the whole machine. Nowadays physical pinball machines have a tilt sensor – if you tilt the machine in order to affect the ball, it ceases operation. Of course, we’re talking about digital hardware in our book, but bypassing security systems is as old as security systems. So, the abundance of digital devices naturally increases the amount of hacking going on. Side-channel attacks are fascinating if you’re into the intersection between electronics, signal processing and cryptography. Beyond being fascinating though, they only become relevant when more straightforward attacks are mitigated.

NSP: Fault injection (FI) attacks – which inject a glitch into a target device that alters its behavior so you can bypass security mechanisms – used to be too “high end” for most hackers to bother with, often requiring expensive tools and intimate technical knowledge of the specific system under attack. But those days are over. Not only are low-cost FI toolkits readily available, the explosion of IoT has led to the rise of new defensive features, like Secure Boot, that can be easily subverted by a well-timed FI attack. What are the potential risks to a larger IoT network once a device is compromised this way?

CO: In the past we’ve seen end devices used as a pivot point into a more sensitive network. When it comes to commercial devices, we’re seeing many proprietary protocols replaced with network devices. For example, recent access-control readers are now simply PoE devices that talk back to a central server. With many of these devices, the original designers haven’t considered what happens if an end node becomes compromised. While the network may be correctly secured, you still see sensitive credentials stored in end devices become accessible to an attacker. And if an attacker is able to access these credentials, it means they may be able to pivot off the external network and into more sensitive internal networks.

JVW: I think the cost of the tools is a common misunderstanding – they can be really inexpensive. In our lab, we’ve done attacks literally by soldering a single wire to a bus, connecting it to a button, and when we pressed the button at the right time, the system boots our own code. The cost usually comes from the many days and weeks spent trying to figure out how to carry out the attack. And yes, some attacks do require high-end equipment, or at least equipment that can bring down the time used to figure out the attack.

One common stepping-stone attack we see is the firmware dump. Typically, embedded-device firmware does not receive a lot of scrutiny, and may have lingering vulnerabilities that can be exploited. This usually means gaining control over a single device, but there have been wormable firmware issues in the past.

NSP: What measures can be taken to harden embedded systems against FI attacks, and do you see this happening throughout the industry (why or why not)?

JVW: We always advise our customers to threat model and see if it makes sense to consider FI in scope. Usually that’s the case for embedded systems that are out in the field and have some sensitive assets to protect. Next is the question of whether faults can be mitigated in hardware and/or software. Both is ideal, but that’s not always feasible. Our book contains a chapter on countermeasures that also has a lab, so people can try out some ideas for FI countermeasures. Finally, verification of countermeasures early and often is critical. It’s virtually impossible, as a human, to predict all the ways a system can fault. Pre-silicon fault simulation and post-silicon fault injection, without exception, turn up surprises. Iteration and adaptation are key.

And then the million-dollar question: why is the hardening not happening throughout the industry? It’s a combination of cost and human nature. There is a real engineering cost to these countermeasures, so typically we only see customers that have had their devices compromised requiring FI resistance. If a compromise hasn’t happened, it’s very easy to write the attacks off as unrealistic or irrelevant. Nobody cares about fault injection until they get hacked with fault injection.

CO: Fault injection can be tricky to prevent, as we see countermeasures applied that aren’t effective. For instance, Jasper and I demonstrate a few examples in the book where compilers might remove the effect of your clever countermeasures. There seems to be a lot more interest in this now – for many companies, they just need some “end customer” to ask about it. I talked to silicon vendors a few years ago who were tracking countermeasure ideas, but basically none of their customers (people who actually build products) cared about FI attacks. So that meant they weren’t going to pay for engineering efforts to add those countermeasures. We seem to be seeing a very fast shift in the last couple of years though, so people who were tracking this early-on are in a good position to quickly offer solutions.

NSP: Speaking of low-cost fault-injection toolkits, Colin, you developed one of the most popular models out there, the ChipWhisperer, and built a company around it (NewAE Technology). Given that just about everything we use in our homes and offices has embedded computing systems and could be vulnerable to attack, how do you pick which devices to test your boards and analysis algorithms on? An example from your book would be smart toothbrushes – are you ever doing something like brushing your teeth when it suddenly occurs to you, “Wow, I could totally hack this thing”?

CO: This is actually a big problem! Unfortunately I tend to buy a lot of devices (microcontrollers, IoT products, industrial control systems, etc.) because I think they will be interesting to poke at! As a result, I’ve got a storage cabinet full of various devices along these lines… I’m slowly working through some of them, and when we get some time at the company, we’ll pick away at one or two of those devices as well.

But as more devices include embedded security, there are more “interesting" targets than there is hope of having time to deal with them. Part of why we design many different target-board devices (our “UFO targets” for ChipWhisperer) is actually to help out other researchers by giving them an easier platform to work with.

NSP: Once you successfully exploit a commonly used product, do you let the manufacturer know or is that generally considered an exercise in futility?

CO: If I plan on talking about the issue publicly I’ll reach out, even if I don’t think it’s a serious issue. Sometimes it takes a bit of time to reach the correct person (or team), but so far they have all generally spun it into positive experiences all around.

With one ongoing disclosure, for example, the engineering team had internally flagged that there could be some issues related to a relatively unsecure microcontroller that they were using in a product, and my report had validated their internal concerns. In this case they were already working on a new design, but I’m sure my report was a nice bonus for the people involved, as they can point to that as proof that the issue would be found eventually “in the wild.” In the meantime it gave them the opportunity to provide an interim fix via a firmware update for existing customers.

NSP: Jasper, one of your specialized areas of interest is combining AI with security research. Would you explain what this entails? And looking into the future, how could AI applications be leveraged to improve hardware and embedded security at the design level?

JVW: What I love about AI is also what I love about hacking: making a computer device do more than the original designer put in. With AI, this is tying a couple of artificial neurons together and getting a cat-and-dog image detector. With hacking, this is sending some weird input into a program and all of a sudden it executes arbitrary code.

The combination, I find fascinating. For instance, we’ve used neural networks to do side-channel analysis and outperform human-designed algorithms. We created an algorithm with colleagues that automatically optimizes fault injection parameters. I’ll work very hard to create some automation so I can be – paradoxically – lazy afterwards.

I firmly believe that most if not all cognitive activities, such as designing or attacking systems, will be better performed using AI rather than brains – the big question is when. I prefer to be on the side of making systems more secure through AI, so my research is going towards automating both the detection and mitigation of vulnerabilities, at scale. For instance, a big push we have currently is in pre-silicon security – detecting side-channel and fault issues before they make it into products. I wouldn’t say we’ve arrived at using AI yet, but the first steps are being made.

NSP: Both of you have advanced degrees, which makes sense given all of the academic knowledge involved with embedded security. Yet, The Hardware Hacking Handbook makes very little assumption about a reader’s background. What was your approach to making this challenging field accessible to novices and newcomers, and why was it important enough that you wrote an entire book on this premise?

CO: My career path on paper seems relatively full of academic love – I was an assistant professor for several years in the Electrical & Computer Engineering department at Dalhousie University. But back at the start, when I was considering applying to start my undergraduate degree in electrical engineering, I came relatively close to not attending university at all. I had self-taught myself a fair amount about electronics in high school, and managed to get a summer job that was effectively an electrical engineering internship, and was considering just continuing to grow with the “on the job” experience instead. In the end I fell onto the academic path, but I’ve always believed that it is not the only path, and part of this shapes my desire to make this as accessible as possible.

While many readers may be undergraduate or grad students, it’s clear that a classic academic textbook would cut out readers coming from other backgrounds (including everyone from high school students to professionals interested in looking at other careers). Practically, what we write down isn’t the only consideration – one of the great things about working with No Starch Press is that the pricing of the books makes them more accessible as well. From academic publishers, this book would have been $150+. And there would never be Humble Bumble sales that make it completely accessible on the level that NSP does!

JVW: I’ve taught courses on side-channel and fault injection for years, and it has taught me that the group of people that has to defend against these attacks is not necessarily interested in all the theory and all the research in this field. They want to focus on their goals of creating a system.

Then there’s the group of people like teenage me. I started hacking software before I had an internet connection, so I know the struggle of having to figure out everything by yourself. Looking around at the amazing blogs, videos, tutorials, etc. that exist for the software space currently, it really made me realize what a gap there is in the hardware space.

So, for both these groups, it’s really about breaking things down into practical tips and tricks, and then some of the unavoidable theoretical background. I really would like to show people that this space isn’t daunting, and that even someone like me – who came off a software background – can learn and enjoy it.

NSP: I’ll end with an easy one (I think) – what is your favorite hacking tool, and has that changed since you first got interested in hardware hacking when you were young?

CO: I should probably say my favourite tool is one of my own more-advanced products. But really, a good DMM is the most important tool! And in that regard, it hasn’t changed much over the years – one of my first “dream gifts” (back when Santa would be responsible for it) was a Fluke 12 multimeter, long before I knew about hardware hacking. I’ve since upgraded to a nicer meter (Fluke 179/EDA2 kit), but as we talk about in the book, there is so much you can do with this tool! Finding where pins go, checking the state of logic levels and voltages – it’s still my most used tool when I’m looking at a new device.

JVW: I started being “creative with technology” in the mid-’90s. What has changed is the amount of information available, and the fact that security is now an actual career – I still don’t always believe people are willing to pay me to do this. What hasn’t changed is my curiosity, and the rush that comes with solving a complex problem.

Favorite hacking tool? Hah. Although I use devices for a significant portion of the day, they are also a source of frustration. So, those are out. I’m going to say: my hammock. When I get stuck on a problem and I sense no more new ideas are being produced, or I get frustrated, I drop the problem for a few hours or days. Then I hop in my hammock for what I call “hammock hacking.” This is where I hang back and relax. I’ll almost always have a new view on the problem, or another way of connecting some dots that I hadn’t considered before. Or I fall asleep. But it’s a win in either case.