You are viewing content from a past/completed QCon Plus - May 2021

The video on-demand of this session is available to logged in QCon attendees only. Please login to your QCon account to watch the session.

Session + Live Q&A

Resilience in Supply Chain Security

Open source usage has exploded in the last decade, but supply-chain practices and hygiene have not kept up. Unfortunately, attackers have started to notice and open source is under attack. We have hardening work to do on our existing delivery pipelines and supply-chains, but it's too late to keep attackers out. This means it's just as important to design resilient systems, those that help us gracefully recover when bad things do happen.

This talk will go over the real-world threats facing open source supply-chains today, and what you can do to architect resilient build and delivery pipelines.

Main Takeaways

1 Hear about security threats existing today for the open source supply chain.

2 Learn how to architect resilient build and delivery systems.

What is the work that you are doing today?

I am the engineering lead for Google Open Source security team. Our role is to make it easy for Google to both consume and produce open source software and software based on open source securely.

What's the security part in this?

Google is well-known for using a monorepo for most of our internal development practices, where all the code, including all of the transitive dependencies, are checked into one place. This falls apart when we move into the cloud and open source environments where we're using the tooling that is familiar by upstream and shipping open source projects to customers who want to use standard open source tooling. We need guarantees around source code, provenance, supply chain security, vulnerability management, secure built systems and all those things for open source, and doing this in open source as well. 

And this is for Google's open source projects, or do you provide this for others, too?

We want to make this as free and easy to use for everybody as possible. We consume open source produced by other people. By making it easy for them to use these tools, it improves the full supply chain for everyone, including us. 

Are you also involved in the various fuzzing services that Google offers?

Yes. This is one of the big parts of our team as well, that's been running for years, finding thousands and thousands of critical security vulnerabilities.

How do you run the fuzzers? Do you just point them at the source code or do you customize them for individual projects? Because fuzzing can be a bit noisy.

We have a couple of different strategies for fuzzing. There is the oss-fuzz, where there's a big open source repo where people can check in configurations for their projects that they want to be fuzzed. And when they do this, we fuzz it for free using our huge compute infrastructure. As part of the fuzzing integration, people have to tell us how to report problems that we find. We do a pretty good job here trying to figure out exactly where things were introduced and looking for false positives. As part of that reporting, we can't just file these things publicly on open GitHub because a lot of them can be critical security issues. So we report this privately to the maintainers and let them deal with it privately at first. Because we do this, we can actually track things from when stuff was actually introduced. When we find a fuzzing crash, all the way to getting the bug fixed and we maintain a high rate, in the 95 percent rate, of things that we find that actually end up getting fixed.

What are your goals for your talk?

The topic is supply chain security in general and then aligning it with the resilience topic for the whole track. I want to give people an overview of what supply chain security means for open source and why it's such a big problem. Then talk about the different ways that people can improve their supply chain security by designing for resiliency in particular. It's actually a pretty good fit for this talk. A lot of approaches to supply chain security assume that no bugs will be found, and you'll do a perfect job preventing malicious code from being introduced and your keys will never be leaked, and your developers won't ever lose their hardware tokens and all this stuff and everything will be signed. Really, that creates a pretty brittle system because as we know, mistakes always happen, and being able to plan for key revocation and mitigation when bad things do happen is just as important as securing the overall supply chain. 

So it's largely about dealing with predicting the things that could go wrong and putting in mitigations. 

Yes, exactly. Trying to encourage people to step back and plan for when things go wrong instead of just trying to prevent things from going wrong. 

In your view, what are the biggest issues of supply chain security nowadays, Is it the NPM, package managers, is it that people bribing their way into open source projects or anything like that? 

I like to say there are two classes of problems. The two problems, the way I like to describe it, with open source security and open source supply chain security are, one, that it's open source, which means anybody can see it, anybody can contribute to it, and not everybody is nice. So you get a lot of attacks like typosquatting and malicious commits, and people attacking build systems because everything is done in the open and all the stuff is exposed. So that's one class of the supply chain attacks. That's pretty serious. 

Dan: The other one is just that the open source software is still software and it has lots of bugs in it, just like all software, not more or less, but there are a ton of them out there. And people have this perception that just because it's a widely used package on NPM, it's bug free and people have hit all the issues so far. Even if you have the most secure supply chain, these bugs are still going to come in. So we need to do fuzzing in programs to fix those bugs, as well as stop the leaks in the supply chain, but also fix everything that is coming in. 

What about new threats? Do you think something will come up that we haven't seen yet? 

I don't think they are really new threats. It's a great question, though. I think we've known about this class of attacks since 1984. Ken Thompson, sometime in the early 80s, wrote the Reflections on Trusting Trust paper. This can go back to a single compiler. Now, everything was built from that compiler and everything from those compilers, all the way down. Now you have no trust. And for some reason that just hasn't been a big worry. It hasn't been a big threat. People have not been taking it seriously until now. I think the real reason it's finally rising now because somehow it got easier. I think it's a good enough job at locking down all the other doors and all the other ways in, things like let's encrypt. SSL finally got deployed everywhere else. Password managers are being used, 2-factor is now a thing. Supply chain attacks haven't really gotten any easier. They've just become the easiest way in and one of the easiest attacks to carry out now because everything else has gotten so much harder. I think it's going to get worse. People are going to do it more and more. Luckily, a lot of it is being done by researchers and testers, and internal red teams. But some of these things can have very, very far-reaching consequences for everybody.


Dan Lorenc

Software Engineer @Google

Dan Lorenc is a Staff Software Engineer and the lead for Google’s Open Source Security Team (GOSST) He’s been working in the Cloud space for eight years and has mostly focused on open source tools related to building containers easily and securely. He founded projects like Minikube,...

Read more
Find Dan Lorenc at:


Thursday May 27 / 10:10AM EDT (40 minutes)


Architecting for Resilience


ResilienceSecurityOpen SourceContinuous Delivery

Add to Calendar

Add to calendar


From the same track

Session + Live Q&A Resilience

Incident Analysis: Your Organization's Secret Weapon

Thursday May 27 / 09:10AM EDT

Learn how you can move faster and focus on the things that matter by using incident analysis as your secret weapon. Operating at speed and at scale tests the capabilities of even the most experienced engineering teams. In this software world, it is inevitable that things will break. When they do,...

Nora Jones

Founder and CEO @jeli_io

Session + Live Q&A Resilience

Making Applications Resilient With a Smart Application Aware Network

Thursday May 27 / 11:10AM EDT

Building resilient highly available applications is not easy since there are so many parts of the hardware and software stack that can fail. This talk will go through three patterns that can be used to make your applications highly available by transparently injecting smart application aware...

Varun Talwar

Co-founder @Tetrateio

PANEL DISCUSSION + Live Q&A Resilience

Architecting for Resilience Panel

Thursday May 27 / 12:10PM EDT

Join us for the Architecting for Resilience Panel to widen your horizons on what architecting for resiliency means, hear ready-to-use examples from our panelists, and gain new ideas you can employ in your own context.

Nora Jones

Founder and CEO @jeli_io

Dan Lorenc

Software Engineer @Google

Varun Talwar

Co-founder @Tetrateio

View full Schedule