The video on-demand of this session is available to logged in QCon attendees only. Please login to your QCon account to watch the session.

Session + Live Q&A

Reversible Debugging with RR

rr is a low-overhead (about 1.2x slowdown) trace-capturing debugger that you can use with your Rust program to investigate tricky bugs. This talk will describe the features of rr, review its overall design and available deployment targets, show some common usage patterns, and end with a demonstration of rr on a Rust program executing on an AWS cloud desktop.

Main Takeaways

1 Hear about RR, a native code debugger: its features, design, and deploying targets.

2 Learn about RR through an interactive session debugging a Rust program running on AWS.

What is the work that you are doing today?

There's the work that I'm doing actively: I'm part of this new Rust platform team at Amazon Web Services, where in some ways the work I'm doing now is a continuation of what I was doing at Mozilla: I'm contributing to the Rust compiler, I'm actually co-lead of the Rust compiler team. But more generally, the Rust platform team's mission at AWS is to enable Rust programmers to do their work joyfully. That is at the heart of what our whole team is trying to do. 

The specific topics that I am looking at, besides the things related to the leadership of the compiler team, the other big one that I'm investing time in is: Rust has this promise of providing you with great performance. We promise that you're supposed to get soundness guarantees and certain levels of productivity due to tools like Cargo, and other other nice things. But we also say you're supposed to get performance that's commensurate with what you'd expect from C or C++. And the problem is that, at least for our user base, they don't always have the ability to evaluate whether the performance actually is meeting the goals they have. They don't necessarily know how to use the profiling tools available on their systems, or even if they do know how to use the profiling tools, the tools don't always work very well with Rust out of the box. 

So a key focus for me right now is on either improving support for the existing profiling tools, or coming up with entirely new ones in cases where there are no options available. The particular case I'm interested in right now in that latter category is for asynchronous code. There are no tools out there for being able to evaluate why is my asynchronous code not behaving as performantly as I would want, or even why is my asynchronous Rust code deadlocking? And so we're investing a lot of energy into developing tools to improve the situation there. And I had a whole series of blog posts that I put out called The Road to TurboWish. That's the name for the suite of tools we're making here in this space called TurboWish, which it's a pun on the TurboFish operator in Rust. That's my big focus, it is compiler team leadership, general language design stuff, and performance tooling.

But my talk is about a particular debugging tool called RR, which I first was exposed to at Mozilla. And I basically just want to advertise it to the world as something amazing to use when you're trying to understand your code's behavior and just explain what it is, how it works, and how it makes life awesome. 

What are your goals for the talk and what do you want the audience to walk away with after the talk?

I want the audience to walk away with an inspiration to go and try out RR, because my belief is that everyone had some experience in the past with some bug that was impossible to figure out, and a better tool for exploring the behavior of your code and being able to spelunking into the caves that are that are formed by your program source is something that's enabled quite well by RR. So I want to get people really excited about debugging. That's the biggest thing, because when I tried to talk to people about using debuggers, I got a lot of resistance, people saying we just don't use debuggers in our space. println!’s are good enough. And I spent a while wrestling with that attitude. It's one I understand myself, and I talk about it in my talk about how I've had a love-hate relationship with debuggers, and it's a part of my talk, getting to the bottom of where does this come from? Why do people not want to use debuggers and how is RR different? How does it change the calculus there? And so I want the people who are used to using println! debugging or looking over logs to consider using this alternative and to see why it could change their lives. 

That seems to be this big schism in the industry of people who use debuggers on a daily basis and then others who pride themselves on not using them..

It is a strange thing. I think part of it is that to become an expert user of a debugger does require a certain amount of investment. And once you've put it in that investment, you might try to apply it to a context where it doesn't even really apply. It's the kind of thing where now you've learned to use this tool very well, so you're just going to use it all over the place, even if it's not helping you solve your problem. And so I can see that there's a reason why. I don't know how to resolve that particular problem. I'm not going to do research personally into how to make debuggers easier to use, but I think that the main thing that RR gives you is deterministic replay to the point where you can step forwards and backwards through your code and inspect the state of it. That enables pure code exploration, nothing too specialized in terms of bells and whistles of a specific debugger. I think that's the crucial thing that can let people who were only used to using println!’s still see a huge benefit here. 

Especially because RR is a time-traveling debugger, that's an interesting feature that many debuggers don't necessarily have.

There's a category of DBs here. I do a survey of a number of them because it's an old idea, and this is a part of my talk, to give a list of them. I will admit I only cover the time-traveling debuggers that were applied to systems programming languages. So things that worked on native code or on the whole hypervisor-based operating systems. So the space that I didn't explore in my survey was the ones that are built on hosted languages. In particular, there are some JVM time-traveling debuggers. And also I think Elm in particular has some sort of time-traveling debugger that I think has gotten a lot of attention; I didn't know about it until pretty recently. So it's not part of my talk. But I think that there is some evidence of interest in the space from other domains or other people in other language communities. And so it's possible there's lots to be learned from looking over there. I just haven't done the research yet.


Felix Klock

Principal Software Engineer @awscloud

Felix Klock is a principal software engineer at Amazon Web Services. He is also a member of the Rust language design team and co-lead of the Rust compiler team. His past programming language work includes: Rust while at Mozilla, ActionScript while at Adobe, and Larceny Scheme while at...

Read more
Find Felix Klock at:


Tuesday May 25 / 11:10AM EDT (40 minutes)


System Level Programming Languages



Add to Calendar

Add to calendar


From the same track

Session + Live Q&A Rust

The Rust Borrow Checker—A Deep Dive

Tuesday May 25 / 09:10AM EDT

The Rust compiler's borrow checker is critical for ensuring safe Rust code. Even more critical, however, is how the borrow checker provides useful, automated guidance on how to write safe code when the check fails. Early in your Rust journey it may feel like you are fighting the borrow...

Nell Shamrell-Harrington

Principal Software Engineer @Microsoft

Session + Live Q&A Rust

Systems Programmers Can Have Nice Things

Tuesday May 25 / 10:10AM EDT

Systems programming presents unique challenges that differ from other areas of software development. However, some of the largest challenges of systems programming aren't inherent to the domain, but rather stem from weak programming language abstractions and overly complex tooling.In this...

Ryan Levick

Principal Rust Developer Advocate @Microsoft


System Level Programming Languages Panel

Tuesday May 25 / 12:10PM EDT

Come to this panel to hear the story of Oxide Computing - a company that builds its systems from the ground up with Rust. We will hear why they chose Rust, information about the operating system they are building on Rust, what they've learned about Rust along the way, and where they'd...

Bryan Cantrill

Co-Creator DTrace, Co-Founder Fishworks Sun Microsystems & Co-founder and CTO @oxidecomputer

Laura Abbott

Engineer @oxidecomputer

Cliff Biffle

Engineer @oxidecomputer

View full Schedule