Resources & Transactions: A Fundamental Duality in Observability

Fundamentally, there are only two types of “things worth observing” when it comes to production systems:

1. Resources, and
2. Transactions

The tricky (and interesting) part is that they’re entirely codependent. “Transactions” are the things that traverse your system and (hopefully) “do something.” The classic example would be an end-user request that propagates across networks and process boundaries. “Resources” are the things in your system that transactions *consume* in order to do their work. Resources survive across transactions. As Transactions transact, they use Resources. And as Resources saturate, Transactions suffer.

You know what that suffering feels like: ordinary DB queries that suddenly get really slow; backend services that don’t respond to requests anymore; OOM flapping; etc. The tricky part is that *end-users don’t care about Resources!* At all. End-users only care about their Transactions, and yet, as operators, we cannot disentagle those from the Resources they consume. It turns out that observability can help here, but only if we move far beyond the misleading telemetry-based silos of “traces, metrics, and logs.” We must reframe the question in terms of *Change*, and the join points between Resources and Transactions. In this talk, we will explore these topics, both theoretically and through some real-world examples, and develop an intuition for how we can understand our systems more completely.