Microservices to Async Processing Migration at Scale

Netflix creates and analyzes operational and analytical data associated with playback of thousands of titles by over 200 Million members worldwide. The data powers product features such as members’ ability to see and manage their viewing history. The data also feeds into the core business analytics as well as to the personalization and recommendation engines.

Previously built systems utilized microservices to ingest playback data in a synchronous manner, which potentially propagates any intermittent back pressure to the edge, and sometimes all the way to the clients on member devices. Utilizing an asynchronous processing model for the playback data, with a durable queue to absorb intermittent back pressure, we have migrated the systems seamlessly with no interruption to, and no changes required from, both the upstream and the downstream services.

We share our experience from the migration along with our design and implementation choices. Asynchronous processing at scale requires attention to managing any data loss with highly available infrastructure, elasticity to handle bursts without a high latency, fault tolerance with graceful degradation, as well as handling out of order and duplicate data. Specifically, we share the lessons learned from migrating the viewing history subsystem that provides the product feature as well as powers the personalization and recommendation engines.