Robust Foundation for Data Pipelines at Scale - Lessons From Netflix

What is the work that you are doing today?

Jun: I work in the Big Data Orchestration team at Netflix. Our team owns multiple Orchestration services. My work focuses on designing and building the Netflix workflow orchestrator, a robust and scalable platform to provide workflow as a service. It has been widely used by thousands of Netflix internal users. With the scale of Netflix, one of my main tasks is to develop the scheduler to not only support a wide variety of use cases but also be able to scale up and scale out to automate hundreds of thousands of data pipelines at Netflix. 

Harrington: I work in the Data Platform Orchestration team at Netflix. My work focused on building the next generation of scheduler tools. This means building all the orchestration layers for our users to be able to execute jobs and schedule DAGs. In addition to this, I also work on building the event-driven platform for the data platform. We want our platform to be smarter, and we have been adopting and leveraging events, to be able to react to anything that happens in the platform. This has allowed us to orchestrate executions based on events instead of simply relying on time-based mechanisms. For example, instead of saying, “I want to run this workflow at midnight because a specific metric I need is likely going to be ready around that time”, we allow our users to say “I want this workflow to run when the metrics I need are available”. By doing this, we make our platform more efficient because instead of running when you think you have to, you run when you actually have to. 

What are the goals for the talk?

Jun: In this talk, we want to share our experiences while building this robust platform. We would like to share the best practices and lessons we have learned while operating the platform for the past several years. For each of the components, we will talk about its design and the technical decisions we made to better serve our users. After the talk, the audiences will see the tradeoffs for building and managing a large data pipeline platform and can apply some of those principles and best practices to their work when they see a fit.

Harrington: We've been working on this for quite a while, several years or so. Over this time, we have learned that by separating everything into components or layers, we have been able to build a solid platform. This has worked out for us quite well. Whenever we have had to make changes or evolve our tools, we have been able to do it causing zero to minimal impact on our users and the applications that are built on top of us. 

I expect that during this talk, I can clearly communicate how the separation of concerns that we have adopted into our implementation has helped us build our platform. I also want people to understand what we have learned, as well as the benefits of each of the main components. Finally, I would like people to walk away understanding how the platform has been able to evolve and keep up with the scale.