The video on-demand of this session is available to logged in QCon attendees only. Please login to your QCon account to watch the session.

Session + Live Q&A

Streaming-first Infrastructure for Real-time ML

Because of data drift, the accuracy of an ML model degrades over time. How well a model performs depends on how often it’s updated. While companies like Alibaba, ByteDance, Google, Facebook have been able to leverage real-time pipelines to continually update many of their models in production and boost their performance, many companies still update their models manually. In this talk, we’ll discuss the state of continual learning for ML, its motivations, challenges, and possible solutions.

Main Takeaways

1 Find out how a streaming-first infrastructure can help you do ML in real-time, both online prediction and continual learning.

2 Discover why real-time ML is getting traction in the industry.

3 Learn what are some of the challenges to implement real-time ML.

What is the focus of your work these days?

My work has been in helping people bring ML into productions. I come from a research background, then I joined NVidia in their applied research team, where I realized a lot of the challenges are in production. It's not that the research is not interesting. I think research is extremely important. But I don't think I have the best skillset for research, whereas I think I can make my impact in production. Currently, I'm working on a startup. We are still in stealth. Our goal is to help companies leverage streaming-first architecture to do real-time machine learning, both for online prediction and continual learning. That's going to be the topic of my talk as well. And the nice thing about streaming infrastructure is that you get a good view of the data transformation through the entire system. So it's really good for model observability. I'm also teaching machine learning systems design at Stanford where I feel like I learn more from students than they learn from me.

How would you describe the persona and the level of the target audience?

Definitely is a technical talk, so it's geared toward both machinery engineers who develop ML models and DevOps engineers who have to deploy and maintain ML models. I'm a huge believer there's a big difference between things that are inherently complex and things that people just make complex because they can't find an easy way to explain. So even though the talk is technical, I hope to make it easier to understand for a less technical audience. We’ll be briefly discussing real-time ML from the business perspective as well. I think the problem with adapting real-time machine learning is that it requires a non-trivial infrastructure investment, so you need buy-in from leadership. I hope that there will be some technologically-progressive business leaders who watch my talk and say, it is an interesting direction and maybe try to experiment to see if it works for the organization.

What would be some of the key takeaways they would get?

I hope that they would have a better understanding of what real-time machine learning means and the challenges around deploying it, adopting the infrastructure for real-time machine learning. Also, I would like to show some of the interesting use cases and motivations for why real-time machine learning is gaining traction in the industry. Some of the answers that I hope they would find interesting to think about, for example, how to evaluate whether ML is right for your use cases.


Chip Huyen

Founder at stealth startup & Teaching ML Sys @Stanford

Chip Huyen is an engineer and founder working to develop tools for ML models to continually learn in production. Through her work with Snorkel AI, NVIDIA, and Netflix, she has helped some of the world’s largest organizations deploy machine learning systems. She teaches Machine Learning...

Read more
Find Chip Huyen at:


Tuesday Nov 9 / 01:10PM EST (40 minutes)


ML Everywhere


Machine LearningData EngineeringArtificial Intelligence

Add to Calendar

Add to calendar


From the same track

Session + Live Q&A Machine Learning

What You Should Know Before Deploying ML in Production

Tuesday Nov 9 / 11:10AM EST

MLOps provides tools that make building, deploying, and maintaining machine learning solutions easier than ever before. However, MLOps is not only a static set of tools that defines the way you operationalize your machine learning models, but is also (and most importantly) about your organization...

Francesca Lazzeri

Principal Data Scientist Manager @Microsoft

Session + Live Q&A Machine Learning

The Unreasonable Effectiveness of Zero Shot Learning

Tuesday Nov 9 / 12:10PM EST

A long and expensive phase of machine learning projects is collecting and labelling data, and training ML models. In this talk I will show how one can get started deploying models without requiring any data. We talk about what foundational models are, and go through examples of them, such as...

Roland Meertens

Product Manager @annotell

PANEL DISCUSSION + Live Q&A Machine Learning

ML Panel: "ML in Production - What's Next?"

Tuesday Nov 9 / 02:10PM EST

The panel will discuss the current lessons learned with putting ML systems into production.What is working and what is not working, from building ML teams, dealing with large datasets, governance and ethics/privacy issues, and what's around the corner for production ML, and ML in computing...

Chip Huyen

Founder at stealth startup & Teaching ML Sys @Stanford

Shijing Fang

Principal Data Scientist @Microsoft

Vernon Germano

Senior Manager of Machine Learning Engineering @zillow

View full Schedule