The video on-demand of this session is available to logged in QCon attendees only. Please login to your QCon account to watch the session.

Session + Live Q&A

Building & Operating High-Fidelity Data Streams

The world we live in today is fed by data. From self-driving cars and route planning to fraud prevention, to content and network recommendations, to ranking and bidding, our world not only consumes low-latency data streams, it adapts to changing conditions modeled by that data. 

While software engineering has settled on best practices for developing and managing both stateless service architectures and database systems, the ecosystem of data infrastructure still presents a greenfield opportunity. To thrive, this field borrows from several disciplines : distributed systems, database systems, operating systems, control systems, and software engineering to name a few. 

Of particular interest to me is the sub field of data streams, specifically regarding how to build high-fidelity nearline data streams as a service within a lean team. To build such systems, human operations is a non-starter. All aspects of operating streaming data pipelines must be automated. Come to this talk to learn how to build such a system soup-to-nuts.


Sid Anand

Chief Architect @Datazoom, PMC @ApacheAirflow

Sid Anand currently serves as the Chief Architect for Datazoom. Prior to joining Datazoom, Sid served as PayPal's Chief Data Engineer, focusing on ways to realize the value of data. Prior to joining PayPal, he held several positions including Agari's Data Architect, a Technical Lead in...

Read more
Find Sid Anand at:


Monday Nov 8 / 11:10AM EST (40 minutes)


Modern Data Architectures, Pipelines, & Streams


Data StreamsData EngineeringDatabaseStreaming Data

Add to Calendar

Add to calendar


From the same track

Session + Live Q&A Data Streams

Microservices to Async Processing Migration at Scale

Monday Nov 8 / 12:10PM EST

Netflix creates and analyzes operational and analytical data associated with playback of thousands of titles by over 200 Million members worldwide. The data powers product features such as members’ ability to see and manage their viewing history. The data also feeds into the core business...

Sharma Podila

Software Engineer @Netflix

Session + Live Q&A Big Data

Protecting User Data via Extensions on Metadata Management Tooling

Monday Nov 8 / 01:10PM EST

In a world where data collection is ever-increasing and new and expanded data protection laws like GDPR and CCPA are introduced yearly, metadata management, the act of storing contextual information about collected and stored data, has become a required staple for many companies. This talk gives...

Alyssa Ransbury

Security Engineer @Square

PANEL DISCUSSION + Live Q&A Data Streams

Managing Data at Scale

Monday Nov 8 / 02:10PM EST

Since the advent of the internet, the need for reliable, low latency access to data has grown at a rapid pace. Data Infrastructure, which was once a single monolithic database, has evolved into a tapestry of point solutions tied together by data movement infrastructure (e.g. data replication...

Mark Grover

Co-founder @Stemma_ai & co-creator of Amundsen

Shirshanka Das

Founder of LinkedIn DataHub, Apache Gobblin, Acryl Data

Chris Riccomini

Distinguished Engineer @WePay

View full Schedule