You are viewing content from a past/completed QCon Plus - May 2021

Track Overview

Modern Data Pipelines and Data Mesh

As data-driven decision-making becomes not just ubiquitous but vital for businesses, new challenges arise to make data available in a sustainable way. The increasing need for quality data, available across multiple teams in a scalable platform, designed with privacy concerns in mind, requires new architectures and techniques.  

In this track, we will discuss emerging solutions for creating modern data pipelines that can solve common challenges in today’s world of data: data engineering bottlenecks, data pipelines deployment in production, data discover, and more. We will also dive into the concept and application of Data Mesh, a new paradigm do organize and distribute data in an organization.


From this track

Session + Live Q&A Architecture

Data Mesh in the Real World: Lessons Learnt From the Financial Markets

Thursday May 27 / 09:10AM EDT

CMC Markets, a FTSE 250 financial services company, has been running a successful trading platform for more than 30 years and is now undertaking a broad and ambitious transformation to take advantage of new technologies and ways of working. One vital aspect of a successful transformation to a...

Tareq Abedrabbo

Core Data Principal Engineer @CMCMarkets

Session + Live Q&A Architecture

Robust Foundation for Data Pipelines at Scale - Lessons From Netflix

Thursday May 27 / 10:10AM EDT

At Netflix, Data/ML pipelines are widely used and have become central for the business. A very wide scenario presents diverse use cases that go beyond recommendations, predictions and data transformations. As big data and ML gains presence and becomes more impactful, the scalability and stability...

Jun He

Sr. Software Engineer in the Big Data Orchestration Team @Netflix

Harrington Joseph

Sr. Software Engineer @Netflix Data Platform Orchestration Team

Session + Live Q&A Architecture

Data Mesh: An Architectural Deep Dive

Thursday May 27 / 11:10AM EDT

Data Mesh is a paradigm shift in how we imagine and build big data management solutions, and most importantly is a shift in how we form our teams around these solutions and govern them. Data Mesh departs from half a century old assumptions on how we need to manage analytical data to be useful. It...

Zhamak Dehghani

Director of Emerging Technologies @thoughtworks & Creator of the Data Mesh concept

PANEL DISCUSSION + Live Q&A Architecture

Data Pipelines & Data Mesh: Where We Are and How the Future Looks Like

Thursday May 27 / 12:10PM EDT

In the panel, experts with different backgrounds will discuss the current challenges for building Modern Data Pipelines and applying Data Mesh in the real world. We will also discuss how the future looks like in terms of new techniques, architectures, and tools to make effective data-based projects.

Zhamak Dehghani

Director of Emerging Technologies @thoughtworks & Creator of the Data Mesh concept

Tareq Abedrabbo

Core Data Principal Engineer @CMCMarkets

Jacek Laskowski

IT freelancer, Java Champion & Author of "The Internals Of"


Speakers from this track

Tareq Abedrabbo

Core Data Principal Engineer @CMCMarkets

Tareq Abedrabbo is a Core Data Principal Engineer at CMC Markets. His main interests are in designing and building data systems that can scale, technically and organisationally. During his career, Tareq worked with a variety of organizations, from startups to large enterprises, on complex...

Read more
Find Tareq Abedrabbo at:

Jun He

Sr. Software Engineer in the Big Data Orchestration Team @Netflix

Jun He is a Sr. Software Engineer in the Big Data Orchestration team at Netflix, where he is responsible for building the big data workflow scheduler to manage and automate ML and data pipelines at Netflix. Prior to Netflix, He spent a few years building distributed services and search...

Read more

Harrington Joseph

Sr. Software Engineer @Netflix Data Platform Orchestration Team

Harrington Joseph is a Sr. Software Engineer at Netflix Data Platform Orchestration team. His work is focused on data orchestration and high-throughput event-driven architectures. Currently, Harrington is actively working on building the next generation of scheduling tools for Netflix Data...

Read more
Find Harrington Joseph at:

Zhamak Dehghani

Director of Emerging Technologies @thoughtworks & Creator of the Data Mesh concept

Zhamak Dehghani works with ThoughtWorks as the director of emerging technologies in North America, with a focus on distributed systems and big data architecture, with a deep passion for decentralized technology solutions - the foundations for democratization: data mesh, decentralized trust and...

Read more
Find Zhamak Dehghani at:

Jacek Laskowski

IT freelancer, Java Champion & Author of "The Internals Of"

Jacek Laskowski is an IT freelancer specializing in Apache Spark, Delta Lake and Apache Kafka (with brief forays into a wider data engineering space, e.g. Dask, ksqlDB and Trino, mostly during Warsaw Data Engineering meetups). He is the author of "The Internals Of" online books series...

Read more
Find Jacek Laskowski at:

Track Date

Thursday May 27 / 09:00AM EDT

Topics

Big Data

Share

Track Host

Fabiane Nardon

Data Scientist, Java Champion & CTO @tail_oficia

Fabiane is a computer scientist with many years of experience in large information systems with huge amounts of data. She was chief architect of the Sao Paulo Healthcare Information System, considered the largest JavaEE application in the world and winner of the 2005 Duke's Choice Award. She...

Read more
Find Fabiane Nardon at: