Session + Live Q&A
Taming the Data Mess, How Not to Be Overwhelmed by the Data Landscape
The data engineering field has evolved at a tremendous pace in the last decade, new systems that enable the processing of huge amounts of data generated enormous opportunities, as well as challenges for software practitioners. All these new tools and methodologies created a new set of requirements for data engineers who tend to live in a constant state of catching up.
In this talk, we review the current data landscape and discuss both technical and organizational ideas to avoid being overwhelmed by the current lack of consolidation of the data engineering world. We will discuss ideas from adopting open source APIs and open standards to more recent data management methodologies around operations, data sharing, and data products that enable us to create and maintain resilient and reliable data architectures.
Speaker
Ismaël Mejía
Senior Cloud Advocate @Microsoft
Ismaël Mejía is a Senior Cloud Advocate at Microsoft working on the Azure Data and AI team. He has more than a decade of experience architecting systems for startups and financial companies. He has been recently focused on distributed data frameworks, he is an active open-source contributor of...
Read moreFind Ismaël Mejía at:
From the same track
Data Versioning at Scale: Chaos and Chaos Management
Wednesday May 18 / 10:10AM EDT
Version control is fundamental when managing code, but what about data? Our data changes over time, first since it accumulates, we have new data points for new points in time. But this is not the only reason. We also have additional data added to past time, since we were able to get additional...
Dr. Einat Orr
Co-creator of @lakeFS, Co-founder & CEO of Treeverse
Modern Data Pipelines in AdTech—Life in the Trenches
Wednesday May 18 / 11:20AM EDT
There are various tasks that the modern data pipelines approach helps us solve in different domains, including advertising. Modern data pipelines allow us to process data in a more efficient manner with a diverse set of data transformation tools for both batch and streaming data processing....
Roksolana Diachuk
Big Data Engineer @Captify
Orchestrating Hybrid Workflows with Apache Airflow
Wednesday May 18 / 12:30PM EDT
According to analysts, 87 percent of enterprises have already adopted hybrid cloud strategies. Customers have many reasons why they need to support hybrid environments, from maximizing the value from heritage systems to meeting local compliance and data processing regulations. As they build...
Ricardo Sueiras
Principal Advocate in Open Source @AWS