Data Versioning at Scale: Chaos and Chaos Management

Version control is fundamental when managing code, but what about data? Our data changes over time, first since it accumulates, we have new data points for new points in time. But this is not the only reason. We also have additional data added to past time, since we were able to get additional data sources, or changed past data in light of new information that was late to arrive.

Since our data is mutable, version control of the data will allow us to ensure we can reproduce a set of results, provide us with the lineage between the input and output data sets of a process or a model, allow us to experiment, provide the relevant information for auditing, and assist us in production management. In this talk we will go over several technologies that version large data sets. We will understand the use cases they support and look under the hood at the technology developed to best support those use cases.