data movement

Everything We’ve Known About Data Movement Has Been Wrong

Data movement remains a perennial obstacle in systems design. Many talented architects and engineers spend significant amounts of time working on data movement, often in the form of batch Extract, Transform, and Load (ETL). In general, batch ETL is the process everyone loves to hate, or put another way, I’ve never met an engineer happy with their batch ETL setup. In this post, we’ll look at the shift from batch to real time, the new topologies required to keep up with data flows, and the...