Monte Carlo reposted this
Great write-up from analyst Matt Aslett at Ventana about the past, present, and future of Monte Carlo and the data observability category. I love this stat Matt cites because I think it really speaks to the endemic nature of the data pipeline issues we’re working so hard to solve. “Data pipeline delays are especially critical given that almost two-thirds (64%) of participants in Ventana Research’s Analytics and Data Benchmark Research cite reviewing data for quality and consistency issues as the most time-consuming task in analyzing data.” While traditional data quality are an obvious MVP solution to this problem, they just as obviously fall short upon any closer examination. While solutions like traditional data quality monitoring and data testing can identify problems in the data itself (if you already know what you’re looking for), they can’t tell you what caused the problem, where it happened, or if it matters in the first place. Which means that—while you may know about the issue in question—you’re still no closer to solving it. I think Matt also summarizes this challenge (and the data observability response) well when he says: “While data quality software is designed to help users identify and resolve problems related to the validity of the data itself, data observability software is designed to automate the detection and identification of the causes of data quality problems. As such, data observability can potentially enable users to prevent data quality issues before they occur.” Of course, no data quality solution can prevent problems on its own. That information needs to be acted upon in one way or another—whether that’s an established incident management workflow, precise triaging, circuit-breakers to stop bad data in production, or some combination of the three—but unlike traditional data quality approaches, data observability gives you the ability to measure pipeline health over time. That means that you can identify and resolve chronic data health concerns at the source and even the cultural level. And THAT is a powerful resource for data teams.