Monte Carlo

Is your organization preparing to dedicate a tremendous amount of time and resources towards a data quality initiative.... that's doomed to fail? 😳 In order for data teams to maximize their investments in their modern data and AI platforms, both data producers and consumers must fully adopt and trust the data being provided. If that's not the case, then your data quality initiative might be at risk of failure before it's even begun. In our latest article, Monte Carlo CEO & Co-founder Barr Moses shares best practices for successful data quality initiatives, including these 4 key lessons for building data quality scorecards: ✅ Know what data matters ✅ Measure the machine ✅ Get your carrots and sticks right ✅ Automate evaluation and discovery Check out the full article: https://lnkd.in/gMWPaUJg #dataquality #dataobservability #datagovernance #dataengineering

It’s no revelation that incentives and KPIs drive good behavior. Sales compensation plans are scrutinized so closely that they often rise to the topic of board meetings. What if we gave the same attention to data quality scorecards? In the wake of Citigroup’s landmark data quality fine, it’s easy to imagine how a concern for data health benchmarks could have prevented the sting of regulatory intervention. But that’s then and this is now. The only question now is how do you avoid the same fate? Even in their heyday, traditional data quality scorecards from the Hadoop era were rarely wildly successful. I know this because prior to starting Monte Carlo, I spent years as an operations VP trying to create data quality standards that drove trust and adoption. Whether it’s a lack of funding or lack of stakeholder buy-in or cultural adoption, most data quality initiatives fail before they even get off the ground. As I said last week, a successful data quality program is a mix of three things: cross-functional buy-in, process, and action.And if any one of those elements is missing, you might find yourself next in line for regulatory review. Here are 4 key lessons for building data quality scorecards that I’ve seen to be the difference between critical data quality success—and your latest initiative pronounced dead on arrival: 1. Know what data matters—the best only way to determine what matters is to talk to the business. So get close to the business early and often to understand what matters to your stakeholders first. 2. Measure the machine—this means measuring components in the production and delivery of data that generally result in high quality. This often includes the 6 dimensions of data quality (validity, completeness, consistency, timeliness, uniqueness, accuracy), as well as things like usability, documentation, lineage, usage, system reliability, schema, and average time to fix. 3. Gather your carrots and sticks—the best approach I’ve seen here is to have a minimum set of requirements for data to be on-boarded onto the platform (stick) and a much more stringent set of requirements to be certified at each level (carrot). 4. Automate evaluation and discovery—Almost nothing in data management is successful without some degree of automation and the ability to self-service. The most common ways I’ve seen this done are with data observability and quality solutions, and data catalogs. Check out my full breakdown via link in the comments for more detail and real world examples.

Data Quality Day is coming up FAST! ⚡ Join us July 30th for Data Quality Day: Scaling Data Trust Across Business Domains. 🔥 You'll hear leading strategies and insights from: ⚡ Zach Wilson, Founder at DataExpert.io ⚡ Seth Y., Global Field CTO Snowflake ⚡ Zachary Lancaster, Data Engineering Manager at Warner Bros. Discovery ⚡ Bronte Baer, Data Platform & Analytics Manager at Earnest ⚡ Siva Veera, Data Engineering Manager at Riot Games ⚡ Barr Moses, CEO & Co-Founder, Monte Carlo Register here: https://lnkd.in/eceYmVsT #dataqualityday #dataengineering #dataanalytics

Bad data isn't just a headache - it's a huge financial risk. As data powers more of the world’s mission critical services—and the data and systems surrounding it become more complex in the process—data quality becomes a non-negotiable. Note: I didn’t say “nice to have.” In 2024 data quality isn’t open for discussion—it’s a clear and present risk and it needs our attention. Citigroup learned this lesson last week when regulators presented the company with a $136M fine for failure to make sufficient progress on a critical data quality initiative. And that’s before you consider the impact to share price. So, what’s the solution? A strong data quality management program is a mix of process, technology, and action. Airbnb’s Clark Wright recently published an article discussing how his team leverages data quality scores to validate their most critical assets. As you read the article (link in the comments), you’ll notice three key things their team did to make that project successful: - They began with the customer in mind - The brought stakeholders into the conversation - And then they made a plan and stuck to it. Delivering fast value for stakeholders is key to building institutional support. So, focus on solving for your most critical data assets first, prove out the value, then scale, scale, scale. Whether we’re talking about data quality scores or rolling out company-wide initiatives, delivering value always starts with solving the right problems. By sticking close to the business and understanding our stakeholders, we can mitigate these risks and drive the adoption of trusted data in the process. https://lnkd.in/gU6SY2bu

We're excited to announce the release of our Validation Monitors! We built no-code Validation Monitors to empower all members of the data team to set deep data monitors, regardless of SQL proficiency. Validation Monitors solve the tedious and time-intensive task of manually crafting SQL statements to validate individual records, helping to democratize data quality across the enterprise. Learn more here: https://lnkd.in/ewVQrq-w #dataquality #datavalidation #SQL #dataengineering #dataobservability

We hear a lot of talk about GenAI chatbots. But what if we could leverage AI to improve the performance of real human CS teams? We spoke with Killian Farrell, Principal Data Scientist at Assurance IQ, about how his team used LLMs to score customer conversations to develop their sales and customer support teams – and how data quality remains fundamental to the performance of their GenAI pipelines. Check out the full story: https://lnkd.in/e2y6uGH2 #GenAI #AI #LLM #dataobservability #dataquality #dataengineering

Bad data affects us all. But when your data is directly correlated to your bottom line, that data downtime takes on a whole new dimension. Financial services companies are becoming increasingly reliant on the accuracy of first-party data to power top-line revenue drivers. And PrimaryBid is just one example of that. As a platform provider that helps companies connect retail investors to capital market transactions like IPOs, PrimaryBid requires an incredible degree of data volume and accuracy to protect the interests of its stakeholders. “Building and maintaining predictive models requires a steady stream of market stock price data,” said Andy Turner, Director Data & AI, “The source of that data needs to be absolutely 100% robust, or the predictions coming out of the models that we’re serving in production will fail.” In preparation for its global expansion, the PrimaryBid team decided to rebuild their data stack from the ground up in order to massively increase the scale of their operations—and that increased scale meant the risk and impact of bad data was about to get even greater. The Monte Carlo team sat down with Andy and three other members of the PrimaryBid team: Ian Harris, Director of Data Engineering, Jonathan Dungay, Analytics & BI Lead, and Rick Wang, Staff Data Engineer — to find out how they were preparing for data quality at scale—and how data observability was helping their team set a new gold standard for financial data reliability. Check out the link in the description for the full story.

We're thrilled to share that Monte Carlo was included in Database Trends and Applications' (DBTA) list of Awesome Companies in AI 2024! We're excited to continue to work to enable enterprises to build trustworthy AI with reliable, high-quality data. Learn more here: https://lnkd.in/ewqD-NNG Congratulations to all included! Check out the full list here: https://lnkd.in/dkjnfzm8

Great session from Matt Luizzi, Director of Business Analytics at WHOOP and Shane Murray, Field CTO at Monte Carlo yesterday at CDOIQ in Boston! 🔥 Learn more about building trustworthy GenAI pipelines with data observability here: https://lnkd.in/ewqD-NNG #CDOIQ2024 #dataobservability #GenAI #dataquality #datatrust

Today's the day! If you're at CDOIQ in Boston today, be sure to tune into Matt Luizzi, Director of Business Analytics at WHOOP and Shane Murray, Field CTO at Monte Carlo's talk on what it takes to successfully implement an LLM, happening at 4:30pm EDT. The'll be discussing how the data team at WHOOP has built and launched an internal GenAI chatbot to surface valuable data insights, enabling more reliable, data-driven decision-making and innovation. If you can't attend in person but you're interested in tuning into the livestream, DM us and we'll send you a code to register! #CDOIQ2024 #dataobservability #GenAI #dataquality