Metaplane

Yesterday, I asked the community, what’s worse? When a stakeholder: 1. Trusts data but it turns out to be wrong 2. Spots a data issue before the data team In both cases, trust is eroded. All because the condition and quality of data aren’t obvious. Until now. 🔎 Introducing our 2nd Summer of Data Observability drop: the Metaplane Chrome Extension. The Metaplane Chrome Extension shows all users a summary of the metrics you monitor in Metaplane in a convenient side panel. Now, everyone has the context they need in a simplified, actionable view – when and where they need it. That means… • Stakeholders know the quality of data as they’re using it • They don’t have to ask the data team if that data can be trusted • They know your team is on top of it (and have quality checks already in place) Read more about the feature in the blog post in the comments! If you use Metaplane already, you can get it installed and running in <5 minutes.

Data movement needs to be scalable, extensible, and reliable. As 6,000 companies move data daily from our 300 data sources, we have also invested heavily in making Airbyte observable through rich metadata available via an API. Today we’re happy to announce that Metaplane, the leading data observability platform used by our own data team, launched an Airbyte integration on top of that API. Data teams can use Airbyte and Metaplane to proactively detect issues like long-running syncs, late-arriving data, and syncs that moved an incorrect amount of data from source to destination. These monitors use machine learning trained on historical metadata, so no configuration is needed. You can also see which Airbyte streams load tables in your warehouse, so you can trace end-to-end lineage from source to destination. Read more here: https://lnkd.in/geAtE7Tw #dataengineering #dataquality #dataobservability #dataintegration

Data teams should be the first to know about data issues. That’s why I’m particularly excited to announce the first 𝑆𝑢𝑚𝑚𝑒𝑟 𝑜𝑓 𝐷𝑎𝑡𝑎 𝑂𝑏𝑠𝑒𝑟𝑣𝑎𝑏𝑖𝑙𝑖𝑡𝑦 drop: Metaplane now integrates with Airbyte. Airbyte is the leading open-source data movement platform, helping 6K companies move data from 300 data sources. Now, both Airbyte Cloud and self-managed users can proactively detect issues like: 𝟏. 𝐋𝐨𝐧𝐠-𝐫𝐮𝐧𝐧𝐢𝐧𝐠 𝐬𝐲𝐧𝐜𝐬 (duration) 𝟐. 𝐋𝐚𝐭𝐞-𝐚𝐫𝐫𝐢𝐯𝐢𝐧𝐠 𝐝𝐚𝐭𝐚 (last succeeded timestamp) 𝟑. 𝐒𝐲𝐧𝐜𝐬 𝐭𝐡𝐚𝐭 𝐝𝐢𝐝𝐧’𝐭 𝐦𝐨𝐯𝐞 𝐭𝐡𝐞 𝐫𝐢𝐠𝐡𝐭 𝐚𝐦𝐨𝐮𝐧𝐭 𝐨𝐟 𝐝𝐚𝐭𝐚 (rows and bytes synced) Our Airbyte integration monitors all of these metrics out-of-the-box with machine learning, so no configuration is needed. We also extract lineage so you can see the Airbyte streams that feed into warehouse tables and trace dependencies end-to-end from ingestion to consumption. Read more about the integration in the blog post in the comments :). And if you use Metaplane already, you can connect in <5 minutes. #dataengineering #dataquality #dataintegration

Imagine you’re relaxing on the beach. The sun is shining, waves are crashing ... life is good. Then suddenly, you get the dreaded Slack message: “𝘛𝘩𝘪𝘴 𝘥𝘢𝘵𝘢 𝘭𝘰𝘰𝘬𝘴 𝘸𝘳𝘰𝘯𝘨.” We don’t want that for you. That’s why we’ve queued up 15 feature releases to help you catch more issues faster and earlier, so you can actually enjoy your vacation. It’s officially the Summer of Data Observability ☀️. But before we press play, can anyone guess what’s Up Next? The cover art is a hint, as was my post yesterday, but here’s another: 🐙. Stay posted to catch the first drop tomorrow! #dataengineering #dataquality #dataobservability  #analytics

Data quality testing without monitoring pipelines is like running a kitchen without inspecting ingredients. No matter how skilled the chef, bad ingredients ruin the dish. #dataengineering #dataquality #analytics

Is AI the next big thing or a passing trend? What is Polaris? Why are street ads the new marketing battleground? Where can a guy find some coffee? If you didn't make it to Snowflake Summit last week, come hang out with me and Shinji Kim this Friday 8am PT / 11am ET as we recap the biggest data event of the year. And if you did make it, come by too! There were so many talks and events that we all got a different slice of the experience, and I'd love to hear what you took away. Join the LinkedIn Live here: https://lnkd.in/e6-q3mek #dataengineering #snowflakesummit #snowflake #analytics

Hey, Boston friends — got plans on Thursday? Snowflake and Sigma Computing are hosting a fireside chat at Bully Boy Distillers from 5-8 PM, and they’ve invited me to join the panel. If you’ve never been to Bully Boy, it’s awesome. They have a fantastic tasting room and their drinks/cocktails are excellent. Really excited about the panel with Jakov Kucan, CTO at Manifold, and Craig S. Stelmach, Sr. Vice President Of Business Development and Sales at Weather Source, to talk about everyone’s favorite topic: AI. The best part? This is a free community event with cocktails and appetizers, so it’s the perfect opportunity to connect with data folks around Boston! Space is filling up fast though, so RSVP at the link in the comments. 👇 #dataengineering #artificialintelligence #data #snowflake

There is a duality between data and code: 1. Business-facing data is the result of continually applying transformations on source data (functions) 2. The validity of those transformations is determined by whether data meets business needs (state) Because of that duality, observing only code *or* data is insufficient. Code is incorrect when the data does not meet spec. When data is incorrect, we have to look at the code. But there remains a gap between these two worlds, like two pieces of a puzzle that don't quite fit together. At the Summit Platform Keynote, Snowflake announced a key bridge between data and code: Snowflake Trail. The Snowflake telemetry team (which includes folks who founded OpenTelemetry!) is bringing the technology and practices of distributed tracing to the data world. Two weeks ago, I shared that Snowflake invested in Metaplane. Our goal is to make observability more powerful, ubiquitous, native to Snowflake — like Airpods to an iPhone. A key part of that vision is extending data observability with metrics, traces, and logs from code execution: Data-Code Observability. I'm excited that Metaplane is named alongside companies we admire like Datadog and Grafana to make that a reality. #snowflakesummit #dataengineering #dataquality #dataobservability

Everyone needs a little help planning out their week at Snowflake Summit. Sessions are always posted early, but what do you do to unwind at the end of every day? Speaking from experience, planning out your post-conference schedule can sometimes be as difficult as planning out your conference session schedule itself. Last year Ethan Aaron and I put together a list of events, and we’re doing it again after many folks asked! Check the comments below for a Google Sheet of 20 events, along with addresses and event sign-up links, to make things a bit easier for everyone. Let us know if we missed any. And don’t forget to share it with your data friends and coworkers! #dataengineering #dataquality #snowflakesummit #snowflake #data

The countdown to Snowflake Summit 2024 is on. Know your plans yet? 🗓️ This year is special for us. Yesterday, we announced that Snowflake is investing in Metaplane. So what better place to celebrate than Summit? Not only will we have a booth (#1116), but we’ll have even more places than ever to meet the Metaplane team to talk about how our platform helps you trust the data that powers your business. 𝐌𝐨𝐧𝐝𝐚𝐲, 𝐉𝐮𝐧𝐞 3𝐫𝐝 - Ramp x Metaplane theater session! I’ll be speaking with Ian Macomber, Head of Data at Ramp, about how data teams can power customer-facing AI products. - Low-Key Data Happy Hour with Portable, Seattle Data Guy, RudderStack, Keboola, Zenlytic, and DataGalaxy. I highly recommend signing up for this one — it will be low-key but epic. With 550 data folks in one place, you can find out what else everyone is up to the rest of the week. - Two happy hours in one night? Let’s do it. Sigma Swing Happy Hour with Sigma Computing and Aimpoint Digital. We'll be celebrating their Series D by sipping cocktails at an underground speakeasy like it’s the 1920s. 𝐓𝐮𝐞𝐬𝐝𝐚𝐲, 𝐉𝐮𝐧𝐞 4𝐭𝐡 - Wherever we go, you know a data dinner follows. We’re almost at capacity and some of my favorite data folks are coming through. Come meet experienced data engineers and leaders, learn something new, and have a fun night out — with no vendor selling allowed. 𝐖𝐞𝐝𝐧𝐞𝐬𝐝𝐚𝐲, 𝐉𝐮𝐧𝐞 5𝐭𝐡 - The Roaring Data After Party with Hightouch, Datafold, Eppo, Heap | by Contentsquare, Hex, and Media.Monks. We’re headed back to our favorite underground speakeasy for more cocktails, noodles, and live performances. Celebrate the last official night of Snowflake Summit 2024 like it’s the prohibition era 😆 Find more details and register for events at the links in the comments. #dataobservability #dataquality #snowflakesummit #data #dataengineering