4. Data is used to make informed decisions
4
Analysts Data Scientists General
Managers
Engineers ExperimentersProduct
Managers
Data-driven decision making process:
1. Search & find data
2. Understand the data
3. Perform an analysis/visualization
4. Share insights and/or make a decision
Make data the heart of every decision
5. What is Data Discovery?
Consider a data-driven decision making process:
1. Search & find data
2. Understand the data
3. Perform an analysis/create a visualization
4. Share insights and/or make a decision
5
Data Discovery
7. • My first project is predict the attendance for IDEAS conference
• Goal: Help the office team make a decision on number of chairs to
provide?
• Idea: Let’s take a look into attendance from previous conferences… but
where do I look?
Hi! I’m a new Analyst!
7
8. • Ask a friend/manager/coworker
• Ask in a wider Slack channel
• Search in the Github repos
Step 1: Search & find data
8
We end up finding tables: hosted_events
that seems to be the right one
9. • You find several columns that might be what you're looking for:
‒ booked, registered, and attendance
• But you still have many questions such as:
‒ Does attendance include staff?
‒ What's the difference between booked and registered?
‒ How accurate are these figures?
Step 2: Understand the data
9
10. Step 2: Understand the data
● Look for further documentation on these columns
○ Where does this documentation live?
● Ask an expert who knows this table
○ Who is an expert?
● Run some queries to try to figure it out at the risk of being wrong
10
SELECT * FROM schema.host_events
LIMIT 100;
11. Nearly 1/3 of Data Scientist time is spent in Data
Discovery
11
• Data discovery is a problem
because of the lack of
understanding of what data
exists, where, who owns it, & how
to use it.
• Data Discovery provides little to
no intrinsic value
• Impactful work happens in
Analysis
13. What is Amundsen?
• Built at Lyft, official launch in late 2018
• Inspired by Google Search, Airbnb Data Portal, and
Apache Gobblin
• Named after Norwegian explorer Roald Amundsen
‒ Led the first expedition to the South Pole
‒ Led the first expedition through the Northwest Passage
13
35. Amundsen’s Impact at Lyft
• Deployed at Lyft for over 1 year
• Over 700 Weekly Active Users
• 90% penetration among Data Scientists
• Reduced mean time to discovery by 75%
• Also used by Data Eng, Software Eng, PMs, Ops, Marketing Managers,
and more
35
41. • github.com/lyft/amundsen
• 200 github stars, 10 companies contributing back
• Slack channel 250 people from 30 companies
• Presented at conferences in San Francisco, Barcelona, Vilnius, Moscow, LA,
NYC by Lyft employees and community
Amundsen is Open Source!
41
44. Alagappan Sethuraman | /in/alagappanut
Daniel Won | /in/danwon
Project Code @ github.com/lyft/amundsen
Icons under Creative Commons License from https://thenounproject.com/
44
Editor's Notes
Name & Role working on an open-source data discovery tool at Lyft. It’s called “Amundsen” -- more on that name later. It leverages Neo4j, glad to share how we’ve been using Neo4j at Lyft to achieve goals of our product Amundsen.
On the agenda for this talk
Now onto challenges with data discovery
Effective data discovery is important because data is at the heart of every decision we make. It is the only way to make informed, objective decisions. Applies to many roles Data-driven decision making process Search & find data Understand the data Perform an analysis Share insights or make a decision
Now onto challenges with data discovery
To highlight some data discover pain points that occur without the proper tools, let’s walk through a hypothetical example
Your experience searching and finding data may involve doing all of the following 3 things.
Your experience understanding the data doesn’t get any easier. Each question leads to further questions - How was this data collected?
⅓ of time on data discovery Difficult to find what exists, understand whether or not it’s what you are looking for, or trust that it is the source of truth for that information We can significantly increase productivity and impact if we can reduce this time...
We’ve talked about some pain points of data discovery and why it’s important, let’s talk about our solution -- Amundsen.
Disclaimer Representative data Amundsen circa March 2019 Our landing page is optimized for search Most common method of data discovery, presented with search bar & help text for some advanced search features We also want the landing page to be able to help users that don’t know what to search for. Created this concept of popular tables
Users presented with ranked search results Not like page-rank but based on relevance and popularity
Now onto challenges with data discovery
However graph databases are not common for many web applications, and so one might ask why choose a graph database.
Well if you remember the diagram of the data ecosystem at Lyft from the beginning of the talk, that can be modeled as a graph. This is a very powerful feature because the alternative to created these kinds of relationships with a RDBMS is joins A NoSQL database isn’t set up for this
As you may remember from the application walkthrough, Amundsen surfaces resource metadata and that is what we are storing in Neo4j
Let’s take a note of some of the features from the table detail page again and see how this is represented in Neo4j Walk through features What’s very beneficial about this is that when we have a new use case and a new piece of metadata to represent, we just have to create the new node and relationship.
Another key characteristic of our system is that neo4j is the source of truth for our editable metadata
This was actually not our original intent, we ran into a roadblock when we were first implementing the description editing feature. We originally had a setup like this
Then we realized we forgot to account for something. Tables can get rebuilt using the source code that generated the table and descriptions will be overwritten
The we thought about whether or not we could do this, update them both!
The answer was no. ...And that’s how Neo4j became the source of truth for editable metadata