IDEAS Amundsen Presentation

Editor's Notes

1. Name & Role working on an open-source data discovery tool at Lyft. It’s called “Amundsen” -- more on that name later. It leverages Neo4j, glad to share how we’ve been using Neo4j at Lyft to achieve goals of our product Amundsen.
2. On the agenda for this talk
3. Now onto challenges with data discovery
4. Effective data discovery is important because data is at the heart of every decision we make. It is the only way to make informed, objective decisions. Applies to many roles Data-driven decision making process Search & find data Understand the data Perform an analysis Share insights or make a decision
5. Now onto challenges with data discovery
6. To highlight some data discover pain points that occur without the proper tools, let’s walk through a hypothetical example
7. Your experience searching and finding data may involve doing all of the following 3 things.
8. Your experience understanding the data doesn’t get any easier. Each question leads to further questions - How was this data collected?
9. ⅓ of time on data discovery Difficult to find what exists, understand whether or not it’s what you are looking for, or trust that it is the source of truth for that information We can significantly increase productivity and impact if we can reduce this time...
10. We’ve talked about some pain points of data discovery and why it’s important, let’s talk about our solution -- Amundsen.
11. Disclaimer Representative data Amundsen circa March 2019 Our landing page is optimized for search Most common method of data discovery, presented with search bar & help text for some advanced search features We also want the landing page to be able to help users that don’t know what to search for. Created this concept of popular tables
12. Users presented with ranked search results Not like page-rank but based on relevance and popularity
13. Now onto challenges with data discovery
14. However graph databases are not common for many web applications, and so one might ask why choose a graph database.
15. Well if you remember the diagram of the data ecosystem at Lyft from the beginning of the talk, that can be modeled as a graph. This is a very powerful feature because the alternative to created these kinds of relationships with a RDBMS is joins A NoSQL database isn’t set up for this
16. As you may remember from the application walkthrough, Amundsen surfaces resource metadata and that is what we are storing in Neo4j
17. Let’s take a note of some of the features from the table detail page again and see how this is represented in Neo4j Walk through features What’s very beneficial about this is that when we have a new use case and a new piece of metadata to represent, we just have to create the new node and relationship.
18. Another key characteristic of our system is that neo4j is the source of truth for our editable metadata
19. This was actually not our original intent, we ran into a roadblock when we were first implementing the description editing feature. We originally had a setup like this
20. Then we realized we forgot to account for something. Tables can get rebuilt using the source code that generated the table and descriptions will be overwritten
21. The we thought about whether or not we could do this, update them both!
22. The answer was no. ...And that’s how Neo4j became the source of truth for editable metadata
23. Now onto challenges with data discovery
24. Now onto challenges with data discovery
25. Now onto challenges with data discovery
26. T