DataExpert.io’s Post

🔔 𝐌𝐲 𝐟𝐢𝐫𝐬𝐭 𝐃𝐚𝐭𝐚 𝐦𝐞𝐝𝐢𝐮𝐦 𝐚𝐫𝐭𝐢𝐜𝐥𝐞 along with 𝐊𝐚𝐠𝐠𝐥𝐞 𝐝𝐚𝐭𝐚𝐬𝐞𝐭 𝐚𝐧𝐝 a complete 𝐄𝐧𝐝-𝐭𝐨-𝐄𝐧𝐝 𝐃𝐚𝐭𝐚 𝐏𝐢𝐩𝐞𝐥𝐢𝐧𝐞 𝐜𝐚𝐩𝐬𝐭𝐨𝐧𝐞 𝐩𝐫𝐨𝐣𝐞𝐜𝐭!! 🔔 It all began with the 6-week live DataExpert.io bootcamp by Zach Wilson this June. Needless to say who he is—one of the most experienced and talented data experts in the industry. During the bootcamp assignments and lecture discussions in the Discord channel, I happened to meet my amazing project partner, Meeta Pandit. Full of ideas, we structured our capstone and combined our expertise and learnings to get the best out of us. Despite being in slightly different time zones, we both beautifully pulled it together. For our capstone project ( https://lnkd.in/dfgRHJaB ), we picked up Citibike data in both real-time and historical formats. Part 1 - Medium Article - https://lnkd.in/dj4cevHi :  Included real-time data ingestion using 𝐊𝐚𝐟𝐤𝐚 clusters followed by the 𝐃𝐚𝐭𝐚𝐛𝐫𝐢𝐜𝐤𝐬 data transformations using Delta tables and 𝐏𝐲𝐒𝐩𝐚𝐫𝐤 for data processing. Data validations are the most overlooked part of a project, which is why we incorporated it at highest priority using the failure recovery, idempotency and data contracts.  Part 2 - Medium Article - https://lnkd.in/de6tpfkN : Involved setting up roles and users setup with permissions and data governance policies based on roles in 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞, followed by DBT setup. We also utilized 𝐒𝐭𝐫𝐞𝐚𝐦𝐥𝐢𝐭 using the data in Snowflake to further analyse the data. Using 𝐃𝐁𝐓, the project showcased numerous transformations using macros and tests making sure clean and structured flow of data for further analytics. Finally, orchestrating DBT using 𝐃𝐚𝐠𝐬𝐭𝐞𝐫 for better dependency visualizations and error handling. But wait, it’s not over yet. We also incorporated the clean data on 𝐊𝐚𝐠𝐠𝐥𝐞 (https://lnkd.in/dhzK-tZ3) so that data enthusiasts like yourself can explore it more and create your own mini projects. This data mainly contains station details in the Citibike_dim_station table and fct_citibike_data, which not only includes the trip details but also the total cost of the trip based on the duration and standard price set—this data is exclusively extracted from the DBT model created for the trip cost calculation. Kudos to our hard work and collective efforts in making the most out of this bootcamp. A big thank you to the entire DataExpert.io team Zach Wilson Mitali Gupta JulieAnn Scherer! #dataengineering #datapipeline #datajobs #dataexpert