Mat Chabros

Do PySpark and Databricks depend on each other? Not necessarily! But here's why it makes totally sense to use them together... Looking for a great Databricks training? Then check out my Academy course Data Engineering on Databricks 👉 https://lnkd.in/egWigU9u #DataEngineer #DataEngineering #LearnDataEngineering #BigData #DataScience #databricks #apachespark

4

I asked Chat GPT what the best tools to learn in data in 2024 are, and this is what it said... 1. Apache Spark - because it's open source, efficient in data processing and supports various languages 2. SQL - because of how fundamental it is across data manipulation, querying, and database management 3. Python - because of it's many different use cases and libraries across data Do you agree with this top 3? The rest of this list was: Airflow, Snowflake, DBT, Google BigQuery, Kubernetes, MLflow, and data viz tools (non-specific) I'd probably reverse the order, and have SQL number one, and Python number two. I'm surprised it didn't tell me to learn Databricks though. 😂 I was also expecting to see ADF on the list. 🤔 What do you think? Is Chat GPT right? #data #engineering #chatgpt #recruitment #sql #python #spark #apache

13

🎉 29 Days of Airflow 2.9: Day 23 🎉 📈 Today's feature update in #ApacheAirflow 2.9 focuses on enhancing data visibility: adding datasets to the DAG graph. This improvement, delivered via pull request #37604, integrates datasets directly into the visual #DAG graph in the #Airflow UI. This allows users to see how datasets are connected to various tasks, providing a clearer understanding of how data flows through and impacts workflows. 🔍 Why is this significant? Incorporating datasets into the DAG graph helps users visualize and comprehend the dependencies and relationships between tasks and data more intuitively. This addition is crucial for troubleshooting and optimizing data pipelines, ensuring a more efficient and effective data handling process. 🔗 Learn more about the benefits of visualizing datasets in your DAG graphs: https://bit.ly/458xvqh #Airflow29 #DataVisualization #29daysofairflow29 #DataEngineering

19

You don't need statistical theory to design A/B tests.

4

😘 Why Data Engineers Can’t Afford to Wing It 💡 Article by: Tim Webster ➡ Link to Article: https://lnkd.in/efhvP8yf - - - - - - - - - - - - - - - - - - - - - - - - - - - 💡 Follow for more great content! ✅ ✍️ Want to contribute? Connect with us. 🔗 Let's create together! ✅ - - - - - - - - - - - - - - - - - - - - - - - - - - - #dataengineering #careergrowth #datascience #dataanalytics

4

Avoid using inferschema when reading large CSV files with PySpark. Tom Reid's article shows you how, read now! #DataScience #Python #PySpark #MachineLearning

14

1 komentarz

No more indexes for your task names with dynamic tasks in Airflow 👇 Why? 🤔 Before 2.9, tasks created dynamically had an index for their name. If you generate 50 tasks, you have tasks 1, 2, 3, 4, 5, etc. Very inconvenient to identufy your tasks. But this is OVER! With 2.9, you can set custom names to dynamically mapped tasks 🤩 How? Easy 1️⃣ Add the map_index_template parameter to your task 2️⃣ Set variable name with Jinja (here custom_name) 3️⃣ In the task, set the custom name to the variable by accessing the context That's it! No more tasks 1, 2, 3, 4, 5 😎 Enjoy ❤️ #airflow #apacheairflow #dataengineer #dataengineering

176

11 komentarzy