Airflow Summit’s Post

Explore the journey of transitioning from an Airflow user to an active project contributor. This talk will cover essential insights, steps, resources, and best practices to effectively engage with the Airflow community and make meaningful contributions around open-source projects. Register at: https://airflowsummit.org/ Briana Okyere Amogh Desai Ryan H. Srabasti Banerjee #AirflowSummit #Airflow #ApacheAirflow #AirflowSummit2024 #dataorchestration #batchprocessing #dataprocessing #datapipelines #data #datasience #opensource Apache Airflow

🎉 29 Days of Airflow 2.9: Day 7 🎉 📊 Today’s feature update in Apache Airflow 2.9 is the addition of dataset conditions to the next run datasets modal. With this update from pull request #38123, #Airflow introduces the capability to specify conditions for datasets directly in the next run modal. This allows for more precise control over how datasets trigger subsequent task runs, enhancing the clarity and customization of workflow triggers. 🌟 Why is this important? Adding conditions to datasets directly in the modal streamlines the setup process, making it easier and more intuitive to manage complex dependencies. This ensures that tasks run only when the necessary data conditions are met, optimizing workflow execution and resource usage. 🔗 Explore the new dataset conditions feature: https://bit.ly/3wpbGFH #29daysofairflow29 #Airflow29 #DataDrivenWorkflow #EfficientDataHandling #DataEngineering

So excited to share my quick glimpse intro to very interesting tool Apache Airflow which provides a huge flexibility of scheduling tasks and control them reliably. #airflow #dataengineering #dataanalytics #data #schduling #opensource

New in Apache Airflow 2.9 the ability to set on_skipped_callback to execute a function when task is skipped. This is a feature I worked on in the project and I am very happy to contribute back to the community. #apacheairflow #airflow #dataengineer #dataengineering 

In this talk Roberto Santamaria & Xiaodong DENG from Apple will explore how adding custom dependency checks into Airflow’s scheduling system can elevate Airflow’s performance. Register at: https://airflowsummit.org/ #AirflowSummit #Airflow #ApacheAirflow #AirflowSummit2024 #dataorchestration #batchprocessing #dataprocessing #datapipelines #data #datasience #opensource Apache Airflow

🚀 Join us for Airflow Summit 2024 to learn how to optimize Apache Airflow performance for complex workflows! Discover with Pankaj Singh & Utkarsh Sharma, strategies to enhance execution speed, reduce resource use, and improve efficiency. Perfect for both seasoned users and newcomers. Don't miss it! https://airflowsummit.org #AirflowSummit #dataengineering #apacheairflow #dataworkflows #airflowdeployments #dataorchestration #batchprocessing #dataprocessing #datapipelines #data #datasience #opensource

Get ready for #Airflow 2.9! The latest version of Apache Airflow brings enhanced data-aware scheduling, dynamic task mapping, broader object storage options, and more! Join our #webinar featuring experts Kenten Danas, Tamara Janina Fingerlin, and Marc Lamberti, as they unpack everything #Airflow29 has to offer. Save your virtual seat here 👉 https://bit.ly/3VIaKGP #datapipelines #dataorchestration #dataengineering #AirflowUpgrade

Soon we are releasing new feature version of Apache Airflow ! There are so many improvements, fixes and new features! I am very excited about 2 of them: 1. Added ability to implement custom Weight Rules - This is a very cool feature that will allow you to define your own strategy for task priority. For example: decrease task priority based on retry number. 2. Added DatasetTimetable integration - It allows making scheduling decisions based on both dataset events and time. For example: schedule run every 1 hour and also when dataset is updated. #apacheairflow #airflow #dataengineering #etl

6 DE tools, I love playing with 👇 Apache Airflow for orchestration (OSS) Soda for data quality (OSS) Airbyte for ingestion (OSS) dbt Labs for transformations (OSS) DuckDB for analytics (OSS) Malloy to query data 😍 What are yours? Anything I should try? #dataengineering #dataengineer #airflow #apacheairflow

Discover the benefits of ELT processes with Apache Airflow! Learn how the new approach optimizes data processing and which tools, such as Singer, Meltano, DLT and Airbyte, can be used in combination with Airflow. Just read our new blog post: Apache Airflow ELT Process for Data Orchestration https://bit.ly/4bYqAmy What data processing challenges do you face? Let us know in the comments. #ELT #ApacheAirflow #DataProcessing #NextLytics