Active Hackathon

Apache Airflow 2.0 Released- New Scheduler, Full Rest API & More

One of the trending open-source workflow management systems among developers, Apache Airflow is a platform to programmatically author, schedule and monitor workflows. Recently, the team at Airflow unveiled the new version of this platform, which is Apache Airflow 2.0. Last year, the Apache Software Foundation (ASF) announced Apache Airflow as the Top-Level Project (TLP).

With substantial changes than the former version, the 2.0 release of the Airflow came with significant upgrade. In order to start using Airflow 2.0, one must need to follow some prerequisites, such as if users are using Python 2.7, they need to migrate to Python 3.6+. Also, when the user is on the latest Airflow 1.10 release, they can use the airflow upgrade-check command to see if they can migrate to the new Airflow version. 


Sign up for your weekly dose of what's up in emerging technology.

Before diving into the significant upgrades, let us take you through the basics of AirFlow first. 

Behind the Basics

Created by Airbnb, Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. The platform is a flexible, scalable workflow automation and scheduling system for authoring and managing Big Data processing pipelines of hundreds of petabytes.

It is a workflow engine that performs several tasks, such as managing scheduling and running jobs and data pipelines, managing the allocation of scarce resources, provides mechanisms for tracking the state of jobs and recovering from failure and more. 

There are four key components of Airflow, which are:

  • Web server: It is the GUI, which remains under the hood of a Flask app where you can track the status of your jobs and read logs from a remote file store
  • Scheduler: Scheduler is a multithreaded Python process and is responsible for scheduling jobs. It uses the DAGb object to decide what tasks need to be run, when and where.
  • Executor: Executor is the mechanism that gets the tasks done.
  • Metadata database: Metadata database powers how the other components interact, stores the Airflow states and all the processes read and write from here.

Some of the intuitive features of AIrflow are mentioned below:

  • One of the main advantages of using a workflow system like Airflow is that all is code, which makes the workflows maintainable, versionable, testable, and collaborative.
  • Airflow is versatile in nature and can be used across various domains, including growth analytics, data warehousing, engagement analytics, anomaly detection, email targeting, among others. 
  • Airflow has built-in support using schedulers.
  • Through smart scheduling, database and dependency management, error handling and logging, Airflow automates resource management, from single servers to large-scale clusters.
  • Written in Python, the project is highly extensible and able to run tasks written in other languages, allowing integration with commonly used architectures and projects such as AWS S3, Docker, Apache Hadoop HDFS, Apache Hive, Kubernetes, MySQL, Postgres, among others.

Coming to the major release of version 2.0, the developers announced that Airflow 2.0 is in the alpha testing stage and is scheduled to be generally available in December of 2020. According to its developers, Airflow 2.0 includes hundreds of features and bug fixes, both large and small, where most of the significant updates were influenced by the feedback from Airflow’s 2019 Community Survey.

Some of the significant updates are mentioned below:

A New Scheduler: Low-Latency + High-Availability  

According to the developers, Scheduler Performance was the most asked for improvement in the Community Survey. With version 2.0, the team introduced a new, refactored Scheduler. The most impactful Airflow 2.0 change in this area is support for running multiple schedulers concurrently in an active/active model. The new functionality includes horizontal scalability, lowered task latency, zero recovery time and easier maintenance.


Airflow 2.0 introduces a new, comprehensive REST API that sets a strong foundation for a new Airflow UI and CLI in the future. This new API includes authorisation capabilities, makes easy access by third-parties and more.

Know more here.

Smart Sensors

Sensors are a special type of Airflow Operator whose purpose is to wait on a particular trigger. Version 2.0 introduced Smart Sensor and is capable of checking the status of a batch of Sensor tasks, storing sensor status information in Airflow’s Metadata DB and other such.

TaskFlow API

Airflow 2.0 introduced the TaskFlow API and Task Decorator to mitigate the issue of scheduling and running the idempotent tasks. The TaskFlow API implemented in 2.0 makes DAGs significantly easier to write by abstracting the task and dependency management layer from users. The functionalities of TaskFlow API include support for Custom XCom backends, automatically creates PythonOperator tasks from Python functions and handles variable passing and more.

More Great AIM Stories

Ambika Choudhury
A Technical Journalist who loves writing about Machine Learning and Artificial Intelligence. A lover of music, writing and learning something out of the box.

Our Upcoming Events

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
21st Apr, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM

Ouch, Cognizant

The company has reduced its full-year 2022 revenue growth guidance to 8.5% – 9.5% in constant currency from the 9-11% in the previous quarter

The curious case of Google Cloud revenue

Porat had earlier said that Google Cloud was putting in money to make more money, but even with the bucket-loads of money that it was making, profitability was still elusive.