Introduction to Airflow

From the Airflow documentation:

Apache Airflow™ is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Airflow’s extensible Python framework enables you to build workflows connecting with virtually any technology. A web interface helps manage the state of your workflows. Airflow is deployable in many ways, varying from a single process on your laptop to a distributed setup to support even the biggest workflows.

Airflow is at the core of IQVIA NLP Data Factory. Airflow provides reliable and scalable orchestration capabilities, and we’ve built an Airflow provider (the IQVIA NLP Airflow provider) for incorporating I2E into your Airflow workflows.

The Airflow documentation is well-written and extensive, and should be the first place to check for any Airflow questions. That said, we’ve documented some of “must know” Airflow concepts here for your convenience:

Concept	Meaning
DAG	DAGs (Directed Acyclic Graphs) are “workflows” or “pipelines” in Airflow. DAGs are comprised of tasks, with dependencies and relationships between tasks to say how they should run. DAGs are defined as Python files, such as those in our Example DAGs section.
DAG Run	DAG runs represent an instance of a DAG. Whenever a DAG is executed, a DAG run is created and the tasks in the DAG are executed. Each DAG run has a status, such as success, failed, queued, or skipped.
Task	Tasks are the basic unit of execution in Airflow. Each task will typically do one thing, such as processing some data, moving a file, or sending a notification.
TaskFlow	The TaskFlow API is an approach to writing DAGs that uses the @task decorator to produce cleaner DAGs. Our Example DAGs often use the TaskFlow API.
Provider	Providers are packages that extend Airflow’s capabilities, often to allow Airflow to communicate with some external system. For example, the IQVIA NLP Airflow provider allows Airflow to communicate with I2E. Providers can contain operators, hooks, sensors, connection types, and more.
Connection	Connections are objects in Airflow for storing credentials and other information needed for connecting to external services. This will typically include usernames, passwords, and URLs.
Operator	Operators are pre-defined templates for a task, making it easy to include commonly-used tasks in your DAGs. For example, the BashOperator can be used in a DAG to run a Bash command as a task. Operators that transfer data from one service to another are often called “Transfer Operators”.
Sensor	Sensors are a special type of operator that wait for something to occur. For example, a sensor might wait for a file to be present, or for an external job to complete.
Hook	Hooks are the high-level interface to external systems, making it easy to communicate with external systems without needing to understand their API or use special libraries. Hooks integrate with connections to gather credentials. Operators typically make use of hooks behind the scenes.
API server	The Airflow 3 component responsible for the web interface and REST API.
Worker	The Airflow component responsible for executing tasks. Production-level Airflow setups will typically have multiple workers.