Data Engineering |

Mastering Task Dependencies in Apache Airflow

Fri, 11 Oct 2024 00:00:00 +0000

Welcome to Day 5! Today we explore defining task dependencies for complex, real-world data workflows.

Basic Dependencies

Linear Dependencies

extract >> transform >> load

Fan-out/Fan-in Patterns

download_launches >> [get_pictures, download_metadata]
[get_pictures, download_metadata] >> notify

Branching

Use BranchPythonOperator for conditional logic:

from airflow.operators.python import BranchPythonOperator

def choose_path():
 return "task_A" if condition else "task_B"

branch = BranchPythonOperator(
 task_id='branch_task',
 python_callable=choose_path
)

Trigger Rules

Trigger Rule	Behavior
`all_success`	(default) All parent tasks completed successfully
`all_failed`	All parent tasks failed
`all_done`	All parents done, regardless of state
`one_failed`	At least one parent failed
`one_success`	At least one parent succeeded
`none_failed`	No parents failed (succeeded or skipped)

task_A.xcom_push(key='data', value=my_data)
task_B.xcom_pull(task_ids='task_A', key='data')

Note: XComs are stored in the metadata database - use for small data only, not large datasets.

Taskflow API

Simplify Python task chaining:

@task
def extract():
 return data

@task
def transform(data):
 return transformed_data

@task
def load(transformed_data):
 print("Loading data")

The @task decorator converts each function into an Airflow task automatically.

Scheduling DAGs in Airflow

Wed, 09 Oct 2024 00:00:00 +0000

Welcome to Day 3! Today, we’re diving into scheduling DAGs, an essential component of automating workflows in Airflow.

Scheduling Options

1. Unscheduled DAGs

For DAGs that only run when manually triggered:

dag = DAG(
 dag_id="01_unscheduled",
 start_date=dt.datetime(2019, 1, 1),
 schedule_interval=None,
)

2. Regular Intervals

Use predefined intervals like @daily, @hourly, @weekly:

dag = DAG(
 dag_id="03_with_end_date",
 schedule_interval="@daily",
 start_date=dt.datetime(2019, 1, 1),
 end_date=dt.datetime(2019, 1, 5),
)

3. Cron-based Intervals

For fine-grained control:

schedule_interval="0 0 * * *" # Every day at midnight

4. Frequency-based Intervals

For custom intervals using timedelta:

dag = DAG(
 dag_id="03_frequency_based",
 schedule_interval=dt.timedelta(days=3),
 start_date=dt.datetime(2019, 1, 1),
)

Backfilling

Control historical runs with the catchup parameter:

dag = DAG(
 dag_id="09_no_catchup",
 schedule_interval="@daily",
 start_date=dt.datetime(2019, 1, 1),
 catchup=False # Only run future tasks
)

Best Practices

Atomicity: Each task should perform a single responsibility
Idempotency: Tasks should produce the same results when run multiple times

Stay tuned for Day 4!

Crafting Your First Real Airflow DAG

Tue, 08 Oct 2024 00:00:00 +0000

Welcome back to Day 2! Today, we’ll explore the anatomy of a DAG and get hands-on by writing a simple ETL workflow.

Writing Your First Workflow (DAG)

Here’s a complete ETL pipeline for an art gallery:

from airflow import DAG
from datetime import datetime
from airflow.operators.python import PythonOperator
import pandas as pd

with DAG(
 dag_id="art_gallery_etl_2024",
 start_date=datetime(year=2024, month=1, day=1, hour=9, minute=0),
 schedule="@daily",
 catchup=True,
 max_active_runs=1,
 render_template_as_native_obj=True
) as dag:

 def extract_art_data_callable():
 print("Extracting art piece data from gallery records")
 return {
 "date_acquired": "2022-09-15",
 "artist": "Vincent van Gogh",
 "title": "Starry Night",
 "details": {
 "type": "Painting",
 "dimensions": "73.7 cm x 92.1 cm"
 }
 }

 extract_art_data = PythonOperator(
 task_id="extract_art_data",
 python_callable=extract_art_data_callable
 )

 def transform_art_data_callable(raw_data):
 transformed_data = [
 [
 raw_data.get("date_acquired"),
 raw_data.get("artist"),
 raw_data.get("title"),
 raw_data.get("details").get("type"),
 raw_data.get("details").get("dimensions")
 ]
 ]
 return transformed_data

 transform_art_data = PythonOperator(
 task_id="transform_art_data",
 python_callable=transform_art_data_callable,
 op_kwargs={"raw_data": "{{ ti.xcom_pull(task_ids='extract_art_data') }}"}
 )

 def load_art_data_callable(transformed_data):
 loaded_data = pd.DataFrame(transformed_data)
 loaded_data.columns = ["date_acquired", "artist", "title", "art_type", "dimensions"]
 print(loaded_data)

 load_art_data = PythonOperator(
 task_id="load_art_data",
 python_callable=load_art_data_callable,
 op_kwargs={"transformed_data": "{{ ti.xcom_pull(task_ids='transform_art_data') }}"}
 )

 extract_art_data >> transform_art_data >> load_art_data

Key Components

DAG: Defines the workflow with properties like dag_id, start_date, and schedule
Tasks: Each task performs a single action (extract, transform, load)
Operators: Define what kind of task is executed (PythonOperator for Python functions)
Dependencies: Using >> to define execution order

Handling Task Failures

Airflow excels at handling failures:

View detailed logs from the UI
Selectively rerun just the failed task
Successful tasks don’t need to be rerun

Stay tuned for Day 3, where we’ll dive into scheduling!

Introduction to Data Pipelines with Apache Airflow

Mon, 07 Oct 2024 00:00:00 +0000

Welcome to the first post of my 15-day series on learning Apache Airflow! Every day, I’ll break down one chapter from the book “Data Pipelines with Apache Airflow” by Bas Harenslak & Julian Rutger de Ruiter.

What is Apache Airflow?

Apache Airflow is an open-source platform that allows you to orchestrate complex data workflows. It was developed to automate the scheduling and monitoring of tasks involved in data pipelines.

Key Concepts

Workflows & DAGs (Directed Acyclic Graphs)

Airflow manages workflows by representing them as DAGs
A DAG is a collection of tasks executed based on dependencies
Example: A data pipeline where raw data is extracted, transformed, and loaded into a database

Tasks in Airflow

Each node in a DAG represents a task
Tasks can be anything from running a Python script to querying a database
Tasks run in parallel or sequentially based on dependencies

Schedulers

Airflow has a scheduler that ensures tasks run at the right time
You can schedule tasks to run at specific intervals (daily, hourly, etc.)

Setting Up Apache Airflow

1. Set Up a Python Environment

python3 -m venv airflow_venv
source airflow_venv/bin/activate

2. Install Apache Airflow

export AIRFLOW_VERSION=2.6.3
export PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
export CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

3. Initialize the Database

airflow db init

4. Create a User

airflow users create \
 --username admin \
 --firstname FIRST_NAME \
 --lastname LAST_NAME \
 --role Admin \
 --email admin@example.com

5. Start Airflow

airflow webserver --port 8080
airflow scheduler

My First DAG

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

dag = DAG('my_first_dag', start_date=datetime(2024, 10, 1), schedule_interval='@daily')

def my_first_task():
 print("Hello, this is my first Airflow task!")

task = PythonOperator(
 task_id='print_hello',
 python_callable=my_first_task,
 dag=dag,
)

Stay tuned for Day 2 where I’ll cover Airflow’s Architecture in detail!