Forem: Anthony Gicheru

Refactoring Airflow Pipelines: From PythonOperator to TaskFlow

Anthony Gicheru — Fri, 24 Apr 2026 10:52:39 +0000

Actually Embracing TaskFlow After a Year of Doing It the “Old Way”

1. Introduction: This Isn’t New… But It Feels New

If you’ve been using Airflow for a while-like I have-you probably didn’t start with the TaskFlow API.

You likely started with the classic Airflow 2.x style:

PythonOperator
**kwargs
ti.xcom_push() and ti.xcom_pull()
Explicit task chaining with >>

I spent over a year building pipelines this way. And to be clear-it works. It’s stable, production-ready, and widely used.

But here’s the interesting part:

The TaskFlow API has existed since Airflow 2.0. I just didn’t fully adopt it.

Honestly, I ignored TaskFlow for a long time because I thought it was just ‘syntactic sugar’. That’s more common than people admit.

Most production systems and tutorials still rely on operators, so you naturally stay in that pattern. It’s only later-when readability and maintainability start to matter-that TaskFlow becomes interesting.

And once it clicks, it changes how you think about Airflow.

2. Core Concepts: Same Engine, Different Experience

TaskFlow doesn’t replace Airflow concepts-it abstracts them.

You still work with:

Tasks
DAGs
Scheduling
XComs

The difference is how you express them.

Traditional Approach

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def extract(**kwargs):
    data = [1, 2, 3]
    kwargs['ti'].xcom_push(key='data', value=data)

def transform(**kwargs):
    ti = kwargs['ti']
    data = ti.xcom_pull(task_ids='extract', key='data')
    return [x * 2 for x in data]

with DAG('traditional_dag', start_date=datetime(2023, 1, 1)) as dag:

    t1 = PythonOperator(task_id='extract', python_callable=extract)
    t2 = PythonOperator(task_id='transform', python_callable=transform)

    t1 >> t2

This works, but it introduces a lot of orchestration boilerplate into your business logic.

TaskFlow Approach

from airflow.sdk import dag, task
from datetime import datetime

@dag(start_date=datetime(2023, 1, 1), schedule='@daily', catchup=False)
def taskflow_dag():

    @task
    def extract():
        return [1, 2, 3]

    @task
    def transform(data):
        return [x * 2 for x in data]

    transform(extract())

dag = taskflow_dag()

This feels simpler because it is.

TaskFlow removes explicit XCom handling and lets function returns define data flow.

3. The Real Shift: From Wiring Tasks to Modeling Data Flow

With the traditional approach, your mental model looks like this:

Task A - XCom - Task B - XCom - Task C

With TaskFlow, it becomes:

data = extract()
result = transform(data)

Same execution engine. Different abstraction.

The shift is from task orchestration to data flow composition.

4. XComs: Manual vs Automatic

Manual XComs

def extract(**kwargs):
    kwargs['ti'].xcom_push(key='data', value=[1, 2, 3])

def transform(**kwargs):
    ti = kwargs['ti']
    data = ti.xcom_pull(task_ids='extract', key='data')

You manage everything explicitly.

TaskFlow XComs

@task
def extract():
    return [1, 2, 3]

@task
def transform(data):
    return [x * 2 for x in data]

Airflow handles:

serialization
storage
retrieval

You focus on logic.

When You Still Need Control

TaskFlow still allows explicit control when needed:

from airflow.models.xcom_arg import XComArg

@task
def extract():
    return {"numbers": [1, 2, 3]}

@task
def transform(data):
    return [x * 2 for x in data["numbers"]]

transform(XComArg(extract()))

5. Real-World Example: Gas Prices ETL Refactor

I didn’t build two versions of this pipeline at once.

I originally built it using the traditional Airflow 2.x approach and later refactored it using TaskFlow.

That’s when the difference became clear.

Pipeline Overview

API - Extract gas prices - Transform - Store in PostgreSQL

GitHub Reference

Full project: Github Link to the project

It includes both the original DAG and the TaskFlow refactor.

Traditional Version

def fetch_gas_prices(**kwargs):
    kwargs['ti'].xcom_push(key='raw_gas_data', value=decoded_data)

def transform_gas_prices(**kwargs):
    raw_data = kwargs['ti'].xcom_pull(
        task_ids='fetch_gas_prices',
        key='raw_gas_data'
    )

This approach tightly couples logic with Airflow internals.

Data must be serialized manually:

json_data = df.to_json(orient='records')

TaskFlow Version

@task
def fetch_gas_prices():
    return decoded_data

@task
def transform_gas_prices(raw_data: str):
    return df.to_json(orient='records')

And the pipeline becomes:

raw = fetch_gas_prices()
cleaned = transform_gas_prices(raw)
store_gas_prices(cleaned)

This reads like standard Python.

What Changed

The logic stayed the same. The structure changed completely.

Instead of manually managing XComs, data flows naturally between functions.

Before vs After

Aspect	Traditional	TaskFlow
Task definition	PythonOperator	@task
Data passing	Manual XCom	Automatic
Readability	Medium	High
Boilerplate	High	Low
Mental model	Wiring tasks	Data flow

6. Lessons From the Refactor

1. TaskFlow doesn’t remove XComs

It only hides them.

You still need to respect serialization limits:

return big_dataframe  # still not ideal

2. Passing data is easier-but not always better

TaskFlow makes it easy to pass data between tasks, but large payloads should still live in external storage.

3. Refactoring was mostly structural

Most of the work was:

removing **kwargs
replacing XCom logic with returns
simplifying task boundaries

4. The biggest change is mental

The shift was not technical-it was conceptual.
From:

How do I connect tasks?
to:

How does data flow through this pipeline?

7. Pitfalls to Avoid

Don’t push large objects through XCom
Don’t mix styles without intention
Don’t overuse TaskFlow just because it’s cleaner
Don’t forget serialization still exists

8. Conclusion

TaskFlow isn’t new-but adopting it after using the traditional approach makes its benefits clearer.

It moves you from writing orchestration-heavy DAGs to writing clean Python workflows.

And that shift improves:

readability
maintainability
reasoning about pipelines

Key Takeaways

TaskFlow simplifies DAG structure without changing Airflow’s core engine
XComs still exist but are abstracted
The real improvement is cleaner data flow modeling
Refactoring old DAGs is one of the best ways to understand it

Data Pipelines Explained Simply (and How to Build Them with Python)

Anthony Gicheru — Fri, 17 Apr 2026 07:34:55 +0000

Data pipelines are the backbone of modern data-driven organizations. They automate the movement, transformation, and storage of data - from raw sources to actionable insights.

Python has become the go-to language for building scalable pipelines because of its rich ecosystem, flexibility, and ease of use.

This guide walks through the fundamentals, tools, and best practices for building robust data pipelines using Python.

Understanding Data Pipelines

Imagine you need to supply clean water to a village. The process involves collecting water from different sources (rivers, wells, rain), purifying it, transporting it, and storing it so people can access it whenever they need it.

A data pipeline works in a very similar way.

It automates the journey of raw, unstructured data from multiple sources (like databases, APIs, or IoT devices) and transforms it into clean, usable data stored in a destination (like a data warehouse) for analysis.

Components of a Data Pipeline

Let’s break it down using the same analogy:

1. Collecting Water (Data Ingestion)

Just like gathering water from lakes or wells, a pipeline starts by extracting data from sources such as databases, APIs, spreadsheets, or sensors.

The goal here is simple: get all the data into one system, no matter how scattered it is.

2. Filtering and Purifying (Data Transformation)

Raw water isn’t clean—and neither is raw data.

At this stage, the pipeline:

Removes duplicates
Handles missing values
Standardizes formats
Enriches data

This is where messy data becomes usable.

3. Transporting Through Pipes (Data Movement)

Once cleaned, water flows through pipes. In data pipelines, this represents the movement of data between systems.

This can involve:

ETL processes
Message queues (like Kafka)
Cloud data transfer services

The goal is to move data efficiently without delays or bottlenecks.

4. Storing in Tanks (Data Storage)

Clean water is stored in tanks. Similarly, processed data is stored in:

Data warehouses (like Snowflake)
Data lakes (like AWS S3)
Databases

This is where data becomes ready for use.

5. Accessing on Demand (Data Consumption)

Finally, people use the water.

In the same way, data is consumed through:

Dashboards
APIs
Machine learning models

This is where insights actually happen.

Essential Python Libraries and Tools

Python supports every stage of a pipeline:

Data Ingestion

requests - API calls
pandas - handling CSV/JSON files

Transformation

pandas - cleaning and aggregation
PySpark - large-scale distributed processing

Storage

SQLAlchemy - database interaction
boto3 - AWS S3 integration

Orchestration

Apache Airflow - workflow scheduling and automation
Dagster - modern pipeline orchestration with observability

Best Practices

Error Handling

Implement retries and proper logging to avoid silent failures.

Monitoring

Track pipeline health using tools like Airflow’s UI.

Documentation

Keep clear documentation for:

Code
Dependencies
Workflow logic

Testing

Test each stage of the pipeline using:

Unit tests
Sample datasets

Popular Frameworks for Advanced Use Cases

Apache Airflow - Best for complex workflows with dependencies
Dagster - Strong focus on testing and data asset visibility
Prefect - Simplifies building fault-tolerant pipelines
Luigi - Good for batch processing and dependency management

ETL vs ELT: Which One Should You Use and Why?

Anthony Gicheru — Sun, 12 Apr 2026 21:36:22 +0000

When I first started learning data engineering, ETL and ELT honestly felt like the same thing with just swapped letters. Everyone kept mentioning them like they were obvious concepts, but I had to sit down and really break them apart before it made sense.

If you’re in the same place, don’t worry, you’re not alone.

Let’s make it simple.

First things first: what do ETL and ELT even mean?

Both ETL and ELT are ways of moving and processing data from one place to another.

ETL (Extract, Transform, Load)

Extract data from a source (like an API or database)
Transform it before storing it (cleaning, filtering, joining, etc.)
Load the final cleaned data into a target system (like a data warehouse)

The key idea: you clean the data before storing it.

ELT (Extract, Load, Transform)

Extract data from the source
Load it directly into the storage system first
Transform it inside the database/warehouse later

The key idea: you store raw data first, then clean it inside the system.

So what’s the real difference?

The biggest difference is where the transformation happens.

ETL → Transform happens outside the warehouse
ELT → Transform happens inside the warehouse

That one shift changes a lot more than you’d think.

When ETL makes sense

ETL is usually used when:

You have smaller datasets
You need strict data control before loading
Your system can’t handle heavy processing
Data quality must be enforced early

Think of it like cleaning your room before putting things in storage.

You don’t want messy data entering your system at all.

When ELT makes sense

ELT is more common in modern systems, especially with cloud platforms.

It works well when:

You have large volumes of data
You’re using powerful cloud warehouses (like Snowflake or BigQuery)
You want flexibility in how data is transformed
You want to keep raw data for future use

Think of it like dumping everything into a warehouse first, then organizing it later when needed.

A simple real-world example

Imagine you’re building a dashboard for an e-commerce app.

With ETL:

You:

Pull order data
Clean it (remove duplicates, fix missing values)
Then load it into your database ready for reporting

Everything is neat before it even arrives.

With ELT:

You:

Pull raw order data
Load everything into a data warehouse
Later write SQL transformations to clean and structure it

This gives you more flexibility if business rules change later.

My key takeaway

When I first learned this, I thought ETL was “old” and ELT was “new,” but that’s not really true.

They both still matter.

Here’s a simple way I now remember it:

ETL = Clean first, store later
ELT = Store first, clean later

Common mistakes beginners make

A few things that confused me at the start:

Thinking ELT means “no cleaning” (it still involves transformation!)
Mixing up where SQL transformations happen
Assuming one is always better than the other (it depends on the system)

So… which one should YOU use?

There’s no universal winner.

If you’re working with traditional systems → ETL is common
If you’re in modern cloud data engineering → ELT is more popular

Most real companies actually use a mix of both, depending on the pipeline.

To make this even more practical, here are some common tools used in real ETL and ELT workflows

ETL Tools (Transformation happens before loading)

Apache Airflow – for scheduling and orchestrating ETL workflows
Informatica PowerCenter – widely used in enterprise ETL pipelines
Talend – open-source tool for data integration and transformation
Apache NiFi – good for real-time data flow and routing
SSIS (SQL Server Integration Services) – Microsoft-based ETL tool

These tools usually handle data cleaning and transformation before sending data to a warehouse.

ELT Tools (Transformation happens after loading)

Snowflake – modern cloud data warehouse with strong ELT support
Google BigQuery – popular for serverless ELT workflows
Amazon Redshift – widely used in AWS-based data stacks
dbt (Data Build Tool) – one of the most popular tools for transformations inside the warehouse
Databricks (Apache Spark) – used for large-scale ELT processing

In ELT setups, tools like dbt handle transformation using SQL after data is loaded.

Final thoughts

Once I understood this difference, a lot of other concepts like data pipelines, warehouses, and analytics started to make way more sense.

If you’re learning data engineering right now, don’t rush it. Build a small pipeline, try both approaches, and you’ll see the difference quickly.

That’s where it really clicks.