Forem: Srinidhi

Data Engineering Series #3: Apache Airflow - the modern Workflow management tool. Getting Started

Srinidhi — Tue, 30 Mar 2021 04:00:13 +0000

Why such attention towards Airflow ?

Interest rate of airflow. Source: Google Trends

"The software industry has seen a notable rise in the development of tools to manage data. These tools range from storage solutions that house data to pipelines that transport data".

Data-driven companies like Airbnb, Quizlet rely on these data pipelines to resolve some tedious tasks like

Scheduling
Testing
Handling errors
Versioning
Scaling

the current data flow set up.

We call these pipelines data workflows. And a well-known tool in this space is Apache Airflow.

Airflow solves those tedious tasks by,

click on a feature if you wish to jump to its section directly.

And the best part of all, Airflow is an Open Source tool and has a rapidly growing user base and contributions 🙂

Without further ado,

Let's deep dive into the workflow framework

And create a minimalistic workflow pipeline that leverages all the Airflow features I had listed above.

✨ DAGS

Every workflow is constructed as a Directed Acyclic Graph (DAG).
DAG is created using a .py file which should ideally have three sections configured

DAG configuration - default_args
DAG Instance DAG()
DAG Tasks - Operators

DAG Configuration - `default_args`

default_args is used to set properties (arguments) that are common for all the tasks.

# Initializing the default arguments that we'll pass to our DAG
default_args = {
    'owner': 'Srinidhi',
    'start_date': datetime(2018, 1, 1),
    'retries': 1, 
    'on_failure_callback': task_failure_alert
}

DAG Instance - `DAG()`

# Creating a DAG instance
my_workflow_dag = DAG(
    'first_dag',
    default_args=default_args,
    description='Sample DAG for DE Series',
    schedule_interval='0 0 * * *',
)

DAG Tasks - `Operators`

Airflow Operators are used in creating individual tasks in the DAG. Properties specific to a task will be configured in the operator.

Some of the most commonly used Airflow Operators

PythonOperator - Calls a python function. An alternative way to create a python function task is to use TaskFlow API which is available from Airflow 2.0
BashOperator - Executes a UNIX command

#Operators
# Bash operator tasks that executes unix command - echo
task_1 = BashOperator(
    task_id='first_task',
    bash_command='echo "First Task"',
    dag=my_workflow_dag,
)

# Python Operator that prints the details of current job using context variable
 task_2 = PythonOperator(
    task_id='second_task',
    python_callable=print_fn,
    provide_context=True,
    on_success_callback = task_success_alert,
    dag=my_workflow_dag,
)

#Python Function

def print_fn(ds):
    """
    Implement your python task logic here.
    """
    #Prints job details passed through provide_context parameter
    print(ds)

Now that we have our DAG ready, let's see where I have configured the features in it.

✨ RETRY

Airflow handles errors and failures gracefully.
If you prefer to re-run a failed task multiple times before aborting the workflow run, use retries argument. So that an erroneous task will be executed up to the defined number of times before it's marked as failed.

✨ ALERTS

on_success_callback and on_failure_callback arguments are used to trigger some actions once the workflow succeeds or fails respectively. This will be useful to send personalized alerts to internal team via Slack, Email, or any other API call when a workflow task succeeds or fails.

👇 Sample Slack notification received from Airflow

View Code - Success, Failure

✨ WEBUI

Executing airflow webserver in CLI starts airflow service which opens up Airflow WebUI in 0.0.0.0::8080 URL.

Since Airflow is made of Flask framework, you can even extend WebUI by creating additional pages using Flask AppBuilder.

✨ SCHEDULING

Scheduler service - the heart of Airflow, needs to be Up and Running to run the DAGS.
Once Airflow is installed, execute airflow scheduler to start scheduler service.

In addition to that, DAGs can be made to run automatically at a particular time by providing a cron notation in schedule_interval

✨ API

Airflow 2.0 has released Airflow API that opens the door to Automation with Airflow. With Airflow API, one can automate DAG triggers, reads, and other operations that are possible using WebUI.

Setting up your Airflow Instance

Here are few great resources :
On Windows with Docker
On Ubuntu Local Setup
On Kubernetes
AWS Using cloudformation

If you don't have the bandwidth to set up and maintain airflow in your own infrastructure, here are the commercial Airflow-as-a-service providers:

Google Cloud Composer - managed service built atop Google Cloud and Airflow.
Astronomer.io - In addition to hosting Airflow in their infrastructure, they provide solutions focused on airflow and support services.

Is Airflow the only option for workflows?

Certainly Not. It depends on different factors.
Say your entire product is hosted in a single cloud provider ( AWS / Azure/ GCP) and you are fine with vendor lock-in.
Then,

For AWS, AWS step function will be a good option.
For Azure, you can opt for Azure Data Factory.
For GCP, Google Cloud Composer will be the best fit.

Whereas if you want to host your product in multi-cloud -
Airflow will be a better fit. There are other workflows in the market similar to Airflow :
Prefect
Luigi
Dagster

Airflow Best practices / Tips:

Once you have airflow installed, modify the configurations of your airflow instance in the master config file - airflow.cfg
Avoid creating complex data processing logic in Airflow, as airflow is not intended to work as a data processing engine. Its focus is to orchestrate data flow jobs. (For data processing, Run the processing script in a batch compute or spark framework and invoke from airflow)
Use macros and templates to avoid hard coding values.
While migrating the airflow metadata db to a new database, use "airflow upgradedb" (For 2.0 airflow db upgrade) instead of "airflow initdb" (For 2.0 airflow db init)
When you want to implement a task out of the box, create custom operators using the BaseOperator Class for re-using them.
Use Airflow Plugins, to connect to third-party tools/data sources.
Use Postgres database as airflow's metadata DB to use Local and CeleryExecutors.

If you have any queries or any more tips that you think might be useful, comment below on the post.

Going forward, I'll publish detailed posts on tools and frameworks used by Data Engineers day in and day out.

Follow for updates.

Data Engineering Series #2: Cloud Services and FOSS in Data Engineer's world

Srinidhi — Wed, 24 Jun 2020 05:25:02 +0000

"Open Source (OSS) frameworks have improved the quality of Big Data processing with its diverse set of tools addressing numerous use cases

In fact, if you are a part of a team working on building a modern data architecture, chances are high you are using an open-source stack.

Similarly, Cloud Computing has been enabling Big Data Solutions in yielding scalable and cost-effective solutions in analytics space.

Open Source and Cloud : The Correlation

In the cloud ecosystem, many of the commercially available cloud services are either

Similar to an OSS ➡ Similar in Features (Eg: AWS Step Functions and Apache Airflow )

Modeled after an OSS ➡ Follows/ Inherits the design principles of an existing Open Source framework. (Eg: AWS Kinesis and Apache Kafka)

Managed service of an OSS ➡ Takes care of deployment & maintenance of the OSS framework and making it ready to use. (Eg: AWS RDS Postgres and PostgresDB)

To understand more, Let's touch upon the basics...

Getting to know the cloud

The first step that many of us go through while getting to know about cloud services is to start wondering where to start from the plethora of services available out there.

So, For the ease of understanding, Irrespective of the cloud provider (AWS, Azure, GCP, etc). let's group the big data related cloud services into these stages.

Now, Let's try to understand the cloud ecosystem by comparing AWS cloud services with its equivalent open source frameworks. (Similar comparison can be drawn with Azure and GCP as well)

📍 Data Ingestion:

AWS Service	What it does	Relation with OSS	OSS Alternative
Kinesis	Stream Processing	Modelled After	Apache Kafka
SQS	Message Queue	Similar to	RabbitMQ
Managed Streaming for Kafka (MSK)	Stream Processing	Managed Service of	Apache Kafka

📍 Data Storage:

AWS Service	What it does	Relation with OSS	OSS Alternative
S3	Object store	Similar to	Minio, Swift, Ceph, ...
RDS	Relational database	Managed Service of	MariaDB, MySQL, Postgres
DynamoDB	NoSQL database	Similar to	Apache Cassandra
ElastiCache	In-memory cache	Managed Service of	Memcached, Redis
Neptune	Graph database	Similar to	Neo4j
Amazon QLDB	Ledger database	Modelled After	Hyperledger
Amazon DocumentDB	Document database	Similar to	MongoDB
AWS Lake Formation	Data lake	Similar to	HDFS
EC2 EBS	Block storage for EC2	Similar to	OpenEBS, Portworx

📍 Data Processing:

AWS Service	What it does	Relation with OSS	OSS Alternative
Elastic Map Reduce	Hadoop	Managed Service of	Hadoop,
Step Functions	Worflow Orchestrator	Similar to	Apache Airflow , Flyte
AWS Glue	ETL	Managed Service of	Apache Spark
Lambda	Serverless	Similar to	Knative, OpenFaaS, Fn
Batch	Batch Job Computing	Similar to	Apache Airflow on Kubernetes

📍 Data Analysis & Visualization:

AWS Service	What it does	Relation with OSS	OSS Alternative
Amazon Redshift	Data warehousing	Similar to	Spark SQL, Apache Hive, Presto
Athena	Data warehousing	Similar to	Spark SQL, Apache Hive, Presto
CloudSearch	Search	Similar to	Elasticsearch
Elasticsearch Service	Search	Managed Service of	Elasticsearch
QuickSight	Business analytics	Similar to	PowerBI

📍 Deployment:

AWS Service	What it does	Relation with OSS	OSS Alternative
Elastic Container Registry (ECR)	Container registry	Managed Service of	Docker Registry, Quay
Elastic Container Service (ECS)	Container orchestration	Managed Service of	Kubernetes, Marathon
Elastic Kubernetes Services (EKS)	Container orchestration	Managed Service of	Kubernetes
Cloud Formation	Infrastructure as a code	Similar to	Terraform

Some of the notable cloud adoptions with respect to Big Data.

- Till now, AWS users have launched more than 15 million Hadoop clusters. (EMR / Containerized versions)
- "container-as-a-service" (EKS, ECS) and "Database-as-a-service" (RDS, DynamoDB) are the most commonly used managed services in 2020.
- Database services usage up 127% year over year.

Next Steps...

You can understand how these services are put to use in real-world use cases in this article
This Whitepaper from AWS on Big Data will be a good place to understand its Services.
And start getting hands-on following this repo

Going forward, I'll publish detailed posts on tools and frameworks used by Data Engineers day in and day out.

Follow for updates.

Data Engineering Series #1: 10 Key tech skills you need, to become a competent Data Engineer.

Srinidhi — Tue, 19 May 2020 04:41:53 +0000

Bridging the gap between Application Developers and Data Scientists, the demand for Data Engineers rose up to 50% in 2020, especially due to increase in investments on AI based SaaS products.

After going through multiple Job Descriptions and based on my experience in the field , I have come up with the detailed skill sets to become a competent Data Engineer.

If you are a Backend Developer, some of your skills will overlap with this list below. Yes, it's quite easier for you to make the jump provided the skill gaps are addressed.

🎯 Must Haves

1️⃣ The Art of Scripting and Automating
Can't stress this enough.
Ability to Write a reusable code and to know the Common Libraries and frameworks used in Python for:

* Data Wrangling operations - Pandas,numpy,re
* Data Scraping - requests/BeautifulSoup/lxml/Scrapy
* Interacting with External APIs and other Data Sources, Logging
* Parallel processing Libraries - Dask, Multiprocessing

2️⃣ Cloud Computing Platforms
The rise of cloud storage and computing has changed a lot for data engineers. So much that, being well versed in at least one of the cloud platforms is required.

*Serverless Computing, Virtual Instances, Managed Docker and Kubernetes Services
* Security Standards, User Authentication and Authorization, Virtual Private Cloud, Subnet

Either start with AWS or GCP services.

3️⃣ Linux OS
Importance of Working with Linux OS is often overlooked.
"90% of the public cloud workloads are running on Linux based OS"

* Bash Scripting concepts in Linux like control flow, looping, passing input parameters
* File System Commands
* Running daemon processes

4️⃣ Database Management - Relational Databases, OLAP vs OLTP, NoSQL

* Creating tables, Read,Write,Update and Delete operations, joins, procedures, materialized views, aggrgated views, window functions
* Database vs Data warehouse. Star and snowflake schemas, facts and dimension tables.

Common Relational Databases preferred - PostgreSQL, MySQL etc

5️⃣ Distributed Data Storage Systems

* Knowledge of how distributed data store works.
* Understanding the Concepts like partitioned data storage, sorting key, SerDes, data replication, caching and persistence.

Some of the mostly used ones - HDFS, AWS S3 or any other NoSQL database (MongoDB, DynamoDB,Cassandra)

6️⃣ Distributed Data Processing Systems

* Common techniques and patterns for data processing such as partitioning, predicate pushdown, sort by partition, maintaining size of shuffle blocks, window function
* Leveraging all cores and memory available in the cluster to improve concurrency.

Common Distributed processing frameworks - Map Reduce, Apache Spark (Start with Pyspark if you are already comfortable with Python) Credits : startdataengineering 7️⃣ ETL/ELT tools and Modern Workflow management Frameworks Different companies will have different ways to pick ETL frameworks, One with an In-house data engineering team would prefer to have ETL jobs set up with properly managed workflow management tools for Batch Processing.
ETL - ETL vs ELT, Data connectivity, Mapping, Metadata, Types of Data Loading
When to use a Workflow Management System - Directed Acyclic graphs, CRON scheduling, Operators
ETL Tools: Informatica, Talend
Workfow Management Frameworks: Airflow, Luigi

🎯Good To Have

8️⃣ JAVA / JVM Based Frameworks
Knowledge of a JVM based language such as Java or Scala will be extremely useful

- Understand both functional and object oriented programming concepts
- Many of the high performance data science frameworks that are built on top of Hadoop usually are written using Scala or Java.

JVM Based Frameworks - Apache Spark, Apache Flink, etc

9️⃣ Message Queuing Systems

* Understanding how data injestion happens in Message Queues
* What are Producer and Consumers and how are they implemented
* Sharding, Data Retention, Replay, de-duplication

Popular Messaging queues: Kafka,RabbitMQ, Kinesis, SQS etc

🔟 Stream Data Processing

*Differentiating between Real-time, Stream and Batch Processing.
*Sharding, Repartioning, Poll Wait time, topics/groups,brokers

Commonly used frameworks: AWS Kinesis Streams,Apache Spark,Storm,Samza etc

If you find any other skill that will be helpful, comment below on the post.

Going forward, I'll publish detailed posts on tools and frameworks used by Data Engineers day in and day out.

Follow for updates.

Forem: Srinidhi

Data Engineering Series #3: Apache Airflow - the modern Workflow management tool. Getting Started

Why such attention towards Airflow ?

Interest rate of airflow. Source: Google Trends

click on a feature if you wish to jump to its section directly.

Let's deep dive into the workflow framework

✨ DAGS

DAG Configuration - default_args

DAG Instance - DAG()

DAG Tasks - Operators

Some of the most commonly used Airflow Operators

✨ RETRY

✨ ALERTS

👇 Sample Slack notification received from Airflow

View Code - Success, Failure

✨ WEBUI

✨ SCHEDULING

✨ API

Setting up your Airflow Instance

Is Airflow the only option for workflows?

Airflow Best practices / Tips:

Data Engineering Series #2: Cloud Services and FOSS in Data Engineer's world

Open Source and Cloud : The Correlation

Getting to know the cloud

📍 Data Ingestion:

📍 Data Storage:

📍 Data Processing:

📍 Data Analysis & Visualization:

📍 Deployment:

Some of the notable cloud adoptions with respect to Big Data.

Data Engineering Series #1: 10 Key tech skills you need, to become a competent Data Engineer.

🎯 Must Haves

🎯Good To Have

DAG Configuration - `default_args`

DAG Instance - `DAG()`

DAG Tasks - `Operators`