Forem: mucio

AWS MWAA and AWS SES integration

mucio — Tue, 02 Nov 2021 10:14:46 +0000

Intro
The standard SMTP configuration
Can we skip the credentials?
A possible solution

Intro

This post is about integrating MWAA with SES using an IAM role and not SMTP credentials. I will try to keep it short and focused.

The assumption here is that you already have your MWAA 2.0.2 environment and its role configured, as per the AWS documentation, and with the SES:* actions allowed on your AWS SES.

The standard SMTP configuration

If you already have your MWAA environment configured and you are trying to send emails, you probably ended up on this documentation page: the default settings there should be good enough to give you an idea of what you need.

The one we are interested in the most is the email.email_backend. As you can see in the documentation the default value is airflow.utils.email.send_email_smtp.

This is the default airflow SMTP integration and works pretty well if you have SMTP credentials (and it works also with SES, see the link above to create the credentials).

Can we skip the credentials?

I do not like to create more credentials than I need. I wanted to be able to use AWS SES without them, just using the IAM Role assigned to my MWAA environment.

What we need is a different email backend, specifically airflow.providers.amazon.aws.utils.emailer.send_email. This will use a small module which is sitting among the ones provided by AWS.

All good and dandy until you try to send the first email.

Here is a quick DAG to copy&paste to test your email:

from datetime import datetime

from airflow.models import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.email import EmailOperator

my_dag = DAG(dag_id="test_email_dag",
             start_date=datetime(2021, 10, 1),
             schedule_interval="0 0 * * *"
            )

start = DummyOperator(dag=my_dag,
                      task_id="start"
                     )

end = DummyOperator(dag=my_dag,
                    task_id="end"
                   )

run_this = EmailOperator(task_id='sent_email',
                         to='mucio@mucio.net',
                         subject="test",
                         html_content="this is a test, nothing to worry"
                        )

start >> run_this >> end

If you run this DAG you will probably see the following error in the logs:

botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Source, value: None, type: , valid types:

What is happening here? Do you remember the email_backend provided by the AWS? For some reasons the mail_from parameter which is passed as Source to boto3 does not contain the correct value.

This problem was already reported in a few Airflow issues and PRs. The fix didn't make the cut for Airflow 2.2 and will be probably there in version 2.3, but because we are talking about MWAA (version 2.0.2), we don't really know when this will be fixed on AWS.

A possible solution

The solution I come up with was to rewrite the emailer.py utility, deploy it in the dags folder and reference it in the MWAA configuration.

Here the new emailer (I put mine in ses_email_fix/emailer.py, with an empty __init__.py):

"""Airflow module fix for email backend using AWS SES"""

from typing import List, Optional, Union

from airflow.configuration import conf
from airflow.providers.amazon.aws.hooks.ses import SESHook


def send_email(to: Union[List[str], str],
               subject: str,
               html_content: str,
               files: Optional[List] = None,
               cc: Optional[Union[List[str], str]] = None,
               bcc: Optional[Union[List[str], str]] = None,
               mime_subtype: str = 'mixed',
               mime_charset: str = 'utf-8',
               conn_id: str = 'aws_default',
               **kwargs,
              ) -> None:
    """Email backend for SES."""

    hook = SESHook(aws_conn_id=conn_id)

    hook.send_email(mail_from=conf.get('smtp', 'SMTP_MAIL_FROM'),
                    to=to,
                    subject=subject,
                    html_content=html_content,
                    files=files,
                    cc=cc,
                    bcc=bcc,
                    mime_subtype=mime_subtype,
                    mime_charset=mime_charset,
                   )

This is taking the mail_from from the smtp.smtp_mail_from MWAA environment setting. Also I changed my email.email_backend to be ses_email_fix.emailer.send_email.

Last thing you don't want to forget is to put the folder ses_email_fix in your .airflowignore file (otherwise Airflow will parse that as a DAG).

Now, after updating your environment, your test DAG should be able to fire emails via SES without using credentials.

Last words are for the Airflow community who came up with a quick workaround for this problem that I just implemented in my own MWAA environment.

Credits: Cover photo by Quoc Nguyen from Pexels

Airflow, Python and String Concatenation

mucio — Mon, 26 Oct 2020 14:51:37 +0000

This is the second part of the series about "Python concepts for people who are in need of using Apache Airflow but have little or no knowledge of Python".

While Airflow concepts will be explained en passant, the main focus of these articles are Python concepts and techniques.

In this article I will focus on string concatenation or to put together a text using multiple pieces like hard coded strings, variables, and/or templates.

Why string concatenation?

Initially I wanted to dedicate this article to Python data structures (like lists and dictionaries), but to show them in practice most of my Airflow related examples where about strings.

Feel free to skip this post, or quickly skim it, if you are familiar with Python strings.

Other reasons why I wanted to focus a bit more on this topic:

over time Python introduced multiple ways to do string concatenation, looking at code written by other people can be sometimes confusing
it is a topic for interview questions, especially for juniors engineers or analysts
it is easy to do it wrong (wrong = in a way which is hard to maintain)

Some people find string concatenation confusing, it can be. I hope at the end of this article you can have the tools to understand string concatenation and formatting. If not, feel free to write your feedback in the comments.

A basic dag with strings

from datetime import datetime

from airflow.models import DAG
from airflow.operators.dummy_operator import DummyOperator

dag_name = "string_sample"
task1_name = "start"
task2_name = "end"

my_dag = DAG(dag_id=dag_name + "_dag",
             start_date=datetime(2020, 4, 29),
             schedule_interval="0 0 * * *"
             )

task1 = DummyOperator(dag=my_dag,
                      task_id="task_{}".format(task1_name)
                      )

task2 = DummyOperator(dag=my_dag,
                      task_id=f"task_{task2_name}"
                      )

task1 >> task2

If we look into the Airflow web UI we will see a DAG called string_sample_dag, which consists of two tasks: task_start and task_end. The operators used are, again, the DummyOperator ones. This DAG is just an excuse to show you how to concatenate and format strings in Python.

The many ways to concatenate strings in Python

The lazy way, the `+`

The simplest way to concatenate multiple strings is to use the plus sign, +:

dag_name + "_dag"

It works well adding strings, here another working example:

>>> text = "world"
>>> print("hello " + text)
hello world

And here is when being lazy doesn't work that well:

>>> text = 123
>>> print("hello " + text)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
TypeError: must be str, not int

The + operator is a simple man, it sees values it tries to concatenate them. Sometimes it works, sometimes it doesn't. In our case, it is not able to join a string with an integer. To make it work we need to transform 123 in a string "123":

>>> print("hello " + str(text))
hello 123

Better formatting with `.format()`

In the first task of the DAG there is a second, better way to concatenate strings. The string method .format() allows us to format a string that we use as template:

"task_{}".format(task1_name)

In our previous "hello world" example the template for our greeting is "hello {}", where instead of {} we want to have world, 123 or maybe John.

>>> text = "world"
>>> print("hello {}".format(text))
hello world

And it works with numbers too:

>>> text = 123
>>> print("hello {}".format(text))
hello 123

With multiple values

>>> greeting = "hello"
>>> text = "world"
>>> print("{} {}".format(text, greeting))
world hello

Uhm, this doesn't look right. The .format() method assigns the values to the {} placeholders in the order they appear. Of course we could switch the order of the passed values, but what if over time we change the template?

To avoid this use ordinal numbers or, even better, name the placeholders.

First with numbers:

print("{1} {0}".format(text, greeting))
hello world

Note The first value has index 0, the second 1.

Naming the placeholders:

>>> print("{hi} {who}".format(who=text, hi=greeting))
hello world

Using names placeholders our template string is more meaningful (which will be nice reviewing this code after a while), but we need to specify which value is hi and which is who.

For more formatting options and examples you can take a look at the Python documentation here. You probably will not need them for your DAGs, but it is good to know where to start (beside using Stack Overflow).

The `f` string

Python 3.6 introduced a new way to format strings:

>>> text = "world"
>>> print(f"hello {text}")
hello world

The f stands for formatted string. But also for fast, because this formatting method is faster to write and faster to produce the final result.

In a formatted string the values in between {} are expressions that will be evaluated at runtime. Therefore it is possible to do things like:

>>> print(f"{5 - 4} hello to {text.lower()}")
1 hello to world

Again, more use case and advanced option can be discovered reading the Python documentation.

Few additional things on Python strings

These are probably details that you will not use writing DAGs, but it is worth to mention to complete the basic overview on the strings.

You probably noticed that formatting a string in Python utilizes the curly braces, then how to format a string that contains curly braces? Double curly, duh:

>>> print(f"In {{text}} is {text}")
In {text} is world

What about the double quote sign "? First, in Python you can use ' and " for a string, and you need to use the same sign to close the string. Another why is to escape the quote sign using a \:

>>> print('This is a double quote: "')
This is a double quote: "
>>> print("This is a double quote too: \"")
This is a double quote too: "

Finally, how to deal with very long strings. Python allows you to break a long line of code into multiple using the backslash \:

>>> print("hello " \
... "world")
hello world

As you can see Python concatenates the two strings ignoring the new line (to achieve that you can do "hello \n").

But you can also create a multi line string using """ or ''':

>>> print("""hello
>>> world""")
hello
world

More additional things

Actually there is much more to say about Python strings, concatenation, and formatting, but this will require another post (or more).

Personally I believe these things are not very interesting for someone approaching Python for the first time. But I have been wrong in the past, vote unicorn to tell me that I am wrong and you want to know more about strings (and maybe add a comment on what you find difficult with strings in Python).

Which one to use?

Using Python 3.6 or above, f-string is the way to go: it is fast to write and easy to maintain.

But I am the first to admit that for quick debugging I still resort to the + (when possible).

The .format() is worth to mention because there are many examples in the wild using it or you can encounter some old code which requires your attention.

That said, I suggest you to take a look at the links to the documentation to be aware of the formatting possibility offered by Python, sooner or later you will need them.

Shameless plug

In case you need support or assistance feel free to reach out to me in the comment or direct messages. On twitter you can find me with the handler @mucio.

If you need more structured help, the nice people at Untitled Data Company (which includes me) will be happy to help you with all your data needs.

Credits

Cover photo by Jess Bailey on Unsplash.

Quickly Setup Airflow for Development with Breeze

mucio — Wed, 30 Sep 2020 19:41:54 +0000

Disclaimer: I have submitted the PRs for two of the Breeze features mentioned in this article (the start-airflow command and the --init-scripts flag). I feel responsible for your user experience using them, so if you have questions or feedback please reach out to me.

TL;DR

To have Airflow running on your machine do the following:

Install Docker and Docker Compose
Clone the Airflow repository git clone git@github.com:apache/airflow.git
In the Airflow folder run ./breeze start-airflow

With the first run Breeze creates the folder files/dags in the repo folder. Adding DAG files in that folder will make them appear in Airflow.

Go to https://localhost:28080 to see your Airflow running.

Intro

If you do not like when food recipes start with pages of blabbing skip this part.

My problem

I started a to write a few blog posts for people who are approaching Python and Apache Airflow for the first time. I needed an quick way for my readers to setup their own Airflow and an even quicker way to explain how to do it.

I wanted something so simple that you and I could focus only on DAG's code. Enter Breeze.

What is Breeze?

Breeze is a command line tool to spin up a dockerized* Airflow instance for development or testing. It can be used to create an environment with specific properties to run tests, before deploying to production. This is pretty cool if you are into CI/CD.

The first time I met Breeze I was working on automating the creation of our own environment to run tests (for a data warehouse, not for Airflow), and I was very intrigued by the idea.

Therefore when I started thinking about how easily have a dev Airflow running, Breeze was on top of my list.

Dockerized stands for "running in a virtual machine with no impact on your computer (called the host, while the vm is the guest)." Well, no impact beside consuming CPU and RAM 😕

Setup

Prerequisites

I run my Airflow/Breeze using WLS2 on Windows 10, people using a Mac or a Linux machine will have probably a smoother experience than me, but WLS2 with Ubuntu is quite good (if you are on Windows 10 the WLS2 setup is covered here) and Breeze runs more easily in a linux box (or a mac).

What do you need:

Install Docker
Install Docker Compose
Install git (this is usually already installed)

In my case I installed all these tools in my Ubuntu WLS.

Installation and first run

Clone the Airflow repository from Github with:



git clone git@github.com:apache/airflow.git

Once the repo is downloaded go to the Airflow folder and run Breeze:



cd airflow
./breeze start-airflow

Breeze will download a number of docker images and will ask you if you want to build some of them, just say "yes" when asked (you can use the flags --assume-yes or --assume-no if you find this annoying). The first build can take few minutes, depending on your internet speed and machine.

If everything goes as expected you should see a screen like this:

"I love it when a plan comes together"

Congratulations, your Airflow is up and running.

If you go to http://localhost:28080/ you will see the Airflow UI. The default credentials are admin/admin.

Username: admin - Password: admin

How to use this?

What you see are three tmux panes (tmux is a Linux tool to create a terminal session and split it in multiple parts, called panes). In the lower left corner you have the Airflow Scheduler, which takes care of running things, on the right the Webserver is waiting for you to visit the Airflow Web UI. The top pane is to run additional commands.

If you press Ctrl+b followed by an arrow key you will be able to move between panes. There is not much you need to do in the bottom panes, you can stop the scheduler and the webserver with Ctrl+C. The top one is use the Airflow CLI commands (run airflow --help if you want to know more).

To get quickly out from tmux run the following command:



./stop_airflow.sh

The purpose of having these three panes is to allow you to observe what is happening in Airflow and in case use the command line interface (although this is for more advanced use cases).

Developing with Breeze

Now that Airflow is running, you can just put your dags in the folder files/dags created in your Airflow repository folder. If the folder is not there, Breeze will create it. The DAGs could take few minutes to appear on the web UI.

In case a DAG syntax is wrong the bottom left pane (the Webserver one) shows the errors.

Few additional notes:

In case you run Breeze using an SQLite database as Airflow backend (see below), that database is recreated with every run. In case you want to store Airflow configuration objects (like connections to your databases, users, etc.) use a different backend or use an initialization script (again see below).
Environment variables can be entered in the file files/airflow-breeze-config/variables.env (create it, if not there), these are set preparing the Airflow environment.
In case you want to initialize Airflow, you can put a file called init.sh in the folder files/airflow-breeze-config. The instructions in this file will be executed before Airflow Scheduler and Webserver start.

Some details and recipes

The start-airflow command provides a simple way to start Airflow and monitor it. Behind the scene Breeze initialize the Airflow backend database and create an admin user that can be used to login into the web UI (credential admin/admin).

Recipe 1 - A persistent backend

As mentioned above the default database is recreated with every execution, if you want to have something more persistent you can use a different backend, for example PostgreSQL:



./breeze start-airflow -b postgres

This is start an additional container with a database dedicated for Airflow. Now your changes will survive a restart.

Recipe 2 - A different Airflow version

By default Breeze will start the most recent version of Airflow (currently 2.0.0dev) which is probably different from what you have in production. The good thing is that Breeze allows you to pick the version you need with another flag:



./breeze start-airflow --install-airflow-version 1.10.10

Of course you can compose multiple flags:



./breeze start-airflow --install-airflow-version 1.10.10 -b postgres

Feel free to go ahead and explore the other possible flags.

Recipe 3 - Initialize Airflow with your own database connection

One way to do it is to use a resilient backend, you can add your connection in the web UI and use it. At least this is what I was doing when I first started using Airflow.

A more interesting approach is to use the optional initialization script for Breeze to create the connection. This will make easier to maintain the connections and other Airflow settings, plus you can store this file in your versioning tool (e.g. git).

Here an example of init.sh file:



# Connections
airflow connections add \
    --conn-login my_user \
    --conn-password my_pwd \
    --conn-type jdbc \
    --conn-host localhost \
    --conn-port 9457 \
    --conn-extra {} \
    my_connection

# Variable
airflow variables set my_variable variable_content

Using this file will create a JDBC connection called my_connection and a Variable called my_variable. You can see them in the Web UI clicking on the corresponding section in the menu Admin.

Additional information

The main point of Breeze was to provide an easy way to run automatic tests for the core Airflow developers, the people building Airflow not with Airflow. Breeze's goal is to lay down the foundation to easily run Airflow, taking care of:

start the needed docker containers
expose the ports for the Airflow components (e.g. webserver and backend database)
provide an convenient way to run new code in Airflow (e.g. put the dags in files/dags)
eventually run tests

These features were too interesting to leave them just to the core developers ;)

But this is not everything, if you want to know more about the possibilities offered by Breeze I suggest you to take a look at this video (Airflow Breeze - Development and Test environment fro Apache Airflow); it will not make your DAGs better, but will give you more ideas on how to use Breeze and your new dev environment.

Final words

If you are still here, feel free to leave a comment and provide your feedback. I will be happy to assist you and answer your questions (if I am able to).

Shameless plug

In case you need support or assistance feel free to reach out to me in the comment or direct messages. On twitter you can find me with the handler @mucio.

If you need more structured help, the nice people at Untitled Data Company (which includes me) will be happy to help you with all your data needs.

How much Python do you need to know to write an Airflow DAG? Part 1 - ANSWERS

mucio — Fri, 29 May 2020 21:56:04 +0000

Answers:

airflow.operators.dummy_operator
Nope, confusingly enough the module datetime contains the class datetime, so datetime.datetime.

To have the follwing DAG working:

my_dag = DAG("sample_dag",
             schedule_interval="0 0 * * *"
             start_date=datetime.datetime(2020, 4, 29)
             )

We would need to import the whole module, import datetime.

In the original code you can thinks that from datetime import datetime creates a shortcut to datetime.datetime.

Nope, Airflow will pick up the Tasks in our DAG following the dependency tree we defined. If we remove start >> end they will be executed together. With start << end we can reverse time and save the dinosaurs. Everything is possible with Python
Yes, because it is after the DAG's start_date.

5.1 Nope, the first argument should be the task_id, which is a string of text, not a DAG.

5.2 Nope, the unnamed argument should be before the named arguments.

Back to the original post.

How much Python do you need to know to write an Airflow DAG? Part 1

mucio — Fri, 29 May 2020 21:55:26 +0000

Not sure about you, but I wonder about this a lot when I need to onboard new colleagues, colleagues who were writing mostly SQL only till the day before.

This post is for people who needs to start writing data pipelines (or DAGs) for Apache Airflow, know already how process their data, but have little or none knowledge of Python.

This post will not tell you how to install Airflow on your machine, this quick start guide will, but check with your colleagues how you do thing in your organization. My assumption is that you work in an environment where Airflow is already used in production and you don't need to worry about that. If this is not the case, you will need something more than this post.

I hope that after reading this you will be able to understand the code written by your colleagues and feel confident enough to start writing your own DAGs.

I will try to keep this as much practical as possible, as if you should work with me. I assume you know nothing about Python, feel free to skip the sections you are familiar with. Also feel free to ask questions, I am always up for a chat.

Boring things

I will write another post about the building blocks of Airflow, for now let me just share these:

a DAG is an ETL process (for now 1 DAG == 1 file)
a DAG is made of multiple Tasks
a Task is an instance of an Operator
an Operator does things (moves data, sends emails, writes a post on dev.to)

If you need to know what DAG stands for, just click the hearth icon and I will tell you.

A Basic DAG

from datetime import datetime

from airflow.models import DAG
from airflow.operators.dummy_operator import DummyOperator

my_dag = DAG(dag_id="sample_dag",
             start_date=datetime(2020, 4, 29),
             schedule_interval="0 0 * * *"
             )

start = DummyOperator(dag=my_dag,
                      task_id="start"
                      )

end = DummyOperator(dag=my_dag,
                    task_id="end"
                    )

start >> end

At the end of this post you should be able to recognize that this DAG runs every day at midnight and does more or less nothing.

But let's try to read it line by line.

Imports

Python can do a lot of things, but to save resources not everything is available all the time. What needs to be loaded in the computer memory is left to the developers.

When you want to use something that is not part of the basic Python, you can import the needed Python module or library.

import antigravity

Now if I want to use a method of antigravity I need to call it using the module name: antigravity.fly()

If the module name is too long, you can use an alias:

import antigravity as ag

ag.fly()

Widely used modules have standardized aliases (but this is another story).

Some modules are quite big and it is possible to import only part of them.

from datetime import datetime

from airflow.models import DAG
from airflow.operators.dummy_operator import DummyOperator

This allows us to avoid the module name (see below).

Not all modules are installed on your machine when you install Python, you can install more module using a tool like pip.

Defining the DAG

my_dag = DAG(dag_id="sample_dag",
             start_date=datetime(2020, 4, 29),
             schedule_interval="0 0 * * *"
             )

This will create our DAG, which Python will store in the variable my_dag that we will use later on.

This DAG will be picked up by the Airflow scheduler and executed depending on its schedule_interval and start_date. The dag_id is the identifier used internally by Airflow, you cannot have another DAG with the same name.

A DAG is created using the arguments we pass to its constructor (DAG()), if this is the first time you pass arguments to a Python method let me highlight a few things:

we pass three arguments with the format param_name=value
we pass three arguments to this DAG, but the DAG class accepts more of them, the missing one are filled with default values. You can the full list of parameters here
dag_id is actually the only mandatory parameter. As you probably guessed mandatory parameters have no default and must be passed every time you call a function
the format param_name=value is not necessary, but allows us to pass values in the order we prefer. Using names allows us to pass only the needed parameters and we are not forced to change our code if the function we use adds more optional parameters.

For this reason you could remove dag_id=, invert the named arguments, and the code will still work:

my_dag = DAG("sample_dag",
             schedule_interval="0 0 * * *"
             start_date=datetime(2020, 4, 29)
             )

While this...

my_dag = DAG("sample_dag",
             "0 0 * * *"
             datetime(2020, 4, 29)
             )

...makes me to open the link above to see what really are the second and third parameters of the DAG constructor.

Additional notes:

datetime(2020, 4, 29) returns the 29th of April 2019 in a format understandable for the DAG method
the schedule_interval uses a format called crontab expression. This DAG will run every day at midnight. You can learn more about crontab expression using something like crontab.guru (or clicking the Unicorn, so I will write a post about crontab expression for you)
technically speaking we are creating an instance of the DAG class (but I will not write a post about this and you should feel bad about thinking that a click will make me your object oriented writer)

At this point we are ready to add tasks to our DAG.

Tasks

start = DummyOperator(dag=my_dag,
                      task_id="start"
                      )

end = DummyOperator(dag=my_dag,
                    task_id="end"
                    )

start >> end

Task definition looks similar to the DAG definition. Here we are using a very important Airflow operator: the DummyOperator which does absolutely nothing, but allows us to focus on other things:

a Task needs a unique task_id
task_id is mandatory, it is my convention to keep it named
remember the DAG variable? A Task needs to know its parent DAG, so we use that variable here: dag=my_dag

Finally we have the line

start >> end

If you guessed that it means the start execution is followed by the end execution you have guessed right. We can also write it as end << start, but of course it is less immediate.

>> and << are called bitshift operator (you can use this new knowledge as icebreaker at the next meetup).

In case you are reading some old Airflow DAGs, you could find this ancient syntax (but we modern people are better off sticking to the bitshift operators):

start.set_downstream(end)

Check questions

If you scroll up you should now be able to understand what this DAG is doing.

Here some questions to check if you got everything right:

From which module do we import DummyOperator?
In the DAG definition can I replace start_date=datetime(2020, 4, 29), with start_date=datetime.datetime(2020, 4, 29),?
Because the DummyOperator does nothing, will the task start and end run together?
My unbirthday this year is on the 7th of May 2020, will this DAG run that day?
Will the two task definitions work?

start = DummyOperator(my_dag,
                      "start"
                      )

start = DummyOperator(dag_id=my_dag,
                      "start"
                      )

Answers.

Conclusions

At this point you should be familiar with Python concepts like imports and how to call methods with more or less arguments. You should have also a general idea about the relationship between Airflow DAGs and Tasks, and how to create dependencies between Tasks.

In the next episode we will focus on some fundamental Python data structures that can make our DAGs' code simpler and easier to maintain, some less-dummy operators and other nice things. Till then stay... scheduled.

Forem: mucio

AWS MWAA and AWS SES integration

Table of Contents

Intro

The standard SMTP configuration

Can we skip the credentials?

A possible solution

Airflow, Python and String Concatenation

Why string concatenation?

A basic dag with strings

The many ways to concatenate strings in Python

The lazy way, the +

Better formatting with .format()

With multiple values

The f string

Few additional things on Python strings

More additional things

Which one to use?

Shameless plug

Credits

Quickly Setup Airflow for Development with Breeze

TL;DR

Intro

My problem

What is Breeze?

Setup

Prerequisites

Installation and first run

How to use this?

Developing with Breeze

Some details and recipes

Recipe 1 - A persistent backend

Recipe 2 - A different Airflow version

Recipe 3 - Initialize Airflow with your own database connection

Additional information

Final words

Shameless plug

How much Python do you need to know to write an Airflow DAG? Part 1 - ANSWERS

How much Python do you need to know to write an Airflow DAG? Part 1

Boring things

A Basic DAG

Imports

Defining the DAG

Tasks

Check questions

Conclusions

The lazy way, the `+`

Better formatting with `.format()`

The `f` string