Forem: Ahmet Can Aydemir

How to Use Apache Airflow

Ahmet Can Aydemir — Wed, 16 Mar 2022 08:26:26 +0000

Airflow is an orchestration tool that ensures that tasks are running at the right time, in the correct order, and in the right way.

Why You Need Apache Airflow

Imagine you have a data pipeline like the one above.

What if an error occurs in any of these stages? There may be an error in the API from which you are pulling the data, there may be an error while processing the data, or there may be an error while saving to the DB.
If you have a lot of data pipelines like this, it will eventually become overwhelming.

Roughly, as in the example above, taking the data from a source and saving it to the target after certain operations are called ETL (Extract Transform Load). Such transactions can be managed in an advanced way by using Airflow.

Benefits of Apache Airflow

Dynamic: What can be done with Python also can be done with Airflow. As a result, Airflow provides tremendous dynamics when creating our tasks.

Scalable: As many tasks as desired can be easily run in parallel.

User Interface: Airflow has a useful UI. Errors that occur in data pipelines and where they occur can be easily observed. Problematic tasks can be restarted etc.

Extensible: No need to wait for Airflow update when a new tool comes out. You can write your plugins and integrate them easily.

Apache Airflow’s Core Components

Web Server: A Flask server that serves the UI.

Scheduler: The daemon that schedules the workflows. It is the most important component of Airflow.

Metastore: Database where metadata is stored.

Executor: Class that defines how tasks should work.

Worker: The process or sub-process executing the task.

DAG (Directed Acyclic Graph)

A DAG (Directed Acyclic Graph) is the core concept of Airflow, collecting Tasks together, organized with dependencies and relationships to say how they should run.

Example DAG above defines four Tasks - A, B, C, and D - and dictates the order in which they have to run, and which tasks depend on what others. It will also say how often to run the DAG - maybe “every 5 minutes starting tomorrow”, or “every day since January 1st, 2020”.
The DAG itself doesn’t care about what is happening inside the tasks; it is merely concerned with how to execute them - the order to run them in, how many times to retry them, if they have timeouts, and so on.

Basic DAG example:

with DAG("my_dag_name",
        start_date=datetime(2020, 1, 1),
    schedule_interval="@daily",
        catchup=False)as dag:
    op = DummyOperator(task_id="task")

Operators

Operators are wrappers that cover the task. By default, there are many different types of operators and can be viewed at this link. In addition, much more can be added as needed.

Action Operators: Operators which executes functions or commands eg. bash scripts, python code
Transfer Operators: Operators for moving data from source to destination eg. ElasticSearch to Mysql
Sensor Operators: Operators which wait for something to happen before moving on to another task, eg. checking the file in the directory and continuing to the other task after that.

What Isn’t Airflow?

Airflow is not a data streaming solution or data processing framework. If you need to process data every second, instead of using Airflow, Spark or Flink would be a better solution. If terabytes of data are being processed, it is recommended to run the Spark job with the operator in Airflow.

Apache Airflow Setup

Although Airflow can be installed with Docker, Kubernetes or different methods, in this article, we will install it locally.

# Airflow needs a home. `~/airflow` is the default, but you can put it
# somewhere else if you prefer (optional)
export AIRFLOW_HOME=~/airflow

# Install Airflow using the constraints file. For this example, we will use Python 3.6 and Airflow 2.2.4
pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.4/constraints-3.6.txt"

# The Standalone command will initialize the database, make a user,
# and start all components for you.
airflow standalone

# Visit localhost:8080 in the browser and use the admin account details
# shown on the terminal to login.
# Enable the example_bash_operator dag on the home page

First Pipeline with Apache Airflow

For example, let's create a data pipeline like the one in the figure using Airflow. For this, let's create a new python file in ~/airflow/dags directory eg: user_processing.py

1. Importing Modules


from datetime import datetime
import json
from pandas import json_normalize

# DAG object; We will need this to create DAG
from airflow import DAG

# Operators
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from airflow.providers.sqlite.operators.sqlite import SqliteOperator
from airflow.providers.http.operators.http import SimpleHttpOperator

# Sensors
from airflow.providers.http.sensors.http import HttpSensor

2. Instantiate a DAG

We’ll need a DAG object to nest our tasks into. Here we pass a string that defines the dag_id, which serves as a unique identifier for your DAG. We also pass a default argument dictionary and define a schedule_interval of 1 day for the DAG.

default_args = {
    'start_date': datetime(2020, 1, 1)
}

with DAG('user_processing', schedule_interval='@daily',
        default_args=default_args,
        catchup=False) as dag:

        # Define tasks/operators

3. Creating the SQLite Table

We can use SqliteOperator for this task. But SqliteOperator will need a parameter named conn_id, so first let's create a connection for SQLite DB.

In Airflow, connections are kept in the metadata database. We will need to do the same for the HTTPOperator.

We can use UI to create a connection

Go to http://localhost:8080/connection/add (Admin / Connections / +)
Type Conn Id field a unique name e.g.: db_sqlite
Select Conn Type field as SQLite
In the Host field, type the path to the SQLite database file. In this example, we can use Airflow's metadata database instead of creating a new SQLite database file. /home/airflow/airflow.db
Save

After this process, we can create the operator. Each operator must have a unique task_id. SqliteOperator also has sqlite_conn_id and sql parameters. We will use the db_sqlite as the sqlite_conn_id (see step 2).

In SQL code; if the users table does not already exist; It creates a users table with fields firstname, lastname, country, username, password and email.

 creating_table = SqliteOperator(
        task_id='creating_table',
        sqlite_conn_id='db_sqlite',
        sql='''
            CREATE TABLE IF NOT EXISTS users (
                firstname TEXT NOT NULL,
                lastname TEXT NOT NULL,
                country TEXT NOT NULL,
                username TEXT NOT NULL,
                password TEXT NOT NULL,
                email TEXT NOT NULL PRIMARY KEY
            );
        ''' 
    )

4. Verify API

We can add a sensor to check if there is a problem with the API by making an HTTP request. HttpSensor will be useful for this, but HttpSensor needs http_conn_id parameter. So we have to create a new connection as we did in the previous step. Unlike before, this time we will create a connection with HTTP type, not an SQLite type.

We can use UI to create a connection

Go to http://localhost:8080/connection/add (Admin / Connections / +)
Type Conn Id field a unique name e.g.: user_api
Select Conn Type field as HTTP
In the Host field, enter the address of the API that brings random users. eg. https://randomuser.me/
Save

Now we can create the sensor. Since the API we use works at [https://randomuser.me/api/](https://randomuser.me/api), we used api/ in the endpoint parameter.

is_api_available = HttpSensor(
        task_id='is_api_available',
        http_conn_id='user_api',
        endpoint='api/'
    )

5. Retrieve Random Users from API

With using SimpleHttpOperator we send a GET HTTP request to the user_api connection we created before. With response_filter we convert the JSON string in the response.text field into a python object.

 extracting_user = SimpleHttpOperator(
        task_id='extracting_user',
        http_conn_id='user_api',
        endpoint='api/',
        method='GET',
        response_filter=lambda response: json.loads(response.text),
        log_response=True
    )

6. Processing the API Result

With using PythonOperator, we will keep only the desired fields of the python object that we obtained in step 5. And we will save the result as csv file.

Any python function can be run directly with the python_callable parameter.

The important part here is how we will get the response from the previous section. For this, we can send the task instance (ti) object as a parameter to the python function and get the results of the task_ids with users = ti.xcom_pull(task_ids=['extracting_user']).

processing_user = PythonOperator(
        task_id='processing_user',
        python_callable=_processing_user
    )

# Somewhere outside the scope of DAG
def _processing_user(ti):
    users = ti.xcom_pull(task_ids=['extracting_user'])
    if not len(users) or 'results' not in users[0]:
        raise ValueError('User is empty')
    user = users[0]['results'][0]
    processed_user = json_normalize({
        'firstname': user['name']['first'],
        'lastname': user['name']['last'],
        'country': user['location']['country'],
        'username': user['login']['username'],
        'password': user['login']['password'],
        'email': user['email']
    })
    processed_user.to_csv('/tmp/processed_user.csv', index=None, header=False)

7. Saving Processed Users to SQLite DB

We can use BashOperator for trying new operators. BashOperator allows us to run bash commands.

storing_user = BashOperator(
        task_id='storing_user',
        bash_command='echo -e ".separator ","\n.import /tmp/processed_user.csv users" | sqlite3 /home/airflow/airflow/airflow.db'
    )

8. Relationship Between Tasks

Simply, the direction of the tasks and their connection with each other can be defined as follows.

# Somewhere outside the scope of DAG
creating_table >> is_api_available >> extracting_user >> processing_user >> storing_user

Final DAG File

from datetime import datetime
import json
from pandas import json_normalize

from airflow import DAG

from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from airflow.providers.sqlite.operators.sqlite import SqliteOperator
from airflow.providers.http.operators.http import SimpleHttpOperator
from airflow.providers.http.sensors.http import HttpSensor

default_args = {
    'start_date': datetime(2020, 1, 1)
}

def _processing_user(ti):
    users = ti.xcom_pull(task_ids=['extracting_user'])
    if not len(users) or 'results' not in users[0]:
        raise ValueError('User is empty')
    user = users[0]['results'][0]
    processed_user = json_normalize({
        'firstname': user['name']['first'],
        'lastname': user['name']['last'],
        'country': user['location']['country'],
        'username': user['login']['username'],
        'password': user['login']['password'],
        'email': user['email']
    })
    processed_user.to_csv('/tmp/processed_user.csv', index=None, header=False)


with DAG('user_processing', schedule_interval='@daily',
        default_args=default_args,
        catchup=False) as dag:
        # Define tasks/operators

    creating_table = SqliteOperator(
        task_id='creating_table',
        sqlite_conn_id='db_sqlite',
        sql='''
            CREATE TABLE IF NOT EXISTS users (
                firstname TEXT NOT NULL,
                lastname TEXT NOT NULL,
                country TEXT NOT NULL,
                username TEXT NOT NULL,
                password TEXT NOT NULL,
                email TEXT NOT NULL PRIMARY KEY
            );
        ''' 
    )

    is_api_available = HttpSensor(
        task_id='is_api_available',
        http_conn_id='user_api',
        endpoint='api/'
    )

    extracting_user = SimpleHttpOperator(
        task_id='extracting_user',
        http_conn_id='user_api',
        endpoint='api/',
        method='GET',
        response_filter=lambda response: json.loads(response.text),
        log_response=True
    )

    processing_user = PythonOperator(
        task_id='processing_user',
        python_callable=_processing_user
    )

    storing_user = BashOperator(
        task_id='storing_user',
        bash_command='echo -e ".separator ","\n.import /tmp/processed_user.csv users" | sqlite3 /home/airflow/airflow/airflow.db'
    )

creating_table >> is_api_available >> extracting_user >> processing_user >> storing_user

Conclusion

To understand the concept, we have defined a simple and parallelism-free data pipeline. Much more complex, parallel tasks also can be created using Airflow. Thank you for reading!

Screenshots

Show Screenshots

References

KrakenD Monitoring with Grafana

Ahmet Can Aydemir — Mon, 17 Jan 2022 12:16:21 +0000

KrakenD is an ultra-performant open-source Gateway that can transform, aggregate, or remove data from multiple services, with linear scalability.

I will explain how to monitor the status of KrakenD services with the Grafana dashboard.

Instead of creating a dashboard from scratch, we can use one of the pre-configured dashboards available on grafana.com. In this article, we will use the KrakenD dashboard with ID 5722, prepared by dlopez. If you want, you can customize the dashboard according to your needs.

By default, this dashboard provides us with the following metrics:

Requests from users to KrakenD
Requests from KrakenD to your backends
Response times
Memory usage and details
Endpoints and status codes
Heatmaps
Open connections
Throughput
Distributions, timers, garbage collection, etc.

Prerequisites

KrakenD

InfluxDB Setup

The dashboard we will use uses InfluxDB to read metric data. You can easily run InfluxDB with Docker.

docker run -p 8086:8086 \ 
  -e INFLUXDB_DB=krakend \ 
  -e INFLUXDB_USER=myusername -e INFLUXDB_USER_PASSWORD=mypassword \ 
  -e INFLUXDB_ADMIN_USER=admin -e INFLUXDB_ADMIN_PASSWORD=myadminpassword \ 
  -d --name=influx-1.8 \ 
  influxdb:1.8

docker exec -it influx-1.8 /bin/bash

Grafana Setup

If Grafana is not installed, you run it with Docker. After the following command, Grafana will be running at http://localhost:3000.

docker run \ 
  -d \ 
  -p 3000:3000 \ 
  --name=grafana \ 
  grafana/grafana

KrakenD Configuration

Add the following configuration to your krakend.json at the root level. After this, your KrakenD metrics will start saving to InfluxDB at <your-influx-db-server-ip>:8086.

{
  "version": 2,
  "extra_config": {
    "github_com/letgoapp/krakend-influx":{
        "address":"http://<your-influx-db-server-ip>:8086",
        "ttl":"25s",
        "buffer_size":0
    },
    "github_com/devopsfaith/krakend-metrics": {
      "collection_time": "30s",
      "listen_address": "127.0.0.1:8090"
    }
  }
}

Importing the Grafana Dashboard

Go to the browser and open http://localhost:3000. Use admin for both username and password
Click Configuration from the side menu and find the button to add the data source. Select InfluxDB as the database and fill in the details you provided when starting InfluxDB:

Query Language: InfluxQL
URL: http://localhost:8086
Access: Browser
Database: krakend
User: admin
Password: myadminpassword
HTTP Method: GET

To import the dashboard: Click the + icon in the side menu from the Grafana interface and then click Import. Select Import via Grafana.com. Enter 5722 as the ID and click Load. Your dashboard is ready, enjoy it.

References

Elasticsearch Snapshots to Minio

Ahmet Can Aydemir — Thu, 23 Dec 2021 18:23:05 +0000

Minio is a high-performance, self-hosted AWS S3 compatible object storage solution.

I will explain how we can store Elasticsearch snapshots in Minio.

Prerequisites

Minio
Elasticsearch

Creating Bucket

First, we need a bucket to store Easticsearch snapshots.

You can create a bucket at Minio UI Console. Click Buckets from the left menu and click Create Bucket.

Creating a Service Account

A service account is also required to establish a Minio connection with Elasticsearch.

You can create a service account at Minio UI Console. Click Service Accounts from the left menu and click Create Service Account.

Save the Secret Key and Access Key, we will use this information during the plugin installation.

Elasticsearch S3 Repository Plugin Installation

Since Minio is compatible with Amazon AWS S3, we will use the S3 repository plugin. To install the plugin, connect to all Elasticsearch nodes and follow the steps below.

First of all, install repository-s3 plugin with following command:

/usr/share/elasticsearch/bin/elasticsearch-plugin install repository-s3

An S3 client named default will be created after the plugin is installed. Actions to be taken can be performed by this client. We will define the access_key and secret_key information to the default client for accessing Minio.

/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key
# Waiting for secret key

/usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key
# Waiting for access key

After this stage, the Elasticsearch service should be restarted.

systemctl restart elasticsearch

Adding Elasticsearch Repository

A repository named my-minio-repository can be created by sending the following request:

PUT _snapshot/my-minio-repository {
   "type": "s3",
   "settings": {
      "client": "default",
      "bucket": "elasticsearch-snapshots",
      "endpoint": "minio.example.com",
      "path_style_access": "true",
      "protocol": "http"
   } 
}

Manual Snapshot

A snapshot named my-first-snapshot can be taken manually by sending the following request:

PUT /_snapshot/my-minio-repository/my-first-snapshot?wait_for_completion=true

Taking Snapshots with Policy

If desired, the snapshot process can be automated using snapshot policy. For example, the policy below will save snapshots of all indexes to the Minio repository at 07:30 every day.

PUT _slm/policy/daily-snapshots
{
    "name": "<daily-snapshot-{now/d}>",
    "schedule": "0 30 7 * * ?",
    "repository": "my-minio-repository"
}

Restoring snapshots

The my-first-snapshot backup can be restored with the following request:

POST /_snapshot/my-minio-repository/my-first-snapshot/_restore

References

Snapshot and restore

GitLab CI: Update MR With Jira Issue Id

Ahmet Can Aydemir — Thu, 02 Dec 2021 16:50:36 +0000

In my previous article, we enforced branch names and commit messages to certain rules. Now I will go a step further. When an MR is opened, if there is a Jira issue id (eg. ISSUE-0000) in the title, we will pull the Jira title and description and update MR. CI will fail if there is no Jira Issue Id in the title.

To do this, we can briefly follow these steps.

Check if there is an issue id in the MR title. Otherwise, return fail exit code. We will use regex for this. $CI_MERGE_REQUEST_TITLE will give us the MR header.
Extract Issue Id and pull data from Jira API. When you make a GET request to the https://jira.<your-hostname>/rest/api/2/issue/ISSUE-0000 Jira address, Issue details will be returned to you as JSON.
Take the summary and description parts from the JSON data returned by Jira API with jq and assign them to variables.
Update MR's title and description using GitLab API.

Curl ve jq Setup

We need curl for our API requests and jq to parse JSON.

apk add --update curl && apk add --update jq && rm -rf /var/cache/apk/*

Issue Id Check

REGEX_ISSUE_ID="^(ISSUE-[0-9]+)"
ISSUE_ID_IN_MR=$(echo "$CI_MERGE_REQUEST_TITLE" | grep -o -E "$REGEX_ISSUE_ID")
[[ -z "$ISSUE_ID_IN_MR" ]]; then exit 1; fi

Jira Issue Parsing

We store new lines (eg. \n) as characters in DESCRIPTION_JSON variable because newlines are the problem when sending curl requests.

JSON=$(curl -s -u <jira-username>:<jira-password> -X GET -H "Content-Type:application/json" "https://jira.<your-hostname>/rest/api/2/issue/$ISSUE_ID_IN_MR")

SUMMARY=$(echo "$JSON" | jq -r ".fields.summary")
DESCRIPTION=$(echo "$JSON" | jq -r ".fields.description")
DESCRIPTION_JSON=$(jq --null-input --compact-output --arg msg "$DESCRIPTION" '$msg')

Updating Merge Request

You need a private token for this check GitLab documentation and learn how to do that.

curl -v \
--data "{\"title\": \"${ISSUE_ID_IN_MR} ${SUMMARY}\"\"description\": ${DESCRIPTION_JSON}}" \
--fail \
--header "Content-Type:application/json; charset=utf-8" \
--header "PRIVATE-TOKEN:<private-token>" \
--output "/dev/null" \
--request PUT \
--show-error \
--silent \
--trace-ascii "/dev/stderr" \
--write-out "HTTP response: %{http_code}\n\n" \
"${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_request$CI_MERGE_REQUEST_IID"

Result

When we combine all this into one GitLab CI yml file it will look like this.

check_mr:
  image: alpine
  stage: test
  script:
    - apk add --update curl && apk add --update jq && rm -rf /var/cache/apk/*
    - IFS=$"\n"
    - REGEX_ISSUE_ID="^(ISSUE-[0-9]+)"
    - ISSUE_ID_IN_MR=$(echo "$CI_MERGE_REQUEST_TITLE" | grep -o -E "$REGEX_ISSUE_ID")
    - if [[ -z "$ISSUE_ID_IN_MR" ]]; then exit 1; fi 
    - JSON=$(curl -s -u <jira-username>:<jira-password> -X GET -H "Content-Type:application/json" "https://jira.<your-hostname>/rest/api/2/issue/$ISSUE_ID_IN_MR")
    - SUMMARY=$(echo "$JSON" | jq -r ".fields.summary")
    - DESCRIPTION=$(echo "$JSON" | jq -r ".fields.description")
    - DESCRIPTION_JSON=$(jq --null-input --compact-output --arg msg "$DESCRIPTION" '$msg')
    - |
      curl -v \
      --data "{\"title\": \"${ISSUE_ID_IN_MR} ${SUMMARY}\", \"description\": ${DESCRIPTION_JSON}}" \
      --fail \
      --header "Content-Type:application/json; charset=utf-8" \
      --header "PRIVATE-TOKEN:<private-token>" \
      --output "/dev/null" \
      --request PUT \
      --show-error \
      --silent \
      --trace-ascii "/dev/stderr" \
      --write-out "HTTP response: %{http_code}\n\n" \
      "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/$CI_MERGE_REQUEST_IID"
    only:
      - merge_requests

You should add this file you created to your GitLab project as a CI file. After all these processes, your MR titles and descriptions will now be much more organized and understandable!

Git Hooks: Enforce Commit Message and Branch Name

Ahmet Can Aydemir — Wed, 01 Dec 2021 19:05:48 +0000

If you want to standardize things among your team, you may need to enforce strict rules. Some examples of rules might be for git are:

Branches must be created as ISSUE-0000/feature-name.
All commit messages must start as ISSUE-0000 (Jira issue id).

In this post, I will show you how we can add the two rules I mentioned using git hooks.

There are multiple git hooks that work in different stages. We will use two client-side hooks for our rules.

pre-commit: This hook is invoked by git-commit, and can be bypassed with the --no-verify option. We will use this hook for checking the branch name before committing.
commit-msg: This hook is invoked by git-commit and git-merge, and can be bypassed with the --no-verify option. It takes a single parameter, the name of the file that holds the proposed commit message. We will use this hook for checking the commit message. And we will modify the commit message if needed.

Creating Hooks Folder

Go to your home directory and create a folder named hooks. And create hook files.

mkdir ~/hooks
cd ~/hooks

touch pre-commit
chmod +x pre-commit

touch commit-msg
chmod +x commit-msg

Restricting Branch Names

We will accept branch names as ISSUE-0000/feature-name but we have to consider also other branch names like main, develop, release etc. For this, we can use the following regular expression.

REGEX_ISSUE_ID="^(ISSUE-[0-9]+\/([a-zA-Z0-9]|-)+|develop|main|release"

We also need the branch name:

git rev-parse --abbrev-ref HEAD

That's it! We have regex and we have branch name. We will check for branch name starts with ISSUE-0000/feature-name or develop or main or release. If our check fails we will echo our error message and exit with exit code 1.

pre-commit

#!/bin/bash

REGEX_ISSUE_ID="^(ISSUE-[0-9]+\/([a-zA-Z0-9]|-)+|develop|main|release"

ISSUE_ID_IN_BRANCH=$(echo $(git rev-parse --abbrev-ref HEAD) | grep -o -E "$REGEX_ISSUE_ID")

if [[ -z "$ISSUE_ID_IN_BRANCH" ]]; then
    echo "[pre-commit-hook] Your branch name is illegal. Please rename your branch with using following regex: $REGEX_ISSUE_ID"
    exit 1
fi

Restricting Commit Messages

Steps are similar to restricting branch names. But if there is no issue id in the commit message, we will look at the branch name and try to get it from there. If the issue id is also not in the branch name, we will throw an error.

REGEX_ISSUE_ID="^(ISSUE-[0-9]+|Merge|hotfix)"

Our first argument $1 gives us the commit-msg file. We can read the commit message with the following command:

cat "$1"

We have regex and we have to commit message name, and from the last step, we know how to get branch name. We will check for commit message starts with ISSUE-0000 or Merge or hotfix.

If the commit check fails we will check our branch name and parse the issue id and insert to commit message.

If our check fails again we will echo our error message and exit with exit code 1.

commit-msg

#!/bin/bash

REGEX_ISSUE_ID="^(ISSUE-[0-9]+|Merge|hotfix)"
ISSUE_ID_IN_COMMIT=$(echo $(cat "$1") | grep -o -E "$REGEX_ISSUE_ID")

if [[ -z "$ISSUE_ID_IN_COMMIT" ]]; then
    BRANCH_NAME=$(git symbolic-ref --short HEAD)
    ISSUE_ID=$(echo "$BRANCH_NAME" | grep -o -E "$REGEX_ISSUE_ID")

    if [[ -z "$ISSUE_ID" ]]; then
        echo "[commit-msg-hook] Your commit message is illegal. Please rename your branch with using following regex: $REGEX_ISSUE_ID"
        exit 1
    fi

    echo "$ISSUE_ID | $(cat "$1")" > "$1"
fi

Usage

Go to the project folder and run the following command to use your hooks. You need to make this for every project that you want to use hooks.

cd <your-team-repository>
git config core.hooksPath ~/hooks

Sharing With Team Members

You need to share your ~/hooks folder. You can share with a git repository or you can send the folder directly to your team.