Forem: Umesh Kumar Dhakar

Trigger Azure Data Factory Pipeline from Event Grid (Using Webhook Endpoint)

Umesh Kumar Dhakar — Wed, 12 Apr 2023 09:25:30 +0000

In this post, I will explain how we can use Azure Event Grid Topics to trigger Data Factory pipeline. We will be using Access key for authentication and POST request for publishing a topic to Event Grid. I have also attached a snapshot for each step which can help to understand the steps easily.

Prerequisite:
Microsoft.EventGrid should be registered under the subscription.
To confirm this, go to your subscription, then click Resource Provider.
Search for Event Grid, if Microsoft.EventGrid status is Registered, we are good to go else you need to select Microsoft.EventGrid and click on the register button.

We will create 2 Azure resources: Event Grid Topic and Azure Data Factory.

Create Event Grid Topic Resource

Here, I created a resource with name eg-tp-useast-dev-01

While creating the resource, make sure that Enable access key and Shared Access Signature (SAS) is enabled.
Remaining configurations you can keep as default.

Once Event Grid Topic Resource is deployed, go to the resource. Under the Overview tab, you can see there is no subscription found at this moment. Once we configure the ADF trigger, we will see a subscription here.

Create Azure Data Factory resource and launch the Studio

I created a Data Factory resource df-useast-dev-01.

Go to Author Tab and create a sample pipeline. For simplicity, I just added a Wait activity inside a sample pipeline PL_Sample.

Once you created the pipeline, Publish it.

Create a custom trigger

Create a custom trigger for the pipeline PL_Sample which we just created. Click on Add Trigger and create a new trigger with the below details:

give trigger name
select type as Custom events
select Subscription
you should see all available event grid topics, select the topic which we just created (eg-tp-useast-dev-01)
type the subject: tp-subject-01
type the event type: event-type-01
check Start trigger on creation
click OK
publish Trigger

subject and event type values are used to filter the events.

NOTE: If Event Grid topic is not visible in the Event Grid topic name drop-down, follow these 4 steps:

Open Event Grid topic resource in a new tab, Go to the overview tab, click on JSON view.

Copy the Resource ID.

Again, switch back to the ADF trigger page and switch to Enter manually for Account selection method and paste the Resource ID in the scope field.

Click on Ok and Publish the trigger.

Once Trigger is published successfully, go to Event Grid topic, you should see 1 event subscription.

So far we have configured a ADF pipeline with the Event Grid topic.

Publish an event to the Event Grid Topic

Now we will publish an event to the Event Grid Topic by sending a POST request. I am using Postman to send requests.

In order to send a request, we need an endpoint and an access key of the topic.

Go to Overview tab and copy the Topic Endpoint value
Go to Access Keys and copy a Key

Create A POST request

URL: Use the Topic Endpoint as request URL
header: Add a header in the request with key aeg-sas-key and access key as value.
body: Add the below body in JSON format



[
  {
    "id": "I01",    
    "eventType": "event-type-01",
    "subject": "tp-subject-01",
    "data":{
      "key1": "val1",
      "key2": "val2"
    },
    "eventTime": "2023-04-06T11:26:07+05:30",
    "dataVersion": "1.0"
  }
]

id: id of the event (must be unique for each request)
eventType: same as eventType mentioned while creating ADF trigger
subject: same as subject mentioned while creating ADF trigger
data: parameters that can be passed to ADF trigger (explained at the end)

Click on Send, If the status is 200 OK, ADF pipeline should trigger.
Go to ADF monitor tab > Pipeline Runs > Triggered.
You should see the Pipeline has been triggered by the custom event trigger

As I mentioned earlier that we can pass the parameter from event topic to ADF pipeline. For that follow below steps:

Create a pipeline parameter (here I created par_input_file)
Edit the trigger and pass @triggerBody().event.data.key1 in Trigger Run Parameters.

When pipeline gets triggered, it should initialize the pipeline parameter with event.data.key1 value.

Note: Resources should be deleted after testing in order to avoid costs.

Thank you for reading this post. Hope it helped you understand and implement this event-driven pipeline execution. I could not find all these steps in a single post, so thought to consolidate all the steps into one post and share them with other developers. Please let me know if you stuck in between while implementing this, I would be happy to help. Happy learning.

How to Create and Publish Python Package.

Umesh Kumar Dhakar — Mon, 28 Jan 2019 08:59:00 +0000

Hello World!
Python has a simple way of creating and publishing packages. Python Package Index is a package repository which maintains all the published package.
Once I was curious about that packaging thing in python, So decided to learn this stuff. Along with that learning journey, I maintained a note and wrote necessary steps. I refined that note and posted here so that it may help someone who is looking into it. If you want to learn this stuff, this post can help you. You need a PyPI account to publish a package, So before starting, please Sign Up on PyPI, then proceed with the below process.

Create a directory for the package.
mkdir demo_car
Create file __init__.py in the directory.
touch demo_car/__init__.py
(This file represents that the directory which contains it is a python package.)
Write 'name = "<package-name>"' in file.
echo 'name="demo_car"' >> demo_car/__init__.py
Create python scripts for the package.
touch demo_car/engine.py
Let's write some code in engine.py script. Open an editor (I prefer Nano), write some python code or paste below sample code and save the file by Ctrl+o.
nano demo_car/engine.py

def start():
    print("Started!")

def stop():
    print("Stopped!")
~~~~
> If no editor is available then install nano with the following commands.  
> `apt-get update`
> `apt-get install nano`
> `export EDITOR=nano`

- Package files structure should be like this.

.

├── demo_car

│ ├── engine.py

│ └── init.py

├── LICENSE

├── README.md

└── setup.py
~~~~

Purpose of these files.
- setup.py Build script for packaging.
- LICENSE Every package should have a License. We use MIT license here.
- README.md This file contains a description of the package which will be shown on PyPI repository.
Create these three files.

touch setup.py LICENSE README.md
Open these files in the editor and update them.
- For setup.py file

import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="demo_car",
    version="0.0.1",
    author="Umesh Kumar Dhakar",
    author_email="kumar886umesh@gmail.com",
    description="A demo_car package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/umeshdhakar/sample-python-package",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
) 
~~~~  

-     For LICENSE file

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

~~~~

For README


## Demo_car
This is a sample package to learn the steps of creating and publishing package.
~~~~

- Now we are ready with all the necessary files so without waiting we are going to packaging it.  

- Install virtual environment package.
  *(If you already have then skip this step.)*
  `pip install virtualenv`

- Create and activate the environment.
  `virtualenv -p python3 .`
  `source bin/activate`

- Install or Update the setuptools and wheel packages in the environment.
  `pip install --upgrade setuptools wheel`

      *setuptools and wheel are the packages which use setup.py file to build 
      package.*

Now all Set! 😊, You are going to create the package, Are you excited? 😉

- Fire below command to build package.

    `python3 setup.py sdist bdist_wheel`

Hopefully, the package should be built and you can see package's compressed file in `dist` directory beside setup.py file.
Well Done my friend(👍), You have created It.
***
**Installing the created package.**

- Create a new environment at some other location and activate it.
  *(Now you know how to do this).*

- Copy that zip file in the new environment.
  `pip install <package>`
  (`pip install  demo-car-0.1.tar.gz`)

*Run* `pip list` *to show the installed package in the activated environment. Your package should be enlisted here.*

Congratulations! (😊) You have installed your package.
***
**It's time to publish the Package to PyPI so that it is publically available.**

- We need to install twine package to upload it but before that go to the path where `setup.py` exists and Activate the first environment you created for packaging then fire up the command to install or update twine.
  `pip install --upgrade twine`

- Final command to send it to **PyPI**.
  `twine upload dist/*`

It will prompt you for the account credentials, 
So enter your username and password, then it will start uploading your package.

**Congratulations! (🎉 ) Dear, You did it.**
Now your package should be available on the [PyPI](https://pypi.org/)
([Reference]( https://packaging.python.org/tutorials/packaging-projects/))

Thank you for reading 😊.
These are the minimum steps to create a package which I know. Please refer official documentation if you want to explore more 💡. 
Feel free to ask query, I am happy to help. Have you found any useful information which I missed here? then please comment below.

Basic commands to start Git and GitHub

Umesh Kumar Dhakar — Tue, 08 Jan 2019 10:43:33 +0000

Hello Folks, Git and GitHub are very useful tools for people like us.
It's a good habit to make notes of anything we are learning and there are many online tools which make note taking easier.
notepad.pw and simplenote are some clean and simple tools for note-taking. Happily, I have also developed this good habit and created a document while learning Git.
Some good Dev.to posts motivated me to pull out those documents and write them here also, which might help someone here and definitely me also.
So here are the basics commands to start Git and host source code on GitHub(remote repository)

Install git.
sudo apt-get install git

Create a git repository locally in a directory.
git init

Set your globle user details.
git config --global user.name <name>
git config --global user.email <email>

Create a README file to store the description of the repository.
touch README.md

Create a file to ignore some files and directories from the repository.
(write the relative path of files and directory, wild cards can also be used here)
touch .gitignore

Create project's files or paste the existing files in current directory.

Add changed files for committing.
git add <filename>
or add all files at once
git add .

Check the status of the repository.
git status

It's time to commit the first changes locally.
git commit -m "initial commit" <file>

Everything we did is stored locally, Now we will link the local repository to a remote repository.

Create a fresh repository on GitHub (now we can create unlimited private repos on GitHub also, Thanks GitHub for this new year gift 😊).
Click that green button 'Clone or download' and copy the URL of the repository.

Link the remote repository with the local repository.
git remote add origin <repo_url>

Push the changes to the master branch of remote repository.
git push -u origin master

One extra command. 💡
If you accidentally commited changes or want to undo the commit these commands can save you time.
git reset --hard HEAD^
git push -f

Thank you for reading 😊.
There are much more commands out there. These are only the basics which can help in getting started. Please let me know if have some doubts, I am happy to help you out.

Auto backup of Databases in Postgres Container.

Umesh Kumar Dhakar — Mon, 10 Sep 2018 07:11:11 +0000

1. Create A file in Container.
touch /path/<file>
(ex. touch /data/bkp_cmd.sh)

2. Write Backup command to that file.
echo "pg_dumpall -U postgres -f /path/file" >> <file>
(ex. eg. echo "pg_dumpall -U postgres -f
/data/files/backup_date +%Y%m%d.sql" >> bkp_cmd.sh )

3. Make file executable.
chmod 777 <file>
(ex. chmod 777 /data/bkp_cmd.sh )

4. Append cron file.
crontab -e
00 00 */15 * * /data/bkp_cmd.sh >> /data/logs/cron_logs.txt 2>&1
You can generate cron command here.
if no editor is installed then
Append command to cron file via
(crontab -l && echo "00 00 */15 * * /data/bkp_cmd.sh >> /data/backup_cron_logs.txt 2>&1") | crontab -

5. Check whether cron service is running or not.
service cron status
If it it is not running, start it by command
service cron start

5. Wait and check the backup file at specified time.

Thanks for reading! 😊
Please leave a comment if you have any suggestions or you have any doubt.

Share file between Host System and Docker Container.

Umesh Kumar Dhakar — Tue, 04 Sep 2018 10:05:11 +0000

1. Create a folder in Host System.
C:\\<directory-name>
(eg. C:\\Tunnel)

2. Add this folder in Shared Folder list of VM.
Go to : Machine > Settings > Shared Folder
(Add folder here.)

3. Create folder in VM.
mkdir /<directory-name>
(eg. mkdir /vm_tunnel)

4. Mount Shared Folder in VM directory.
mount -t vboxsf <host-directory-name> /<vm-directory-name>
(eg. mount -t vboxsf Tunnel /vm_tunnel)

5. Test sharing by creating file in Tunnel or /vm_tunnel and check in
/vm_tunnel or Tunnel.

6. Share file between Host and Conatiner.
docker cp <container>:/path/to/file/in/container /path/of/vm-direcory/where/host-folder/is/mounted
(eg. docker cp foo:/data/bar.txt /vm_tunnel/)

Thanks for reading! 😊

How to build a docker image which can restore databases whenever a new container is created from it

Umesh Kumar Dhakar — Wed, 29 Aug 2018 10:18:22 +0000

1. Run a postgres container.
docker container run --name <name> --detach -p 5432:5432 -e POSTGRES_PASSWORD=<my-pass> postgres
(ex: docker container run --name store --detach -p 5432:5432 -e POSTGRES_PASSWORD=my-pass postgres)

2. Create a directory in container to copy backup file.
docker exec <container_name> mkdir /<dir_name>
(ex: docker exec store mkdir /data)

3. Copy a backup file in container from Host or from another container.
docker cp /path/of/backup_file/database_bkp.sql <container_name>:/path/to/copy/file
(ex: docker cp /volume1/database_bkp.sql store:/data)

4. Install an editor in container (nano or vim etc...).
apt-get update apt-get install vim export EDITOR=vim

5. Create file in /docker-entrypoint-initdb.d directory
touch /docker-entrypoint-initdb.d/<file_name>
(ex: touch /docker-entrypoint-initdb.d/restore_db.sh )

6. write restore script in restore_db.sh

#!/bin/bash set -e psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" -f /data/database_bkp.sql

(if you dumped only a single database using pg_dump then please mention database name also in script)

psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" -d <database-name> -f /data/database_bkp.sql)

7. Make file executable
chmod 755 /docker-entrypoint-initdb.d/restore_db.sh

8. Build a new image from running container.
docker commit <running-container> <new-image>
(ex: docker commit store store_new)

9. Verify that image has built properly.
docker image ls

10. Run container from newly build image on a different port.
docker container run --name <name> --detach -p 4432:5432 -e POSTGRES_PASSWORD=<my-pass> <new-image>
(eg: docker container run --name store_postgres --detach -p 4432:5432 -e POSTGRES_PASSWORD=my-pass store_new)

This image has a restore file in it, a database is created when a new container is created from this image

Thanks for reading! 😊, This is my first post. Please let me know if you have any issue and also please write feedback.