Introduction
Creating your own Python package is one of the most satisfying steps as a developer/engineer. For me, it all started with a goal to simplify ETL (Extract, Transform, Load) operations using tools I already used for ETL pipeline development, pandas, requests, and SQLAlchemy. In this post, I'll walk you through how I created eazyetl, a lightweight and modular ETL package for data projects, and published it to both TestPyPI and the official PyPI.
Like many data analysts and engineers, I often repeat similar steps or write many lines of codes for reading from APIs, cleaning data, and loading it into databases or files. Rather than rewriting the same code, I built a simple, reusable library with:
Extract: Get data from CSV, JSON, APIs, or databases
Transform: Clean, rename, convert, and format your data
Load: Export to CSV, JSON, Excel, or PostgreSQL
All of it wrapped in a Pythonic, static-method class structure.
Sample Usage.
Installation
pip install eazyetl
Extracting data from various sources.
from eazyetl import Extract, Transform, Load
df = Extract.read_csv('data/data.csv')
df_api = Extract.read_api('https://www.fantasypremierleague.com/api')
df_db = Extract.read_db(database='employees', user='postgres', password='postgressuperuser', host='localhost', port='5432')
Transforming and cleaning of data
df = Transform.drop_na(df, columns=["name", "price"])
df = Transform.to_datetime(df, "release_date")
df = Transform.rename(df, columns={"old_name": "new_name"})
Loading data into various data sources.
Load.load_csv(df, "cleaned_data.csv", overwrite=True)
Load.load_to_excel(df, 'weather_data.xlsx', overwrite=False)
Load.load_to_db(df, name="salaries", url="postgresql://user:pass@localhost:5432/mydb")
Testing and uploading package to PyPI.
a. Building package and Testing
I created a pyproject.toml file which contained project details, setup and build settings for my package and installed build, setuptools and wheel which will be vital for package build and testing.
pip install build setuptools wheel
After installing the required packages, I built my package for local testing.
python -m build
This process created dist/ and eazyetl.egg.info folders. The dist/ folder contains a eazyetl-0.1.5-py3-none-any.whl file which we will use for local package testing. To install the package locally on your machine, run:
pip install "/Users/user/projects/eazyetl-0.1.5-py3-none-any.whl"
# pip install <filepath to .whl file>
Note: Make sure that you are installing packages inside your project's dedicated virtual environment to avoid conflicts
The file is now installed into another project and ready for use!
Sample usage:
b. Uploading to PyPI.
Install the twine package which will help us to upload our package to PyPI.
pip install twine
Then, create an account on PyPI, enable two-factor authentication, create an API token, name the project your package name and copy the API token.
In your terminal, in the same root diretory as dist/ and package_name.egg.info run
twine upload dist/* --verbose
--verbose is used to log messages into the terminal for efficient error logging and debugging using twine
You will be prompted for your API token as a password, and the package will be uploaded to PyPI for global installation and can be installed using pip install.
Conclusion.
Building my first Python package easyetl helped me in understanding the fundamentals of OOP in Python for example encapsulation, inheritance, package development and deployment to PyPI.
If you would want to check out the code on GitHub, you can find it here. Any suggestions, contributions and collaborations will be highly appreciated.
Check out the package on PyPI here.
Top comments (1)
This is nice