Forem: Thomas Epelbaum

EcoAct releases ecodev-app!

Thomas Epelbaum — Tue, 03 Sep 2024 10:31:49 +0000

In line with its recent contributions to the Open Source world, EcoAct is making its ecodev-app (and its associated documentation available to all developers wishing to create complete and secure web applications in Python.

Ecodev-app is the culmination of over a year's work by Thomas Epelbaum, Olivier Gabriel, Amaury Salles, Yoann Diep, Dorian Kodelja and Japheth Yates (all members of EcoAct's Climate Data Analytics (CDA) team) to facilitate the development of websites in Python from scratch.

As explained in our previous publication on open source, developers perpetually build on the volunteer work of their generous predecessors. The various missions undertaken by CDA would never have seen the light of day without various tools to manage site creation, interaction with a backend engine, and database creation.

CDA decided to add its contribution to this open source environment, in line with EcoAct's values of sharing.

Ecodev-app benefits from all the previous work already carried out and shared by CDA in 2024: a technical library for backend tasks (the hidden part of a website, its engine), a library for reading and writing files stored in the cloud, a library for turning your code into a secure web site deployed on the Internet and a library of graphic components for websites.

Thanks to Ecodev-app and its ecosystem, CDA has been able to develop over thirty applications in a very short space of time.

We hope that all Python developers will be able to benefit from this work and create modern, secure websites in record time. Feel free to reuse, copy, and even contribute!

CDA's ambition is to continue contributing to the open source universe in the future, whether through the publication of scientific articles, white papers or other open source libraries. We might also produce a video detailing the setup from scratch of Ecodev-app if enough people show interest.

To find out more, don't hesitate to send your questions to Thomas Epelbaum, Head of python development and Machine Learning, CDA, via this form.

EcoAct has released it's third open source library!

Thomas Epelbaum — Tue, 28 May 2024 16:00:01 +0000

If you've ever had to handle a large volume of data, chances are that you've had to interact with boto3 / azure.storage.blob.

Did you love the experience? We didn't totally either. We needed some high-level functions that could be relied on to work 98% of the time, and that allowed us to be agnostic in terms of which cloud provider/object storage protocol we're working on.

ecodev-cloud (and it's associated documentation) does just that, with some (relatively... 😊) cloud-agnostic load/save/search/copy/move methods.

By setting some env variables and adding ecodev-cloud to your usual requirements, it should be easier to interact with cloud storages (this was the case for us, and we surely hope that it will be the case for you too!).

In any case, do not hesitate to take what you need, read the code, learn from what we did (and share your own insights, we would be delighted )...

The local dev setup relying on minio and/or azurite might prove useful to you as well (we found very few relevant information on azurite).

Special thanks to Olivier Gabriel , Amaury Salles , Yoann Diep , Dorian KODELJA and Japheth Yates for the support!

Happy coding from the EcoAct CDA Python team! 😊

How to quickly setup an infra [Part 1]

Thomas Epelbaum — Wed, 20 Mar 2024 17:37:05 +0000

This is the first part of a serie where I would like to present a simple way to setup an infra for either

an ambitious personal project
a small SME/a startup in it's infancy.

Why this serie?

Good question! 😁 You could go with the heroku-like approach, or the hyperscaler one. I am sure there are tons of other tutorials out there to help you do so.

But the approach that I am going to present is ultra cheap (only 1 VM needed) and has some pedagogical merits 😊.

Plus in an era where companies are more and more moving away from the cloud, you never know, it might proves useful 😋.

What are we aiming at?

By the end of this serie, you should have a stack looking like so

This stack includes:

Traefik as a reverse proxy
PostgreSQL as a sql db and pgAdmin as its admin interface
Bookstack and its associated db for documenting your product
keycloak and traefik forward-auth to protect apps that cannot implement authentication without keycloak
minio back and front to stored/exchange data
Ofelia as a job scheduler
Private pypi to store your precious private python libraries between collegues.
dozzle for log aggregation
uptime-kuma for monitoring and alerting

I'll talk (python) web app setup in another subsequent serie 😊.

Where to start?

Well, thanks to the work we did at EcoAct, we got you covered:

This documentation page and the ones related are the content I am going to present in this serie. This post corresponds to this page.
The associated code can be found in this open source (MIT licence) repo.

What are we going to learn today?

Setting up a VM with one simple script! Yes I know, it surely could be done with ansible, but fur such a simple use case it seems overkill (plus one has to install ansible before being able to use it, by contrast with bash 😂).

TL DR: clone the repo, launch the setup script. The end! 😂 Now the details.

Actual setup

There are countless blog posts to help you decide which cloud provider is the best for you, and I won't add to this literature here 😊.

I therefore just assume you have a Virtual Machine (VM) Cloud setup with ssh access.

Here I provide a simple bash script to setup a freshly bought VM, assuming an Ubuntu Operating System (this OS choice is important for the firewall setup).

Launching the setup script

After having accessed with ssh to your VM, just git clone ecodev-infra. Then in the main folder, launch

make setup-vm <YOURUSER>

Where <YOURUSER> is the username you used for connecting to the VM. The bash script executed will

Install docker and related components
Add <YOURUSER> to the sudo and docker group of the VM (might need to disconnect and reconnect at the end of the setup for this to take effect)
Setup ufw (uncomplicated firewall) to work with docker (more on that below)
block all but 22, 80 (tcp only) and 443 (tcp only) ports.
Setup a decent history (english resource on the topic) memory size.

🚨 Technical topic incoming 🚨

Why going into all this trouble with ufw?

As explained here way better than I could, the standard ufw setup won't forbid access to docker exposed ports on your VM, even ports explicitely blocked with an ufw rule!!.

This has to do with technicalities related to iptables explained in the linked posts.

To make ufw work with docker, one has to use this (awesome!) solution. This is the reason of the sudo cp after.rules /etc/ufw/after.rules line in the setup.sh script.

You might be as curious as I was and wondering why there is no Open Source docker container responsible for dealing with firewall issues.

To the best of my knowledge, this has to do with the fact that firewell related stuff is really low level and needs to live close to the VM kernel. One example out of the numerous conversations on the topic.

The script in its whole splendor

If you're too curious 😋 to wait to go inspect the EcoAct infra repo (though I still highly encourage you to do so), here is the srcipt

#!/bin/bash

# This is a VM setup for ubuntu only OS
# Provide as argument your username (in order to add you to docker and sudo group)

echo "Installing docker"
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg make
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo   "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
       "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |   sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

echo "Add access rights for current user" $1
sudo usermod -a -G docker $1
sudo usermod -a -G sudo  $1

echo "Setting up ufw to be docker compatible"
sudo ufw enable

echo "Edit ufw rules to be docker compatible"
# ref: https://github.com/chaifeng/ufw-docker?tab=readme-ov-file#solving-ufw-and-docker-issues
sudo cp after.rules /etc/ufw/after.rules

sudo ufw reload
echo "Setup ufw: block all but 22, 80 (tcp only) and 443 (tcp only) ports"
sudo ufw default deny
sudo ufw allow 22
sudo ufw route allow proto tcp from any to any port 80
sudo ufw route allow proto tcp from any to any port 443
echo "Turning ufw log"
# ufw is known to be glutton in terms of memory
sudo ufw logging off

echo "Restarting docker"

sudo systemctl restart docker

echo "Setup decent history size"

echo "export HISTSIZE=100000" >> .bashrc
echo "export HISTFILESIZE=100000" >> .bashrc
echo "export HISTCONTROL=ignoreboth:erasedups" >> .bashrc

echo "All done!"

I like what I read!

That's awesome! If you want to encourage me to continue the serie, do not hesitate to ⭐ ecodev-infra, best way to aknowledge the hard work we put at EcoAct to create this repo.

What's the (hopefully no so the distant) future?

We would like to do a full pythonic equivalent of Tiangolo's work. Of course Tiangolo repo is the place to start if you're new to modern python, but the work we hope to build will rely on dash for the frontend part, to have the full web stack within python! Plus the infra bricks are a little more numerous here, since we want to do more than just we app development (remember: build a full-fledged infra stack for a small structure)

The first brick is there and the associated documentation there. But one step at a time! Let's first setup the infra, then I'll talk python! 🥰

2nd EcoAct OS release: ecodev-infra

Thomas Epelbaum — Tue, 05 Mar 2024 10:04:25 +0000

Writing code is a developer's day-to-day job, but getting that written code to its end-user is quite a different matter!

EcoAct is once again contributing to the open source world, this time with a brick aimed at helping devops 😊.

ecodev-infra is in fact a code base that enables an enthousiastic programmer or a small structure (SME, startup in its infancy) to create a production environment in a few simples actions (for those that master docker and docker-compose).

Available in ecodev-infra :

a reverse proxy (traefik)
a database (postgresql) and its administrator interface (pgadmin)
a job scheduler (cron ofelia)
a private pypi (pypiserver)
internal documentation (bookstack)
a storage server (minio)
SSO application (keycloak)

As well as a procedure for setting up an ubuntu Linux VM securely, and a few cybersecurity tips.

Documentation can be found here! 😊

Please note: this brick is really intended for users who want to quickly iterate and test these new technologies. Once the business model has been established, we strongly recommend that you make the transition to managed services (we talk about this in the documentation).

As with ecodev-core, feel free to use whatever you need!

The next brick will complete the lego construction that we aimed to give to the developer community at the start of the year: a secure dashboard website developed in python dash, integrating the previous bricks, and all usable in less than 5 minutes. Stay tuned! 😊

ecodev-core: low level helpers around FastAPI and SQLModel

Thomas Epelbaum — Tue, 05 Mar 2024 10:01:01 +0000

Hi all! 🙂

my collegues and I recently released ecodev-core . As explained in the associated documentation, we make thorough use of Pydantic/FastAPI/SQLModel in our work and we were ending up repeating the same boiler plate code in our 20+ applications. We just decided to factorize this code in a library, that we are happy to share (MIT licence) to everyone that might find it useful. Install it, copy what you need in your projects... Whatever you see fit 😊

This is the first out of 4/5 libraries/cookiecutter that we plan on releasing this year, the end goal being to have a full python equivalent of Tiangolo stack (with the frontend made with dash). Obviously Tiangolo project is 10.000 times better than ours, but if you do not want to learn react/vue... Stick with python and still produce a decent dashboard application with a decent associated infra, you might want to say tune to our future release!

Part 1 (what this post is all about! 😆) is essentially a set of helper methods around FastAPI and SQLModel which helps us quicken the setup our web-apps, with authentication, logging, monitoring, and other basic database and API interaction work.
Part 2 will focus on an infra in our opinion suitable for a small startup
Part 3 (small) on wrappers around dash components
Part 4 will be an advanced object storage read/write library aimed at being blob/s3 protocol agnostic.
Part 5 will be a cookie cutter app template in the spirit of Tiangolo stack (again, as mentionned, surely 10.000 inferior to Tiangolo work but fully in python), linking all parts together.

Stay tuned!