Forem: Guionardo Furlan

Python Virtual Environment Setup

Guionardo Furlan — Mon, 12 Dec 2022 13:48:04 +0000

Python Virtual Environment Setup

Post created at 2022-12-12 10:30

Setting your virtual environment with another python version.

While I'm writing this post, I'm using Ubuntu 22.04 LTS Jammy Jellyfish, wich has python 3.10.

But, I need to work on a project that runs on python 3.8. To assure that the development environment will be the same to the production environment, I have to setup this. I don't want to use docker in this case.

To start, we need to install required packages:

sudo apt install dirmngr ca-certificates software-properties-common apt-transport-https -y

We will use the launchpad PPA's to get things done easely.

Next, we need to install the GPG keys the repositories.

sudo gpg --list-keys

Output


gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created

Now, wee need to import the GPG key.

sudo gpg --no-default-keyring --keyring /usr/share/keyrings/deadsnakes.gpg --keyserver keyserver.ubuntu.com --recv-keys F23C5A6CF475977595C89F51BA6932366A755776

Output

gpg: key BA6932366A755776: public key "Launchpad PPA for deadsnakes" imported
gpg: Total number processed: 1
gpg: imported: 1

If no errors, you can continue importing PPA. Use the command below to your installed version:

Ubuntu 22.04 LTS Jammy Jellyfish

echo 'deb [signed-by=/usr/share/keyrings/deadsnakes.gpg] https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy main' | sudo tee -a /etc/apt/sources.list.d/python.list

Ubuntu 20.04 LTS Focal Fossa

echo 'deb [signed-by=/usr/share/keyrings/deadsnakes.gpg] https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu focal main' | sudo tee -a /etc/apt/sources.list.d/python.list

Complete the python3.8 installation:

sudo apt update && sudo apt install python3.8 python3.8-venv python3.8-python3.8-distutils python3.8-dev python3.8-dbg

Update pip

python3.8 -m pip install --upgrade pip

Output

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in /usr/lib/python3/dist-packages (22.0.2)
Collecting pip
 pip-22.3.1-py3-none-any.whl (2.1 MB)
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 2.5 MB/s eta 0:00:00
Installing collected packages: pip
Successfully installed pip-22.3.1

Access your project folder on terminal:

cd ~/dev/my_project

Create your virtual environment

I like to use .venv as virtual environment folder name, but you can use venv or whatever you want.

python3.8 -m venv .venv

Checking the folder/files created:

✦ ❯ tree -L 2 .venv
.venv
├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── Activate.ps1
│ ├── pip
│ ├── pip3
│ ├── pip3.8
│ ├── python -> python3.8
│ ├── python3 -> python3.8
│ └── python3.8 -> /usr/bin/python3.8
├── include
├── lib
│ └── python3.8
├── lib64 -> lib
└── pyvenv.cfg

Activating the virtual environment

Visual Studio Code and another editors can automatically detect the virtual environment and activate it. In the terminal you should do this:

source .venv/bin/activate

And now, you can use the python that you need:

❯ python --version
Python 3.8.16

Post-setup

After this, I usually install autopep8 and flake8 for linting/formatting.

pip install autopep8 flake8

Thanks for reading! Leave a comment and click the heart icon!

Creating animated gifs from your terminal

Guionardo Furlan — Fri, 19 Aug 2022 14:35:00 +0000

Creating animated gifs from your terminal

Post created at 2022-08-19 08:36

I need to generate some videos to demonstrate console applications running.

source

Screen casting or capture is boring, and generates large files.

Then, I found two command line tools that do the job.

First, we need to capture the console interation, and this is done using the Terminal Session Recorder asciinema. Installation instructions can be found here.

I'm using Ubuntu, so the installation was easy-peasy.

❯ sudo apt install asciinema
[sudo] senha para guionardo: 
A ler as listas de pacotes... Pronto
A construir árvore de dependências 
A ler a informação de estado... Pronto
Serão instalados os seguintes NOVOS pacotes:
 asciinema
0 pacotes actualizados, 1 pacotes novos instalados, 0 a remover e 0 não actualizados.
É necessário obter 35,0 kB de arquivos.
Após esta operação, serão utilizados 125 kB adicionais de espaço em disco.
Obter:1 http://br.archive.ubuntu.com/ubuntu focal/universe amd64 asciinema all 2.0.2-1 [35,0 kB]
Obtidos 35,0 kB em 7s (4.877 B/s) 
A seleccionar pacote anteriormente não seleccionado asciinema.
(A ler a base de dados ... 435184 ficheiros e directórios actualmente instalados.)
A preparar para desempacotar .../asciinema_2.0.2-1_all.deb ...
A descompactar asciinema (2.0.2-1) ...
A instalar asciinema (2.0.2-1) ...
A processar 'triggers' para man-db (2.9.1-1) ...

For our first test, we can command:

❯ asciinema rec first.cast
asciinema: recording asciicast to first.cast
asciinema: press <ctrl-d> or type "exit" when you're done

After that, you can run your commands normally and all your key strokes and console outputs will be recorded into the first.cast file.

To finish the capture press CTRL+D or type exit. Check the official usage documentation for more options.

Now, we need to generate the animated gif file. The tool to this job is the might agg - asciinema gif generator.

It's a rust application, and you can clone the repository and build it, or you can download a binary file compatible to your environment surfing to the latest release link. Download the correct version to your system, put the executable in some directory you can reach and execute.

If you have rust installed, you can simply run this to install agg:

cargo install --git https://github.com/asciinema/agg

Check if the installation is OK.

❯ agg --version
agg 1.1.0

If your shell has some fancy unicode characters (like mine), you can reference a TTF font to render the gif.
I like "JetBrains Mono", and you can download it from here, or if you like another dev font, you can use.

In my case, I saved the TTF files into a folder (./fonts) to use in next step.

Generate a gif from the first.cast file created .

❯ ./agg --font-dir ./fonts --font-family "agave Nerd Font" first.cast first.gif
31 / 31 [=================================================================] 100.00 % 47.71/s

And now, we have our first.gif file in current folder.

Using local mail as notification tool

Guionardo Furlan — Wed, 04 May 2022 18:50:55 +0000

Using local mail as notification tool

Post created at 2022-05-04 14:25

I used to run commands and develop into linux terminal, and have some scheduled tasks running in the background.

I need to be notified on terminal about some finished processes in a generic form, without depending GUI notifications.

By default, every cron job thats writes to console is sent to user local email. So, local email will be my notification messages repository.

Setup

I used this setup in Ubuntu, but it must works in any distro.

Adding current user to mail group

$ sudo adduser $USER mail
Adding user 'guionardo' to group 'mail' ...
Adding user guionardo to group mail
Done.

Install mailtools, postfix, and mutt

$ sudo apt install mailtools postfix mutt

Send test mail

$ echo "This is a test" | mail "--subject=Test email" $USER@localhost

Test new mail

$ mail
"/var/mail/guionardo": 1 message 1 new
>N 1 Guionardo Furlan qua mai 4 17:41 14/464 Test email
? q
Held 1 message in /var/mail/guionardo

Ok, we have an e-mail into our inbox. Lets read with better tool.

$ mutt

q:Quit d:Del u:Undel s:Save m:Mail r:Reply g:Group ?:Help
 1 O F mai 04 To guionardo@lo ( 2) Test email



---Mutt: /var/mail/guionardo [Msgs:1 Old:1 0,6K]---(threads/date)----(all)---

i:Exit -:PrevPg <Space>:NextPg v:View Attachm. d:Del r:Reply j:Next ?:He
Date: Wed, 4 May 2022 17:41:20 +0000 (UTC)
From: Guionardo Furlan <guionardo@furlan-server>
To: guionardo@localhost
Subject: Test email
X-Mailer: mail (GNU Mailutils 3.7)

This is a test

-O F- 1/1: Guionardo Furlan Test email -- (all)

Setup for notifications

Add this content to your ~/.profile or equivalent file (.bashrc, .zshrc, etc):

MAIL_CHECK_TIME=0
mail_prompt() {
 local pwd='~'
 local MAIL_SECONDS_DIFF=$MAILCHECK

 local MAIL_ELAPSED_SECONDS=$((SECONDS - MAIL_CHECK_TIME))

 [ "$PWD" != "$HOME" ] && pwd=${PWD/#$HOME//~/}

 printf "033]0;%s@%s:%s033\%s" "${USER}" "${HOSTNAME%%.*}" "${pwd}"

 if [[ "$MAIL_CHECK_TIME" -eq "0" || "$MAIL_ELAPSED_SECONDS" -gt "$MAIL_SECONDS_DIFF" ]]; then
 local MAILX="$(mailx 2>/dev/null &)"
 UNREADEN_REGEX="s([0-9]{1,4})sn"
 [[ $MAILX =~ $UNREADEN_REGEX ]] && UNREADEN=$(echo "${BASH_REMATCH[1]}") || UNREADEN=0
 local COUNT=$((UNREADEN))
 local MESSAGE_TEXT="message"
 if [ "$COUNT" -gt "0" ]; then
 if [ "$COUNT" -gt "1" ]; then
 MESSAGE_TEXT="messages"
 fi
 echo "$COUNT unreaden $MESSAGE_TEXT. Run mutt"
 fi
 MAIL_CHECK_TIME=$SECONDS
 fi

}

if [[ $(which mailx) ]]; then
 PROMPT_COMMAND="mail_prompt"
fi

Close and reopen your terminal to load this configuration
Now, when you access the terminal, messages not readden will show a prompt:

1 unreaden message. Run mutt

In your custom scripts, you can send email to your local user and get notified.

HTTP Hello World server for testing purposes (part 1)

Guionardo Furlan — Sat, 12 Mar 2022 13:08:21 +0000

HTTP Hello World server for testing purposes

Post created at 2022-03-12 09:53

I needed a http server, small, minimalistic, just for proxy testing.

There are many options to do this, but I wanted to create my own solution.

A basic golang project, with only native components, generating a docker image with 10MB. Just K.I.S.S. !

The repository for this project is in github.com/guionardo/http_helloworld

All what we need to do to use this tool is run a docker command:

docker run --rm -p 8080:8080 guionardo/http_helloworld:latest

Fast folder iteration in python

Guionardo Furlan — Fri, 21 Jan 2022 00:26:12 +0000

Fast folder iteration in python

Post created at 2022-01-19 07:53

I have a problem to deal here.

I need to iterate over a big tree of folders and files and run process over each file.

In python, we have some options to deal to folders and files.

glob
iglob
os.walk
os.scandir
pathlib.Path

Running this benchmark, we can read some implementation details:

glob

import glob

folder='some_folder/another_folder/**/*'
for file in glob.glob(folder, recursive=True):
 print(file)

some_folder/another_folder/f1/file1
some_folder/another_folder/f1/file1
some_folder/another_folder/f1/file2

Pros

Easy to use, less code to write
Easy to apply filters using masks (fnmatch based)
Returns data as a list

Cons

Time to scan is the worst (more than 11x the quickest method)
Data will be available only after all files/folders been scanned

iglob

import glob

folder='some_folder/another_folder/**/*'
for file in glob.iglob(folder, recursive=True):
 print(file)

some_folder/another_folder/f1/file1
some_folder/another_folder/f1/file1
some_folder/another_folder/f1/file2

Pros

Easy to use, like glob
Same filtering
Returns data as iterator, so you can have your data without need to wait all iteration ending.

Cons

Time to scan is a little better than glob (10x the quickest method)
Time to first file could be mutch better

os.walk

import os

folder='some_folder/another_folder/**/*'

for root, _, files in os.walk(folder):
 for file in files:
 print(os.path.join(root, file))

Pros

Good performance. Second quickest method (less than 2x slower than the quickest)
Time for the first file almost imediattely (better result of all)
Explicit code in loops gives more visibility and control if you need validations or another nasty processes

Cons

More code needed, with nested loops.
If you need to nest os.walk in another os.walk loop, some strange things can ocurr. But probably your code needs some refactoring.

os.scandir

import os
from typing import Generator

folder='some_folder/another_folder/**/*'

def get_files(folder: str) -> Generator:
 with os.scandir(folder) as scan:
 for item in scan:
 if item.is_file():
 yield item.path
 else:
 for subitem in get_files(item.path):
 yield subitem

Pros

Best performance of all
Context based, assure resources are released after processing
Less verbose than os.walk
Easy to implement your custom business rules

Cons

A little more complex. No big deal.

pathlib.Path.rglob

import pathlib

folder='some_folder/another_folder'

for path in pathlib.Path(folder).rglob('*'):
 if path.is_file():
 yield str(path)

Pros

Easy to implement your custom business rules
Returns a iterator
You can have your files soon and don't need to wait for all scan (7x more time than os.scandir)

Cons

Average time to scan (almost 8 times greater than os.scandir)
Memory consumption is 4 times more than the other methods

Some data from tests

System Information

System	Release	Version	Machine	Processor
Linux	5.11.0-46-generic	#51~20.04.1-Ubuntu SMP Fri Jan 7 06:51:40 UTC 2022	x86_64	x86_64

CPU Info

Physical cores	Total cores	Max frequency	Min frequency	Current frequency
4	8	3400	400	1.892

CPU Usage Per Core

0	1	2	3	4	5	6	7	Total
15.6	16.5	10.4	14.1	15.2	14.4	15.2	12.4	14

Memory Information

Total	Available	Used	Percentage
15.51GB	4.77GB	9.56GB	69.2%

SWAP

Total	Free	Used	Percentage
15.26GB	3.28GB	11.98GB	78.5%

 Creating sample files
 + Creating folder ./test_files
 + Creating files LEVELS=6 FOLDER_COUNT=6 FILE_COUNT_BY_FOLDER=20
 + Created 933120 files in 0:02:08.928680

* Running iterators: GlobFolderIterator IGlobFolderIterator OSWalkFolderIterator ScanDirIterator PathLibFolderIterator

MEMORY USAGE

Iterator	RSS	VMS	DATA
IGlobFolderIterator	111280128	96468992	96468992
OSWalkFolderIterator	111280128	96468992	96468992
ScanDirIterator	111280128	96468992	96468992
GlobFolderIterator	117309440	102498304	120254464
PathLibFolderIterator	469721088	455081984	455081984

ELAPSED TIME

Iterator	Elapsed time	X
ScanDirIterator	0:00:01.317526	1
OSWalkFolderIterator	0:00:02.496639	1.9
PathLibFolderIterator	0:00:10.464794	7.9
IGlobFolderIterator	0:00:14.358165	10.9
GlobFolderIterator	0:00:15.308936	11.6

TIME FOR FIRST FILE

Iterator	Elapsed time	X
ScanDirIterator	0:00:00.000098	1
OSWalkFolderIterator	0:00:00.000247	2.5
PathLibFolderIterator	0:00:00.000690	7
IGlobFolderIterator	0:00:00.001135	11.6
GlobFolderIterator	0:00:07.831005	79908.2

You can check the source code for this here.

Image from this wikipedia article.

Automatizando a publicação de artigos no dev.to com github actions

Guionardo Furlan — Wed, 03 Nov 2021 13:11:04 +0000

Automatizando publicações de artigos no dev.to com github actions

Recentemente decidi começar a escrever sobre problemas que encontro diariamente no desenvolvimento e as soluções encontradas.

Entre as opções de plataformas de publicação, uma que eu gosto bastante é a dev.to por alguns motivos:

É uma comunidade de desenvolvedores
É construída com base em software open source (Forem)
Não custa nada para quem escreve e nem para quem lê.

Eu já tinha alguns assuntos para começar a tratar aqui, mas resolvi organizar as postagens em uma plataforma centralizada, onde eu poderia de qualquer lugar, escrever, pensar, unir material e manter os dados em um local que eu sempre tenha acesso. Como nem tudo são flores no mundo open-source, não posso confiar que meus posts no dev.to ficarão lá para sempre.

Então, parti da ideia de criar um repositório no github para guardar meus artigos.

E já que os artigos ficariam no github, porque não automatizar a tarefa de publicação usando o github actions?

Ok, primeira coisa a ser verificada. O dev.to tem alguma API?

Tem! DEV API

Vamos precisar de três operações dessa API.

Então, precisamos ter um usuário registrado na plataforma. Como você é um desenvolvedor safo, não vou me preocupar com estas instruções, ok?

Após logar no dev.to, acesse as configurações da conta e crie uma API key. Guarde essa chave para usarmos em seguida.

No próximo post, vamos configurar o repositório que usaremos para guardar os artigos e que automatizará a formatação e publicação.

Guardando seus segredos com segurança

Guionardo Furlan — Tue, 02 Nov 2021 13:01:49 +0000

Segredos

O gerenciamento de senhas pode ser complicado quando temos vários serviços (e-mail, redes sociais, github, API´s, certificados digitais, orkut :-) , etc).

A regra é usar senhas complexas, pass phrases, cuidando pra não incluir dados pessoais, como aniversário, apelidos, nomes de conhecidos, etc que possam ser "adivinhados" por engenharia social.

E quando você cria sua senha super exclusiva e segura, encontra sistemas que expiram as senhas periodicamente e impedem que a reutilize.

Então, em nosso socorro, existem ferramentas para criar, gerenciar senhas, como o KeepassXC. Além de armazenar usuários e senhas, a ferramenta permite documentar e adicionar arquivos, como exemplos de requests/responses de API, arquivos de chaves, certificados digitais, etc. O KeepassXC armazena todas essas informações em um arquivo local criptografado (database .kdbx).

Existem outros, mas esse é um que eu gosto de usar e me atende muito bem. KeepassXC é open-source, roda em Windows, Linux, e Mac. É facilmente instalado. Siga o link acima e sucesso!

Ok, já resolvi o meu problema com muitas informações de autenticação e esbarrei em outro: Quando eu uso em mais de uma máquina, preciso me preocupar com a distribuição e sincronização do meu database por todas as minhas máquinas.

Dropbox, Google Drive, OneDrive poderiam ser uma opção, mas eu gostaria de algo mais hipster. Brincadeira. Se estamos falando em segurança de informação, dá pra confiar nos grandes players? Talvez.

Aí, encontrei o Keybase, que, entre outras coisas, permite um sistema de arquivos em nuvem e criptografado.

Então, vamos por mãos à obra.

Instalação

Acesse o site do Keybase, escolha seu ambiente (macOS, Windows ou Linux), faça o download e instale normalmente. Siga as instruções de instalação e depois crie seu usuário com uma senha segura ;-)

Depois do keybase devidamente instalado, acesse as opções de arquivo:

E clique em "Yes, enable" para instalar o Dokan Library, que permitirá montar o sistema de arquivos em uma unidade do sistema (K: geralmente, no Windows, e /keybase no Linux e no macOS)

Forem: Guionardo Furlan

Python Virtual Environment Setup

Python Virtual Environment Setup

Ubuntu 22.04 LTS Jammy Jellyfish

Ubuntu 20.04 LTS Focal Fossa

Complete the python3.8 installation:

Update pip

Access your project folder on terminal:

Create your virtual environment

Activating the virtual environment

Post-setup

Links:

Creating animated gifs from your terminal

Creating animated gifs from your terminal

Using local mail as notification tool

Using local mail as notification tool

Setup

HTTP Hello World server for testing purposes (part 1)

HTTP Hello World server for testing purposes

Fast folder iteration in python

Fast folder iteration in python

glob

Pros

Cons

iglob

Pros

Cons

os.walk

Pros

Cons

os.scandir

Pros

Cons

pathlib.Path.rglob

Pros

Cons

Some data from tests

System Information

CPU Info

CPU Usage Per Core

Memory Information

SWAP

MEMORY USAGE

ELAPSED TIME

TIME FOR FIRST FILE

Automatizando a publicação de artigos no dev.to com github actions

Automatizando publicações de artigos no dev.to com github actions

Guardando seus segredos com segurança

Segredos

Instalação

TODO: Continuar