Forem: Micheal Ojemoron

The Environment Variables Pattern

Micheal Ojemoron — Sun, 17 May 2020 18:39:12 +0000

An Introduction to Environment variables and how to use them

During software development, there are things we shouldn't share with our code. These are often configurations like secret keys, database credentials, AWS keys, API keys/tokens, or server-specific values.
According to 12 Factor methodology, it is wrong to store config as constants in the code because config varies substantially across deploys, code does not.

What Are Environment Variables?

When I started writing code it took quite some time to figure out what an environment variable is and how they can be set-up.
In a nutshell, an environment variable is a combination of values, called key/pair values. They hold information that other programs and applications can use.
One common environment variable described by the operating system is PATH which contains a set of directories where executable programs are located.
Using Git Bash shell:

echo $PATH
/c/Users/MICHAEL/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/c/Users/MICHAEL/bin:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/WINDOWS/System32/OpenSSH:

How to Set-up Environment Variables

Aside from built-in environment variables defined by our operating system or third party programs, there are several ways to create environment variables:

the Windows Environment Variables setup: you can use this to configure global variables
.bashrc file via Bash shell
the export command in Bash like environment or set command in windows command-line:

#in bash shell
export API_KEY=https://dev.to/mojemoron #key=value
echo $API_KEY #echo key
https://dev.to/mojemoron #value

#in windows cmd
set API_KEY=https://dev.to/mojemoron
echo %API_KEY%

.env file: this is the most common way to create environment variables per project especially during development. To do this, you create a .env file in your project's root directory and set the various environment variables you need in your project and afterward, you use a library for your respective programming language to load the file which will dynamically define these variables. For example, using python-decouple in a Django project to load environment variables.

touch .env #create this file in the root directory

#add this to the .env file
DB_HOST=localhost
DB_NAME=root
DB_USER=root
DB_PWD=root
DB_PORT=5432

#in your settings.py file, load the environment variables like so
DATABASES = {
    'default': {
        'ENGINE': 'django_postgres_extensions.backends.postgresql',
        'HOST': config('DB_HOST'),
        'NAME': config('DB_NAME'),
        'USER': config('DB_USER'),
        'PASSWORD':config('DB_PWD'),
        'PORT': config('DB_PORT'),
    }
}

There are several libraries that can help you achieve this:
- Node js
- Python
- Ruby
- Java

Cloud service providers like AWS, GCP, Heroku, Azure, and Digital Ocean recommend the use of environment variables for configuration and have built-in features for setting and managing them.

Summary

It's very important to note that separating application configurations such as credentials, API keys from code will remove unnecessary roadblocks/headaches when deploying your apps to multiple environments.

Finally, remember not to check in your .env file into a version control system by using git .ignore file.
You don't want to expose your environment variables to the world!

How do you configure environment variables for your projects?

Kindly follow me and turn on your notification.
Thank you! Happy coding! ✌

Productivity 101: Git Aliases for Lazy Developers

Micheal Ojemoron — Sun, 03 May 2020 16:18:55 +0000

According to Larry Wall, the original author of the Perl programming language, one of the qualities of a great program is laziness...
"The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don’t have to answer so many questions about it. Hence, the first great virtue of a programmer."

Some years ago, I stumbled upon an interesting article- Eliminate Stupid Mental Effort (ESME). I can't remember the author's name. The article explains why some developers are unproductive because they spend too much time on repetitive tasks that can easily be avoided.

As a lazy developer, I started using aliases in my git workflow to save myself from typing long commands. So instead of typing "git checkout master", my lazy self simply types "gom", which saves precious human fingertip power by allowing the shorter keystroke form.

What are aliases?
Aliases are used to create shorter commands that map to longer commands.
They are commonly found in popular utilities like bash.

How to create aliases
Creating an alias is as simple as typing:

alias ALIAS_NAME="ALIAS_COMMAND" # generic form
alias gom = "git checkout master" # git example

To get started, make sure you have installed git bash or any bash program on your system.
Open git bash, and type "cd" to get to your home directory.
Type the following commands to create a .bashrc file that automatically runs whenever you open git bash.

touch .bashrc # create a  .bashrc

You can either open the .bashrc file with any code editor to start adding your aliases or use shell output redirection.

Using Output Redirection
For example, let's add an alias for "git rebase" command

cd ~ # to enter your home directory
echo alias gre=\"git rebase\" >> .bashrc

To have this change taken into account you should then either source this file (ie: run source .bashrc) or restart your terminal.

Common examples of lazy git aliases

alias gs="git status" # to see changes that have been staged and which haven't
alias gac="git add . && git commit -m" # to stage and commit changes
#git push and pull
alias gp="git push" # + remote & branch names
alias gl="git pull" # + remote & branch names

# Pushing/pulling to origin remote
alias gpo="git push origin" # + branch name
alias glo="git pull origin" # + branch name

# Pushing/pulling to origin remote, master branch
alias gpom="git push origin master"
alias glom="git pull origin master"

alias gcb="git checkout -b" # To create a new branch and checkout into it
alias go="git checkout"
alias gom="git checkout master"
alias gre="git rebase"

alias gd='git diff' 
alias glo="git log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit"

In conclusion, using git aliases in your git workflow will increase your productivity as an Engineer and eliminate stupid mental effort.

Also, note that aliases are not only meant for git workflow, you can also create aliases for any command you use frequently e.g Docker, Laravel, Ruby, Python, WordPress and Django commands, etc.

What aliases are you using?

Kindly follow me and turn on your notification.
Thank you! Happy coding! ✌

What the heck is Unpacking and Packing of Sequences in Python?

Micheal Ojemoron — Sun, 29 Mar 2020 16:52:18 +0000

If you are new to python or have been programming in python and you haven't started using Packing and Unpacking this article is for you.

Python provides additional convenience involving the treatments of tuples and other sequence types.
What the heck is Packing

In Python, a series of comma-separated objects without parenthesis are automatically packed into a single tuple. For example, the assignment

fruits = apple,orange,lemon

results in the variable fruits being assigned to the tuple (apple, orange, lemon). This behavior is called the automatic packing of a tuple.
Also, another common use of packing is when returning multiple values from a function. like so

return x,y,z

this packs the return values into a single tuple object (x,y,z).

So what the heck is Unpacking in Python?

In the same light, Python can automatically unpack a sequence,
allowing one to assign a series of individual identifiers to the elements
of the sequence. As an example, we can write

a, b, c, d = range(1, 5)
a, b, c, d = [1,2,3,4]

which has the effects of a=1,b=2,c=3,d=4. For this syntax, the right-hand
side expression can be any iterable type, as long as the number of variables on the left-hand side is the same as the number of elements in the iteration.
if they aren't equal an error will be raised:

a, b, c = [1,2,3,4]
ValueError: not enough values to unpack (expected 3)

Unpacking can also be used to assign multiple values to a series of identifiers at once

a, b, c = 1, 2, 3

The right-hand-side is first evaluated i.e packed into a tuple before unpacking into the series of identifiers. This is also called a simultaneous assignment.

Another common usage of simultaneous assignment which can greatly improve code readability is when swapping the values of two variables like so:

a, b = b, a

With this command, a will be assigned to the old value of b, and b will be assigned to the old value of a. Without a simultaneous assignment, a swap typically requires more delicate use of a temporary variable, such as

temp = a
a = b
b = temp

You can also get the rest of a list like so:

a, b, c, *d = [1, 2, 3, 4, 5, 6] 
a => 1
b => 2
c => 3
d => [4, 5, 6]

In conclusion, using packing and unpacking during development can greatly improve the readability of your code and also solve some common issues with sequence assignment.

If you have any questions, kindly leave your comments below.

Kindly follow me and turn on your notification. Thank you!
Happy coding! ✌

Can you recommend interesting software engineering blogs you follow?

Micheal Ojemoron — Sat, 28 Mar 2020 12:06:23 +0000

These are my interest:

System design
Application design/scaling
Database design/scaling
Software Architecture
Data structures and Algorithm
Languages like Python, Javascript, PHP, Golang and Rust
Cloud computing, DataScience, etc

It can either be written by individuals or companies.
Something a beginner can understand.
Thanks in advance
Update:
You can also add podcasts too 🤗

How to Connect your Django App to a dockerized PostgreSQL and pgAdmin

Micheal Ojemoron — Sun, 22 Mar 2020 15:08:02 +0000

Docker has been my default environment set-up for deploying most of my web projects quickly. It is lightweight and it has also helped me solve package dependencies and environment configuration issues because it provides a consistent environment across different servers and also makes continuous delivery and deployment enjoyable.

Recently, I felt the need to connect my Django application to a dockerized Postgres database and also manage the database with a dockerized pgAdmin(a web app. for managing Postgres databases).

I did this because I wanted my application layer to stay outside the docker environment of my database without the need to install Postgres and pgAdmin on my local machine for a quick app. prototype.
I had several issues setting this up until I successfully got it up and running.
Let me show you how I did it 😊.

In this post, I am assuming that you have successfully installed Docker and Django. I will also be using docker-compose (this will enable you to run multiple containers).
Visit this link to understand what docker-compose is: https://docs.docker.com/compose/

To dockerize Postgres and pgAdmin:

create a postgres_docker directory in the root folder that contains your Django project dir.
cd into the postgres_docker dir. and create a docker-compose file that will contain the Postgres and pgAdmin images: copy the following code and paste it into your docker-compose file

version: "3.1"

services:

  db:
    restart: always
    image: postgres
    container_name: demo-postgres #you can change this
    environment:
      - POSTGRES_USER=demo
      - POSTGRES_PASS=demo
      - POSTGRES_DB=demo
      - POSTGRES_PORT=5432
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data/

  pgadmin:
      image: dpage/pgadmin4
      container_name: demo-pgadmin #you can change this
      depends_on:
        - db
      ports:
        - "5051:80"
      environment:
        PGADMIN_DEFAULT_EMAIL: pgadmin4@pgadmin.org
        PGADMIN_DEFAULT_PASSWORD: root
      restart: always


volumes:
  postgres_data:

open your command line, cd into postgres_docker dir. and run

docker-compose up

This command builds, (re)creates, starts, and attaches to containers for a service. to confirm if there are no errors, visit http://localhost:5051 to see pgAdmin Interface.

To connect pgAdmin to Postgres:

 PGADMIN_DEFAULT_EMAIL: pgadmin4@pgadmin.org #you can change this
 PGADMIN_DEFAULT_PASSWORD: root #you can change this

enter the following settings from the images below to your pgAdmin interface Please note
your hostname db is the name of your Postgres service in the docker-compose file.

To connect Django to Postgres:

In your settings.py, configure database settings like so:

DATABASES = {
    'default': {
        'ENGINE': 'django_postgres_extensions.backends.postgresql',
        'NAME': 'demo',
        'USER': 'demo',
        'HOST': 'localhost',
        'PORT': 5432,
        'PASSWORD':'demo'
    }
}

To see that all went as plan, startup your Django application like so:

python manage.py runserver

If everything works fine, you should see your application without errors.
visit http://localhost:8000

In conclusion, there are several ways to achieve this but this is my preferred way and it also easy to understand.
If you have any issue with your setup leave your comments below.
Kindly follow me and turn on your notification. Thank you!
Happy coding! ✌

A Beginner's Guide to Multithreading and Multiprocessing in Python - Part 1

Micheal Ojemoron — Sun, 15 Mar 2020 12:42:32 +0000

As a Backend Engineer or Data Scientist, there are times when you need to improve the speed of your program assuming that you have used the right data structures and algorithms. One way to do this is to take advantage of the benefit of using Muiltithreading or Multiprocessing.
In this post, I won't be going into detail on the inner working of Muiltithreading or Multiprocessing. Instead, we will write a small Python script to download images from Unsplash. We will start with a version that downloads images synchronously or one at a time. Next, we use threading to improve execution speed.

I am sure you are excited to learn this...

Multithreading

In a nutshell, threading allows you to run your program concurrently. Tasks that spend much of their time waiting for external events are generally good candidates for threading. They are also called I/O Bound tasks e.g writing or reading from a file, network operations or using an API to download stuff online.
let take a look at an example that shows the benefit of using threads.

Without Threading
In this example, we want to see how long it takes to download 15 images from Unsplash API by running our program sequentially.

import requests
import time
img_urls = [
    'https://images.unsplash.com/photo-1516117172878-fd2c41f4a759',
    'https://images.unsplash.com/photo-1532009324734-20a7a5813719',
    'https://images.unsplash.com/photo-1524429656589-6633a470097c',
    'https://images.unsplash.com/photo-1530224264768-7ff8c1789d79',
    'https://images.unsplash.com/photo-1564135624576-c5c88640f235',
    'https://images.unsplash.com/photo-1541698444083-023c97d3f4b6',
    'https://images.unsplash.com/photo-1522364723953-452d3431c267',
    'https://images.unsplash.com/photo-1513938709626-033611b8cc03',
    'https://images.unsplash.com/photo-1507143550189-fed454f93097',
    'https://images.unsplash.com/photo-1493976040374-85c8e12f0c0e',
    'https://images.unsplash.com/photo-1504198453319-5ce911bafcde',
    'https://images.unsplash.com/photo-1530122037265-a5f1f91d3b99',
    'https://images.unsplash.com/photo-1516972810927-80185027ca84',
    'https://images.unsplash.com/photo-1550439062-609e1531270e',
    'https://images.unsplash.com/photo-1549692520-acc6669e2f0c'
]

start = time.perf_counter() #start timer
for img_url in img_urls:
    img_name = img_url.split('/')[3] #get image name from url
    img_bytes = requests.get(img_url).content
with open(img_name, 'wb') as img_file:
     img_file.write(img_bytes) #save image to disk 

finish = time.perf_counter() #end timer
print(f"Finished in {round(finish-start,2)} seconds") 

#results
Finished in 23.101926751 seconds

With Threading
Let's see how the threading module in Pyhton can significantly improve our program execution.

import time
from concurrent.futures import ThreadPoolExecutor

def download_images(url):
    img_name = img_url.split('/')[3]
    img_bytes = requests.get(img_url).content
    with open(img_name, 'wb') as img_file:
         img_file.write(img_bytes)
         print(f"{img_name} was downloaded")

start = time.perf_counter() #start timer
with ThreadPoolExecutor() as executor:
    results = executor.map(download_images,img_urls) #this is Similar to map(func, *iterables)
finish = time.perf_counter() #end timer
print(f"Finished in {round(finish-start,2)} seconds")

#results 
Finished in 5.544147536 seconds

To have a better understanding of how to use the Threading module in Python visit this link: https://docs.python.org/3/library/concurrent.futures.html

We can see that with threading code speed improved significantly compared with without threading code i.e from 23 secs to 5 secs 💃.
For this example, please note that there is an overhead in creating threads so it makes sense to use threads for multiple API calls, not just a single call.

Also, for intensive computations like data crunching, image manipulation Multiprocessing performs better than thread.

In conclusion, for I/O bound tasks, anytime our program is running synchronously it is actually not doing much on the CPU. It's probably waiting around for some input. That's actually a good sign that we can get some benefits running our program concurrently using Multithreading.

Next week, In part 2, we'll learn how to use Multiprocessing for CPU heavy tasks to speed up our programs :).
Next, we learn how to connect a Django application to a dockerized PostgreSQL and pgAdmin 4 image running on your local machine😎

Please follow me and turn on your notification. Thank you!
Happy coding! ✌

What does this mean in Python: *args, **kwargs?

Micheal Ojemoron — Sun, 01 Mar 2020 11:26:22 +0000

If you are new to python one stumbling block you will face during your learning experience is understanding the *args and **kwargs magical variables. I also had a hard time figuring how they work 😅 when I started.

 def args_kwargs(*args, **kwargs):
      pass

Python *args and **kwargs arguments are everywhere. You can find them in the source code of one of the most popular Python frameworks (Django).
The variable *args and **kwargs can be written in any form. e.g *vars and **kvars are also valid. Only the * (asterisk) is necessary.
In this lesson, I will explain what they are and how to use them 😎.

Using *args in a function definition:

*args can be used in function definitions. They allow the user to call a function with a number of variable arguments. Variable here means that you do not know beforehand how many arguments that can be passed to your function by the user so in this case, you use the *args keywords. What *args allows you to do is take in more arguments than the number of formal arguments that you previously defined. Internally they are represented using tuple data structure.
Note that before the variable number of arguments, zero or more normal arguments may occur.

An interesting example for usage of *args:

def positions(first_arg, *args):
    print(f"Type of *args is {type(args)}") #print type of args
    print(f"first position arg:{first_arg}")
    for arg in args:
        print(f"another arg through *args: {arg}")

positions('first','second','third','forth')
#results
Type of *args is <class 'tuple'>
first position arg:first
another arg through *args: second
another arg through *args: third
another arg through *args: forth

Using **kwargs in a function definition:

The special syntax **kwargs in a function definition is used to pass a keyworded, variable-length argument list.
**kwargs allows you to pass keyworded variable length of arguments to a function. Internally, they are represented using dictionary data structure.
Here is a basic example to explain this concept:

def items(**kwargs):
    print(f"Type of **kwargs is {type(kwargs)}") #print type of **kwargs
    #iterating over the keyword argument
    if kwargs is not None:
        for key, value in kwargs.items():
            print(f"{key} = {value}")

items(a="basket", b="plate", c="soap")
#results
Type of **kwargs is <class 'dict'>
a = basket
b = plate
c = soap

From the above examples, we can see that using *args and *kwargs gives our function definitions the flexibility of accepting an unlimited number of arguments.
Let's look at some examples of using **args** and **kwargs when calling a function.

*args can be used to unpack values from iterable objects(list, tuple) in Python.

def test_args_kwargs(arg1,arg2,arg3):
     print(f"{arg1},{arg2},{arg3}")

#using tuple
test_args_kwargs(*('first','second','third'))
#OR
test_args_kwargs(*['first','second','third'])

#results
first,second,third #unpacking tuple
first,second,third #unpacking list

While the **kwargs can only be used to unpack a dictionary.

# now with **kwargs:
kwargs = {"arg1": "first", "arg2": "second","arg3":"third"}
test_args_kwargs(**kwargs)
#results
arg1: first
arg2: second
arg3: third

Order of using *args **kwargs and formal args

The correct order for your parameters is:
standard arguments
*args arguments
**kwargs arguments

Let's see how to combine the various types of parameter:

#right
def my_func(args, *args, **kwargs):
    pass

#wrong
def my_func(args, **kwargs, *args):
    pass

Bonus
We can also use * to collect values during an assignment.

first,*others = ["first","second","third"]
print(first,others, sep="\n")
#results
first
['second', 'third']

I hope you now have a clear understanding of how to use *args and **kwargs when defining or calling a function in Python.
If you have any questions, please leave your comments below.

Next week, I will explain how to use Multithreading and Multiprocessing to speed up your programs :). Please follow me and turn on your notification. Thank you!
Happy coding! ✌

Pythonic way to aggregate or group elements in a list using dict.get and dict.setdefault

Micheal Ojemoron — Sat, 22 Feb 2020 15:47:42 +0000

Using dict.get

How often have you aggregated an item by its group like this:

ar= [1,1,3,1,2,1,3,3,3,3]
pairs = {}
for i in ar:
    if i not in pairs:
         pairs[i] = 0
    pairs[i] = pairs[i] + 1


print(pairs)
{1: 4, 3: 5, 2: 1}

The above code shows how to get the frequency of elements in a list.
While this code is good, the time complexity of in(membership operator) is O(n) worse case.
We can make this code better and more pythonic by using python's dict.get(k,d) method.

Using the same list, we have:

pairs = {}
for i in ar:
    pairs[i] = pairs.get(i, 0) + 1
print(pairs)
{1: 4, 3: 5, 2: 1}

the trick here is that dict.get(k,d) returns None by default or the specified default value if a key doesn't exist.

Using dict.setdefault

Similarly when grouping things together we usually do this:

items_by_type = {}
for item in items:
    if item.type not in items_by_type:
        items_by_type[item.type] = list()
    items_by_type[item.type].append(item)

Let's make it better and more pythonic:

items_by_type = {}
for item in items:
    items_by_type.setdefault(item.type, list()).append(item)

The trick here is that setdefault(k,d) only sets the item if it does not
exist. If it does exist then it simply returns the item.

Final Thoughts

You can use the Counter class from the collection module to get the frequency of elements in a list.
Just do this:

from collections import Counter
#using our above example:
print(Counter(ar))

Happy coding! ✌

Please drop your comments :)