Forem: Samuel Abolo

How to ship code fast: Like a 10x developer

Samuel Abolo — Sat, 18 May 2024 19:04:33 +0000

Introduction

We all want to ship features as fast as possible. Especially when we have a long wishlist of other features waiting to be shipped. The catch is our shipped code must be clean, scalable, and working as expected. The big question is, can we have the best of both worlds? This article explores how to release quality code and still have time to watch the Champions League finals.

Why do developers fail to ship fast?

Developers fail to ship fast for a couple of reasons

Developers often struggle with understanding complex features. They spend a significant amount of time analyzing ambiguous feature requests and translating them into detailed specifications and algorithms. Breaking down a feature into smaller, logical components can be a challenging process.
They spend too much time on things that don't matter much. Spending hours or even days trying to make a function 1% faster or turning that for loop to a list comprehension.
It can be demotivating to write unit tests, documentation, and change logs, often resulting in taking longer than necessary.
Developers often get sidetracked by fixing bugs unrelated to the feature.
Fixing bugs takes a lot of time, especially when found by QA instead of yourself.

Luckily for us, all these problems have solutions.

Build Early, Validate Continuously

Instead of spending so much time worrying about whether you designed the right feature, build what you understand. Keep the client in the loop. Design a version or prototype and demo it to the client. Let the client tell you how far away you are from what they want. Always remember that success is just iterative failure.

Lay the foundation: Write clean code from the onset

Avoid technical debt by writing code that's clean and reusable from the very beginning of the project. Apply design patterns that support extensibility and reduce coupling. You shouldn't have to refactor your entire codebase for every new feature. Having a good project directory also goes a long way.

The Zen of Testing

Always write your tests before you code, while you're still full of energy. It keeps you focused on the task at hand. Every piece of code you write should aim at passing those test cases. Start by designing integration tests (tests that cover the entire feature) then decompose them into smaller units, testing each function or module to be written. If the requirements change, remember to adjust the tests first before updating the actual code.

Solve first, optimize later

Most times, the client doesn't care about how fast your product is, as long as it is fast enough. They don't care if it is multithreaded or runs on the GPU. Don't waste everyone's time trying to make the code faster when the code doesn't even exist. It doesn't hurt for your product to be the fastest bunny in town, but fast code that's full of bugs or doesn't work means nothing. Focus on getting it working first, and spend time speeding it up later.

Conclusion

In this article, we've looked at the various reasons why we developers may fail to deliver on time and how we can tackle each of these reasons. Overall, it promotes an agile culture and a Test Driven Development methodology. It also encourages developers to embrace clean code and proper use of design patterns throughout their projects. This article is helpful to all developers from interns, to seniors across all niches.

Relevant Reads and Watches

Mastering PR Messages: Why worry about them when there's AI (Part 2)

Samuel Abolo — Fri, 08 Dec 2023 14:41:43 +0000

Content

Introduction
Scenario
PR messages with GitHub Copilot
PR messages with CodiumAI
Which is better?
Conclusion

Introduction

Let's face it. We're developers. We all want to ship fast. When that new feature is requested or that issue is raised, we want to get it checked out ASAP and focus on other things. We hate the extras; standups, emails, testing. We want to focus on writing the code.

In the last article, we looked at the qualities of a good PR message and how to write one. Crafting the perfect PR message takes a lot of work. But what if you don't have to do it yourself?

Today, we've observed a boom in generative AI and LLMs (Large Language Models). We have generative AI models like GPT-4, DALL-E, Stable DIffusion, LLaMA, and more. We've also seen tools that use these models to solve problems and automate tasks like never before. If you're reading this, you have probably used ChatGPT at some point in time to do things that would take you hours in seconds.

In this article, we will be exploring how we can take advantage of some of these tools to help us write better PR messages and allow us to focus on what truly matters, the code.

Scenario

To illustrate how we can let AI handle our PR messages, I'm going to be using an open-source project written by a good friend of mine (Alfred) on Microservices; a simple illustration of how to build, deploy, and scale (locally) microservices using Docker and Nginx.

Let's head over to the repo and fork it.

It's a system that helps you keep track of popular TV shows. I'm going to be making changes to one of the microservices called "search-service" because it's written in my favorite programming language (Python). "search-service" handles searching for TV shows using a search query.

We clone our fork to our local computer and create a new branch called "feature/subquery-search" using the following commands.

git clone https://github.com/your-username/tv-shows-microservice

git branch feature/subquery-search

git checkout feature/subquery-search

We're going to be making a PR to enhance the search service. If you take a look at the code, you will observe that it takes in a search parameter, and searches for tv-shows with that parameter using regular expressions. What happens when what you searched doesn't match any of the TV shows?

We'd also like to return some shows that closely match. A simple way to do this is to break down the search query into different queries and search for those instead. For example, if a user searches for "The Walking Nun", our algorithm will also search for "The Walking", "The Nun", "Walking Nun", "The", "Walking", and "Nun".

To make things neater, let's create a function called search_for_shows to handle the regular expressions search for us.

Finally, here's what the search endpoint looks like now.

Once that has done, we commit our changes and push back to GitHub so we can make our PR.

PR messages with GitHub Copilot

One of the tools we can use to help us create our PR message is GitHub Copilot. GitHub Copilot is an AI-powered peer programmer developed by GitHub. It has recently launched a new feature called GitHub Copilot Chat on VS Code To set up GitHub Copilot, you can check this link. After that, you can ask GitHub Copilot to give you a good PR message based on the changelog.

PR messages with CodiumAI

Another tool to structure your PR message is CodiumAI's PR Agent. I recommend CodiumAI's PR Agent for a couple of reasons.

CodiumAI's PR Agent is easy to integrate. You don't need to install CodiumAI on VS Code (though that option is also available).

You can mention the CodiumAI's PR Agent in the comment section of the Pull Request. it automatically makes the necessary changes to the title and description. To use this feature, you will need to add CodiumAI either as a GitHub App or GitHub Action in your repository to make it work. You can check out this guide on how to get that done.

CodiumAI works on both sides of the table. It's not just a great tool for writing good PR messages, but it's also a great tool for reviewing the PRs of your teammates. With the CodiumAI command "/review", you get to see an analysis of the branch's changelog, including things like a general summary of the changes, how long it will take you to go through the PR, the type of PR, and suggestions on the PR. So even if your teammates didn't use CodiumAI to structure their PR messages, you can use CodiumAI to review the PRs.

CodiumAI also provides an "/ask" command which lets you ask questions about the PR and what has changed. Think of this as having ChatGPT inside the PR.

You can ask any questions about the code and what has changed. You also get an "/improve" command that shows code suggestions for the PR.

Here's a link with all the recommendations from CodiumAI You also have the option to just commit the changes directly to the pull request. (Pretty cool right?)

GitHub Copilot VS CodiumAI's PR Agent

In this article, we have discussed two AI tools that can assist us in creating our PR messages. The question now arises as to which one should we use.

GitHub Copilot is an excellent tool for generating code, as it is its primary function. It is a pair programmer AI trained on all open-source code repositories on GitHub, making it extremely powerful for coding tasks. However, its strength in writing effective PR messages may not be as comprehensive or tailored specifically for PR needs.

CodiumAI's PR Agent, on the other hand, specializes in structuring PR messages. It automatically adjusts titles and descriptions based on the content of the PR. Since this AI agent is designed specifically for PRs, it is easier to work with and has more commands that help in other aspects of code collaboration.

Conclusion

In conclusion, AI-powered tools like CodiumAI's PR Agent and GitHub Copilot can help us write better PR messages and save us time. With these tools, we can focus on writing quality code and let the AI handle the tedious tasks of structuring and reviewing PR messages. Consider using one of these AI-powered tools to help you out next time you're creating a PR message.

Mastering PR Messages: How to get your PRs merged (Part 1)

Samuel Abolo — Mon, 04 Dec 2023 06:18:15 +0000

Content

Introduction
The Art of Writing Great PR Messages
Conclusion

Introduction

As a computer programmer, you've probably been in a situation where you need to work with others on projects. Whether it's a project at work, school, an open-source project, or even a side project you and your friends decided to work on it.

As a computer programmer in this category, you most likely used a tool called Git as a VCS (Version Control System) and GitHub, Bit Bucket, or any of the others. If so, you must be familiar with the term called "pull request" or "PR", related to something called a branch.

Branches and PRs

A branch is like a duplicate of the main code folder. You can edit its content and then merge it with the main folder. You can also collaborate with others by merging your folder with theirs.

When merging your branch with others, you can add a nice message stating why your code should be merged. In really chill environments (say your school project or side project), this message helps the team know what exactly has changed.

It's a slightly different case if it's a work or open source project. You don't just merge your branch with the main one as that can break things. What you do instead is to create a Pull Request. In the case of a pull request, the message tells the team what will change if they merge your code and whether or not they should let it through. In short, everyone else looks at your changes and decides if they should be merged.

The Art of Writing Great PR Messages

When writing a pull request, a clear and straightforward message is important. A poorly written message can lead to confusion among team members about code changes, making them reluctant to adopt your changes. There are various things to look for when writing a great PR

Write good PRs

Great PR messages require great PRs. Ensure your PRs contain clear, straightforward, and well-written code with context. Avoid adding too many features in one PR. PRs should focus on a new feature, a bug fix, a refactored feature, feature tests, or documentation changes.

When submitting a pull request (PR), it's important to ensure that the changes made are neither too little nor too much. If the changes made are too little, it might not be considered worthy of a PR. On the other hand, if it's too much, it should be broken down into multiple PRs for better clarity.

A useful tip is to create a new branch for each PR or proposed change. Once the objective of the branch has been completed and the PR has been merged, the branch should be deleted.

For instance, if you create a branch called "login-feature", after creating and merging the login feature to the main branch, the branch should be deleted. Even if there's a change or bug fix to the login feature, the change should have its branch. This helps keep things organized and makes the process a lot cleaner.

Plan your PR message before creating the branch

Now that you've got a great PR, you need a great PR message to explain what's in the PR. A great way to always have a clear and brief PR message is to write it before writing code. Let the PR message be your guide to the changes you'll make. This makes your work easy and helps you stay on track while coding, thus making your PR message a perfect description of PR.

Use Clear Language

Always remember that not everyone who reads your PR message will have technical knowledge. So, it's best to avoid using technical terms as much as possible. Only use programming words when necessary, and always write your message as if it's being read by someone who is not a coder. This will ensure that your message is clear and easily understood by everyone.
Lastly, when writing a PR in English, use common words instead of uncommon ones. For instance, use "brief" instead of "succinct".

Add Images if necessary

There's this saying that an image is worth a thousand words. If your changes also affect design, it's not a bad idea to include pictures of the new design. Whether it's the database design, system design, or even changes to some form of front end, images go a long way in explaining what has changed.

Tag your teammates

When you want your teammates to review your code, make sure to tag them. If your teammates are also working on the same feature, they should also be tagged. Additionally, it's important to mention everyone who is involved in your change in your PR message. This will help draw attention to your change and ensure that everyone stays informed.

Follow the crowd

When writing PR messages, it's important to ensure they follow the same structure and format that other team members use. If you are contributing to an open-source project, likely, there are already guidelines in place for how PR messages should be structured. It's essential to follow these guidelines strictly to ensure clarity and consistency across the project.

Conclusion

Writing clear and concise PR messages is crucial for getting your PRs merged
Avoid using technical jargon and use simple language that is easy to understand
Tag your teammates in the message if necessary
Consider including images to help explain the changes you've made
Following these tips will help your teammates understand your changes and make the review process more efficient.

Watch out for Part 2 and if you have any questions, feel free to reach out to me via E-mail: ikabolo59@gmail.com or Twitter

Celery: Python's Silent Hero

Samuel Abolo — Sun, 01 Oct 2023 22:00:50 +0000

Introduction

A few days ago, I was working on this cool personal project of mine (no spoilers till I release it). I needed to work on a microservice that read image data from Kafka and perform operations on them. These operations involved heavy image processing and machine learning tasks that could take about 1 second to finish. Sounds pretty small right? Here's the problem. These messages coming from Kafka were arriving in real-time. Imagine getting ready to spend 1 second on one image when the other has already arrived.

One solution I considered (which I will still implement) was to horizontally scale this microservice since I'm already using a Publisher/Subscriber architecture with Kafka. It means that I have multiple instances of this microservice subscribed to Kafka, so while one is busy with an image, the other has already picked it up and is working on it. But this wasn't enough. I needed this microservice to be as scalable as possible. That's when it hit me. Why don't I use Celery?

What is Celery?

I needed some way to receive these messages as quickly as possible and perform these computations on each image. That's where Celery came in. Celery is a Python-based task queue that helps us queue up tasks in a separate thread allowing the main process of our app to run smoothly. In other words, it's an asynchronous task queue. These tasks get executed in the order that they arrive.

Why Celery?

Now, I know many of you are probably wondering, why Celery? Why didn't Samuel use multithreading? After all, he has written an article on how to use multithreading and multiprocessing in Python.

The answer is the queue in the task queue. Celery allows us to use robust message brokers such as RabbitMQ and Redis making our lives better. Celery then lets us create Workers. You can think of these workers as threads that keep checking the message brokers for jobs to execute.

Even if our celery workers aren't running, the jobs get queued until a worker gets back up. Also, since we can spawn multiple celery workers connected to the same RabbitMQ or Redis Queue, we've achieved horizontal scalability within each instance of my awesome microservice.

Using Celery in Python

If you're still reading this, you'd also like to use Celery in your project. To use Celery, you only need four things installed on your computer.

A message broker (RabbitMQ, Redis, Amazon SQS, or Zookeeper)
Python3 installed
The Celery Framework
A celery backend (optional) for queuing up the results from our celery workers. The best choice here is Redis.

The first thing you'd want to do is to get a message broker installed on your computer or server. I recommend RabbitMQ because it's the default message broker for Celery and should work best, but you should use a message broker that you're most comfortable with. Here are some links on how to install these brokers

I would assume you already have Python3 installed on your computer so I won't say so much about it. I'd also recommend you create a virtual environment rather than using your global Python environment.

Lastly, you can install Celery using your favorite package manager in Python (pip, easyinstall, and lots more)

The first step to using Celery is to create a Celery object in a file called tasks.py (it's just a convention. You can name it anything you like). This object tells Python how we'd like our Celery worker to behave. This is where we specify what message broker Celery should use.


from celery import Celery

app = Celery(
    'tasker',
    backend='redis://localhost',
    broker='amqp://127.0.0.1:5672',
)

The next step is to copy our computationally expensive code and paste it into a function in our tasks.py file. You'll then decorate this function with the @app.task decorator from our Celery object. This decorator tells Celery that the code is a possible task in Celery.


import time

@app.task 
def awesome_task(time: int):
    # We pretend as if the task is taking some time.
    time.sleep(time)
    logging.info(f"Time taken: {time}")


import random
from tasks import awesome_task

while True:
    # We create a new task and fire it to celery
    awesome_task.delay(random.randint(1, 3))

We can then add tasks to our Celery task queue that can be picked up by any of the Celery workers and executed.

To create a Celery worker, run the command in the terminal.

celery -A tasks worker --loglevel=INFO

When we run our app, you will notice that we aren't seeing the output in the app's shell but rather, in the shell that's running the Celery task.

Conclusion

I hope I've been able to show you how awesome Celery is at helping scale your apps.

Of course, this isn't the only way you can use Celery. You can use Celery to make your Flask or Django app asynchronous. This means that the requests made to your app don't have to wait for it to be less busy to get a response. A lot of people also use Celery to run background tasks like sending successful E-mails or storing application logs.

If you have any questions about using Celery in Python, feel free to reach out to me through my E-mail or the comment section.

How to NOT overfit in Deep Learning

Samuel Abolo — Mon, 23 Jan 2023 22:34:46 +0000

Introduction: What is overfitting
Why overfitting happens in neural networks
Techniques to preventing overfitting: L2 Regularization
Techniques to preventing overfitting: Dropout Regularization
Other things that can help: Batch normalization.
Other things that can help: Diversify your dataset Summary and Conclusion

Introduction

When you're using deep learning, sometimes your neural network can do really well on the data you're training it with, but not as well when you test it or use it in real life. This can be really frustrating because it's not always easy to fix. In this article, we'll talk about what overfitting is in neural networks and what you can do to fix it when it happens in your model.

What Overfitting is

Overfitting is when your neural network is doing really well on the data you're training it with, but not as well when you test it or use it in real life. It happens when the neural network has learned patterns that only appear in the data you're training it with, but not in the actual problem you're trying to solve. These patterns are like noise, and they're not helpful for your final solution.

In the above image, notice how the model does really well on the data that was used to train it but performs poorly on the evaluation data. We could say in simple terms, that the model has gotten used to the training data.

It's kind of like how students can get so caught up in practicing past questions for an exam, that they can't answer any question that didn't fall into past questions training set.

Why overfitting happens

Complex network structure

The simplest and most obvious reason why overfitting occurs is the complexity of our network. The more layers we add, the more our neural network tries to derive insights from our data.

Insufficient data

Another reason why overfitting happens is when you're trying to learn from not enough data or data that all looks the same, or when you're running too many "epochs" (repeated training cycles) with a big network on not enough data. For example, let's say you're training a neural network to recognize the difference between a cat and a dog. If all the dogs in your training set are bulldogs, it might have trouble recognizing a chihuahua.

Another problem that can happen when you have little or similar data is that your network will stop learning too soon and will only find patterns that exist in that specific data set and not in the problem you're trying to solve.

Poorly Synthesized data

A technique to fix the problem of "homogeneous" data is called data synthesis. This means creating new data by combining different sources. For example, if you're training a neural network to recognize a specific word (like "Hey Siri") but you want it to also work well in a noisy environment, you can make new data by combining sounds of that specific word with sounds of different noisy places. This will make the network better at recognizing the word in different environments.

Using data synthesis can be helpful in making sure your data is diverse and not too similar, but it can also cause problems if not done correctly. One way it can lead to overfitting is if the network starts to learn the noise that was used to create the new data. In other words, if the network starts to focus on the wrong things instead of the main feature you're trying to train it on.

Techniques for preventing Overfitting

L2 regularization

L2 regularization is an idea borrowed from classical machine learning and regression. It works by adding a regularization term to the loss function. This term is the Frobenius/Matrix Norm of the weights of the neural network

The lambda value is a number that can be adjusted to change how much regularization is used to prevent overfitting. Regularization is a technique that helps the network not to focus too much on the noise in the data. If the lambda value is zero, then regularization is not applied at all. If the lambda value is very high, then regularization becomes too strong, and the network will not learn enough from the data, which is called underfitting.

So the intuition is that the higher the value of lambda, the stronger the effect L2 regularization has on our network

Dropout Regularization

The main Idea behind Dropout Regularization is randomly temporarily shutting off a small percentage of units within layers of our neural network to reduce the chances of our neural network depending on strong signals from any of these units.

We should however be careful not to shut off a large portion of our network layers because it can cause our network to underfit our data.

Other things that can help

Batch Normalization

Batch Normalization has this regularization effect that can help prevent overfitting.

The first benefit it gives is that it helps to set all your features to be on the same scale (i.e having the same standard deviation), helping your Neural Network to converge faster and better. Your neural network would also be less likely to depend on features that are on a higher scale than others.

Diversify your Data

One simple way to fix overfitting is to add more diverse data to your dataset. However, this might not always be possible or easy. By having more varied data, your model will learn to recognize features that appear throughout the dataset instead of just in a small part of it.

Transfer Learning

This is a way of addressing the issue of overfitting caused by not having enough data. By using a neural network that has been trained with a lot of data, you can use it to solve similar problems and make it better. Often, all you need to do is retrain the input and output layers of the network.

Thanks for Reading!

Thanks for reading this article! I would love to hear your thoughts, questions, or any other feedback you may have. Don't hesitate to share your thoughts in the comments section.

Reach out to me

Multiprocessing in Python (Part 2)

Samuel Abolo — Sun, 07 Aug 2022 00:01:00 +0000

FACE DETECTION WITH PYTHON AND MULTIPROCESSING

In this post, we will be writing a simple script to illustrate how handy multiprocessing and multithreading can be in real world situations.
We will be writing a simple program that loads images from our computer, detects and draws rectangles on all human faces and upload to a server.

Tools

Python
OpenCV
Flask for our server
The Python requests module to push our images

OUR FACIAL RECOGNITION ALGORITHM

So this code basically handles all the local processing, To understand more about how the face detection works, check out
realpython.com


import cv2

from requests import post

# Create the haar cascade
cascPath = "haarcascade_frontalface_default.xml"
faceCascade = cv2.CascadeClassifier(cascPath)

def draw_face(image):

    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    # Detect faces in the image
    faces = faceCascade.detectMultiScale(
        gray,
        scaleFactor=1.1,
        minNeighbors=5,
        minSize=(30, 30)
    )

    # Draw a rectangle around the faces
    for (x, y, w, h) in faces:
        cv2.rectangle(image, (x, y), (x+w, y+h), (0, 255, 0), 2)

    return image

def read_image(image_path):
    return cv2.imread(image_path)

def write_image(image, image_path="output.jpg"):
    fn = f"out-{image_path}"
    cv2.imwrite(fn, image)
    return fn

def upload_image(fn):
    url = "http://localhost:5000/upload"
    files = {'file': open(fn, "rb")}

    return post(url, files=files).status_code

THE SIMPLE SERVER

This is a simple WSGI server written in Flask, to allow us to upload our files.

# app.py
from flask import Flask, request

app = Flask(__name__)

@app.post("/upload")
def upload_file():

    file = request.files['file']

    file.save(f"./temp/{file.filename}")

    return {"status": True}


if __name__ == '__main__':
    app.run()

Run this in a new terminal window

$ python app.py

THE NORMAL WAY OF PROCESSING (Synchronously)

If we were to test this function with 1000 images, running our flask server locally, it’s gonna take A LOT of time because it’s processing it synchronously.

Synchronous programming or execution basically means that the next process has to wait for the current process to finish before it starts.

Synchronous execution is only suitable if the current process needs the data generated by the previous process.


import time

def normal_way(image_paths):
    for image_path in image_paths:
        image = read_image(image_path)
        image = draw_face(image)
        fn = write_image(image, image_path)
        upload_image(fn)

if __name__ == '__main__':
    # Trying this 100 times to simulate 100 images
    image_paths = ["a.jpg"] * 100

    start = time.time()
    normal_way(image_paths)
    stop = time.time()

    # Time taken for synchronous: 309.1052303314209
    print(f"Time taken for synchronous: {stop-start}")

If we were to have 1000 samples, the time taken would be approximately 3091.05 seconds.

OUR OPTIMISED CODE (ASYNCHRONOUS EXECUTION)

Asynchronous programming is basically the opposite of synchronous programming, the next process can start concurrently or in parallel with the current process.

There are many ways to achieve this: An Event Loop, Multithreading, Multiprocessing, etc. For the purpose of this tutorial, we will be using Multithreading and Multiprocessing only.

We will be using multithreading for the IO-based functions: reading the images from the disc, writing to the disc, and uploading to the server.

We will be using multiprocessing for the CPU-intensive function, which is the function that detects and draws rectangles on the faces.


from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def optimised_solution(image_paths: list):

    # Define our pool executors
    with ThreadPoolExecutor() as thread_pool:
        with ProcessPoolExecutor() as process_pool:

            # We use threading to read our images
            images = thread_pool.map(read_image, image_paths)

            # We use multiprocessing to process our images
            processed_images = process_pool.map(draw_face,images)

            # We use multithreadfing to write our images to disk
            files = thread_pool.map(write_image, processed_images)

            # And finally, we use multithreading to push our images to our server
            responses = thread_pool.map(upload_image, files)

            print("Processed: ", all([response == 200 for response in responses]))


if __name__ == '__main__':
    # Trying this 100 times to simulate 100 images
    image_paths = ["a.jpg"] * 100

    start = time.time()
    optimised_solution(image_paths)
    stop = time.time()

    # Time taken for Optimised 1: 147.17485308647156
    print(f"Time taken for Optimised 1: {stop-start}")

KEY TAKEAWAYS

As we observe, notice that our solution optimised with multiprocessing took about half the time to finish executing.
Our program now has the ability to scale horizontally with the number of CPUs because of the concept of parrallel computing discussed in part 1.
If we where to have an Asynchronous server (ASGI), our program would be more optimised, but lets leave that for another day.

THE FULL CODE

import cv2
import time

from requests import post

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor



cascPath = "haarcascade_frontalface_default.xml"

# Create the haar cascade
faceCascade = cv2.CascadeClassifier(cascPath)

def draw_face(image):
    # More Info: https://realpython.com/face-recognition-with-python/

    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    # Detect faces in the image
    faces = faceCascade.detectMultiScale(
        gray,
        scaleFactor=1.1,
        minNeighbors=5,
        minSize=(30, 30)
    )

    # Draw a rectangle around the faces
    for (x, y, w, h) in faces:
        cv2.rectangle(image, (x, y), (x+w, y+h), (0, 255, 0), 2)

    return image

def read_image(image_path):
    return cv2.imread(image_path)

def write_image(image, image_path="output.jpg"):
    fn = f"out-{image_path}"
    cv2.imwrite(fn, image)
    return fn

def upload_image(fn):
    url = "http://localhost:5000/upload"
    files = {'file': open(fn, "rb")}

    return post(url, files=files).status_code


def normal_way(image_paths):
    for image_path in image_paths:
        image = read_image(image_path)
        image = draw_face(image)
        fn = write_image(image, image_path)
        upload_image(fn)

def optimised_solution(image_paths: list):

    # Define our pool executors
    with ThreadPoolExecutor() as thread_pool:
        with ProcessPoolExecutor() as process_pool:

            # We use threading to read our images
            images = thread_pool.map(read_image, image_paths)

            # We use multiprocessing to process our images
            processed_images = process_pool.map(draw_face,images)

            # We use multithreadfing to write our images to disk
            files = thread_pool.map(write_image, processed_images)

            # And finally, we use multithreading to push our images to our server
            responses = thread_pool.map(upload_image, files)

            print("Processed: ", all([response == 200 for response in responses]))


if __name__ == '__main__':
    # Trying this 100 times to simulate 100 images
    image_paths = ["a.jpg"] * 100

    start = time.time()
    normal_way(image_paths)
    stop = time.time()

    # Time taken for synchronous: 309.1052303314209
    print(f"Time taken for synchronous: {stop-start}")

    start = time.time()
    optimised_solution(image_paths)
    stop = time.time()

    # Time taken for Optimised 1: 147.17485308647156
    print(f"Time taken for Optimised 1: {stop-start}")

CONCLUSION

Multiprocessing and Multithreading can be combined to boost our code performance by a lot.
Synchronous execution can only scale up if we have faster CPUs, but multiprocessing allows us to scale up with more CPUs.

Feel free to ask any questions in the comment section, or reach me at ikabolo59@gmail.com and on Twitter

Thanks

Multiprocessing in Python (Part 1)

Samuel Abolo — Fri, 29 Jul 2022 20:43:00 +0000

A lot of you have probably been in a situation where you need to carry out multiple tasks, or a repetitive action on multiple items, like doing your homework, or even something as little as doing your laundry. It’s so much easier when we have the ability to do multiple things at the same time. Like have multiple washing machines for our laundry, or 5 people do your homework.

The same principle also applies to computing. There are times when we have lots of data and we would like to perform the same action on all of our data. The problem now is, that it’s the same action and we have lots of data. This slows down our program.

import time

def our_function():
    print("Processing stuff...")
    time.sleep(5)
    print("Done")

def normal_linear_method():

    our_function()
    our_function()
    our_function()

normal_linear_method()
# Time taken: about 15 seconds

Let’s assume it takes exactly 5 seconds to complete the action or function on the data. If we have 100 units of data to process, it's going to take us 500 seconds, which is about 8 minutes of our time. What if I told you there was a way we could speed things up from 8 minutes back to our unit time of 5 seconds?

Multithreading in Python

The first technique we will use to solve our problem is something called multithreading. Multithreading works by constantly switching the context (basically the state of the task it’s working on at the moment) such that an illusion of parallel processing is achieved. This concept is also known as concurrency.

# Example of task speed up using multithreading

from threading import Thread
import time

def using_multithreading():

    # Our threads
    t1  = Thread(target=our_function)
    t2 = Thread(target=our_function)
    t3 = Thread(target=our_function)

    # Starting our threads
    t1.start()
    t2.start()
    t3.start()

    # We join the threads/processes so our main thread/process
    # can wait for it to be completed before terminating

    t1.join()
    t2.join()
    t3.join()

using_multithreading()
# time taken: about 5 seconds

Multiprocessing in Python

The second technique we will use to solve our problem is multiprocessing. While multithreading in python makes use of context switching, multiprocessing in python runs each of the processes in parallel. Each process has its own copy of the entire program's memory and runs on its own core.

# Example of task speed up using multiprocessing
import time
from multiprocessing import Process

def using_multiprocessing():
    # Our processes
    p1  = Process(target=our_function)
    p2 = Process(target=our_function)

    # Starting our processes
    p1.start()
    p2.start()
    p1.join()
    p2.join()

if __name__ == '__main__':

    start = time.perf_counter()
    using_multiprocessing()
    stop = time.perf_counter()

    print("Time taken {}".format(stop-start))

Multiprocessing vs Multithreading: Parallelism vs Concurrency

Both multiprocessing and multithreading come in handy. The question is, when should we use what.

We use multithreading for IO-bound operations, like reading data from a file, or pooling data from a server.
We use multiprocessing for CPU-bound operations, like image processing, training a machine learning model, big data processing, etc.

Running multiple processes at once

There are times when we want to run a function on a sequence of data. Say we have a list of 100 units of data, and we would like to apply our function to all of them in parallel or concurrently. There are different approaches we can take:

Approach 1: iteratively create processes and start them

In this approach, we’ve used a loop to create a process for all our data and start them. The problem with this approach is that we can’t really get the output of the processes easily.

import time
from multiprocessing import Process

def multiple_processes():

    # Spawn our processes iteratively
    processes = [
        Process(target=operation, args=(x,)) 
        for x in data
    ]

    for process in processes:
        # Iteratively start all processes
        process.start()

    for process in processes:
        process.join()

    return 

if __name__ == '__main__':

    start = time.perf_counter()
    multiple_processes()
    stop = time.perf_counter()

    print("Time taken {}".format(stop-start))
    # time taken: about 8 seconds

Approach 2: The ProcessPoolExecutor

In this approach, we’ve used something called a pool, which is an easier and neater way to manage our computing resources. Although this is slower than spawning the processes iteratively, its way neater and allows us to use the output of those processes in our main process.

# Using multiprocessing with ProcessPoolExecutor
import time
from concurrent.futures import \
    ProcessPoolExecutor, as_completed


def multiple_processes_pooling():

    with ProcessPoolExecutor() as executor:
        process_futures = [
            executor.submit(operation, x) 
            for x in data
        ]
        results = [
            p.result() 
            for p in 
            as_completed(process_futures)
        ]

        print(results)


if __name__ == '__main__':

    start = time.perf_counter()
    multiple_processes_pooling()
    stop = time.perf_counter()

    print("Time taken {}".format(stop-start))
    # time taken: about 50 seconds

Approach 3: ProcessPoolExecutor().map

In this approach, instead of iteratively submitting a process to our pool executor, we’ve used the executor.map method to submit all of the data in the list at once. The output of this function is the result of all the completed processes.

import time
from concurrent.futures import ProcessPoolExecutor

# Using the executor.map
def pooling_map():

    with ProcessPoolExecutor() as executor:
        results = executor.map(operation, data)

        print([res for res in results])

if __name__ == '__main__':

    start = time.perf_counter()
    pooling_map()
    stop = time.perf_counter()

    print("Time taken {}".format(stop-start))
    # time taken: about 50 seconds

Very Important to remember

If you look at the time output, you’d notice that the time taken isn't exactly the unit time, there are four main factors that affect this.

The computer in use can affect its time, as well as other programs running on your PC. The code was tested using an intel Core i5 7th generation computer.
It takes a few microseconds for our program to properly set up our processes and start it.
When there are more processes than we have CPU cores, our system automatically queues the pending processes and helps us manage them properly.
And finally, it takes a few microseconds for our program to properly close processes.

That being said, it’s important to note that we only use multiprocessing when there’s a lot of data and the operation takes a lot of time to be completed.

Conclusion

Multiprocessing and Multithreading help us to speed up our programs.
Multiprocessing is most suited for CPU-bound operations, like machine learning and data processing.
Multithreading is most suited for IO-bound operations, like communicating with servers, or the file system.
Multiprocessing is not a magic wand; Don't use it unless you have to, or it could actually slow down your code.

Will NoSQL Databases Replace SQL Databases in the future

Samuel Abolo — Mon, 25 Jul 2022 22:08:58 +0000

WILL NOSQL REPLACE SQL DATABASES IN THE FUTURE

In this article, we will be discussing The following

What SQL is (Pros and Cons)
What NoSQL is (Pros and Cons)
When to use SQL and to use NoSQL (ACID vs BASE)
Will NoSQL replace SQL in the future? Why (not)?
What the future of databases looks like (Databases supporting both SQL and NoSQL databases)

What is SQL?

SQL stands for the structured query language. It’s the standard programming language for data definition, insertion, and manipulation in relational databases.
An SQL database is basically a term used to classify relational databases that use SQL.
The most popular ones are MySQL, PostgreSQL, Oracle SQL, and T-SQL. Data in SQL databases are represented in rows and columns, making it easier to write intuitive queries

Pros

ACID properties (Atomicity, Consistency, Isolation, and Durability)
SQL databases follow ACID principles in transactions.
- Atomicity means that the data change in a transaction is interpreted as one. This means that if the transaction fails halfway for some reason, the database state is rolled back.
- Consistency means that the data from the transaction is still checked against constraints placed on it in the database schema.
- Isolation means that any changes in a transaction is isolated and would'nt affect any changes in another transaction that's running concurrently. This means that two transactions manipulating the same data resource can't run at the same time but must be queued.
- Durability means that once the transaction is complete, it's changes will permanently reflect on the database.

Easier to query
Data is normalized in rows and columns in SQL, making SQL databases perfect for complex queries.
Larger community
It’s quite popular, so knowing how to work with SQL databases could improve your chances of landing a job as a data scientist, data analyst, or software engineer.

Cons

Consistency for Performance tradeoff
SQL databases achieve atomicity, consistency, and isolation of data through something called data locks, which basically prevents a transaction from modifying or reading from a resource (row or column) while it’s being used by a different transaction. While this ensures consistency, it takes its toll on the performance of SQL databases compared to NoSQL databases.
Difficult to scale
Horizontally scaling an SQL database can be painful. Techniques such as data sharding/partitioning and master-slave replication can be implemented, but they aren’t as easy as they sound making them difficult and expensive to scale.

What is NoSQL?

NoSQL is a term used to classify databases that don’t store data in a relational manner. While SQL databases only support storing data in rows and columns, we have different kinds of NoSQL databases that we can use, depending on our business needs.

Pros

Easier to scale
Database Sharding in NoSQL is a lot easier than Partitioning in SQL databases. Most NoSQL databases, in fact, do this for you, allowing you to focus on the business logic
Faster queries
NoSQL databases do not follow ACID principles, instead, follow BASE principles. NoSQL databases also, don’t have to worry about joins, making NoSQL database queries faster.
More diverse
Ranging from key-value databases like dynamo DB to document-based databases like MongoDB, to graph databases like neo4j and Tigergraph, NoSQL databases could be a perfect fit for our business needs.
For example, a road network database could make use of Graph databases, and a social media app could use MongoDB to store posts. A business can decide to use multiple NoSQL databases in its architecture

Cons

Performance for Consistency tradeoff
NoSQL databases don’t implement data locks, while this makes transactions/queries a lot faster than in SQL databases, data is not immediately consistent in the database, making it inappropriate for systems like FinTech applications or MedTech (systems that require data to be immediately consistent after a modification).
Smaller user community
SQL is more popular than NoSQL databases. Also, NoSQL is relatively new in the market, meaning most existing companies already have their database systems implemented in SQL.

When to use SQL or NoSQL

When to use SQL

We use SQL in systems whose data needs can be solved with ACID properties—finance Applications, Invoice Applications, and systems that value data consistency over faster queries.

When to use NoSQL

We use NoSQL databases in systems that require a high volume of data to be processed and stored. This is because NoSQL databases are highly scalable both in storage and in read-write performance. E.g Social media, Blogs, Streaming services.

Will NoSQL replace SQL in the future?

Both databases at this point in history, can’t replace each other, and it’s looking like it’ll stay that way.

The only way NoSQL databases will surface as a replacement for SQL databases is if NoSQL can find a way to ensure that data is immediately consistent and still maintain its query speed.

On the other hand, SQL could potentially replace NoSQL databases in some use cases in the future if they find ways to ensure ACID properties and at the same time not trade its query performance.

Another key point is that most developers and data professionals find querying data with SQL easier and more intuitive than with any other method implemented by NoSQL databases. NoSQL databases like Apache Hive and Tigergraph have solved this problem by providing an SQL interface for querying data stored in its NoSQL databases

The future of DBMS

The future of database systems is the best of both worlds. Combining the ACID properties of SQL with the BASE properties of NoSQL databases

This can be achieved in our present day by implementing a microservices architecture. That way, services that require their database to possess ACID properties could use SQL, and those that require BASE properties could use NoSQL. A good example is Facebook, which uses MySQL, Apache Cassandra, and a lot more NoSQL and SQL databases

Forem: Samuel Abolo

How to ship code fast: Like a 10x developer

Introduction

Why do developers fail to ship fast?

Build Early, Validate Continuously

Lay the foundation: Write clean code from the onset

The Zen of Testing

Solve first, optimize later

Conclusion

Relevant Reads and Watches

Mastering PR Messages: Why worry about them when there's AI (Part 2)

Content

Introduction

Scenario

PR messages with GitHub Copilot

PR messages with CodiumAI

GitHub Copilot VS CodiumAI's PR Agent

Conclusion

Mastering PR Messages: How to get your PRs merged (Part 1)

Content

Introduction

Branches and PRs

The Art of Writing Great PR Messages

Write good PRs

Plan your PR message before creating the branch

Use Clear Language

Add Images if necessary

Tag your teammates

Follow the crowd

Conclusion

Celery: Python's Silent Hero

Introduction

What is Celery?

Why Celery?

Using Celery in Python

Conclusion

How to NOT overfit in Deep Learning

CONTENTS

Introduction

What Overfitting is

Why overfitting happens

Techniques for preventing Overfitting

Other things that can help

Thanks for Reading!

Reach out to me

Multiprocessing in Python (Part 2)

FACE DETECTION WITH PYTHON AND MULTIPROCESSING

Tools

OUR FACIAL RECOGNITION ALGORITHM

THE SIMPLE SERVER

THE NORMAL WAY OF PROCESSING (Synchronously)

OUR OPTIMISED CODE (ASYNCHRONOUS EXECUTION)

KEY TAKEAWAYS

THE FULL CODE

CONCLUSION

Multiprocessing in Python (Part 1)

Multithreading in Python

Multiprocessing in Python

Multiprocessing vs Multithreading: Parallelism vs Concurrency

Running multiple processes at once

Approach 1: iteratively create processes and start them

Approach 2: The ProcessPoolExecutor

Approach 3: ProcessPoolExecutor().map

Very Important to remember

Conclusion

Will NoSQL Databases Replace SQL Databases in the future

WILL NOSQL REPLACE SQL DATABASES IN THE FUTURE

What is SQL?

Pros

Cons

What is NoSQL?

Pros

Cons

When to use SQL or NoSQL

When to use SQL

When to use NoSQL

Will NoSQL replace SQL in the future?

The future of DBMS