Forem: Mustafa Abdallah

Azure Cloud Infrastructure: Step-by-Step Guide

Mustafa Abdallah — Tue, 11 Jul 2023 16:42:12 +0000

Introduction

After I wrote about how to build a cloud infrastructure, I decided to implement real-world projects to demonstrate how you could build your own and follow best practices. Today, I am writing about how to build a cloud infrastructure for a backend API on Microsoft Azure.

Our Project
Building The Cloud Infrastructure
- Define
- Choose
- Plan
- Design
- Implement
- Deploy
Conclusion

Our Project

The project we will work on is a backend API built with Python, Flask, and PostgreSQL, and containerized with Docker. You can find it in this repository, clone it to your local computer, and let's begin building.

Note: This project could be built with any tech stack, but the steps will remain the same.

Building The Cloud Infrastructure

So, we have the project set up and ready to go for production. Now, our goal is to deploy it to the cloud. To achieve that, we need to build the cloud infrastructure. Let's follow these steps to build it.

Define

Our goal here is to deploy this simple backend API, making it available on the internet for others to access.

To accomplish this, we need to choose a deployment method. I have written about some of them in this article, but let's assume we want to deploy it using Docker on a remote server.

This outlines our requirements from the cloud, and based on the information above, we need:

Docker registry to push and pull Docker images.
Remote server to host our backend API.
PostgreSQL database for our backend API.
Storage for saving and retrieving static and media files used by our backend API.

Choose

For this article, we have chosen Microsoft Azure as the cloud platform to deploy our backend. You will need to create an Azure account as we will be using it during the implementation step.

Plan

Let's align the services we require with the services provided by Azure and document the initial configuration we need for each of those services.

Our Service	Azure Service	Configration
Docker Registry	Azure Container Registry (ACR)	Tier or SKU: Basic
Remote Server	Azure Virtual Machine	OS: Ubuntu, Size: DS1 v2, Storage: LRS
PostgresSQL Database	Azure Database PostgreSQL Server	SKU: B_Gen5_2, Storage: 5120
Storage	Azure Blob Storage	Tier: Standard, Type: LRS

Design

Our design is intentionally kept simple for now, without considering networking or security aspects, to avoid unnecessary complexity. Let's focus on the services we currently need.

The design, as shown below, involves users accessing the IP address of the Azure Virtual Machine instance. The Azure Virtual Machine instance runs a Docker container from the Docker image pulled from the Azure Container Registry repository. This Docker image contains our code, which is configured to access the PostgreSQL database in Azure Database PostgreSQL Server and the storage in Azure Blob Storage .

I used draw.io to do this diagram.

Implement

We have multiple options for implementing this cloud infrastructure on Azure, all of which are effective. However, the best approach will depend on your specific requirements and the complexity of your project.

Microsoft Azure Portal: The web interface provided by Azure, where you can manually create the resources one by one through their portal.
Azure CLI: The Azure Command Line Interface allows you to create resources using the command line in a terminal.
Azure Resource Manager: Azure Resource Manager is an Infrastructure as Code (IaC) tool provided by Azure, where you can provision services using JSON.
Terraform: Terraform is an IaC tool provided by HashiCorp. It has an Azure provider that allows you to provision Azure resources.

In this article, we will use Terraform to create the Azure resources. Make sure to install it.

The infrastructure folder

Let's create a new folder named infrastructure in the root directory of our project. Here is what it will contain:

├── secrets.auto.tfvars              # contains secrets, make sure to ignore it
├── main.tf
├── azure                            # azure module that contains azure resources
│   ├── main.tf
│   ├── blob_storage.tf
│   ├── container_registry.tf
│   ├── postgres_database.tf
│   ├── virtual_machine.tf
├── ssh                               # bring your ssh keys, make sure you ignore them
│   ├── id_rsa.pub
│   ├── id_rsa
└── scripts
    └── install_docker.sh

Set up Azure provider on Terraform

We need to configure the Azure provider so that we can use it in Terraform. In the main.tf file, we will add the following configuration:

# main.tf

terraform {
  required_version = "1.3.1"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.30.0"
    }
  }
}

provider "azurerm" {
  features {}
  subscription_id = var.azure.subscription_id
  tenant_id       = var.azure.tenant_id
  client_id       = var.azure.client_id
  client_secret   = var.azure.client_secret
}

To securely store Azure credentials, we will create a file named secrets.auto.tfvars. Please ensure that this file is ignored and not committed to version control. In the secrets.auto.tfvars file, we will add the following content:

# secrets.auto.tfvars

azure = {
  subscription_id = ""
  tenant_id       = "" 
  client_id       = ""
  client_secret   = ""
}

Run the following command to initialize Terraform and ensure that everything is working correctly:

terraform init

We will need to create a resource group, so it hold all our resources that we create, so in the azure/main.tf file add those:

# azure/main.tf

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
    }
  }
}

resource "azurerm_resource_group" "main" {
  name     = "example-project-resource-group"
  location = "East US"
}

Provision Azure Virtual Machine

# azure/virtual_machine.tf

resource "azurerm_linux_virtual_machine" "main" {
  name                            = "example-project-machine"
  computer_name                   = "example-project-machine"
  admin_username                  = "example-project-user"
  size                            = "Standard_DS1_v2"
  resource_group_name             = azurerm_resource_group.main.name
  location                        = azurerm_resource_group.main.location
  network_interface_ids           = [azurerm_network_interface.main.id]
  disable_password_authentication = true

  admin_ssh_key {
    username   = "example-project-user"
    public_key = file("${path.module}/ssh/id_rsa.pub")
  }

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-focal"
    sku       = "20_04-lts-gen2"
    version   = "latest"
  }

  provisioner "remote-exec" {
    connection {
      type        = "ssh"
      user        = "example-project-user"
      host        = self.public_ip_address
      private_key = file("${path.module}/ssh/id_rsa")
    }
    scripts = ["${path.module}/scripts/install_docker.sh"]
  }

}

output "MACHINE_SSH_CONNECT" { 
      value = "ssh example-project-user@${azurerm_linux_virtual_machine.main.public_ip_address}" 
}
output "MACHINE_USER" { value = "example-project-user" }
output "MACHINE_IP" { value = azurerm_linux_virtual_machine.main.public_ip_address }

Provision Azure Database PostgreSQL Server

# azure/postgres_database.tf

resource "azurerm_postgresql_server" "main" {
  name                = "example-project-database-server"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location

  administrator_login          = "example-project-db-user"
  administrator_login_password = var.database_server_password

  sku_name   = "B_Gen5_2"
  storage_mb = "5120"
  version    = "11"

  ssl_enforcement_enabled          = true
  ssl_minimal_tls_version_enforced = "TLS1_2"
}

resource "azurerm_postgresql_database" "main" {
  name                = "example-project-db"
  resource_group_name = azurerm_resource_group.main.name
  server_name         = azurerm_postgresql_server.main.name
  charset             = "UTF8"
  collation           = "English_United States.1252"
}

output "POSTGRES_DB" { value = "example-project-db" }
output "POSTGRES_USER" { value = "example-project-db-user@${azurerm_postgresql_server.main.fqdn}" }
output "POSTGRES_PASSWORD" { value = var.database_server_password }
output "POSTGRES_HOST" { value = azurerm_postgresql_server.main.fqdn }
output "POSTGRES_PORT" { value = "5432" }

Provision Azure Blob Storage

# azure/blob_storage.tf

resource "azurerm_storage_account" "main" {
  name                     = "example-project"
  account_tier             = "Standard"
  account_replication_type = "LRS"
  resource_group_name      = azurerm_resource_group.main.name
  location                 = azurerm_resource_group.main.location
}

resource "azurerm_storage_container" "main" {
  name                  = "example-project-container"
  container_access_type = "container"
  storage_account_name  = azurerm_storage_account.main.name
}

output "AZURE_ACCOUNT_NAME" { value = "example-project" }
output "AZURE_ACCOUNT_KEY" { value = azurerm_storage_account.main.primary_access_key }
output "AZURE_CUSTOM_DOMAIN" { value = azurerm_storage_account.main.primary_blob_host }
output "AZURE_CONTAINER" { value = "example-project-container" }

Provision Azure Container Registry

# azure/container_registry.tf

resource "azurerm_container_registry" "main" {
  name                = "example-project-registry"
  sku                 = "Basic"
  admin_enabled       = true
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
}

output "ACR_URL" { value = azurerm_container_registry.main.login_server }
output "ACR_USERNAME" { value = azurerm_container_registry.main.admin_username }
output "ACR_PASSWORD" { value = azurerm_container_registry.main.admin_password }

Add them to the main

# main.tf

module "azure" {
  source = "./azure"
}

output "azure" {
  value     = module.azure
  sensitive = true
}

Apply our infrastructure

Now that all the necessary resources are provisioned, we can apply them using Terraform to create them on Azure.

terraform apply

Get the output

If everything is working correctly, your infrastructure should be set up. You can now retrieve the output values from Terraform, which can be used when deploying your code.

terraform output azure

Deploy

You can deploy the project manually or automatically using any CI/CD tool. For instructions on how to do this, please refer to this section in this project's README file.

Conclusion

Azure is a popular cloud provider and an industry trend. Many organizations utilize Azure to build their cloud infrastructure. Knowing how to build on Azure is valuable, but it's crucial to follow best practices and defined steps to ensure high-quality work. I hope this article has been helpful for you, and feel free to follow me for more articles like this or ask me anything related on my LinkedIn.

AWS Cloud Infrastructure: Step-by-Step Guide

Mustafa Abdallah — Mon, 10 Jul 2023 21:16:34 +0000

Introduction

Our Project
Building The Cloud Infrastructure
- Define
- Choose
- Plan
- Design
- Implement
- Deploy
Conclusion

Our Project

The project we will work on is a backend API built with Python, Django, and PostgreSQL, and containerized with Docker. You can find it in this repository, clone it to your local computer, and let's begin building.

Note: This project could be built with any tech stack, but the steps will remain the same.

Building The Cloud Infrastructure

Define

Our goal here is to deploy this simple backend API, making it available on the internet for others to access.

To accomplish this, we need to choose a deployment method. I have written about some of them in this article, but let's assume we want to deploy it using Docker on a remote server.

This outlines our requirements from the cloud, and based on the information above, we need:

Docker registry to push and pull Docker images.
Remote server to host our backend API.
PostgreSQL database for our backend API.
Storage for saving and retrieving static and media files used by our backend API.

Choose

For this article, we have chosen AWS as the cloud platform to deploy our backend. You will need to create an AWS account as we will be using it during the implementation step.

Plan

Let's align the services we require with the services provided by AWS and document the initial configuration we need for each of those services.

Our Service	AWS Service	Configration
Docker Registry	AWS ECR	Repository Type: Public
Remote Server	AWS EC2	OS: Ubuntu, CPU: t2.micro, Volume: 20G
PostgresSQL Database	AWS RDS	Engine: Postgres, CPU: db.t3.micro, Volume: 20G
Storage	AWS S3	Access Type: Public

Design

Our design is intentionally kept simple for now, without considering networking or security aspects, to avoid unnecessary complexity. Let's focus on the services we currently need.

The design, as shown below, involves users accessing the IP address of the AWS EC2 instance. The EC2 instance runs a Docker container from the Docker image pulled from the AWS ECR repository. This Docker image contains our code, which is configured to access the PostgreSQL database in AWS RDS and the storage in AWS S3.

I used draw.io to do this diagram.

Implement

We have multiple options for implementing this cloud infrastructure on AWS, all of which are effective. However, the best approach will depend on your specific requirements and the complexity of your project.

AWS Management Console: The web interface provided by AWS, where you can manually create the resources one by one through their dashboard.
AWS CLI: The AWS Command Line Interface allows you to create resources using the command line in a terminal.
AWS CloudFormation: AWS CloudFormation is an Infrastructure as Code (IaC) tool provided by AWS, where you can provision services using YAML.
AWS CDK: AWS CDK is another IaC tool by AWS that enables you to provision services using programming languages like Python, TypeScript, or Java.
Terraform: Terraform is an IaC tool provided by HashiCorp. It has an AWS provider that allows you to provision AWS resources.

In this article, we will use Terraform to create the AWS resources. Make sure to install it.

The infrastructure folder

Let's create a new folder named infrastructure in the root directory of our project. Here is what it will contain:

├── secrets.auto.tfvars  # contains secrets, make sure to ignore it
├── main.tf
├── aws                  # aws module that contains aws resources
│    ├── EC2.tf
│    ├── ECR.tf
│    ├── RDS.tf
│    ├── S3.tf
├── ssh                  # bring your ssh keys, make sure you ignore them
│    ├── id_rsa.pub
│    ├── id_rsa
└── scripts
    └── install_docker.sh

Set up AWS provider on Terraform

We need to configure the AWS provider so that we can use it in Terraform. In the main.tf file, we will add the following configuration:

terraform {
  required_version = "1.3.1"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

provider "aws" {
  region     = var.aws.region
  access_key = var.aws.access_key
  secret_key = var.aws.secret_key
}

To securely store AWS credentials, we will create a file named secrets.auto.tfvars. Please ensure that this file is ignored and not committed to version control. In the secrets.auto.tfvars file, we will add the following content:

aws = {
  region     = "" # your aws region
  access_key = "" # your aws access key
  secret_key = "" # your aws secert key
}

Run the following command to initialize Terraform and ensure that everything is working correctly:

terraform init

Provision AWS EC2

# aws/EC2.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

resource "aws_key_pair" "main" {
 key_name   = "example-project-ssh-key"
 public_key = file("${path.module}/ssh/id_rsa.pub")
}

resource "aws_instance" "main" {
 tags          = { "Name" = "example-project-instance" }
 ami           = "ami-08c40ec9ead489470"
 instance_type = "t2.micro"
 key_name      = aws_key_pair.main.key_name

 root_block_device {
   volume_size = "20"
   volume_type = "gp2"
 }

 connection {
   type        = "ssh"
   user        = "ubuntu"
   host        = self.public_dns
   private_key = file("${path.module}/ssh/id_rsa")
 }

 provisioner "remote-exec" {
   scripts = ["${path.module}/scripts/install_docker.sh"]
 }

}

output "INSTANCE_USER" { value = var.instance_user }
output "INSTANCE_IP" { value = aws_instance.main.public_dns }
output "INSTANCE_SSH_CONNECT" { value = "ssh ${var.instance_user}@${aws_instance.main.public_dns}" }

Provision AWS RDS

# aws/RDS.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

resource "aws_db_instance" "main" {

  identifier = "example-project-database"

  db_name  = "example-project-db"
  username = "example-project-user"
  password = var.database_password # save it in the secrets.auto.tfvars file
  port     = 5432

  engine         = "postgres"
  engine_version = "13.7"

  instance_class    = "db.t3.micro"
  storage_type      = "gp2"
  allocated_storage = "20"

  publicly_accessible      = true
  delete_automated_backups = true
  skip_final_snapshot      = true
  deletion_protection      = false
}

output "POSTGRES_DB" { value = aws_db_instance.main.db_name }
output "POSTGRES_USER" { value = aws_db_instance.main.username }
output "POSTGRES_PASSWORD" { value = aws_db_instance.main.password }
output "POSTGRES_HOST" { value = aws_db_instance.main.address }
output "POSTGRES_PORT" { value = aws_db_instance.main.port }

Provision AWS S3

# aws/S3.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

resource "aws_s3_bucket" "main" {
  bucket        = "example-project-bucket"
  force_destroy = true
}

resource "aws_s3_bucket_acl" "main" {
  bucket = aws_s3_bucket.main.id
  acl    = "public"
}

resource "aws_s3_bucket_public_access_block" "main" {
  bucket                  = aws_s3_bucket.main.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

output "AWS_ACCESS_KEY_ID" { value = var.bucket_access_key }
output "AWS_SECRET_ACCESS_KEY" { value = var.bucket_secret_key }
output "AWS_STORAGE_BUCKET_NAME" { value = aws_s3_bucket.main.bucket }
output "AWS_BUCKET_DOMAIN_NAME" { value = aws_s3_bucket.main.bucket_domain_name }

Provision AWS ECR

# aws/ECR.tf

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

resource "aws_ecr_repository" "main" {
  name = "example-project-repository"
  force_delete = true
}

output "REPOSITORY_URL" { value = aws_ecr_repository.main.repository_url }

Add them to the main

# main.tf

module "aws" {
  source = "./aws"
}

output "aws" {
  value     = module.aws
  sensitive = true
}

Apply our infrastructure

Now that all the necessary resources are provisioned, we can apply them using Terraform to create them on AWS.

terraform apply

Get the output

If everything is working correctly, your infrastructure should be set up. You can now retrieve the output values from Terraform, which can be used when deploying your code.

terraform output aws

Deploy

You can deploy the project manually or automatically using any CI/CD tool. For instructions on how to do this, please refer to this section in this project's README file.

Conclusion

AWS is a popular cloud provider and an industry trend. Many organizations utilize AWS to build their cloud infrastructure. Knowing how to build on AWS is valuable, but it's crucial to follow best practices and defined steps to ensure high-quality work. I hope this article has been helpful for you, and feel free to follow me for more articles like this or ask me anything related on my LinkedIn.

How to Build a Cloud Infrastructure

Mustafa Abdallah — Fri, 16 Jun 2023 14:41:20 +0000

Introduction

With the age of cloud computing and public clouds like AWS, GCP, and Azure, building a cloud infrastructure has become a regular practice in software development. However, doing it correctly with defined steps is what will enable you to produce high-quality and efficient work. Today, I am writing about those steps that help us build an efficient cloud infrastructure.

What is Cloud Infrastructure?

It's basically the resources that your app needs to run your code on the internet. These resources are provided as services by cloud computing providers like AWS, GCP, or Azure. So, to make your app work, you need to build a cloud infrastructure.

Steps of Building a Cloud Infrastructure

These steps define what we should follow when building cloud infrastructure, and they are applicable to any cloud provider. Here is a detailed overview of each one:

Define

You first need to define your requirements and goals, so that you can have an outline of what you want to achieve. You should study your application and decide how you want to deploy it. There are various deployment methods, some of which I discussed in previous sections of this article. After choosing the deployment method, you will need to determine the cloud services you will require.

For example, if you have a Python backend API built with Django and you have chosen to deploy it using Docker on a remote server, you will need to consider services such as:

For the remote server, you will need a virtual machine on the cloud. This can be achieved using services like AWS EC2, Google Compute Engine, or Azure Virtual Machine.
For Docker, you will require a Docker registry where you can push and pull your Docker image. Services such as Docker Hub, AWS ECR, or Azure ACR can fulfill this need.
For Django, you will need database and storage services. Services like AWS RDS, Google SQL, or Azure Database can provide the necessary database services, while storage needs can be met using services like AWS S3, Google Storage, or Azure Blob Storage.

So far, you have defined your deployment method and identified the cloud services you may require. The next step is to choose the cloud provider.

Choose

After you define your requirements and goals, you have to choose one of the public cloud providers that satisfies your requirements and fits your budget. While each provider may follow the pay-per-use payment strategy, the prices of services can vary, with some being more expensive than others. You can also benefit from using multiple or hybrid clouds, leveraging the strengths of different providers. The crucial step is to select a cloud provider that allows you to provision the services you need for your application.

The three current popular public cloud providers are AWS, GCP, and Azure. While the underlying concept of their services is the same, their configurations may differ. For instance, if you need a remote server, you can find it on AWS as part of the AWS EC2 service, on GCP as Google Compute Engine, and on Azure as Azure Virtual Machine. These providers offer similar functionality with slight variations.

For our earlier example of the Python Backend API discussed in the previous step, based on our requirements, we could choose AWS as it provides all the necessary services, and the pay-per-use payment model aligns well with our needs, allowing us to pay only for our usage.

It's worth noting that all of these cloud providers offer a free tier, which you can take advantage of if you're unsure about which one to choose. Visit their respective websites for more information.

Plan

After defining the desired services and selecting the cloud provider, the next step is to plan the cloud infrastructure. This involves choosing the specific cloud services that align with our needs from the chosen provider and configuring them accordingly.

For instance, in the case of a backend API, we require a remote server, a Docker registry, a database, and storage. Since we have chosen AWS as our cloud provider, let's explore the AWS services that meet our requirements.

Remote Server: AWS provides the AWS EC2 service, which allows us to configure an Ubuntu server to host our code. We can choose Ubuntu as the operating system, t2.micro as the CPU configuration, and allocate 20GB of general-purpose SSD for the server volume storage.
Docker Registry: AWS offers the AWS ECR service, which enables us to push and pull Docker images. We have the option to choose between a public or private repository for our Docker registry.
Database: AWS provides the AWS RDS service, allowing us to provision a managed PostgreSQL database. We can configure the database engine as PostgreSQL, choose the db.t3.micro instance type for CPU configuration, and allocate 20GB of storage for the managed database.
Storage: AWS offers the AWS S3 service, which enables us to store static and media files. We can configure the storage capacity and choose whether to have public or private access to the stored data.

Design

After planning, we need a visual representation to provide an overview of our cloud infrastructure's big picture. This is crucial as it allows for changes to be made before implementation. Designing our solution based on the chosen cloud provider demonstrates how the services are interconnected and how the workflow of our cloud infrastructure will function. This step requires additional work and must take networking and security into consideration. Frameworks like AWS Well-Architected can assist in effectively accomplishing this task, but they are beyond the scope of this article.

For example, in the case of our backend API, we can design and create a diagram of our cloud infrastructure using a tool like draw.io. As seen in the picture below, our infrastructure is relatively simple. Users access the IP address of the AWS EC2 instance, which runs a Docker container using an image retrieved from AWS ECR. Our Docker image, stored and pushed to AWS ECR, connects to AWS S3 and AWS RDS, allowing our code to utilize these services.

Implement

All the previous steps are just the basic blocks that we need to complete before proceeding further. These steps ensure that we approach the tasks correctly. After completing these initial steps, we can move on to implementing our solution and provisioning the necessary services on the chosen cloud provider. For AWS, we have several options such as using the console dashboard, the AWS CLI, or utilizing Infrastructure as Code tools like AWS CloudFormation or Terraform.

Based on the planning stage, we have identified the required services and their configurations. Now, we can proceed with their implementation. The design we created earlier helps us understand the relationships between services and how the workflow will function in the cloud environment. For instance, you might consider aspects like Virtual Private Cloud (VPC) and security groups for the EC2 instance. However, for simplicity, let's focus on keeping things straightforward for now.

Deploy

Is everything set up correctly? Great! Now, let's utilize the configured services to deploy our code. In the case of our backend API, our code is containerized in a Docker image. We will push this image to ECR and update our code to utilize AWS RDS for the database and AWS S3 for storage. By using SSH to access our server, we can pull the Docker image and create a container to run our backend API. To access the API, we simply visit the IP address of the EC2 instance.

For deployment, we have the option to do it manually or leverage a CI/CD tool such as GitHub Actions or Jenkins for automated deployments. Both methods are effective, and we can even combine them. Initially, performing a manual deployment ensures that everything is functioning correctly. Then, we can proceed to write a CI/CD pipeline that automates the steps performed during manual deployment.

Monitor

We may need to set up monitoring tools so that you can gain an overview of your code and the performance of your cloud infrastructure. This will be beneficial if you aim to make improvements and require data-driven insights to make informed decisions.

Maintain

You will need to do this because cloud providers frequently update and remove features from their services. Therefore, it is important to regularly maintain your cloud infrastructure to ensure its continued functionality.

What is Next in This Series?

I will continue and write other articles about how to build cloud infrastructure on AWS, Azure, GCP, and Digital Ocean. I will use real-world examples and projects, following the steps mentioned above, and providing a more detailed approach so that you can learn how to build your own infrastructure.

Conclusion

Cloud computing has changed the game of software development, and companies are moving towards adapting it in their organizations. Knowing how to use it effectively will help you create efficient solutions. Therefore, relying solely on the cloud provider is not enough. You should have a detailed plan and defined steps to follow when building cloud infrastructure solutions for your applications. I hope this article has been helpful for you, and feel free to follow me for more articles like this or ask me anything related on my LinkedIn.

Methods of Deployment with Examples

Mustafa Abdallah — Fri, 12 May 2023 12:57:50 +0000

Introduction

Deployment is an important step in the software development lifecycle, as it involves shipping our code to the outside world. Making the process smooth, seamless, and error-free is crucial for tech people. During my journey as a Software Engineer, I've worked on various deployment methods - from the basic to the most modern and advanced ones. Today, I am writing about these different ways of deployment and how to implement them.

Deploy using Nginx
- What is Nginx?
- Deploying Manually
- Deploying Automatically With Linux Bash Script
Deploy using Docker
- What is Docker?
- How to deploy Docker Image?
Deploy using Kubernetes
- What is Kubernetes?
- How to deploy to Kubernetes?
Conclusion

Deploy using Nginx

What is Nginx?

Nginx is a web server that searches for our code in a specific folder and executes the command to run it, before serving it to the internet through port 80.

Deploying Manually

To deploy our code, we need a VPS or virtual machine, which are available on many cloud platforms such as AWS EC2, Azure Virtual Machine, Google Cloud Compute Engine, or Digital Ocean Droplet. These platforms provide a fresh Ubuntu server that can be customized as needed. In our case, we will need to install Nginx and any required packages for our code. If our code requires a database, we can also set it up on the same server and connect it to our code.

For instance, I recently worked on a project that involved building a backend API using Django and PostgresSQL as the database. To deploy the API, I first set up an Ubuntu server on a Digital Ocean Droplet, installed Nginx, required Python packages, and PostgresSQL. Then, I uploaded my code to the server using Git, configured PostgresSQL and Nginx, and voilà! Our backend API was live and accessible via the Digital Ocean Droplet IP address. For more information and a step-by-step guide, I highly recommend reading this Digital Ocean tutorial, as it was my reference for all similar deployments.

Deploying Automatically With Linux Bash Script

All of the above is great, but when entering the world of automation, we want everything to be automated! That's exactly what I did when I built a new project. Instead of following a step-by-step guide every time, I wrote a Linux Bash script to automate these steps for me!

My project was a backend API built with Django and PostgresSQL, and I needed to deploy it with Nginx on Google Cloud Compute Engine. I set up an instance using Terraform as shown here, and then wrote a Linux Bash script that automates all the necessary steps, as shown here. I attached the script to the instance so that when it launches, it runs the script and sets everything up for me automatically.

Deploy using Docker

What is Docker?

Docker is a containerization technology that allows us to package our code and required packages into an image called a Docker image. We can then launch containers from this image as many times as we want.

How to deploy Docker Image?

There are many ways to deploy Docker images, but in this section, I will talk about deploying a Docker image to a virtual machine. As mentioned above, we could launch an Ubuntu server or virtual machine on any public cloud provider like AWS, GCP, or Azure. What we care about is that we need to install Docker on this Ubuntu server, then we push our Docker image to the Docker registry. After that, we pull the Docker image on our server and run some scripts to create a container from it, which makes our code accessible via port 80. Then, we can access the IP address of the server and see our code served.

For example, when I was developing the CrewTech project, I needed to deploy it on AWS EC2 using Docker. First, I built and packaged a Docker image from the code and then pushed it to AWS ECR. After that, I set up AWS EC2 to install Docker once it launched using Terraform. Finally, I just ran a script that pulled this Docker image and created a container from it that was accessible via port 80. This made our backend API or Docker image accessible via the AWS EC2 IP address.

This process could be done manually, as I did here, or by writing a CI/CD pipeline that does all of these steps automatically every time we push new code, as I did here with CircleCI.

Deploy using Kubernetes

What is Kubernetes?

Kubernetes is an orchestration tool that manages multiple containers, and it is very handy when it comes to microservices because each service has its own Docker image, and the number of containers can become larger as the number of microservices increases.

How to deploy to Kubernetes?

Kubernetes requires that the Docker image be pushed to any Docker registry, and it takes care of all the other parts. While this may seem simple, the Kubernetes system is more complex. Kubernetes is availabe on my cloud providers as a manged service like AWS EKS, Azure AKS, and GCP GKE. All of them gives you the same Kubernetes instance.

For example, when I built Docorvter, which is a PDF-to-HTML document converter, I needed to deploy it on AWS using Kubernetes. After building and pushing the Docker image of the backend, I wrote deployment and service files that pulled this image. Then, Kubernetes took care of the rest and gave me an accessible IP address that I could visit to see my backend API.

Kubernetes is a very efficient tool when it comes to microservices, as we often have many microservices, each with its own Docker image, and we need a container for each one. As our services grow, things can become complicated, but with Kubernetes, we can manage them easily. For example, in the Fuilder project, which is a dynamic form builder, each service has its own Kubernetes deployment and service files that are managed by one Kubernetes instance.

Conclusion

All methods of deployment work, and any one of them could be ideal for you based on your requirements. I have seen many projects that only require Nginx and an Ubuntu server to be deployed and don't actually need advanced technologies like Docker and Kubernetes. I hope this article has been helpful for you, and feel free to follow me for more articles like this or to ask me anything related on my LinkedIn.

Terraform and Microservices

Mustafa Abdallah — Thu, 11 May 2023 12:20:06 +0000

Introduction

When I was improving my skills and learning about Microservices, Cloud, and DevOps, I faced a problem when migrating from Monolithic Architecture to Microservices Architecture, specifically in the part where I had to provision the cloud infrastructure with Terraform. The issue was how to write the cloud infrastructure successfully for each service in our microservices program without duplicating code and ensuring that each service’s infrastructure was standalone, without affecting other services. This was challenging because microservices follow the Single Responsibility Principle. Today, I am writing this article to share how I solved this issue for anyone who may wonder how to use Terraform with Microservices.

What is Terraform?

Terraform is an infrastructure-as-code (IaC) tool that allows us to provision cloud infrastructure using code instead of relying on manual configuration through the cloud console. This can be very convenient, especially if you have a coding background. To learn more about Terraform, you can visit their website.

What is Microservices?

Microservices is a backend architecture that follows the Single Responsibility Principle. It involves breaking down a monolithic application into small services, each of which performs a specific function and communicates with each other via APIs and messaging services such as RabbitMQ. The benefits of microservices include the ability to use different programming languages to build these services and the flexibility of having different teams with different backgrounds and technical skills work on them. However, implementing and debugging microservices can be complex due to its distributed nature.

Terraform and Monolith

When it comes to regular backend architecture or monolithic architecture, using Terraform is relatively straightforward. We can write the infrastructure code for one backend API in a single file or use modules if we have a more complex infrastructure. Ultimately, we’re dealing with just one thing, so it’s much simpler compared to implementing microservices architecture.

For example, in my latest project, I built a PDF-to-HTML document converter using a monolithic architecture. I deployed this backend API on AWS infrastructure using Terraform, and I only needed to use AWS S3 and AWS RDS. As a result, my Terraform folder was quite simple, as shown below:

├── aws.tf
├── main.tf
└── modules
    └── aws
        ├── RDS.tf
        ├── S3.tf
        └── main.tf

Terraform and Microservices

On the other hand, when we divide a monolithic service into many microservices, each service has its own infrastructure. This can make things more complicated, especially if we try to do it the old-fashioned way, which can lead to code duplication and difficulties in managing each service’s infrastructure.

The solution to this problem was to use modules to make each service standalone. First, we created a module for the common cloud infrastructure and then created a separate module for each service that used this common module. With this approach, we were able to avoid code duplication and manage services more easily.

For example, in my latest project, I built a form builder using microservices architecture, and I needed to deploy the infrastructure on Google Cloud Platform (GCP). I had three services, each of which needed a Google Storage Bucket and a Google SQL Database. To accomplish this, I created modules for GCP to provision these common GCP services, and then created a separate module for each service under the services’ folder. These modules used the common GCP modules, as shown below:

├── README.md
├── main.tf
└── modules
    ├── gcp
    │   ├── sql_database
    │   │   └── main.tf
    │   └── storage_bucket
    │       └── main.tf
    └── services
        ├── form_builder_service
        │   └── main.tf
        ├── forms_service
        │   └── main.tf
        └── users_service
            └── main.tf

Conclusion

Overall, Terraform is a great tool, and when it comes to microservices, it can become more complicated. However, with Terraform modules, things become easier to implement. I hope this article has been helpful for you, and feel free to follow me for more articles like this or ask me anything related on my LinkedIn.

Building a Document Converter with the Factory Method pattern using Python

Mustafa Abdallah — Tue, 28 Feb 2023 16:33:24 +0000

Introduction

I am implementing real-world projects using the design patterns to see how we could benefit from them when building our own software, in this article I am building a Document Converter using the Factory Method design pattern, so let's get started!

What is the Factory Method pattern?

Explaining the pattern is out of scope of this article, and you could take a look at the references to know more about this pattern, how we use it and the UML diagram for it. Please make sure you understand the pattern before moving on.

What is the Document Converter?

A Document Converter is a software tool that allows converting documents from one format to another. in this we will build a Document Converter that can convert between various document formats (e.g. PDF, DOCX, HTML) using the Factory Method pattern. Each converter (e.g. PdfConverter, DocxConverter, HtmlConverter) could be implemented as a separate factory that takes in the input document and returns the corresponding output format.

UML diagram for the Document Converter

According to the Factory Method UML diagram, the Creator will be the Converter and the Concrete Creators will be (PDFConverter, DocxConverter, HTMLConverter) and the Product will be the Document and the Concrete Products will be (PDFDocument, DocxDocument, HTMLDocument), so let's see how we implement that in code.

Code implementation

- Step 1: Create the Document class

# document.py

class Document:

    def __init__(self):
        self._file = ""
        self._name = ""

    def read_file(self):
        print(f"Reading {self._file}")

    def process_file(self):
        print(f"Processing {self._file}")

    def convert_file(self):
        print(f"Converting {self._file} to {self._name}")

- Step 2: Create the PDFDocument, DocxDocument and HTMLDocument

# document.py

class PDFDocument(Document):

    def __init__(self, file):
        super().__init__()
        self._file = file
        self._name = "pdf document"

class DocxDocument(Document):

    def __init__(self, file):
        super().__init__()
        self._file = file
        self._name = "docx document"

class HTMLDocument(Document):

    def __init__(self, file):
        super().__init__()
        self._file = file
        self._name = "html document"

- Step 3: Create the Converter class

# converter.py

from document import Document
from abc import ABC, abstractmethod

class Converter(ABC):

    def convert(self, file, to):
        document: Document = self._create_document(file, to)
        document.read_file()
        document.process_file()
        document.convert_file()
        return document

    @abstractmethod
    def _create_document(self, file, to):
        pass

- Step 4: Create the PDFConverter

# converter.py

from document import PDFDocument, HTMLDocument, DocxDocument

class PDFConverter(Converter):

    def _create_document(self, file, to):
        if to == "pdf":
            return PDFDocument(file)
        elif to == "html":
            return HTMLDocument(file)
        elif to == "docx":
            return DocxDocument(file)
        return None

# The DocxConverter and HTMLConverter will be the same

Testing the Document Converter

We could test the PDF Converter as follows:

pdf_converter = PDFConverter()
document = pdf_converter.convert(file="file.pdf", to="docx")

And here is the result:

Reading file.pdf
Processing file.pdf
Converting file.pdf to docx document

Conclusion

So this is just a simple Document Converter that show us how to use the Factory Method design pattern in a real-world application, I hope this article was helpful for you and follow me for more articles like this.

References

How to implement a cache manager with the singleton pattern using Python

Mustafa Abdallah — Mon, 27 Feb 2023 16:24:17 +0000

Introduction

In this article we will build a cache manager that stores frequently used data in memory using the Singleton pattern where the Singleton instance can ensure that the data is accessed efficiently and that memory usage is optimized.

UML Diagram of the Cache Manager

Code Implementation

The cache manager is very simple approach, we need to save data with predefined key and then use this key to retrieve this data and this approach could be achieved through the set and get methods as shown below, we will also use the singleton pattern that require a static method and private constructor, so we could get only one instance from the class every time we call it

class CacheManager(object):

    _instance = None
    _cache_store = []

    def __init__(self):
        raise RuntimeError("Call get_instance() instead")

    @classmethod
    def get_instance(cls):
        if cls._instance is None:
            cls._instance = cls.__new__(cls)
        return cls._instance

    def set(self, key, value):
        self._cache_store.append((key, value))

    def get(self, key):
        value = None
        for item in self._cache_store:
            if key in item:
                value = item[1]
        return value

This is just a simple example of cache manager, where it has two methods for setting and getting cache values.

Testing the Cache Manager

We could test the cache manager as follows:

cache1 = CacheManager.get_instance()
cache1.set("name", "Mustafa")

cache2 = CacheManager.get_instance()
cache2.set("role", "backend engineer")

print(cache2.get("name")) # note we use cache2 to get the name
print(cache2.get("role"))

And here is the result:

Mustafa
backend engineer

As you can see, even creating multiple instances from the cache manager, but we still access only one instance.

Conclusion

This cache manager could be useful in any Python application, where it caches data in memory and optimize the application performance.

References

Head First Design Patterns: A Brain-Friendly Guide
Singleton Pattern – Design Patterns (ep 6) by Christopher Okhravi
I used draw.io for drawing the UML diagram

How to implement a logging system with the singleton pattern using Python

Mustafa Abdallah — Mon, 27 Feb 2023 00:28:01 +0000

Introduction

I am implementing real-world projects for each design pattern to see how they are useful when building projects, here is an article about how to benefit from the singleton pattern to build a logging system.

Logging is an important aspect of software development, as it allows developers to track what their applications are doing and to identify and fix issues as they arise. One design pattern that can be useful in implementing a logging system is the Singleton pattern, which ensures that there is only one instance of a class in the system.

UML Diagram of the Logging System

Code Implementation

Implementing a logging system with the Singleton pattern is to define a Logger class that will handle all the logging functionality. The Logger class should have a private constructor and a static method that returns the single instance of the class.

class Logger(object):

    _instance = None
    _logs = ""

    def __init__(self):
        raise RuntimeError('Call get_instance() instead')

    @classmethod
    def get_instance(cls):
        if cls._instance is None:
            cls._instance = cls.__new__(cls)
        return cls._instance

    def debug(self, message):
        self._logs += f"[DEBUG] {message}\n"

    def info(self, message):
        self._logs += f"[INFO] {message}\n"

    def warning(self, message):
        self._logs += f"[WARNING] {message}\n"

    def error(self, message):
        self._logs += f"[ERROR] {message}\n"

    def get_logs(self):
        print(self._logs)

This is a just a simple example of logging system, where it has four methods for each log level and one more method for getting the logs.

Testing the logger

We could test the logger as follows:

logger = Logger.get_instance()
logger.debug("my error")
logger.get_logs()

logger2 = Logger.get_instance()
logger2.info("my error2")
logger2.get_logs()

And here is the result:

[DEBUG] my error

[DEBUG] my error
[INFO] my error2

As you can see the new logs added to the old logs which means that the logger is instanced only once.

Conclusion

Implementing a logging system using the Singleton pattern can be a useful approach for managing logging functionality in a Python application. By defining a Logger class with a private constructor and a static method that returns the single instance of the class, we can ensure that there is only one instance of the class in the system, and simplify the process of logging messages.

References

Head First Design Patterns: A Brain-Friendly Guide
Singleton Pattern – Design Patterns (ep 6) by Christopher Okhravi
I used draw.io for drawing the UML diagram

About Mustafa

A Backend Engineer with a Computer Engineering degree. Experienced in collaborating remotely with international teams across Ireland, Canada, and Kenya. Also, I've helped clients in Germany, Sweden, Switzerland, and the USA with freelance projects. With strong professional English communication skills and a flexible attitude towards learning new technologies, I am seeking an opportunity to enhance my experience by joining a skilled international and professional team and tackling challenging problems. Learn more about me on UpWork and LinkedIn

Forem: Mustafa Abdallah

Azure Cloud Infrastructure: Step-by-Step Guide

Introduction

Contents

Our Project

Building The Cloud Infrastructure

Define

Choose

Plan

Design

Implement

The infrastructure folder

Set up Azure provider on Terraform

Provision Azure Virtual Machine

Provision Azure Database PostgreSQL Server

Provision Azure Blob Storage

Provision Azure Container Registry

Add them to the main

Apply our infrastructure

Get the output

Deploy

Conclusion

AWS Cloud Infrastructure: Step-by-Step Guide

Introduction

Contents

Our Project

Building The Cloud Infrastructure

Define

Choose

Plan

Design

Implement

The infrastructure folder

Set up AWS provider on Terraform

Provision AWS EC2

Provision AWS RDS

Provision AWS S3

Provision AWS ECR

Add them to the main

Apply our infrastructure

Get the output

Deploy

Conclusion

How to Build a Cloud Infrastructure

Introduction

Table of Contents:

What is Cloud Infrastructure?

Steps of Building a Cloud Infrastructure

Define

Choose

Plan

Design

Implement

Deploy

Monitor

Maintain

What is Next in This Series?

Conclusion

Methods of Deployment with Examples

Introduction

Contents

Deploy using Nginx

What is Nginx?

Deploying Manually

Deploying Automatically With Linux Bash Script

Deploy using Docker

What is Docker?

How to deploy Docker Image?

Deploy using Kubernetes

What is Kubernetes?

How to deploy to Kubernetes?

Conclusion

Terraform and Microservices

Introduction

What is Terraform?

What is Microservices?

Terraform and Monolith

Terraform and Microservices

Conclusion

Building a Document Converter with the Factory Method pattern using Python