Forem: Daniel Vigueras

Best Practices Setting up Your Local Development Environment

Daniel Vigueras — Wed, 22 Jun 2022 08:51:12 +0000

Best Practices Setting up Your Local Development Environment

You are about to start a new project and then you wonder what's the best option to set up your local development environment. Your OS could be Linux, Mac, or Windows. However other people working on the same project may be running a different OS. There are many alternatives to install and use all the tools you need but, what pros and cons have each one of them?

This post presents different methods and best practices when setting up your local development environment along with the advantages and drawbacks of each option.

Services in your Machine

This is the first alternative when setting up a local environment and is based on manually installing on your computer all the required software. A simple web environment usually includes the Web Server (Apache and Nginx are the most common), the Language Interpreter (PHP/Node/Python/etc.), and a Database Engine (usually MySQL or PostgreSQL).

So you start downloading, installing, and setting up every component. This can be done by visiting the web page for every component needed and downloading the installer, but the best option is to use a package manager. Package managers will make this process easier and will allow you to keep your software updated to the latest version.

The specific package manager depends on your Operating System. In macOS the most popular tool is Homebrew. In Linux distributions like Ubuntu/Debian, you can use apt, and in RedHat/CentOS distributions you can use yum. In Windows the most popular tool is Chocolatey.

Once all the required software is installed you need to do extra steps to set up the environment:

Create the Virtual Host in the Web Server and point it to your language interpreter service
Create a new User in the Database Engine, a new Database and the give the proper permissions to that User
Configure your code to use the Database credentials
Repeat the process for every extra service required in the project (NoSQL databases, caches, etc.)

This process is not hard once you know how to do it, however, the following problems arise.

All the devs in the project don't share the same environment, they may have different operating systems and even different software versions. It may trigger situations where tests pass locally but not in CI/CD and vice-versa. Also, a bug that is happening in the computer of a developer may not happen on the computer of another developer.

Another issue is that if you reinstall your computer you have to repeat every step again. This is boring and tedious, also the installation and configuration method of these packages depends on your operating system. The instructions to set up your local dev environment won't be the same for every member of the project and then they must be written and duplicated for every OS.

There is a better alternative when setting up a local development environment: using Virtual Machines.

Virtual Machines

Not every developer in a project will use the same Operating System so the solution is for everyone to run a Virtual Machine (VM) on their computer with the same OS (e.g: Ubuntu Linux 22.04). With this method, all the developers will have the same environment and many problems from the previous section will be solved.

A free alternative to run Virtual Machines on your computer is VirtualBox. VirtualBox runs on Mac, Windows, and Linux. Using the interface you can create, start, stop, and destroy Virtual Machines. Once the Virtual Machine is created we can install all the software needed for our development environment. Also, we need to share (mount) the directory where the project files are located on our computer with the Virtual Machine.

However, creating and setting up a Virtual Machine manually is not the best in terms of productivity. The solution is to use a tool that will allow you to create and set up these environments automatically. This tool is Vagrant.

With Vagrant you just need to create a Vagrantfile in the root directory of your project and then run vagrant up in the terminal to create and start your Virtual Machine. This is an example of a simple Vagrantfile after running vagrant init ubuntu/jammy64:

Vagrant.configure("2") do |config|

  #
  # Use Ubuntu 22.04 LTS (Jammy Jellyfish) 
  #
  config.vm.box = "ubuntu/jammy64"

end

Vagrant needs boxes, which is the package format of installed Operating Systems images. In this example, we are using ubuntu/jammy64 which is an image of Ubuntu's latest LTS version (22.04). Just with this file, we have solved the problem of creating and installing a new Virtual Machine and sharing our project files in the /vagrant path inside the VM. Boxes can be versioned too, so you can update to the latest version of a box at any moment.

Now we need to install and set up (provision) the services needed inside the VM. The first option would be to do this manually, but, as in the previous section, there are many issues that we want to avoid. The idea is to do it automatically with a command.

Vagrant supports provisioning automatically using provisioners. A provisioner could be something like a shell script with every step to install and set up all the software dependencies needed.

A shell script could work, but the best option is to use an orchestration/automation tool like Ansible, Chef, Puppet, or Salt. Using these tools you can define the steps needed to set up your project in a set of files, and just by invoking vagrant provision the VM will be set up. The advantage is that setting up a new development environment with this method is easy, fast, and can be shared with all the members of the project.

Example of a Vagrantfile using Ubuntu 22.04 and running Ansible with the playbook.yml file as the initial playbook:

Vagrant.configure("2") do |config|

  #
  # Use Ubuntu 22.04 LTS (Jammy Jellyfish) 
  #
  config.vm.box = "ubuntu/jammy64"

  #
  # Run Ansible from the Vagrant Host
  #
  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "playbook.yml"
  end

end

This scenario is much better than the previous one where everything is installed directly on your machine. Now there is only one environment and everything is shared within the team and automated. However, there are still some issues that can be improved:

Machines are slow to create: The first time you create a new VM it will take some time to download, import, and create the VM in Virtualbox.
Machines are slow to provision: The first time you provision a VM it will also take some time to download and set up all the packages required by the provisioner.
High disk space usage: every project requires a new and independent VM, which can take many Gigabytes of storage. It means that if you are working on multiple projects the disk space needed could be very high.
High memory usage: if you need to have more than one project running at the same time the different VMs needed will consume a lot of memory in your computer.

Because of all these reasons, there is a better alternative than using directly Virtual Machines: using Containers.

Containers

Instead of having to install and set up every service, you can use containers. With containers, you can take profit from existing images built officially. You only need to run a container with the software you need and it will start in a matter of seconds. It will take an extra time the first time because Docker needs to download the image from Docker Hub.

Docker Hub is a container image library where you can find interesting images to use in your projects like the Apache image, Nginx image, PHP image, Node.js image, MySQL image, etc. These images are ready to use, and they come with instructions about how to use them. Usually, you have to pass environment variables to the container and/or tweak a couple of config files.

Linux is the only Operating System supporting containers natively, so if you are using macOS or Windows you'll need software to run a VM with Docker in the background for you. Compared with the previous section the advantage is that we will need just one VM for all of our projects, not a VM for every project. The alternatives to running Docker in these systems are:

Docker for Desktop: The most popular option.
Podman: An alternative container engine with the same CLI interface as the docker command.

Imagine a project where you need two containers: a Node.js container and a Database container. One option would be to start both containers from the command line using the docker run command and passing every argument needed (files to mount, environment variables, ports to publish, etc). However, this is a manual process and is error-prone. There is a better solution to manage a set of containers: Docker Compose.

Docker Compose lets you define a set of containers using a docker-compose.yml file. This file is in YAML format and defines every container you need along with the configuration parameters.

Example of docker-compose.yml file with a PHP 8.1 installation, an Apache Web Server and a MySQL Database:

version: '3'
services:
    app:
        image: 'php:8.1-apache'
        ports:
            - '80:80'
        volumes:
            - '.:/var/www/html'
        depends_on:
            - mysql
    mysql:
        image: 'mysql:8.0'
        environment:
            MYSQL_ROOT_PASSWORD: 'rootpassword'
            MYSQL_ROOT_HOST: "%"
            MYSQL_DATABASE: 'appdb'
            MYSQL_USER: 'app'
            MYSQL_PASSWORD: 'appsecret'
            MYSQL_ALLOW_EMPTY_PASSWORD: 1

Having this file in the root directory of your project and running docker compose up would start both containers (app and mysql) in Docker. After working with the project you just need to type docker compose stop and both containers will be stopped. This process is very easy to learn and to apply to any developer in your team and it just requires having Docker installed, no extra dependencies needed.

It also has an extra advantage: a Dockerfile could be created at the root of the project and then you will have a way of building and publishing an image of your application in your CI/CD service ready to deploy.

Conclusion

Installing the services directly on your machine has the problem of having a different environment for every developer in the project.

This problem can be solved using Virtual Machines with the same environment for every member of the project, but it has the drawback of speed and resources needed when working with multiple projects.

Using containers solves this problem by having just one Virtual Machine on your computer (or even zero if you are using Linux). Working with containers also prepares your application to be deployed in container-based services in staging and production environments, like Kubernetes.

Boosting Your Development Feedback Loop with Preview Environments

Daniel Vigueras — Tue, 07 Jun 2022 09:20:28 +0000

Boosting Your Development Feedback Loop with Preview Environments

The process of developing a new feature for a project usually involves the same steps:

Create a new branch in the repository for the new feature (feature branch)
Commit the code and create a new Pull Request
Review the code changes
Merge the Pull Request into the base branch
Deploy the base branch to the Staging environment
Test the feature (QA) in Staging
Approve and deploy to Production or request changes opening the PR again.

If you follow git-flow (or an alternative flow based on feature branches) this will sound familiar to you. The main point here is that until you don't deploy your new feature to the Staging environment it can't be tested in a realistic environment (local environments don't count 🙃 ).

Also, this workflow leads to the following questions:

What happens when the team is developing multiple features at the same time?
Is the code in the base branch always deployable (in terms of business logic)?
What if two new features are merged and tested but the team only wants to release one of them and delay the other one?
Which feature is responsible for a bug found in Staging if two or more feature branches were merged into the base branch?

As you imagine this way of working with branches and environments produces situations which would be better to avoid. The solution is to implement Preview Environments (sometimes also known as Preview Deployments).

A Preview Environment allows us to have a new complete and isolated environment for each new feature. The advantages of having Preview Environments are:

Test features independently: preview features
Improves feedback loop
Avoid merging into the base branch until the feature is approved
Avoid mixing new features in the Staging environment
Preview changes as they would look in Production

The following diagram reflects different environments: Production, Staging, and one Preview Environment for every active Pull Request

Lifecycle

Preview Environments are disposable environments, which means that they are created, updated, and destroyed. They have the purpose of testing a feature, and once the task is done it doesn't make sense to keep them alive. This process can take from minutes to months, it depends on the size of the feature, the time it takes to test it, if it makes sense to merge the Pull Request at the moment due to business requirements, etc.

When a new Pull Request is created a new Preview Environment is created automatically with the name of the Pull Request.

If the Pull Request is updated with new commits then it is deployed again to show the latest changes.

When a Pull Request is merged or closed the environment is destroyed automatically.

To be able to test a Preview Environment, every environment must have a unique preview URL. This URL can be automatically generated from the branch name. e.g.: for a branch named feature/user-register in a project named Manhattan the preview URL could be https://feature-user-register.manhattan.com.

Environment Data

Another advantage of Preview Environments is that data can be seeded and updated without affecting other environments. It means that every environment must create a new exclusive database when the environment is created, and delete the database once the environment is destroyed.

The initial data of this database can be cloned from the Staging database or seeded with default values. This makes the data in this environment independent from other environments, so QAs can create, update and delete anything they need without worrying about affecting other QAs testing other features. Also, the data in the environment can be reset to the initial values as needed.

Service Dependency

When we think of a new feature in a project we tend to think about a new branch in a repository, this can be usual in projects with a single repository and a single service, but this is not the common case in modern projects where multiple services are involved.

A new feature could concern a backend service and a frontend service at the same time. For the new feature, the frontend could depend on a new endpoint developed in the backend. Therefore we need a way to coordinate both repositories if we want to test the new feature. We can't just create a new Preview Environment for the frontend and then use the API available in the Staging environment because the new endpoints in the backend won't exist.

A good practice is to use the same branch name in both repositories, e.g.: feature/new-feature. Then we could deploy both backend and frontend to the same Preview Environment. By doing this we could be able to test the new feature completely and the frontend will use the backend endpoints of the new feature.

Setting up Environments

Creating a Preview Environment can be done in many different ways. Creating the environment depends on the design of the project infrastructure and the way of deploying the application to Production. In this article, I am going to cover two options: deploying the app code directly to a server and using Docker containers.

Deploy code to servers

This process deploys the code to a server using methods like uploading the code using SFTP via SSH. As the environment doesn't exist it requires some steps before being able to deploy the code. Supposing we already have an existing server dedicated to Preview Environments, the step to create a new Environment would be:

Install and set up everything needed to run the code to be deployed: Web server, Database, Language Runtime, Virtual hosts, etc.
Create DNS records for the preview URL(s)
Deploy the code using SFTP

This way of creating Preview Environments has the following problems:

Slow: Installing and setting up services takes a lot of time. Also, if the code of the project has hundreds of files the deployment process will be slow too. We want to have these Preview Environments ready as soon as possible.
Complex: There are a lot of steps to maintain. We need an orchestration tool (like Puppet, Chef, or Ansible) to execute all the steps required and maintain that code in the future.
Error-prone: Sometimes setting up a virtual host can fail due to a misconfiguration of the webserver or a repository with an external dependency can be down. If the process fails the environment won't be created.

Containers

A much better alternative is to use Docker containers. Supposing we already have an existing server dedicated to Preview Environments with Docker installed, the steps to create a new Environment would be:

Create DNS records for the preview URL(s)
Build the application container in the CI/CD service
Upload the container to a container image registry
Connect to the server and run the Docker container

This process seems to be more agile than before, but there are still some questions to solve:

How are the Docker services exposed as external services? We would need something like a container to act as a reverse proxy with forwarding rules.
How do we request the issue and manage the renovation of SSL certificates using Let's Encrypt?
How many servers do we need for Preview Environments in our organization?
How do projects map to Preview Environment servers in the organization?

It would be great if we could have a cluster of servers and we could forget about the specific server where the code must be deployed. If we need more resources we can just add more servers to the cluster. The best alternative to achieve this is to use a container scheduler like Kubernetes. We just need:

A Kubernetes cluster in a provider (like DigitalOcean Kubernetes, Civo, Google GKE, Amazon EKS, etc)
Write Dockerfiles for every repository
Write the Kubernetes manifests to deploy every service

Writing Dockerfiles and Kubernetes manifest is out of the scope of this article.

Instead of using a provider to launch a maintain your own Kubernetes cluster you can use existing services to achieve the same result. One of my favorite options is Okteto Kubernetes service and their feature to create Preview Environments from our code. This allows us to forget about creating and maintaining the Kubernetes cluster, dealing with SSL certificates, and creating DNS records so that we can focus on what provides value for the project.

Conclusion

Using only environments like Staging and Production is not a good idea
Having an independent environment for every new feature improves the feedback loop of the team
Getting feedback fast means you decrease the time needed to deploy a new feature to Production, and also the costs associated
Using containers and schedulers like Kubernetes simplifies the creation of Preview Environments
Working with external services like Okteto makes the whole process easier to create and maintain