Forem: Alfonso Domenech

Build Faster: Your Guide to a Quick-Start Project Template

Alfonso Domenech — Mon, 18 Dec 2023 14:20:47 +0000

Every time I've wanted to embark on a personal project - to learn a new technology or develop an idea - I've found myself redoing configurations for code structure and maintenance, not to mention execution and deployment. It can be frustrating, but I've come to realize that these steps are crucial for success. That's why I'm sharing a minimal template with the most common configurations - the essentials that I believe will help you achieve your goals. We are going to introduce and configure the following technologies.

NestJS
NVM
Commitlint
Husky
Docker
Docker compose
Github Actions

If you want to take a look at it while you read, you can check out the official repo.

NestJS

NestJS is our backend superhero in the template repo. It's like a magic trick for building solid server-side applications effortlessly in TypeScript or JavaScript. Check out more at NestJS Official Website.

Creating .nvmrc file

In this part of the process, the whole team must work with the same version of NodeJS to ensure that all changes made to the application are compatible. This can also be done with Docker, but we will see it later.

In the .nvmrc file we have to put the version of node we want to use.

Then we have to install nvm.

Configure commitlint

If you've worked on projects with a team, you may have noticed that everyone creates different branches and commits changes with random messages. This can make things confusing. That's where commitlint comes in. It helps make sure that commit messages are more consistent and easier to understand.

💡 Here we have to explain the commit convention and if we want to add our custom commit configuration. Also, the benefits of including commitlint in the projects.

Now we have to set up commitment in our project following the instructions of the library commitlint.

First, we must install the library and the convention we want to follow. In our case, we are going to use conventional commits.



npm install --save-dev @commitlint/cli @commitlint/config-conventional

Then we have to configure commitlint to use the conventional config. Let´s create our commitlint.config.js.



module.exports = {
  extends: ['@commitlint/config-conventional'],
};

Set up Husky

Husky is a library which allows developers to execute some commands in the different Git hooks. Why do we want this? As we said we need to ensure a little bit of homogeneity in our project with husky so we can achieve this. Let´s see how.
So, there's this neat library called husky that developers can use. It helps you run specific commands in various Git hooks(https://git-scm.com/docs/githooks), which is a fancy way of saying it keeps your project looking consistent. Why is that a good thing? Well, it makes everything easier to understand! Want to know more? Let me break it down.

First, we need to install husky in our project. For this we are going to follow husky automatic installation which is recommended.



npx husky-init && npm install

This command executes the next actions.

Add the prepare script to package.json.
Create a sample pre-commit hook that you can edit (by default, npm test will run when you commit).
Configure Git hooks path.

If you want manual installation and customize the husky configuration you can read the corresponding documentation.

Now we have to configure husky to execute some commands in the git hooks. We are going to add three that from my point of view are the most important.

To add a new hook we have to follow the next structure.



npx husky add .husky/{gitHook} "{command}"

This command is going to create different bash files in the .husky directory. The names of these files are going to be named with the correspondent git hook.

The first one is the commit-msg hook. Here we are going to ensure that our commit messages follow our commitlint configuration.

So, let´s add our new hook! Following the command structure mentioned above.



npx husky add .husky/commit-msg  'npx --no -- commitlint --edit ${1}'

The second one is the pre-commit hook. With this hook, we are going to lint and format the files we want to add to our commit. This action is executed before committing our changes.



npx husky add .husky/pre-commit  'npm run lint && npm run format'

The last one but not the least, is the pre-push hook. This hook is going to execute our test before pushing our changes.



npx husky add .husky/pre-push  'npm run test'

Docker

Now we want to create our development environment for our NestJS project. In this part of the article we are going to see how to include Docker and Docker compose.

First, we need to configure our Docker container creating our Dockerfile.

Our Dockerfile employs multi-stage builds, which facilitate the setup of our docker container for both local and development settings. This method offers a range of advantages. It's a two-step process, compile dependencies in one stage and then keep only the essentials in the final image improving our Docker image performance. To learn more about multi-stage builds you can check the official documentation



# BUILD FOR LOCAL DEVELOPMENT

FROM node:20-alpine As development

# Create app directory
WORKDIR /usr/src/app

# Copy application dependency manifests to the container image.
# A wildcard is used to ensure copying both package.json AND package-lock.json (when available).
# Copying this first prevents re-running npm install on every code change.
COPY --chown=node:node package*.json ./

# Install app dependencies using the `npm ci` command instead of `npm install`
RUN npm ci

# Bundle app source
COPY --chown=node:node . .

# Use the node user from the image (instead of the root user)
USER node

# BUILD FOR PRODUCTION

FROM node:20-alpine As build

WORKDIR /usr/src/app

COPY --chown=node:node package*.json ./

# To run `npm run build` we need access to the Nest CLI which is a dev dependency. In the previous development stage we ran `npm ci` which installed all dependencies, so we can copy over the node_modules directory from the development image
COPY --chown=node:node --from=development /usr/src/app/node_modules ./node_modules

COPY --chown=node:node . .

# Run the build command which creates the production bundle
RUN npm run build

# Set NODE_ENV environment variable
ENV NODE_ENV production

# Remove husky from the production build and install the production dependencies
RUN npm pkg delete scripts.prepare \
    npm ci --omit=dev

USER node

# PRODUCTION

FROM node:20-alpine As production

# Copy the bundled code from the build stage to the production image
COPY --chown=node:node --from=build /usr/src/app/node_modules ./node_modules
COPY --chown=node:node --from=build /usr/src/app/dist ./dist

# Start the server using the production build
CMD [ "node", "dist/main.js" ]

Docker-compose configuration

Docker-compose is our dev sidekick, making local coding a cakewalk. One config file and that´s it! No more "it works on my machine" drama. Our docker-compose file is located at the root level of our project.



services:
  api:
    build:
      dockerfile: Dockerfile
      context: .
      # Only will build development stage from our dockerfile
      target: development
    volumes:
      - .:/usr/src/app
    env_file:
    - .env
    # Run a command against the development stage of the image
    command: npm run start:dev
    ports:
      - 3000:3000

Now it's a piece of cake to develop our app with the container system we have. It's super easy and hassle-free.

Github Action

Continuous Integration (CI) is a software development practice that allows developers to automatically build, test, and validate their code changes in a centralized and consistent environment. GitHub Actions is a powerful CI/CD (Continuous Integration/Continuous Deployment) platform integrated directly into GitHub repositories, enabling developers to automate their workflows seamlessly. Checkout their documentation to learn more!

In this section, I'll guide you through the process of setting up a basic CI pipeline using GitHub Actions. This pipeline will automatically build and push your docker image to the Docker Hub account when a Pull Request is merged.

Great! So, to push our Docker images, we need to authenticate our GitHub action. Don't worry, it's quite simple. First, let's add your Dockerhub username and token. After that, we just need to store the credentials in our GitHub repository settings. You can see how to do this in the image below. Let me know if you need any help with this.

Awesome! Let's create our workflow now. We need to place it in the root of your directory at .github/workflows/build.yml. This file will help us set up our CI steps. Don't worry, I'll guide you through the different steps so you can easily follow along!



# This workflow is triggered when a pull request is closed (merged or closed without merging) into the main branch.
on:
  pull_request:
    branches: [main]
    types: [closed]
jobs:
  #This defines a job named "build" that runs on the latest version of the Ubuntu environment.
  build:
    name: Build Docker image
    runs-on: ubuntu-latest
    steps:
      - # This step checks out your Git repository content
        name: Checkout
        uses: actions/checkout@v4

      - # Uses the docker/login-action to log in to Docker Hub using the provided username and token.
        # The credentials are stored as secrets (DOCKERHUB_USERNAME and DOCKERHUB_TOKEN).
        name: Log in to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - # Extracts the repository name from the GitHub repository full name and sets it as an environment variable (REPO_NAME). 
        # This information can be useful for later steps.
        name: Get repository name
        run: |
          repo_full_name="$GITHUB_REPOSITORY"
          IFS='/' read -ra repo_parts <<< "$repo_full_name"
          echo "REPO_NAME=${repo_parts[1]}" >> $GITHUB_ENV

      - # Uses the docker/metadata-action to extract metadata such as tags and labels for Docker. 
        # This metadata can be used for versioning and labelling Docker images.
        name: Extract metadata (tags, labels) for Docker
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: |
            ${{ secrets.DOCKERHUB_USERNAME }}/${{env.REPO_NAME}}
          tags: |
            type=sha,format=short 

      - # Uses the docker/build-push-action to build and push the Docker image. It specifies the context as the current directory (.), the Dockerfile location (./Dockerfile), tags from the metadata, and labels from the metadata. 
        # The push: true indicates that the image should be pushed to the Docker Hub registry.
        name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          push: true

Now each time we merge our PRs a GitHub action is going to build a docker image and push it to our registry in DockerHub as you can see in the following images.

Hope you found it cool and picked up something useful. Stay curious, keep learning, and rock on! Cheers to your next adventure!

Bibliography

Deploying a Web App in AWS China

Alfonso Domenech — Wed, 06 Sep 2023 08:02:53 +0000

The Chinese market is becoming more and more important for companies due to the volume of customers this market represents. Therefore, companies with digital products want to focus their products on the Chinese population. Companies using AWS services have the opportunity to do so, but with a number of limitations to consider when planning to move your services to China.

Things to consider when using AWS services in China

Signing up for AWS China Regions

There are currently two AWS regions in China, Beijing and Ningxia. Global AWS accounts are not suitable for accessing services in China and vice versa. You have to create a separate account to access AWS services in China. But first we have to identify our identity to be able to use AWS services in China. For this we have to provide our business license issued by the Bureau of Industry and Commerce of People’s Republic of China (PRC) or a valid government agency. To do this, Amazon offers us the AWS China Gateway Portal which will guide us through the process of creating an account in this region.

ICP Filing for Internet Information Services in Mainland China

In accordance with Chinese regulations. We have to fill in the Internet Content Provider (ICP) permit and there are two types. The ICP Recordal is the process for hosting a website providing non-commercial internet information services and ICP License to host a website providing commercial internet information services.

Network Connections for AWS China Regions

The AWS China and AWS global regions are not directly connected. To overcome packet loss and latency between the Chinese and global AWS regions, AWS works with local internet service providers such as China Mobile. The DirectConnect service is used to establish this solution by signing a contract with the provider as well as following Chinese regulations regarding the transfer and localization of information.

Relevant Compliance and Regulatory Requirements in Mainland China

It is important that if we plan to move our services to China, we have to be aware of the legislation. AWS can facilitate this process by contacting their helpdesk.
Apart from the ICP and depending on the type of business, the following regulations must be taken into account security obligations under China Cybersecurity Law, Multi-level Protection Scheme (MLPS) Certification, and cryptographic related regulations.

Our particular use case

In our particular use case, we had to deploy a fairly simple web page where users would have to fill in a series of forms to enter a competition. Their personal data would be stored in a database for further processing. As a fundamental requirement, the application had to be highly scalable, as we expected to receive high traffic peaks at certain times after running promotional campaigns on social media. These projects usually have very short development times (between 1 month and 2 weeks) so the developments tend to be quite hectic and results-oriented rather than maintainable.

All the projects we do for this client have very similar requirements, so we always use a very similar architecture and technology stack to optimize the development process:

Front end app using React and NextJS framework.
API REST using NodeJS and Express. It connects our front end app and our database.
PostgreSQL as a database to store user data.

We deployed these projects on a Kubernetes cluster using the AWS EKS service. We use the same template to create the necessary resources in the cluster, so the deployment and configuration of a new project is quite straightforward.

However, in the case of this application we encounter a big problem: by Chinese regulations, no personal data of a Chinese citizen can leave China.

Neither our Kubernetes cluster nor any of our resources are located in China (but in the Frankfurt region). Therefore, due to Chinese regulatory restrictions it was not going to be possible to reuse the infrastructure we already had in place for this project.

Fortunately, AWS has a couple of special regions within China: Ningxia and Beijing. These regions, which make up AWS China, are segregated from the rest of the regions (AWS Global), have a much more limited number of services than the other regions and their interconnectivity with the rest of the AWS regions is practically non-existent.

We decided to simplify our infrastructure as much as possible. Instead of using EKS, we decided to use Elastic Beanstalk, an AWS service that we had not used so far but that seemed to fit perfectly with our use case. Elastic Beanstalk allows to deploy scalable web applications in a very simple way, taking care of creating all the necessary AWS resources for its proper implementation (ec2, auto-scaling group, load balancer, etc). Therefore, it made deployment and configuration very easy for us.

Once we managed to get it up and running (setting up infrastructure, continuous integration, etc.), we ran into another problem related to the Chinese regulation.

As discussed in the previous section, any website needs to have an ICP license. This license, among other things, links our website to an IP. Our client, right from the start of the project, provided us with an Elastic IP (fixed IP) for which he had already requested and obtained this license.

At this point we realized one basic thing: how are we going to build a highly scalable application only accessible with a single Elastic IP? If we want the application to be highly scalable, we will have to be able to scale the application horizontally (i.e. increase the number of servers that host the application). However, the Elastic IP can only be associated to a single EC2 machine, which does not work for us. No problem, let's just use a very large server. The big problem with this is that it is not elastic at all: the capacity does not increase/decrease depending on the processing load at any given time, which leads to higher unnecessary costs.

Well, but Beanstalk can provide us with a load balancer (ELB), allowing us to have several servers hosting the application but only one entry point. Why not associate the IP with the ELB? It turns out, however, that assigning an Elastic IP to an ELB is not possible. Internally, an ELB uses a pool of IPs that allows it to elastically scale its processing capacity, so assigning a single IP to it is not possible.

Therefore, we had to improvise an alternative solution, which would not be optimal but could help us to solve the problem.

Rapid Solution. Configuring an EC2 instance as a reverse proxy

If you are in a hurry to move your services in China as a temporary solution you can set up an EC2 instance as a reverse proxy. The first thing to do is to create an EC2 instance from the AWS China console which you have to create beforehand. Here is a link that shows you how to do it. Remember to configure the security groups to be able to access the instance via SSH and to be able to access port 80 and 443 from anywhere. We have installed certbot to get the SSL certificates for free with Let's Encrypt.
Once you have created the EC2 instance, in our case we selected Amazon Linux 2, and you can access it via SSH you have to install the Nginx web server and certbot. Let's do it!
First we need to update the OS (Amazon Linux 2) and install the Nginx web server.

sudo yum update
sudo amazon-linux-extras install -y nginx1
sudo yum install certbot python2-certbot-nginx

Certbot is able to automatically configure SSL for Nginx but needs to find the server_name directive that matches the domain you are requesting the certificate for. This directive is found inside the server block. Let's modify the nginx.conf file.

sudo vi /etc/nginx/nginx.conf

server_name example.com www.example.com;

We save our changes and verify that we have respected the nginx syntax.

sudo nginx -t

If no error has occurred, we must restart our server for the changes to take effect.

sudo systemctl reload nginx

Now we are able to automatically configure our server to use HTTPs. Now we have to run the following command to make certbot update our configuration file.

sudo certbot --nginx -d example.com -d www.example.com

Then certbot will communicate with the Let's Encrypt server and check that you control the domain for which you are requesting the certificate. If everything is successful, certbot will give you two options for configuring the https settings.

Easy - Allow both HTTP and HTTPS access to these sites
Secure - Make all requests redirect to secure HTTPS access

Once you have chosen one of these options certbot will modify the nginx configuration and restart it. It will also give you a message saying that the process was successful and where the certificates are stored.

Now you just need to make sure that the certificates are renewed automatically as let's encrypt certificates expire after 3 months. To do this we are going to configure a cron job with the certbot command that allows us to renew the certificates.

sudo crontab -e

Our editor will open the default cron task tab and we will write the following task.

30 5 * * * /usr/bin/certbot renew --quiet

This task will be executed at 5:30 am everyday. The renew command for Cerbot will check if installed certificates are set to expire in less than one month and update them. And the --quiet option tells certbot not to wait for user acceptance on certificate renewal.

Let's create the configuration file for our Nginx to act as a reverse proxy. Now we are going to create a file in the following path /etc/nginx/default.d/proxy.conf
The proxy_pass directive indicates where the traffic has to be redirected, in our case it will be the url of our load balancer. Once this is done we will load our main configuration file. The file will look like this.

location / {
 proxy_pass http://your-load-balancer-url;
 proxy_pass_header Server;
 proxy_hide_header X-Powered-By;
}

Then we restart our server and check that the syntax is correct (as we have done in the previous steps) and we would have our nginx server ready with https traffic enabled and acting as a reverse proxy.

The Stable Solution

Once the problem was solved in time and the day of the launch of the application arrived,we had the feeling that there should be a better way to solve this problem. We thought it was a too common situation. It should have happened to someone before.

After searching and searching in Google, we found a document from SINNET, the company that manages the Beijing region in AWS China (link). This document redirected to another document with information on how to manage ICP licenses when using ELBs (link)

Summarizing the content of the document, it seems that, when applying for an ICP license, it is possible to request several IPs, not just one. On the other hand, there is the possibility to request AWS China to only assign IPs to an ELB from a fixed set of IPs (the ones requested in the ICP license).

However, starting to use this approach at this stage is no longer possible. A new ICP request can take between 1 and 3 weeks to be accepted, so the current system will have to remain in place for at least a month.

We decided to write this article mainly to save the lives of those who, in the future, will encounter this problem and will be able to solve it in time. But we would also like to make some conclusions from this:

Rushing is never good, leading to improvising in case of any problem.
The client should tell you what he wants to do, not how, especially if he lacks technical knowledge of the domain. I'm not saying that they cannot contribute to it, but not to impose it without a previous discussion.