Forem: HakamRaza

[Laravel] Solving Bottleneck in Laravel.

HakamRaza — Mon, 18 Dec 2023 08:03:28 +0000

Intro

this is based on my experience in solving bottlenecks in Laravel projects hosted in AWS EC2. Most of this is simple steps that can be taken on the dot.
this will not cover everything, but at least give you some solutions that maybe useful.

Memory Bottleneck

Scaling your server vertically may not be the solution.
There maybe a situation when your project running slow, but the EC2 CPU% chart is still showing below 50%. Then, it is time to check your server memory. You see, EC2 does not monitor your memory (RAM) usage by default.
EC2 also does not 'shared' your memory to another EC2 although behind the same load balancer. i.e, a process can only be compute in one server that execute it. This is an issue if you load everything (eg: User::all()) into memory.
you usually only able to check memory available by using command inside the ec2 itself:

cat /proc/meminfo | grep Mem

Try to estimate the memory you needed and scale appropriately:

1 php-fpm process size: ~ 40mb - max 80mb RAM
1 queue worker running: ~ 140mb RAM
Standard laravel process: ~ 2mb - 16mb RAM

Database Bottleneck

although there is automated server auto-scaling, there is none for AWS RDS, which require manual scaling/upgrade.
new spawn server means additional connection to your DB.
Amount of connection a DB can handle mainly depend on the amount of RAM the DB has.
Any RDS upgrade/scaling operations require 30 mins DB down time. Which means you cannot access your DB during this time.
Estimate appropriately, for example a db.t3.large (8GB) can handle up to 600 connections
RDS Monitoring tools can be use to monitor CPU usage, query depth and so on.
Other steps that can be taken includes:

Index columns with frequent filter / querying / joining
Using aggregate tables (for aggregate summary) instead of query multiple tables for multiple aggregate metrics
Using aggregate function such as withCount to get directly the count instead of loading the whole relationship children.
Read and write using separate database that syncing
Cache common and frequent queries such as list of countries, phone codes, etc.

Application Improvement

Select only columns required to reduce memory consumption
Use eager load moderately as not everything are actually needed on the same time. Utilize lazy loading instead, where for additional information, forward user to another page (new request) instead of eager loading all at once.
Using raw queries instead of eloquent maybe a hassle but that depending on what information you want to fetch. Raw queries use less memory due to no need to load additional methods or properties come with eloquent
Utilize jobs for faster response (depend on your design) and control your application compute and memory consumption when executing something.

[Linux] Setting Up MailHog

HakamRaza — Fri, 08 Sep 2023 03:52:44 +0000

MailHog is an email testing tool for developers. It almost like Mailtrap but this one is hosted in your own server. This give more privacy and freedom when you are testing your mail notifications.

Caution : This project has been abandoned but it's okay, we have alternative, MailPit which are forked from MailHog and the installation should be almost similar.

This installation is based on following:

Linux ubuntu server to host your mailserver
Laravel to test send your mail
Nginx to create webproxy to MailHog port

Okay, lets go !

1. Installing MailHog into your ubuntu server.


# Golang is require to install MailHog
sudo apt-get -y install golang-go

# Downloading MailHog binary to current directory
go install github.com/mailhog/MailHog

# moves downloaded binary to '/usr/local/bin/mailhog' directory
sudo mv ~/go/bin/MailHog /usr/local/bin/mailhog/MailHog

2. Configure MailHog using JSON file

# we will store related mailhog file here
cd /usr/local/bin/mailhog

# create a json file 'mailhog-outgoing.json'
sudo nano mailhog-outgoing.json

and paste the following json :

{
    "server name": {
        "name": "development mail",
        "host": "...",
        "port": "1025",
        "email": "...",
        "username": "test",
        "password": "test123",
        "mechanism": "PLAIN"
    }
}

here you can set up your password, username, etc. for access by laravel application which you can refer here.

3. Running MailHog

You can running mailhog manually by CLI provided but since we want to run it 'forever', we can create a linux service for that utilising linux 'systemctl'

# create a service file in this case, i named it as MailHog.service (capital 'M, H' here)
sudo nano /etc/systemd/system/MailHog.service

and paste the following:

[Unit]
Description=Mailhog Local Mail SMTP
After=network.target

[Service]
Type=simple
User=ubuntu
ExecStart=/usr/bin/env /usr/local/bin/mailhog/MailHog -api-bind-addr 0.0.0.0:8025 -ui-bind-addr 0.0.0.0:8025 -smtp-bind-addr 0.0.0.0:1025 -storage maildir -maildir-path /usr/local/bin/mailhog/mails/ -outgoing-smtp /usr/local/bin/mailhog/mailhog-outgoing.json > /dev/null 2>&1 &

[Install]
WantedBy=multi-user.target

the important part here is which user you want to run this service and the command execStart which calling Mailhog CLI with the mailhog-outgoing.json configuration. Other flag are kept default and can be refer here

Start the service by:

# the name of service mail is sensitive

# check does systemctl detect the new service file
sudo systemctl status MailHog

# if yes, run the new service
sudo systemctl start MailHog

4. Sending a Mail

By now, the MailHog should be running and can be test. Since I'm already has a Laravel project hosted using mailtrap beforehand, I just need to update my .env to use the local mailhog mail server like this:

MAIL_DRIVER=smtp
# localhost
MAIL_HOST=0.0.0.0
# depending on value you set up json
MAIL_PORT=1025
MAIL_USERNAME=test
MAIL_PASSWORD=test123
MAIL_ENCRYPTION=null

Then try to send an email.

5. Opening MailHog Inbox

MailHog inbox is hosted on port 8025 (default). To open the inbox by browser, we can directly go to this port using server-ip:8025. But since my server is behind a domain, I need to proxy through port 80 to port 8025. To do this, use this nginx server block template file:

server {
    server_name mailtrap.mydomain.com;
    listen 80;
    listen [::]:80;

    if ($host != "mailtrap.mydomain.com") {
       return 404;
    }

    location / {
       proxy_pass      <http://localhost:8025>;
       proxy_set_header    Host             $host;
       proxy_set_header    X-Real-IP        $remote_addr;
       proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
       proxy_set_header    X-Client-Verify  SUCCESS;
       proxy_set_header    X-Client-DN      $ssl_client_s_dn;
       proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
       proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;
       proxy_read_timeout 1800;
       proxy_connect_timeout 1800;
       chunked_transfer_encoding on;
       proxy_set_header X-NginX-Proxy true;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_http_version 1.1;
       proxy_redirect off;
       proxy_buffering off;
    }
}

where mailtrap.mydomain.com is the subdomain you set for mailhog inbox.

[Linux] SSH to GitHub

HakamRaza — Sun, 21 May 2023 14:05:02 +0000

Some intro

GitHub do provide option to use SSH to connect and do many things same as using the standard CLI.
And there are some functionalities offered by using SSH that make life easier, for example no need for token to pull your branch.
This is how to change your repo connection to use SSH. This is done in linux ubuntu OS.

Setting Up SSH keys and Permission

For this, you can use existing user or create a new user specifically for deployment/pull/push through SSH.

cd ~

# create .ssh folder inside user home directory
mkdir .ssh

# generate key private and public (.pub) using ssh-keygen, and give name like "github_dev".
# Optional, dont assign passphrase if you want to use it for CLI.
ssh-keygen -t ed25519-sk -C "your_email@example.com"

# check ssh agent running
eval $(ssh-agent -s)

# register only private key (no .pub) generated to ssh agent
ssh-add ~/.ssh/<private_key_file>

for secure linux and current user to read private key to execute GitHub CLI, update the permission for access.

# change permission key file for read-only
cd ~/.ssh
chmod 644 <private_key>
sudo chgrp <username> <private_key>

# change permission to folder .ssh
cd ~
chmod 700 .ssh
sudo chgrp <username> .ssh

you also can directly define any SSH connection to GitHub to use the private key generated by creating 'config' file

# config connection using ssh and correct key file
nano .ssh/config

# paste the following into the 'config' file
Host github.com
Hostname ssh.github.com
Port 443
User git
IdentityFile ~/.ssh/<private_key> #generated private key location

Register public key generated to GitHub account.

Go to intended GitHub account setting to register this public SSH key.
Give any name you want.

Check Signature Github.

use the command below to cross check signature of the ssh agent is the same as displayed by GitHub

ssh-add -l -E sha256

Testing GitHub SSH Connection

test successful connection through SSH to GitHub by:

# testing connection to github
ssh -T git@github.com

# testing with more details for troubleshooting
ssh -vT git@github.com

Update Repository Remote URL

After successful connection through SSH, change current local repository remote to use SSH
usually the remote url is like this:

# Check connection profile setup
git remote -v

# result 
# origin  https://github.com/****.git (fetch)
# origin  https://github.com/****.git (push)

you can either update original 'origin' or add new one like 'myssh'
the address can be refer back here:

# add ssh connection profile 'myssh'
git remote add myssh ssh://git@****.git

# result 
# origin  https://github.com/****.git (fetch)
# origin  https://github.com/****.git (push)
# myssh     git@github.com:****.git (fetch)
# myssh     git@github.com:****.git (push)


# update existing connection profile 'origin' to use SSH
git remote set-url origin git@github.com:****.git

# result 
# origin     git@github.com:****.git (fetch)
# origin     git@github.com:****.git (push)

# update back to use HTTP
git remote set-url origin https://github.com/****.git

example using GitHub CLI through SSH connection profile different than default 'origin'

# using 'myssh' connection profile
git checkout myssh/<branch name> -b <new branch name>

User Connection Issue

There will be an issue sometimes with linux system user especially when using 'sudo'. In this case, the one executing CLI is not the current user but by the 'root' system user.
In this case, to maintain user profile, use '-E' flag:

sudo -E git fetch

Bash Script

Example a bash script to pull the 'develop' branch.
The ssh-agent may need to be started again in linux.

#!/bin/bash

# start ssh-agent and register back private key
eval $(ssh-agent -s)
ssh-add ~/.ssh/<private_key_file>

# Go to project epo
cd /var/www/<local repository directory>

# Checkout 'develop' branch
git checkout develop

# Update connection profile
# git remote set-url origin git@github.com:*****.git

# Pull latest changes
git pull

# Checkout by latest tag
# git fetch --tags
# tag=$(git describe --tags `git rev-list --tags --max-count=1`)
# echo $tag
# git checkout $tag -b latest

# Set back to HTTP
# git remote set-url origin https://github.com/*****.git

#
# Additional steps
#

echo Done

[Linux] Setting Up Nginx Block

HakamRaza — Wed, 17 May 2023 07:25:35 +0000

What is nginx block ?

is a config file on how to serve your app. It is part of Nginx webserver settings file. They are located at /etc/nginx/sites-available/
Same as Apache which is another webserver, it also has its own config 'block' to serve your app.

Example a Standard Block for PHP project

server {
    # listen to port 80 for IPv4
    listen 80;
    # listen to port 80 for IPv6
    listen [::]:80;
    # the address domain of your app 
    server_name dev.mydomain.com;`

    # location of your SSL certificates if in use
    #ssl_certificate /ssl/crt/file.crt;
    #ssl_certificate_key /ssl/key/file.key;

    # location of 'myproject' folder where index.php located
    root /var/www/development/myproject/public;
    # name of index file entry point
    index index.php index.html index.htm index.nginx-debian.html;

    # add additional header before request reach your app
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Content-Type-Options "nosniff";

    charset utf-8;

    # optional condition to block request coming from other address
    if ($host != "dev.mydomain.com") {
       return 404;
    }

    # Optional, prevent Direct Access To Protected Files
    location ~ \.(env|log) {
        deny all;
    }

    # Optional, prevent Direct Access To Protected Folders
    location ~ ^/(^app$|bootstrap|config|database|overrides|resources|routes|storage|tests|artisan) {
        deny all;
    }

    # Optional, prevent Direct Access To modules/vendor Folders Except Assets
    location ~ ^/(modules|vendor)\/(.*)\.((?!ico|gif|jpg|jpeg|png|js\b|css|less|sass|font|woff|woff2|eot|ttf|svg|xls|xlsx).)*$ {
        deny all;
    }

    # passing address parameters from request
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    error_page 404 /index.php;

    # Pass PHP Scripts To FastCGI Server
    location ~ \.php$ {
        # passing address parameters from request
        try_files $uri $uri/ /index.php?$query_string;

        # set version of php fpm to serve the php project
        fastcgi_pass unix:/var/run/php/php8.0-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
    }

    # deny all other requests that not match
    location ~ /\.(?!well-known).* {
        deny all;
    }
}

Example a Standard Block for PHP project with multiple subdomains

server {
    listen 80;
    listen [::]:80;
    server_name dev.mydomain.my dev2.mydomain.my;

    root /var/www/production/myproject/public;
    index index.php index.html index.htm index.nginx-debian.html;

    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Content-Type-Options "nosniff";

    charset utf-8;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    location ~ \.php$ {
        try_files $uri $uri/ /index.php?$query_string;
        fastcgi_pass unix:/var/run/php/php8.0-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
    }

    location ~ /\.(?!well-known).* {
        deny all;
    }
}

Example a Standard Block for Reverse-Proxy to Port

this is usually use to serve Node projects

server {
    listen 80;
    listen [::]:80;
    server_name dev.mydomain.com;

    if ($host != "dev.mydomain.com") {
       return 404;
    }

    location / {
       # example proxy-ing port 80 to port 1337 in server localhost
       proxy_pass      http://localhost:1337;

       # set up headers towards proxy
       proxy_set_header    Host             $host;
       proxy_set_header    X-Real-IP        $remote_addr;
       proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
       proxy_set_header    X-Client-Verify  SUCCESS;
       proxy_set_header    X-Client-DN      $ssl_client_s_dn;
       proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
       proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;
       proxy_read_timeout 1800;
       proxy_connect_timeout 1800;
       chunked_transfer_encoding on;
       proxy_set_header X-NginX-Proxy true;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "upgrade";
       proxy_http_version 1.1;
       proxy_redirect off;
       proxy_buffering off;
    }
}