Forem: Bryan

Create a Basic Framework with Express (Part 2)

Bryan — Mon, 01 May 2023 02:15:27 +0000

In this part 2, we will define routes while incorporating Handlebars.

Installing Handlebars

Install express-handlebars with the command
```
npm install express-handlebars
```

Require express-handlebars in index.js

...
const app = express()
const handlebars = require('express-handlebars')
...

Set the template engine in express

use .hbs file extension instead of .handlebars
default layout name as app
add helper functions for displaying full routes and env variable

...
app.engine(
    '.hbs',
    handlebars.engine({
        // use hbs instead of handlebars extension
        extname: '.hbs',
        // default layout name
        defaultLayout: 'app',
        // simple template functions
        helpers: {
            // return the full route path
            route(path) {
                return process.env.APP_URL + ':' + process.env.APP_PORT + path
            },
            // returns the env variable value
            config(name) {
                return process.env[name]
            },
        },
    })
)
app.set('view engine', '.hbs')
// use views folder
app.set('views', __dirname + '/views')
// static folder
app.use(express.static('public'))
...

Create Views

Add APP_URL to .env file
```
APP_URL=http://localhost
```

Create the main layout app.hbs in views/layouts

route helper function is used here to get the full path of the favicon
config helper function is used to get the env variable APP_NAME

<html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <meta http-equiv="X-UA-Compatible" content="ie=edge">
        <link rel="icon" type="image/png" href="{{route  '/favicon.png' }}" />
        <title>{{config 'APP_NAME' }}</title>
        <script src="https://cdn.tailwindcss.com"></script>
    </head>
    <body class="bg-gray-100">
        {{{ body }}}
    </body>
</html>

Add a favicon in the public folder named favicon.png

Create the homepage content index.hbs in views folder

<div class="min-h-screen flex flex-col">
    <div class="min-h-screen flex justify-center flex-wrap content-center">
        <div class="bg-gray-100 rounded-md w-10/12 md:w-1/2 lg:w-4/12 flex flex-col shadow">
            <div class="w-full p-8 flex flex-wrap content-center bg-white">
                <div class="flex w-full justify-center">
                    <a href="{{route '/' }}">Homepage</a>
                </div>
            </div>
        </div>
    </div>
</div>

Router

DELETE the following code since we will be adding controller and router files in index.js

...
const { ServerError } = require('./exceptions/ServerError')
...
app.get('/', async (req, res, next) => {
    // res.send('Hello World!')
    try {
        // res.send('hello')
        throw new ServerError()
    } catch (error) {
        return next(error)
    }
})

Create index.js in routes folder

const router = require('express').Router()

// home page route
router.get('/', async (req, res) => {
    // this will render the index.hbs with the default app.hbs layout
    return res.render('index')

    // to switch the layout file called main.hbs (make sure it is created in views/layouts)
    /**
    return res.render('index', {
        layout: 'main',
    })
    */
})

module.exports = router

Now to make sure the app knows to use the routes/index.js

...
app.use('/', routes)

app.use(errorHandler.logger)
...

Create a Basic Framework with Express (Part 1.1)

Bryan — Sun, 02 Apr 2023 18:57:54 +0000

In Part 1, error handling was created. Let's make it easier to throw errors.
Instead of

let error = new Error('This is a major issue!!')
error.statusCode = 500
throw error

We will make it something like this

throw new ServerError({ message: 'This is a major issue!!' })

Create a new folder exceptions

Create ServerError.js in the exceptions folder

class ServerError extends Error {
  constructor(resource) {
    super()
    this.name = this.constructor.name // good practice
    this.statusCode = 500
    this.message = resource ? resource.message : 'Server Error'
  }
}
module.exports = {
  ServerError
}

Edit index.js, on how to use it

// index.js
...
const { ServerError } = require('./exceptions/ServerError')
...
app.get('/error', async (req, res, next) => {
    try {
        throw new ServerError()
        // or
       // throw new ServerError({ message: 'A huge mistake happened!' })
    } catch (error) {
        return next(error)
    }
})
...

Now go to localhost:3000/error to see the 500 server error

No space left on device error

Bryan — Sun, 26 Mar 2023 23:27:12 +0000

No space left on device error
Clear up old php sessions

rsync -a --delete /home/{user}/public_html/temp/ /var/lib/php/sessions/

Create a Basic Framework with Express (Part 1)

Bryan — Mon, 20 Mar 2023 16:15:00 +0000

We will be using Docker, Node 18, MySQL 8, and Redis. Following Setting Up Node App using Docker post express, nodemon, dotenv packages should be installed.

In this application, we will not be writing a rest app, it will be a basic full stack framework. Packages will be installed once we reach the point where we need to use them.

Create Project Structure

node_modules
public
controllers
database
helpers
middlewares
routes
storage
views

Setup Logging

Install Winston
```
npm install winston
```
To be able to write daily logs instead of a single log, install Winston Daily Rotate File
```
npm install winston-daily-rotate-file
```

Create a logger.js helper in helpers directory

We want to have it log into daily log files of max size of 1GB and will be kept for 10 days.

const winston = require('winston')
const { format } = winston
const { combine, timestamp, printf, splat } = format
require('winston-daily-rotate-file')

const dailyTransport = new winston.transports.DailyRotateFile({
    // file name format
    filename: 'application-%DATE%.log',
    datePattern: 'YYYY-MM-DD',
    zippedArchive: true,
    // limit each log to 1GB
    maxSize: '1g',
    // format the log
    format: combine(
        timestamp(),
        splat(),
        printf((info) => {
            return `[${info.timestamp}] ${info.level}: ${info.message}`
        })
    ),
    level: 'info',
    // save logs into storage/logs
    dirname: 'storage/logs',
    // save logs for 10 days
    maxFiles: '10d',
})

const logger = winston.createLogger({
    transports: [dailyTransport],
})

module.exports = logger

Create a errorHandler.middleware.js helper in middlewares directory

Log the error and send a response to the user

// import logger helpers
const loggerx = require('../helpers/logger')

const errorHandler = {
    // log the error
    logger: (error, request, response, next) => {
        loggerx.error(`${error.message} %o`, error)
        next(error) // calling next middleware
    },

    responder: (error, request, response, next) => {
        const status = error.statusCode || 500
        // send status and error message
        return response.status(status).send(error.message)
    }
}
module.exports = errorHandler;

Add the error handler middleware to index.js and test it out!

...
const express = require('express')
const app = express()

// import the logger helper so we can test the logging
const logger = require('../helpers/logger')
// import the error handler
const errorHandler = require('./middlewares/errorHandler.middleware.js')

...

// Cause an error
app.get('/error', async (req, res, next) => {
    try {
        let error = new Error('This is a major issue!!')
        error.statusCode = 500
        throw error
    } catch (error) {
        return next(error)
    }
})

// Just logging
app.get('/log', async (req, res, next) => {
    logger.info('hello from /log')
    res.send('hello')
})

// log the error
app.use(errorHandler.logger)
// send error to user
app.use(errorHandler.responder)

...

- Go to `localhost:3000/error` and `localhost:3000/log`, the log file can be found in **storage/logs** (example below)

```log
[2023-03-20T00:41:29.652Z] error: This is a major issue!! Error: This is a major issue!!
    at /app/index.js:28:25
    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
    at next (/app/node_modules/express/lib/router/route.js:144:13)
    at Route.dispatch (/app/node_modules/express/lib/router/route.js:114:3)
    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
    at /app/node_modules/express/lib/router/index.js:284:15
    at Function.process_params (/app/node_modules/express/lib/router/index.js:346:12)
    at next (/app/node_modules/express/lib/router/index.js:280:10)
    at expressInit (/app/node_modules/express/lib/middleware/init.js:40:5)
    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5) {
[stack]: 'Error: This is a major issue!!\n' +
    '    at /app/index.js:28:25\n' +
    '    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)\n' +
    '    at next (/app/node_modules/express/lib/router/route.js:144:13)\n' +
    '    at Route.dispatch (/app/node_modules/express/lib/router/route.js:114:3)\n' +
    '    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)\n' +
    '    at /app/node_modules/express/lib/router/index.js:284:15\n' +
    '    at Function.process_params (/app/node_modules/express/lib/router/index.js:346:12)\n' +
    '    at next (/app/node_modules/express/lib/router/index.js:280:10)\n' +
    '    at expressInit (/app/node_modules/express/lib/middleware/init.js:40:5)\n' +
    '    at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)',
[message]: 'This is a major issue!!',
statusCode: 500
}
[2023-03-20T00:41:59.271Z] info: hello from /log
```

Setting Up Node App using Docker

Bryan — Mon, 16 Jan 2023 08:00:00 +0000

Create a docker setup to build an Express app with MySQL and Redis

Create a directory and go switch to it
```
mkdir nodeapp
cd nodeapp
```

Create package.json

To avoid installing Node locally, we will use this docker command to npm init

docker run --rm -v "$PWD:/$(basename $PWD)" -w "/$(basename $PWD)" -it node:current-alpine sh -c "npm init"

Install dotenv and express packages

docker run --rm -v "$PWD:/$(basename $PWD)" -w "/$(basename $PWD)" -it node:current-alpine sh -c "npm install dotenv express"

docker run --rm -v "$PWD:/$(basename $PWD)" -w "/$(basename $PWD)" -it node:current-alpine sh -c "npm install --save-dev nodemon"

Edit package.json to add to scripts section

"start": "node index.js",
"dev": "nodemon index.js"

Create .env

APP_PORT=3000
DOCKER_PORT=3000

#db is the service name in docker-compose.yml
DB_HOST=db
DB_PORT=3306
DB_NAME=nodeapp

#can be left blank if using root username
DB_USERNAME=
DB_PASSWORD=
DB_PORT=3306

#redis is the service name in docker-compose.yml
REDIS_HOST=redis
REDIS_PASSWORD=null
REDIS_PORT=6379

Create Dockerfile

#change node version to your liking
FROM node:18
WORKDIR /app/
COPY ./package.json .
RUN npm install
COPY ./* .
EXPOSE $DOCKER_PORT

Create docker-compose.yml

version: '3'
services: 
  web:
    build:
      context: .
    env_file: ./.env
    tty: true
    volumes: 
      - ./:/app/
      - node_modules:/app/node_modules
    ports:
      - '${APP_PORT:-3000}:${DOCKER_PORT:-3000}'
    depends_on: 
      - db
    networks:
      - nodeapp
  db:
    image: mysql:latest
    env_file: ./.env
    environment: 
      MYSQL_ROOT_PASSWORD: '${DB_PASSWORD}'
      MYSQL_ROOT_HOST: "%"
      MYSQL_DATABASE: '${DB_NAME}'
      MYSQL_USER: '${DB_USERNAME}'
      MYSQL_PASSWORD: '${DB_USERNAME}'
      MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
    ports:
      - '${DB_PORT:-3306}:3306'
    volumes:
      - mysql:/var/lib/mysql
    networks:
      - nodeapp
  redis:
    image: 'redis:alpine'
    ports:
      - '${REDIS_PORT:-6379}:6379'
    volumes:
      - 'redis:/data'
    networks:
      - nodeapp
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      retries: 3
      timeout: 5s
networks:
  nodeapp:
    driver: bridge
volumes:
  node_modules:
    driver: local
  mysql: 
    driver: local
  redis:
    driver: local

Create index.js

require('dotenv').config();

const express = require('express');
const app = express();

// Constants
const PORT = process.env.DOCKER_PORT || 3000;
const HOST = '0.0.0.0';

// App
app.get('/', (req, res) => {
    res.send('Hello World!');
});

app.listen(PORT, HOST, () => {
     console.log(`Running on http://${HOST}:${PORT}`);
});

Run docker-compose up -d
Run docker-compose exec web npm run dev
Go to http://localhost:3000 3000 will be APP_PORT

BONUS

To easily run npm commands, do the following

Create a file with the name soar

#!/bin/bash

path=$(printf '%s\n' "${PWD##*/}")

# get first arg
arg1=$1

if [[ $arg1 == "up" ]]; then
    command="docker-compose up -d"
    $command
elif [[ $arg1 == "down" ]]; then
    command="docker-compose down"
    $command
else
    # remove first arg
    shift

    # get rest of args
    argrest=$@

    # web is the service name in docker-compose.yml
    ccommand="docker-compose exec web $arg1 "$argrest""

    # run the command
    $command
fi

Make soar runnable
```
chmod 700 soar
```
Test it with
```
./soar npm -h
```
Create an alias for ./soar to soar
```
alias './soar'
```
To make sure this is always available, you may add this to your shell configuration file in your home directory, such as ~/.zshrc or ~/.bashrc, and then restart your shell.
Now you can start docker, run npm, or npx

Start docker container
```
soar up
```
Stop docker container
```
soar down
```
Run npm
```
soar npm list
```
Run npx
```
soar npx ...
```

Querying for Data in MySQL is Slow

Bryan — Sun, 18 Dec 2022 17:00:00 +0000

Trying to query the MySQL database is slow (millions of records in table)

Example query:

SELECT * FROM my_table WHERE code='X87L9D82NB';

Just add an index to the column that is frequently used to query for data

Query to add index:

CREATE INDEX index_name ON table_name (column_name);

SFTP Only Access on Laravel Forge Server

Bryan — Sun, 04 Dec 2022 16:05:00 +0000

Requirements

Let's say you want to create 3 SFTP only users

mainuser : manage other users' folders and files
user01 : can only view/edit own folder and files
user02 : can only view/edit own folder and files

Creating Users and Directories

Create the 3 users

sudo adduser mainuser
sudo adduser user01
sudo adduser user02

Create folders for user01 and user02

sudo mkdir -p /var/sftp/user01/uploads
sudo mkdir -p /var/sftp/user02/uploads

Give root write permissions to the same directory, and give other users only read and execute rights
```
sudo chmod 755 /var/sftp/user01
sudo chmod 755 /var/sftp/user02
```

Change the ownership for the uploads directory to the user you just created

sudo chown user01:user01 /var/sftp/user01/uploads
sudo chown user02:user02 /var/sftp/user02/uploads

Change ownership to mainuser to view all user folders
```
sudo chown root:mainuser /var/sftp
```

Restricting Access to Directory

Note 1: We are creating a new sshd file because it would be cleaner and /etc/ssh/sshd_config automatically includes .conf files in /etc/ssh/sshd_config.d/

Note 2: user01/user02 will default to its own folder, but cannot create folders or files outside the uploads folder

Create mainuser sshd config file

sudo nano /etc/ssh/sshd_config.d/mainuser.conf

Add to the newly created mainuser conf file

Match User mainuser
ForceCommand internal-sftp
PasswordAuthentication yes
ChrootDirectory /var/sftp/
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

Create user01 sshd config file

sudo nano /etc/ssh/sshd_config.d/user01.conf

Add to the newly created user01 conf file

Match User user01
ForceCommand internal-sftp
PasswordAuthentication yes
ChrootDirectory /var/sftp/user01
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

Create user02 sshd config file

sudo nano /etc/ssh/sshd_config.d/user02.conf

Add to the newly created user02 conf file

Match User user02
ForceCommand internal-sftp
PasswordAuthentication yes
ChrootDirectory /var/sftp/user02
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

Restart sshd service
```
sudo systemctl restart sshd
```

Resource

https://www.digitalocean.com/community/tutorials/how-to-enable-sftp-without-shell-access-on-ubuntu-20-04

Create Node app using Docker without installing Node Locally

Bryan — Mon, 21 Nov 2022 00:59:11 +0000

Almost all tutorials shows how to initialize a node app with Node installed locally. Here is a way to create a node app in Docker without installing Node locally.

Initialize a node app in the current directory with Docker:

Replace npm init with whatever npm command is needed

docker run --rm -v "$PWD:/$(basename $PWD)" -w "/$(basename $PWD)" -it node:current-alpine sh -c "npm init"

Info

-rm remove
-v volume
$PWD path to current directory
basename last element of file path
-w working directory
-it interactive docker terminal
-c context, without it sh will not work

TikTok Business API Tutorial - Part 1 (Account Setup)

Bryan — Fri, 18 Nov 2022 03:49:35 +0000

Register for TikTok's Business Developer Account
- https://ads.tiktok.com/marketing_api/homepage
Go to My Apps
Create New App
Fill in all the information and submit (It takes a few days to get verified)

Once the Verification Status is Approved, we are are to use the api

Docker, NodeJS, and MySQL 8

Bryan — Mon, 17 Oct 2022 00:46:58 +0000

Seen this error when using Docker for NodeJS and MySQL 8?

Error: ER_NOT_SUPPORTED_AUTH_MODE: Client does not support authentication protocol requested by server; consider upgrading MySQL client

It is most likely the wrong package is being used.

mysql package does not support MySQL 8 authentication method (More info here)
Use mysql2 package

Note

Make sure the MySQL server is fully running before trying to connect, otherwise an error like below

Error: connect ECONNREFUSED 172.17.0.1:3306

Download HelloSign Documents using Laravel

Bryan — Sun, 09 Oct 2022 05:40:23 +0000

Ever need to download all HelloSign documents with Laravel without the SDK? It can be done by basic HTTP requests with an API key generated by HelloSign.

Things used

Laravel's built in HTTP Client (Guzzle)
Jobs to avoid HelloSign API rate limits
Functions will be placed in routes file instead of a controller (to prototype faster)

Steps

Get API key from HelloSign
Add the api to the .env file
```
HELLO_SIGN_API_KEY=
```

Create hellosign.php in the config folder

<?php

return [
  'api_key' => env('HELLO_SIGN_API_KEY')
];

Create a migration create_signature_requests_table for the HelloSign document ids and add the following columns
```
$table->string('signature_request_id');
$table->boolean('downloaded');
```

Create a model SignatureRequest and add the following

$table->string('signature_request_id');
$table->boolean('downloaded');

Go to routes/web.php and add the following

use App\Jobs\DownloadJob;

Route::get('/', function () {
  $api_key = config('hellosign.api_key');

  $page = 1;
  $total_pages = 99; // just a temporary high number

  while($page <= $total_pages) {
      // max out the per page and documents from all accounts
      $response = Http::get("https://{$api_key}:@api.hellosign.com/v3/signature_request/list?account_id=all&page_size=100&page={$page}");

      $object = $response->object();

      // set the number of total pages of documents
      if($page == 1) {
          $total_pages = $object->list_info->num_pages;
      }

      // loop through each result and get the signature_request_id if it exists
      foreach($object->signature_requests as $sig) {
          if(!isset($sig->signature_request_id)){
              SignatureRequest::create(['signature_request_id' => $sig->signature_request_id]);
          }
      }

      // increase the page
      $page++;
  }
});

Route::get('download', function() {
  // get all signature ids that are not downloaded yet
  $sigs = SignatureRequest::where('downloaded', false)->get();

  // chunk it into 20, HelloSign API limits 25 requests per minute
  $chunks = $sigs->chunk(20);
  $chunks->all();

  // wait in minutes
  $wait = 0;
  foreach($chunks->all() as $chunk) {
      foreach($chunk as $sig) {
          // dispatch the download job and delay it by now + minutes
          dispatch(new DownloadJob($sig))->delay(now()->addMinutes($wait));
      }
      // increase wait time
      $wait++;
  }
});

Create a job DownloadJob and update with the following

namespace App\Jobs;
...
use Illuminate\Support\Facades\Http;
use Illuminate\Support\Facades\Storage;
use App\Models\SignatureRequest;

...
public $sig;

  public function __construct(SignatureRequest $sig)
  {
      $this->sig = $sig;
  }

  public function handle()
  {
      $api_key = config('hellosign.api_key');

      // request the document
      $response = Http::get("https://{$api_key}:@api.hellosign.com/v3/signature_request/files/{$this->sig->signature_request_id}");

      // only download if response code is 200
      if($response->status() == 200) {
        // save the document into a `documents` folder in storage with the document id as it's file name
        Storage::put("documents/{$this->sig->signature_request_id}.pdf", $response->body() );

        // update db to downloaded
        $this->sig->downloaded = true;
        $this->sig->save();
      }
  }

Run the migrations with php artisan migrate
Start the queue with php artisan queue:work
Go to localhost in the browser to start getting the all the document ids
Once #10 is finished, in the browser go to localhost/download to add put the download into the queue
Wait until the queue is clean and go back to #11 to make sure it's finished (repeat as many times as needed)

2 Way Syncing between 2 Servers using lsyncd

Bryan — Sat, 01 Oct 2022 22:58:04 +0000

Requirements

At least 2 servers
lsyncd installed on both servers
ssh access between both servers

Assumptions

username for both servers is: forge
IP for server #1: 111.111.111
IP for server #2: 222.222.222
Folder to sync between the 2 servers (can be different, this example both same path and directory): syncro
SSH key (can be different, this example same key name): syncro_key

Setup

This setup is for server #1, to setup server #2 it's the same. Just see the comments in the code in step #4

Install lsyncd
- sudo apt install lsyncd
Create lsyncd log folder
- sudo mkdir /var/log/lsyncd
Create log and status files
- sudo touch /var/log/lsyncd/lsyncd.{log,status}

Create config file

sudo nano /etc/lsyncd.conf.lua
Note: Anything after -- are comments in the code

-- server #1 setup
settings {
      logfile = "/var/log/lsyncd/lsyncd.log", -- location of log file created in step 3
      statusFile = "/var/log/lsyncd/lsyncd.status", -- location of status file created in step 3
}

targetlist = {
    "forge@222.222.222:/home/forge/syncro", -- server #2 will have target forge@111.111.111
}

for _, server in ipairs( targetlist ) do
    sync {
        default.rsync,
        source = "/home/forge/syncro", -- folder which to sync from
        target = server,
        delete = 'running',
        rsync = {
            rsh = "ssh -i /home/forge/.ssh/syncro_key", -- ssh key
            update = true,
            archive = true,
            compress = true,
        }
    }
end

Start the lsyncd service
- sudo service lsyncd start
Set the config lsyncd service will use
- sudo lsyncd /etc/lsyncd.conf.lua
- IMPORTANT When the server restarts run this again

Notes

There is a limit to how many files/folders that can be watched

To check the current limit:
- cat /proc/sys/fs/inotify/max_user_watches
To permanently alter the limit : (500000 is an example number)
- sudo sysctl fs.inotify.max_user_watches=500000