Forem: Kyle Foo

Host ArcticDB on S3 and connect with Lambda

Kyle Foo — Thu, 13 Feb 2025 07:03:58 +0000

ArcticDB has been the modern database for Pandas' dataframe. It can handle billions of rows at scale, making it efficient for quantitative analysis. Therefore, I've decided to give it a spin in my data scrapping project.

Besides, Serverless Framework has been my top choice when it comes to developing Lambda function and its deployment to AWS. In this project, I wrote a data scrapping function that gets triggerred every min to scrap news and store them into ArcticDB.

First I created a S3 Bucket named devto-arctic, then connect locally with Jupyter Notebook to set up its library. I have opted to use AWS access key method to gain connection to the storage bucket.

# Jupyter Notebook
import pandas as pd
import arcticdb as adb
import os
import dotenv

dotenv.load_dotenv()
ac = adb.Arctic(f"s3://s3.us-east-2.amazonaws.com:devto-arctic?region=us-east-2&access={os.getenv('AWS_ACCESS_KEY_ID')}&secret={os.getenv('AWS_SECRET_ACCESS_KEY')}")
ac.create_library('intro')
ac.list_libraries() # output list of library in the db
df = pd.DataFrame()
ac.write('news_frame', df) # writing a empty df to a table

You will notice in your S3 bucket, a file *prefix*Intro will be created within.

Then, we should proceed to setting up Lambda Function with serverless framework. After npm install serverless, we can then initialize a Python project. Run serverless login to login to your serverless account before initialization. Next, execute serverless to choose a scheduled task python template as starter.

Once initialized, you'll get a Python project folder with all the necessary files. In the handler.py, it should be your function codes to make connection to ArcticDB for performing data read and write.

# handler.py
import datetime
import logging
import arcticdb as adb
import requests
import pandas as pd
import json
from dotenv import load_dotenv
import os

load_dotenv()

ac = adb.Arctic(f"s3://s3.us-east-2.amazonaws.com:devto-arctic?region=ap-southeast-1&access={os.environ['AWS_ACCESS_KEY_ENV']}&secret={os.environ['AWS_SECRET_ACCESS_KEY_ENV']}")
lib = ac.get_library('intro', create_if_missing=True)
ac.list_libraries()
ac.list_symbols() # symbols are equivalent to tables in a library

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

def fetch_news():
    url = "https://news.endpoint.com/api?limit=500" # dummy endpoint
    try:
        response = requests.get(url)
        response.raise_for_status()  # Raise an exception for bad status codes
        return response.json()
    except requests.RequestException as e:
        logger.error(f"Error fetching news: {str(e)}")
        return None

def run(event, context):
    symbol = 'news_frame'
    current_time = datetime.datetime.now().timestamp() * 1000
    logger.info("Your cron function ran at " + str(datetime.datetime.now().time()))

    # Fetch news data
    news_data = fetch_news()
    if news_data is None:
        return {
            'statusCode': 500,
            'body': json.dumps('Failed to fetch news data')
        }
     df = pd.DataFrame([{
            'time': datetime.datetime.fromtimestamp(int(news['time'])/1000),  # Convert ms to datetime
            'title': str(news.get('title', '')),
            'source': str(news.get('source', '')),
            'news_id': str(news.get('news_id', '')),
            'url': str(news.get('url', '')),
            'icon': str(news.get('icon', '')),
            'image': str(news.get('image', ''))
        } for news in news_data])

     try:
          print(f"\nWriting DataFrame for {symbol}:")
          lib.append(symbol, df) # use append so it doesn't overwrite old data
          print(f"Successfully wrote {symbol} to ArcticDB")
      except Exception as e:
          print(f"Error writing {symbol} to ArcticDB: {str(e)}")

    logger.info(f"Successfully processed news articles")
    return {
        'statusCode': 200,
        'body': json.dumps({
            'message': 'Successfully processed news data',
            'time': str(current_time)
        })
    }

Now we can deploy the Lambda function, but first make sure requirements.txt has all the dependencies:

# requirement.txt
arcticdb; sys_platform != "darwin"
requests
pandas
numpy
python-dotenv

Note that we skip arcticdb from pip install because of binary support for Mac machine is not yet ready at the time of writing. Running pip install locally could fail without the sys_platform != "darwin" syntax. This is a work around so that Mac would skip installing arcticdb via pip. You don't need the syntax on Windows or Linux.

If you are on Mac and want to test the code locally, do activate a virtual python env and use conda install -c conda-forge arcticdb to install arcticdb, run serverless invoke local to execute the the function.

See ArcticDB 4.3.1

In the serverless' package.json, I have made sure plugin serverless-python-requirements is included so that during serverless deployment, python dependencies in the requirements.txt will be packaged as Layer for Lambda function to import the dependent modules from.

Next, if you are on Windows or Linux, you may deploy straight from local by running serverless deploy. Deploy from Mac machine could fail as arcticdb would spit error for not finding its binary distribution as mentioned.

The workaround will be using cloud CI/CD to package and deploy the Lambda.

The scripts install-plugin and deploy in package.json will be used on CI/CD. In this case, let's use Github Actions as deployment tool, deployment script as followed:

# deploy.yml
name: deploy serverless
on:
  push:
    branches:
      - main
jobs:
  deploy:
    name: deploy
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    permissions:
      contents: read
      deployments: write
    strategy:
      matrix:
        node-version: [18.x]
        python-version: [3.9]
    steps:
      - uses: actions/checkout@v3
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}
          architecture: x64
      - run: npm ci --include=dev
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-2
      - name: Install Plugin and Deploy
        run: npm run install-plugin && npm run deploy
        env:
          SERVERLESS_ACCESS_KEY: ${{ secrets.SERVERLESS_ACCESS_KEY }}

The step to configure your AWS credentials allow serverless to deploy accordingly to your AWS environment. Please make sure the IAM for such access key is given admin permission to Lambda and S3.

Above Github Action will be triggered on push to main branch. You may configure to how you would like the deployment trigger.

After the deployment, you can see the Event Bridge is auto set up as the scheduler, and a Layer is uploaded and attached to the Lambda.

Hooray, here we go with the serverless approach to scrap data and save into ArcticDB! You may then use Jupyter Notebook to read data locally and analyze them while Lambda doing its thing in the background.

Scrapping Bybit Kline data with Pybit

Kyle Foo — Wed, 29 May 2024 18:45:40 +0000

At the time of writing this, I'm using python Pybit v5 library for scrapping Bybit kline data. Following python codes work for 1min kline scrapping, do modify the code accordingly based the interval you wish to scrap.

import pandas as pd
from datetime import date, datetime, timedelta

from pybit.unified_trading import HTTP
import numpy as np
import dateparser
import time
import math

session = HTTP(testnet=False, api_key=API_KEY, api_secret=SECRET_KEY)

def GetHistoricalData(currency, start_date, end_date, interval):
    start_time = dateparser.parse(start_date)
    end_time = dateparser.parse(end_date)
    start_ts = int(start_time.timestamp()*1000) #ms
    end_ts = int(end_time.timestamp()*1000) #ms

    page = 0
    # we will query for 2 pages per day, with limit of 720 bar returned every call, we then get 1440min for every 2 pages
    # note that API call below will use 720 as limit because that's easier for pagination
    # hence total pages = num of days * 2
    totalPage = math.ceil(start_ts - end_ts / (1000 * 60 * 60 * 24)) * 2 
    df = pd.DataFrame(columns=['startTime', 'openPrice', 'highPrice', 'lowPrice', 'closePrice', 'volume'])

    while True:
        # Kline API call, 1min interval is the smallest bar we can get, limit max at 1000 but I use 720 here
        bars = session.get_kline(symbol=currency, interval=str(interval), start=start_ts, category="linear", limit=720)
        # counter starts from end of list then goes backward since oldest data appear last
        idx = len(bars['result']['list']) - 1

        while True:
            bar = bars['result']['list'][idx]
            new_row = {'startTime': pd.to_datetime(int(bar[0]), unit= 'ms'), 'openPrice': bar[1], 'highPrice': bar[2], 'lowPrice': bar[3], 'closePrice': bar[4], 'volume': bar[5]}
            # if the kline bar exceed end_time, means reaches target date, break out of loop
            if int(bar[0]) > end_ts:
                start_ts = end_ts
                break
            # append the new row into dataframe
            df.loc[len(df)] = new_row
            # check if we reaches first item in array
            if idx == 0:
                start_ts = start_ts + 43200*1000 # increment by num of ms for a day to get start time for tmr
                break
            idx -= 1
        page += 1
        # check if the end_date is reached
        if start_ts >= end_ts:
            # exit the while loop
            break
        if page > totalPage:
            break
        # time delay to prevent api rate limit
        time.sleep(0.02)

    return df

then execute the function with symbol, intended start time and end time, as well as interval of 1min

data = GetHistoricalData("BTCUSDT", "May 01, 2023 00:00 UTC", "July 31, 2023 23:59 UTC", 1)
data.to_csv("./{}_May-01-2023-July-31-2023.csv".format("BTCUSDT"), index=False)

See https://bybit-exchange.github.io/docs/v5/market/kline for API documentation.

Electron Code Signing: Using SSL.com codeSignTool for EV code signing

Kyle Foo — Thu, 09 May 2024 14:23:41 +0000

Once EV Code Signing Certificate from SSL.com is purchased, you can then use the codeSignTool offered by them for doing application code signing. Based on what I've tested, for signing a Windows app with CodeSignTool-v1.2.7-windows, it would work on a Windows Machine. Unfortunately, I failed to get their unix codeSignTool to work on a Mac machine.

The app that I'm signing is an Electron App, with Forge (v7.4.0) configured to sign the Windows App during postMake. Hence, Forge's postMake hook was used in this example.

In your windows machine, run electron-forge make command so that MakerSquirrel can generate the .exe for signing during post hook.

const config: ForgeConfig = {
  packagerConfig: {
    asar: true,
    icon: './build/icon',
    name: 'My App',
    appBundleId: 'com.myApp.electron',
    usageDescription: {
      Camera:
        'App requires access to your camera for complete experience.',
    },
  },
  rebuildConfig: {},
  makers: [
    new MakerSquirrel((arch) => ({
      // Make sure app name without space, see @https://github.com/electron/forge/issues/3462
      name: appName.replace(' ', '-'),
      authors: 'Myself Inc',
      description: 'My Desktop app'
    })),
  ],
  hooks: {
    postMake: async (config, makeResults) => {
      if (process.platform === 'darwin') {
        return makeResults;
      } else {
        // Window machine runs this block
        makeResults.map((result) => {
          const TEMP_DIR = path.join(__dirname, 'release', 'temp');
          if (!fs.existsSync(TEMP_DIR)) {
            fs.mkdirSync(TEMP_DIR, { recursive: true });
          }

          result.artifacts.forEach((artifact) => {
            if (artifact.endsWith('.exe')) {
              console.log('===> Signing', artifact);
              const { name, dir } = path.parse(artifact);
              // CodeSignTool can't sign in place without verifying the overwrite with a
              // y/m interaction so we are creating a new file in a temp directory and
              // then replacing the original file with the signed file.
              const tempFile = path.join(TEMP_DIR, name);
              const setDir = `cd ./CodeSignTool/CodeSignTool-v1.2.7-windows`; // depending on where your codeSignTool is located
              const signFile = `CodeSignTool sign -input_file_path="${artifact}" -output_dir_path="${TEMP_DIR}" -credential_id="${process.env.WIN_SIGN_CREDENTIAL_ID}" -username="${process.env.WIN_SIGN_USERNAME}" -password="${process.env.WIN_SIGN_PASSWORD}" -totp_secret="${process.env.WIN_SIGN_TOTP}"`;
              const moveFile = `mv "${tempFile}" "${dir}"`;
              childProcess.execSync(`${setDir} && ${signFile} && ${moveFile}`, {
                stdio: 'inherit',
              });
            }
          });
          return result;
        });
        return makeResults;
      }
    }
  }
};

Note that WIN_SIGN_CREDENTIAL_ID, WIN_SIGN_USERNAME, WIN_SIGN_PASSWORD, and WIN_SIGN_TOTP shall be obtained from SSL.com, get the authorized signer from your organization account to give them to you. Have fun!

Install p12 Certificate on the CI/CD's macOS executor

Kyle Foo — Thu, 09 May 2024 13:52:38 +0000

Here I give an example for CircleCI, in a step prior to using the certificate, let's say for code-signing, make sure you have installed the certificate on your macOS executor, otherwise you may get error while executing code signing.

See sample commands below for your reference:

  sign-app-on-macOS:
    macos:
      xcode: 15.2.0
    resource_class: macos.m1.medium.gen1
    steps:
      - run:
          name: Download and install P12 certificate on macOS
          command: |
            aws s3 cp s3://bucket/certificates/Certificates.p12 Certificates.p12
            security create-keychain -p "password" ci.keychain
            security default-keychain -s ci.keychain
            security unlock-keychain -p "password" ci.keychain
            security import Certificates.p12 -k ci.keychain -P $CSC_KEY_PASSWORD -T /usr/bin/codesign
            security set-key-partition-list -S apple-tool:,apple:,codesign: -s -k "password" ci.keychain
            security find-identity -p codesigning -v
     - run: codesign --deep blah blah ...

Steps are as followed:

Download Certificates.p12 from cloud store
Create keychain and import certificate into it
Verify installed certificate that's valid

Note that CSC_KEY_PASSWORD environment variable is set for importing the password protected cert.

Pybit v5 how to subscribe to websocket topics

Kyle Foo — Thu, 06 Jul 2023 16:24:56 +0000

Well, in one of my previous post, I talked about Pybit's v2 API, and we are now at v5!

Quite some breaking updates we are expecting especially the parameters' format change for most APIs, basically it is changed from under_score to snakeCase. Upgrading it would make life easier down the road since the v5 documentations that are being maintained are much readable and informed. And of course, good amount of improvements come with it. See change log

Below is an example of using Pybit API to subscribe to account's order stream, so that we can trigger callback handler base on its socket events:

from pybit.unified_trading import HTTP
from pybit.unified_trading import WebSocket

api_key = os.environ.get("BYBIT_API_KEY")
api_secret = os.environ.get("BYBIT_API_SECRET")

class AlgoOrderStream:
    def __init__(self, trading_pair):
        self.trading_pair = trading_pair
        self.session = HTTP(testnet=True, api_key=api_key, api_secret=api_secret)
        print("Initialized ORDER_WEBSOCKET_STREAM")

    # callback handler when socket event triggers
    def handle_order_stream(self, message):
        data = message["data"]
        print(data)
        # You can see the changed order data here and perform action, below is just a example action
        # If Long position filled, place a conditional Short order
        if data[0]['side'] == 'Buy' and data[0]['orderStatus'] == 'Filled' :
            self.session.place_order(
                category="linear",
                symbol=self.trading_pair,
                side="Sell",
                orderType="Market",
                qty=0.01, # quantity
                triggerDirection=1, #triggered when market price rises to triggerPrice
                triggerPrice="YourPriceTarget",
                triggerBy="LastPrice",
                positionIdx=0,
                stopLoss="CutLossPrice",
                slTriggerBy="LastPrice",
            )

class AlgoBot:
    def __init__(self, trading_pair):
        self.trading_pair = trading_pair
        # initialize websocket instance and point to private topic -> your order stream
        self.ws = WebSocket(
            testnet=True,
            channel_type="private",
            api_key=api_key,
            api_secret=api_secret,
            trace_logging=True,
            restart_on_error=True
        )

    def start():
        orderStream = AlgoOrderStream(trading_pair=self.trading_pair)
        self.ws.order_stream(
            callback=orderStream.handle_order_stream
        )

algoBot = AlgoBot(trading_pair="BTCUSDT")
algoBot.start()

** Notice that the callback will only fire when there's updates to your order stream, do trigger order change manually for testing purpose.

NodeJs Sequelize - Update Enum column's default value

Kyle Foo — Thu, 06 Jul 2023 06:47:01 +0000

Intend to modify a Enum column's default value? we can modify it without losing the original data. Here's an use case when original enum value for size column is small, we're changing it to big.

'use strict';

/** @type {import('sequelize-cli').Migration} */
module.exports = {
  async up(queryInterface, Sequelize) {
    // Create a new enum type, make sure its name isn't the same as current column's enum type
    await queryInterface.sequelize.query(`
      CREATE TYPE enum_new_size AS ENUM ('big', 'medium', 'small');
    `);

    // Add a new temp column with the enum type, setting the new default value as 'big'
    await queryInterface.addColumn('Shirts', 'sizeTemp', {
      type: 'enum_new_size',
      allowNull: false,
      defaultValue: 'big',
    });

    // Copy over the old data to new column and cast to Enum values
    await queryInterface.sequelize.query(`
      UPDATE "Shirts"
      SET "sizeTemp" = CASE
        WHEN "size" = 'small' THEN CAST('small' AS enum_new_size)
        WHEN "size" = 'medium' THEN CAST('medium' AS enum_new_size)
        WHEN "size" = 'big' THEN CAST('big' AS enum_new_size)
        ELSE CAST('big' AS enum_new_size)
      END;
    `);

    // Remove the old column
    await queryInterface.removeColumn('Shirts', 'size');

    // Rename the temporary column
    await queryInterface.renameColumn(
      'Shirts',
      'sizeTemp',
      'size',
    );
  },

  async down(queryInterface, Sequelize) {
    /**
     * Add reverting commands here.
     */
    await queryInterface.removeColumn('Shirts', 'sizeTemp');
  },
};

Handy AWS S3 CLI commands

Kyle Foo — Mon, 26 Sep 2022 16:28:54 +0000

Some of the AWS S3 CLI commands that are handy for getting & writing objects to/from the S3 buckets.

To list data from a S3 folder:
aws s3 ls s3://bucket/folder/subfolder/ --recursive --profile kylefoo

To get object from a S3 bucket:
aws s3api get-object --bucket prod-bucket --key folder/subfolder/remote-file.txt --profile kylefoo local-file.txt

To write object to a S3 file:
aws s3api put-object --bucket prod-bucket --key folder/subfolder/remote-object.json --body local-object.json --profile kylefoo

To remove a specific file type inside a S3 subfolder
aws s3 rm s3://bucket/folder/subfolder/ --recursive --exclude "*" --include "*.txt" --profile kylefoo

Bybit's Pybit: How to subscribe to kline stream via WebSocket

Kyle Foo — Sat, 30 Apr 2022 03:16:31 +0000

As a beginner crypto trader, I often look at automating my trades so I can sleep at ease. Bybit is one of the exchanges that offers open-sourced API, but its documentation can be outdated at times. Hence, I'm learning via trial and error and finally figured out how to use its WebSocket API.

Bybit's python library had been released with a new version. At the time of writing, Pybit's latest version is v2.1.0

Now, let's look at how to consume Pybit WebSocket API to stream realtime kline data:

from datetime import date, datetime, timedelta
import dateparser
import time
import hmac
import json
from pybit import usdt_perpetual

# Apply for API keys and secrets at  https://testnet.bybit.com/
api_key="xxx"
api_secret="xxx"

# Initialize web socket connection instance
ws = usdt_perpetual.WebSocket(
    test=True,
    api_key=api_key,
    api_secret=api_secret,
)

# Initialize http connection instance
session = usdt_perpetual.HTTP(endpoint="https://api-testnet.bybit.com", api_key=api_key, api_secret=api_secret)

# Define a target price that you want to enter your position
target_price = 40000

# handle_position is a callback that will be triggered on every new websocket event (push frequency can be 1-60s)
def handle_position(message):
    data = message["data"][0]
    if data:
        # check for target_price for entering position, if confirm=True, the data is the final tick for the interval. Otherwise, it is a snapshot.
        if data['close'] >= target_price and data['confirm'] == True:
            print("Buy Order | {} | {}".format(data['close'], time.strftime("%m/%d/%y, %H:%M:%S", time.localtime())))
            tp = target_price + (target_price * 0.05)
            sl = target_price - (target_price * 0.05)
            # Buy at market price to enter Long position
            ret = session.place_active_order(
                symbol="BTCUSDT",
                side="Buy",
                order_type="Market",
                qty=0.001, # amount to buy
                time_in_force="GoodTillCancel",
                reduce_only=False,
                close_on_trigger=False,
                take_profit=tp,
                tp_trigger_by="LastPrice",
                stop_loss=sl,
                sl_trigger_by="LastPrice",
                position_idx=0,
            )
            print("Buy Order Id | {}".format(ret["result"]["order_id"], time.strftime("%m/%d/%y, %H:%M:%S", time.localtime())))

# start kline stream here and pass in callback, symbol and desired kline interval
ws.kline_stream(callback=handle_position, symbol="BTCUSDT", interval="5")

Notes: Do request for free credits on testnet.bybit.com so that you can have imaginary $$ to test your algorithm. Request can be done via live chat support.

Reference to official Pybit's WebSocket example: https://github.com/bybit-exchange/pybit/blob/master/examples/websocket_example.py

Upgrade to Gradle JDK 11 on Github Actions

Kyle Foo — Mon, 21 Feb 2022 08:15:11 +0000

Getting a compiler exception while building Android on Github Actions?

> Task :react-native-device-info:compileReleaseJavaWithJavac FAILED
An exception has occurred in the compiler (1.8.0_322). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.lang.AssertionError: annotationType(): unrecognized Attribute name MODULE (class com.sun.tools.javac.util.UnsharedNameTable$NameImpl)

In my case, react-native-device-info requires newer Gradle JDK during build. Hence, we need to update the JDK reference to version 11, not the default JDK 8 that uses version 1.8 compiler. See: https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md#java

I'm using macos-11 Image on Github Actions.

In our local machine, we can do the upgrade here in Android Studio > Preferences > Build, Execution, Deployment > Build Tools > Gradle > Select Gradle JDK version:

On CI/CD, we can configure the environment to upgrade to JDK 11. For example, in Github Actions deployment steps, we add a upgrade step (Set up JDK 11):

jobs:
  deploy:
    name: Deploying to Playstore
    runs-on: macos-11
    steps:
      - name: Checkout Code
        uses: actions/checkout@v2
      - name: Install Dependencies
        run: yarn
        working-directory: ${{ env.working-directory }}
      - name: Set up JDK 11
        uses: actions/setup-java@v2
        with:
          distribution: 'liberica'
          java-version: '11'
          cache: 'gradle'

See https://github.com/actions/setup-java for more configuration regarding distribution, version and caching.

React App served via S3 + Cloudfront getting 403 Access Denied

Kyle Foo — Tue, 28 Sep 2021 13:25:57 +0000

Serving the React App in S3 bucket and caching it with Cloudfront on its edge network will help speeding up the web app load time. All you need to do is just setting up S3 bucket as Cloudfront's origin and mapping a custom domain url to the Cloudfront url on DNS for it's distribution to work.

Ideal setup isn't it? But when you access the path to React App, and you see 403 Access Denied as followed:

<Error>
  <Code>AccessDenied</Code>
  <Message>Access Denied</Message>
  <RequestId>FYHS010933j</RequestId>   
  <HostId>iboleABhadhISamHT+je56AQHlSGAdTRIkFmGAXIbGpvd/PixA9KinXZVOYv1OOBKa7saKYtmuCkuELle=</HostId>
</Error>

It could mean that you bucket policy does not allow access to the bucket files. Hence, please verify your bucket policy to allow the GetObject permission.

In my case, I have also turned off Block public access so that public can request for the objects.

Other cause of access issues please visit https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-403/ for solutions.

Then, make sure your bucket has enabled static web hosting. Find the configuration in bucket > properties > scroll to bottom.

Next, check your Cloudfront distribution settings. To clarify, let's create a new Cloudfront distribution with Origin settings as followed:

Note on the HTTPS settings if you plan to use one, otherwise default to HTTP setup.

Then, input the alternate domain name, this is the domain url that you're going to map on the DNS after this Cloudfront distribution is created.

Secondly, choose a SSL cert if you have one. Then, make sure you have index.html as default root object.

DNS CName mapping:

Once all of the above has been performed, you should be able to access the root path of your React App. However, if you are still getting 403 access denied on a specific React route, it is because S3 will try to locate that object in the bucket with the path, and clearly that object does not exist.

This can be resolved by returning a custom error response in Cloudfront that handles the HTTP 403 Response, the custom response should direct to the path at index.html and return 200 code. This way, HTTP request will not error out, instead, react will route correctly within the app.

403 should be gone by now and you're good to go now :)

Environment Variable in JSON - Compiling ENV without Webpack on Electron

Kyle Foo — Fri, 30 Jul 2021 15:54:21 +0000

In Electron App development, it is optional to use Webpack for bundling the App. If your Electron code base isn't that big to deserve a bundler, especially when it is just a shell to render a Web App, it makes sense to skip it. The Web App itself can have its own bundler, for ex. React App.

However, we won't be having the default support of dotEnv provided by Webpack bundler, as well as hot reload and other cool features. Since React App already hot reload the web contents, is not that big of a deal for Electron to have hot reload since it is just a frame. Then, ENV compilation becomes a major problem when we don't bundle the codes with the respective env for electron-builder to build.

Electron-builder is a packager that package the code, it does not compile your env file for you. Thus, your env inside the codes will become undefined in the packaged build if we skip the bundling process. not cool :(

While not using Webpack or other bundling tool, we created a json file to host the env that we need and update them during CI/CD process. That way, we can just inject ENV that are set in the CI/CD tool into the json dynamically according to build pipeline.

First, create a config.json file that stores the list of env that you want to update during build. It should default to dev environment variable so that you do not need to update them every-time during development. For example,

{
  "env": "development",
  "appUrl": "http://localhost:8080/",
  "bugsnagAppId": ""
}

In your code,

// main.js
const fs = require('fs');
const config = JSON.parse(fs.readFileSync(path.join(__dirname, './build/config.json'), 'utf8'));
const isDevelopment = config.env === 'development';

if (isDevelopment) window.loadUrl(config.appUrl);

Then, set up your ENV in your CI/CD tool, you may have staging build pipeline and production build pipeline that has its own set of ENV. Once all is set, we use json library to do the key-value update accordingly. Below is the script that we execute during CI/CD.

# build-script.sh
yarn install

json -I -f build/config.json -e "this.env='$(echo $NODE_ENV)'"
json -I -f build/config.json -e "this.appUrl='$(echo $APP_URL)'"
json -I -f build/config.json -e "this.bugsnagAppId='$(echo $BUGSNAG_APPID)'"

yarn build:mac

Remember to add the library https://github.com/trentm/json to your package.json for it to work :)

Now, you have a dynamic config.json in different build of your Electron App.

Scaling GraphQL Subscription with Apollo Server and ElastiCache for Redis

Kyle Foo — Wed, 28 Apr 2021 07:51:23 +0000

As the GraphQL application grows and backend architecture requires to scale into high availability structure such as Dockers, you will need to make sure your Apollo Server support scaling horizontally. Otherwise, your GraphQL Subscription will break due to WebSocket Connection failure.

GraphQL subscription on a single instance architecture is pretty straight forward as you only have single WebSocket server so subscription events are centralized by nature, client connection always will be listening to the same PubSub source.

But, when you have multi instances spun up for GraphQL Server (ex. Apollo), you will turn out having multiple WebSocket connection with its own PubSub. It is not ideal and problem arises when u have multiple PubSub server.

For example, Client A and Client B in the same chat room subscribing to the same Channel might be connecting to different WebSocket instance. Thus, you won't be receiving event in the other instance when either client publishes event to the WebSocket that he/she is on.

Hence, you will need to centralize the publication of the subscription events in a PubSub server so that all instances can subscribe to and receiving all them across instances.

A multi instance architecture helps to increase reliability and create elasticity to horizontally scale your application as the number of clients grow.

To test locally, set up redis-server and redis-cli locally so you can connect your local server to local Redis PubSub instance. See https://gist.github.com/tomysmile/1b8a321e7c58499ef9f9441b2faa0aa8 for local Redis Setup

Then, in your GraphQL Server application add a PubSub file.

// PubSub Server (server/pubsub.ts)

import Redis from "ioredis";
import { RedisPubSub } from 'graphql-redis-subscriptions';
import { PubSub } from "apollo-server-express";

const redisOptions = {
  host: REDIS_HOST_URL, // set to localhost for local redis
  port: 6379
};

const dateReviver = (_key, value: string) => {
// By default, Javascript objects are serialized using the JSON.stringify and JSON.parse methods. 
// This means that not all objects - such as Date or Regexp objects - will deserialize correctly without a custom reviver, that work out of the box with the default in-memory implementation.
// For handling such objects, you may pass your own reviver function to JSON.parse.
};

export const pubsub = process.env.NODE_ENV === 'development' ? new PubSub() : new RedisPubSub({
  publisher: new Redis(redisOptions),
  subscriber: new Redis(redisOptions),
  reviver: dateReviver
});

Add PubSub to your Apollo Context, so that subscription resolvers can use use it conveniently, to perform publication.

// Apollo Server (root/index.ts)

import { pubsub } from './server/pubsub';
import { ApolloServer } from "apollo-server-express";

const server = new ApolloServer({
  typeDefs,
  resolvers,
  context: async ({ req, connection }) => {
    if (connection) {
      return {
        ...connection.context,
        pubsub,
      };
    } else {
      return { pubsub };
    }
  },
  subscriptions: {
    path: '/subscriptions',
    onConnect: () => console.log('Connected'),
    onDisconnect: () => console.log('Disconnected'),
  },
)};

Subscription Resolver:

// chatSubscription resolver/subscription/chat.ts

const resolvers = {
  Subscription: {
    chatCreated: {
      subscribe: (_parent, _args, { pubsub }) => pubsub.asyncIterator(['CHAT_CREATED']),
    },
  },
  // ...other resolvers...
};

For production you may use Amazon ElastiCache for Redis as managed service. Here's how to set it up: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/GettingStarted.html. Once created, you will refer to the generated URL and use it for REDIS_HOST_URL env.

Before deploying the changes, make sure you have update ElastiCache's security group to accept inbound traffic from your Dockers/EC2 instances. Otherwise, you might get TIMEOUT error while pubsub.ts trying to connect. Simply do this by white-listing Dockers/EC2 security group in ElastiCache's inbound rules.

Note that you cannot connect ElastiCache to local server for testing due to default security configuration. Unless you set up VPN for secured connection to the ElastiCache: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/accessing-elasticache.html#access-from-outside-aws