Forem: JL

Batect in a nutshell

JL — Wed, 03 Apr 2024 23:24:50 +0000

Batect is an alternative Docker Compose. For my own experience it is easier to use in terms of built-in features such as cross platform compatibility.

Unfortunately it is no longer maintained.

Its core is a yaml file that defines what containers to use and what tasks to execute. For example, I have one container called run-env and a task test-local which runs a line of command:

containers:
  run-env:
    build_directory: .
    volumes:
      - local: .
        container: /code
      - type: cache
        container: /code/node_modules
        name: node_modules
    working_directory: /workdir
tasks:
  test-local:
    description: Run Playwright e2e Tests
    run:
      container: run-env
      command: |
        npx playwright test
      environment:
        ENVIRONMENT: local

Quick command to run locally:
./batect test-local

Troubleshooting

Issue: permission denied: ./batect

Run
chmod +x ./batect

Issue: Checksum unmatched for the download behind firewall

When running './batect ' for the first time, batect will download jar from remote server.
Downloading Batect version 0.84.0 from https://updates.batect.dev/v1/files/0.84.0/batect-0.84.0.jar
To
/Users/{user}/.batect/cache/0.84.0/batect-0.84.0.jar

You might run into an issue:

The downloaded version of Batect does not have the expected checksum. Delete '/Users/{user}/.batect/cache/0.84.0/batect-0.84.0.jar' and then re-run this script to download it again.

In this case, the solution is to paste the url of jar into Chrome to download directly, and overwrite the local jar file.

Testing Spring Application (kotlin) with kotest, jacoco and mockk

JL — Tue, 12 Sep 2023 07:07:03 +0000

Install Kotest

In build.gradle.kts

Add a task for Test

tasks.withType<Test> {
    useJUnitPlatform()
}

Under test dependencies section

Kotest on the JVM uses the JUnit Platform gradle plugin.
Kotest offers a Spring extension that allows you to test code that uses the Spring framework for dependency injection.

    testImplementation("io.kotest:kotest-runner-junit5-jvm:5.6.2")
    testImplementation("io.kotest.extensions:kotest-extensions-spring:1.1.3")

Installing the Kotest IntelliJ IDEA Plugin It makes things convenient when you want to run tests from editor via a click.

On IntelliJ, go to File > Settings... > Plugins. While on the Marketplace tab, search for "kotest," then select the first result and click on Install.

Ref: https://www.waldo.com/blog/kotest-get-started

Pick Testing Styles
It seems the creator of kotest tried the best to cater for all sorts of users.
These are just regular classes that extend one of the Kotest spec styles.

import io.kotest.core.spec.style.FunSpec
import io.kotest.matchers.shouldBe

class StringCalculatorTest : FunSpec({

    test("empty string results in zero") {
        add("") shouldBe 0
    }
})

fun add(test: String) : Int {
    return 0
}

Test Coverage
Add jacoco in plugins section in gradle.
Add following tasks and configs

/// Test Coverage
jacoco {
    toolVersion = "0.8.8"
    reportsDirectory.set(layout.buildDirectory.dir("jacoco"))
}

tasks.test {
    finalizedBy(tasks.jacocoTestReport, tasks.jacocoTestCoverageVerification)
}

tasks.jacocoTestReport {
    dependsOn(tasks.test)
    classDirectories.setFrom(
        sourceSets.main.get().output.asFileTree.matching {
            exclude("**/somepackage/**")
        }
    )
    reports {
        xml.required.set(true)
        html.required.set(true)
    }
}

tasks.jacocoTestCoverageVerification {
    //TODO update it to 0.9
    val minimumThresholdValue = "0.0".toBigDecimal()

    violationRules {
        classDirectories.setFrom(
            sourceSets.main.get().output.asFileTree.matching {
                exclude("**/somepackage/**")
            }
        )
        rule {
            limit {
                minimum = minimumThresholdValue
            }
        }
    }
}

Ref: https://www.baeldung.com/kotlin/kotest

Use Custom CA to Bypass SELF_SIGNED_CERT_IN_CHAIN Error

JL — Tue, 15 Aug 2023 05:59:41 +0000

As a developer you might need to install packages that are hosted on artifact repositories that your VPN / fireware does not allow to access. E.g.

Downloading Chromium 116.0.5845.82 (playwright build v1076) from https://playwright-akamai.azureedge.net/builds/chromium/1076/chromium-mac-arm64.zip
Error: self signed certificate in certificate chain
    at TLSSocket.onConnectSecure (node:_tls_wrap:1539:34)
    at TLSSocket.emit (node:events:513:28)
    at TLSSocket._finishInit (node:_tls_wrap:953:8)
    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:734:12) {
  code: 'SELF_SIGNED_CERT_IN_CHAIN'
}

In such case, you can set Custom CA while downloading.

Install OpenSSL:
brew cleanup && brew update brew install openssl -f

Generate the private key to become a local CA:
openssl genrsa -des3 -out myCA.key 2048

mkdir certs
cd certs

Generate a root certificate:
openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.pem

You should now have two files:
myCA.key (your private key) and myCA.pem (your root certificate).

Ref: https://deliciousbrains.com/ssl-certificate-authority-for-local-https-development/

Now you can set CA while installing from remote:
NODE_EXTRA_CA_CERTS="{path}/certs/myCA.pem" npx <forbidden_pacakge> install

AWS IAM Role and Policies

JL — Tue, 09 May 2023 05:18:39 +0000

AWS IAM Role and Policies
IAM role and policies can be composed either via Visual Editor in AWS, or via plain JSON.

To follow the Least Privilege Principal, we can further restrict actions via Resource and Conditions.

To find the required Resource/Conditions for particular action, see:
https://iam.cloudonaut.io/reference/#/

For condition, you can use the service-specific string matching, such

      "Condition": {
         "StringEquals": {
            "aws:ResourceTag/Department": "Test"
         }
      }

Ref:
https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_ec2_securitygroups-vpc.html

jq - Json Query Tool in bash

JL — Fri, 14 Apr 2023 04:10:00 +0000

Cheatsheet (with explanation on what the params do):
https://stedolan.github.io/jq/manual/

Common Operations

Iterate all elements (and override some) within a JSON object


          echo $temp > json-override.json

          for key in $(jq '. | keys' json-override.json); do

              # remove starting and trailing quotes
              ref_key=`echo $key | tr -d '"'`
              ref_key=`echo $ref_key | tr -d ','`
              if [ $key != '[' ] && [ $key != ']' ]
              then
                  ref_val=`jq \
                  --arg target_key $ref_key \
                  '.[$target_key]' json-override.json`
                  ref_val=`echo $ref_val | tr -d '"'`
                  # echo $ref_key = $ref_val

                  case $ref_val in
                    OLD VALUE)
                      ref_val='NEW VALUE'
                      ;;
                    *)
                      echo -n "Unknown secret"
                      exit 1
                      ;;
                  esac

                  temp=`echo $temp | \
                  jq \
                  --arg target_key $ref_key \
                  --arg target_val "$ref_val" \
                  '.[$target_key] |= $target_val'`
              fi
          done
          echo $temp

GitHub Workflow - Data

JL — Fri, 14 Apr 2023 03:46:13 +0000

Data can be held in various places in GitHub workflow runs.

Job-Level Variable

      - name: Test accessing var in bash
        env:
          temp: ${{ github.event.inputs.json-override }}
        run: |
          echo $temp

Output
Can be used to pass information between sequential jobs.
https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

- name: Save state
  run: echo "{name}={value}" >> $GITHUB_STATE

- name: Set output
  run: echo "{name}={value}" >> $GITHUB_OUTPUT

Refer to output from previous job
https://docs.github.com/en/actions/using-jobs/defining-outputs-for-jobs

jobs:
  job1:
    runs-on: ubuntu-latest
    # Map a step output to a job output
    outputs:
      output1: ${{ steps.step1.outputs.test }}
      output2: ${{ steps.step2.outputs.test }}
    steps:
      - id: step1
        run: echo "test=hello" >> "$GITHUB_OUTPUT"
      - id: step2
        run: echo "test=world" >> "$GITHUB_OUTPUT"
  job2:
    runs-on: ubuntu-latest
    needs: job1
    steps:
      - env:
          OUTPUT1: ${{needs.job1.outputs.output1}}
          OUTPUT2: ${{needs.job1.outputs.output2}}
        run: echo "$OUTPUT1 $OUTPUT2"

Variables

Environment variables
Repository variables
Organization variables

Note: to use Environment variables, you have to specify the environment at job level.
job-abc:
environment: NONPROD

Refer to as

 ${{ vars['ADMIN_PASSWORD'] }} 
 ${{ vars.ADMIN_PASSWORD }}

Secrets
It is similar to variable. However they are better protected:
In the logs, any output form of secrets will be masked.

Refer to as

 ${{ secrets['ADMIN_PASSWORD'] }} 
 ${{ secrets.ADMIN_PASSWORD }}

Other restrictions found are:

The bash shell being called in the job cannot access GitHub’s secret variables. The viable path is to assign them to env of the job and access the env in the bash.
GitHub secret variables cannot be accessed by specifying the key dynamically. You must provide a static string. The best you can do is:
Matrix cannot be a solution as there is no way to share a writable variable between each run of matrix jobs.
Github does not support saving the content which has masked secrets

The only way to see the content of a secret is to output to a file and save it in artifact

      - name: Temp Verify
        if: false
        run: |
          echo ${{ secrets[github.event.inputs.single-param-name] }} > temp.txt

      - name: Archive production artifacts
        if: false
        uses: actions/upload-artifact@v3
        with:
          name: dist-without-markdown
          path: |
            temp.txt

Matrix
Can have one template job run multiple times based on an array (or all possible combinations from multi-dimensional arrays).

jobs:
  matrix-poc:
    strategy:
      matrix:
        secret_name: ['A', 'B']

    steps:
      - name: Iterate
        run: |
          echo ${{ secrets[matrix.secret_name] }}

Event Inputs
This is a read-only value which can only be assigned by user input when triggering the workflow manually. There are 4 data types: In addition to the default string type, we now support choice, boolean, and environment.

    inputs:
      environment-string:
        type: choice
        required: true
        description: Please select your deployment environment. Purely a literal string.
        default: dev
        options:
          - dev
          - tst

      environment-repo:
        description: 'Environment to run against. From what you specify in the repo'
        type: environment
        required: true

      debug:
        description: Debug mode or not
        required: false
        default: boolean

      # refer to as 
      # echo '${{ github.event.inputs.json-override }}' > json-override.json
      json-string-example:
        description: JSON string
        required: false
        default: '{
                      "key1": "value1",
                      "key2": "value2"
                  }'

GitHub Workflow - Action

JL — Fri, 14 Apr 2023 03:11:45 +0000

Github Action
Custom Github Action is for performing task which is re-used. The implementation can be in JS. Repository of the example:
https://github.com/otter13/override-with-secrets-action

Example structure of Github Action folder

.github/actions/override-with-secrets-action/action.yaml
This is the interface to GitHub, specifying inputs.

name: 'override-with-secrets-action'
description: 'loverride values in input json with secrets'
inputs:
  jsonString:
    required: true
runs:
  using: 'node16'
  main: 'dist/index.js'

Caller side in workflow

Code snippet for using the action in workflow

      - name: Checkout Source
        with:
          persist-credentials: false
        uses: actions/checkout@v3

      - name: aws-ssm-to-env
        uses: ./.github/actions/override-with-secrets-action
        with:
          jsonString: $

      - name: log envs
        env:
          output_secrets_override: ${{ env.output_secrets_override }}
        run: |
          echo $output_secrets_override

.github/actions/override-with-secrets-action/src/index.js

const core = require('@actions/core');

const configureInputs = () => {
  const inputjsonString = core.getInput('jsonString');

  return {
    jsonString: inputjsonString,
  }
}
const override = async ({
  jsonString
})=> {
  var overridenJsonString = ''
  const obj = JSON.parse(jsonString)
  for (var key in obj) {
    if (obj.hasOwnProperty(key)) {
        // TODO: override here
        console.log(key + " -> " + obj[key]);
    }
  }
  return overridenJsonString
}


const saveToEnv = (overridenJsonString) => {
    core.exportVariable('output_secrets_override', overridenJsonString);
}

const run = async () => {
  try {
    const { jsonString } = configureInputs();
    const overridenJsonString = await override({jsonString});
    saveToEnv(overridenJsonString);
  } catch (error) {
    core.setFailed(error.message);
  }
}

run();

Package.json
Specifying what packages and the build command

{
    "name": "override-with-secrets-action",
    "version": "1.0.0",
    "description": "",
    "main": "src/index.js",
    "scripts": {
        "build": "ncc build src/index.js -o dist"
    },
    "keywords": [],
    "author": "",
    "license": "ISC",
    "dependencies": {
        "@actions/core": "^1.6.0",
        "@actions/github": "^5.0.0",
        "build": "^0.1.4"
    },
    "devDependencies": {
        "@vercel/ncc": "^0.33.4"
    }
}

Terraform Notes

JL — Sat, 11 Mar 2023 08:09:51 +0000

Install the binary on Windows and set Path
https://developer.hashicorp.com/terraform/downloads

Find the example codes for provider (e.g. AWS)
https://registry.terraform.io/providers/hashicorp/aws/latest/docs

Steps:

init
will parse the script and download the necessary plugins
When Terraform initializes a new Terraform directory, it creates a lock file named .terraform.lock.hcl and the _.terraform _directory.

plan
Will have a preview of what resources to be created (by comparing configuartion and local state file)

apply
makes the changes defined by your Terraform configuration to create, update, or destroy resources.

Example codes:
first_ec2.tf

provider "aws" {
  region     = "us-west-2"
  access_key = "PUT-YOUR-ACCESS-KEY-HERE"
  secret_key = "PUT-YOUR-SECRET-KEY-HERE"
}

resource "aws_instance" "myec2" {
   ami = "ami-082b5a644766e0e6f"
   instance_type = "t2.micro"
}

Commands:

terraform init
terraform plan
terraform apply

Quick steps for configuring provider AWS for Terraform

Create a free-tier account in AWS
Log in using root user
Go to IAM > create a user > attach policy as Administrator
Use following instructions to create access key and secert pair
https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html
Then add this element in your main.tf

provider "aws" {
  region     = "us-west-2"
  access_key = "my-access-key"
  secret_key = "my-secret-key"
}

https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html

============
BASIC

Variable
Allowing you to specify value other than hardcoded in the main.tf. E.g. terraform apply -var "instance_name=YetAnotherName"

 resource "aws_instance" "app_server" {
   ami           = "ami-08d70e59c07c61a3a"
   instance_type = "t2.micro"

   tags = {
-    Name = "ExampleAppServerInstance"
+    Name = var.instance_name
   }
 }

https://developer.hashicorp.com/terraform/tutorials/aws-get-started/aws-variables

Outputs

output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.app_server.id
}

output "instance_public_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.app_server.public_ip
}

Then you can see in the log after running apply

Changes to Outputs:
  + instance_id        = "i-0bf954919ed765de1"
  + instance_public_ip = "54.186.202.254"

https://developer.hashicorp.com/terraform/tutorials/aws-get-started/aws-outputs

Version of project and Terraform
https://developer.hashicorp.com/terraform/tutorials/configuration-language/versions#inspect-the-terraform-state-file
The state file is generated/updated after running a plan/apply. It is a snapshot of the actual remote cloud resource.
"version": 4,
"serial": 8,
"terraform_version": "0.15.0",

Note: when you update your project's main.tf and execute terraform apply, the serial number will bump up by 1 or 2.

The terraform.tf file (similar to Package.json) contains the required version constrains of Terraform Core and plug-ins (=providers). There is no convention that terraform.tf must be the one. Sometimes you will see in the tutorial that versions.tf_ is used. So treat all *.tf as *.js in a node.js project.

This configuration sets required_version to ~> 0.12.29. The ~> symbol allows the patch version to be greater than 29 but requires the major and minor versions (0.12) to match the version that the configuration specifies.

Versions of providers in lock files

constraints
side effects of version upgrade

terraform.lock.hcl (HashiCorp configuration language) is a middle product of terraform.lock.hcl is a middle product of init. It is similar to Package.json.lock. It is similar to Package-lock.json in npm.

Terraform automatically creates or updates the dependency lock file each time you run the terraform init command.

Re-initialize your configuration with the -upgrade flag. This tells Terraform to download the new version of the provider, and update the version and signature in the lock file.

https://developer.hashicorp.com/terraform/tutorials/configuration-language/provider-versioning

Providers
https://registry.terraform.io/browse/providers

Define Infrastructure with Terraform Resources
Full run of a typical EC2 with security group + user data script to initiate a web application:
https://developer.hashicorp.com/terraform/tutorials/configuration-language/resource

2 key learnings from the above are:

Know how to use docs of the plugin

Output information of each 'apply'

output "domain-name" {
  value = aws_instance.web.public_dns
}

output "application-url" {
  value = "${aws_instance.web.public_dns}/index.php"
}

The previous 2 tutorials and this following one are basically talking about the same:
https://developer.hashicorp.com/terraform/tutorials/cli/init

The above one is more exploring the process of how Terraform gets the providers and modules, into local file system.

The main.tf contains reference to 1 local module and 1 remote module. Module is more like function for quick implementation to do some task, relying on other providers.

Providers, which are plugins for Terraform that extend it with support for interacting with various external systems.
Modules, which allow splitting out groups of Terraform configuration constructs (written in the Terraform language) into reusable abstractions.

module "nginx-pet" {
  source = "./nginx"

  container_name = "hello-${random_pet.dog.id}"
  nginx_port = 8001
}

module "hello" {
  source  = "joatmon08/hello/random"
  version = "3.0.1"

  hello = random_pet.dog.id

    secret_key = "secret"
}

Terraform Cloud (doing execution and storing states in remote)
After registration, use token (if you configure remote in main.tf you will be asked to use login with token) and creating an organisation:

All the apply will be executed remotely (as github workflow)

https://developer.hashicorp.com/terraform/tutorials/aws-get-started/aws-remote

Import
Command import is to syncrhonise from an existing infrastructure into terraform state. The aim is to bring that infrastructure under terraform's control.
https://developer.hashicorp.com/terraform/tutorials/state/state-import

Note: The tutorial uses a hardcoded host value docker.sock in docker provider.
The solution is here: you will need to find out what a working docker.sock is in your own machine, and replace the docker.sock in docker provider.

Commands for state management

terraform.tfstate contains the resource state of last sync. If you run terraform plan, Terraform will calculate the difference of configuration and state, to figure out what to apply.

The relationship of 3 entities is:
Configuration (via apply)-> Physical resource (after apply or via refresh)-> Local state
(tutorial: https://developer.hashicorp.com/terraform/tutorials/state/state-cli)

replace
Terraform usually only updates your infrastructure if it does not match your configuration. You can use the -replace flag for terraform plan and terraform apply operations to safely recreate resources in your environment even if you have not edited the configuration, which can be useful in cases of system malfunction.

terraform plan -replace="aws_instance.example"

show
Get a human-friendly output of the resources contained in your state.

terraform show

state list
Run terraform state list to get the list of resource names and local identifiers in your state file. This command is useful for more complex configurations where you need to find a specific resource without parsing state with terraform show.

 terraform state list
data.aws_ami.ubuntu
aws_instance.example
aws_security_group.sg_8080

state mv
The terraform state mv command moves resources from one state file to another. You can also rename resources with mv. Moving resources is useful when you want to combine modules or resources from other states, but do not want to destroy and recreate the infrastructure.

Note: The move command will update the resource in state, but not in your configuration file. You have to add the corresponding resource in the configuration as well, sperately.

terraform state mv -state-out=../terraform.tfstate aws_instance.example_new aws_instance.example_new

state rm
The terraform state rm subcommand removes specific resources from your state file. This does not remove the resource from your configuration or destroy the infrastructure itself.

terraform state rm aws_security_group.sg_8080

terraform refresh
Updates the state file when physical resources change outside of the Terraform workflow.
The -refresh-only flag was introduced in Terraform 0.15.4, and is preferred over the _terraform refres_h subcommand.
In previous versions of Terraform, the only way to refresh your state file was by using the terraform refresh subcommand. However, this was less safe than the -refresh-only plan and apply mode since it would automatically overwrite your state file without giving you the option to review the modifications first. In this case, that would mean automatically dropping all of your resources from your state file.

Use AWS cli to modify the physcial infrastructure outside terraform flow. E.g. delete an EC2 instance
aws ec2 terminate-instances --instance-ids $(terraform output -raw instance_id)

And then use terraform refresh to sync from physcial infrastructure into local state file

And then use terraform plan to check the difference between configuration and the state

terraform destroy

Note:
The unsync situation is defined as Resource Drift. The same command set (refresh and import) are covered in this tutorial:
https://developer.hashicorp.com/terraform/tutorials/state/resource-drift

Trouble-shoot (logging, support ticket)

To switch on logging:
https://www.phillipsj.net/posts/how-to-configure-logging-for-terraform/

PowerShell

> $env:TF_LOG="TRACE"
> $env:TF_LOG_PATH="terraform.txt"

Bash

$ export TF_LOG="TRACE"
$ export TF_LOG_PATH="terraform.txt"

Tutorial: https://developer.hashicorp.com/terraform/tutorials/configuration-language/troubleshooting-workflow#enable-terraform-logging

Modules

Tutorial:
https://developer.hashicorp.com/terraform/tutorials/modules/module-create
This tutorial contains a consuming project and a local module within it. The module encapsulates the actions to create a website hosting-pueposed s3 bucket which allows public access. The project uses the module by passing some inputs such as bucket name as the website name.

In summary, for a module:

Create scaffolder contains tf files and LICENSE + README.md
Define proper input vars and outputs
A consumer terraform to use the module to do stuff

The commands used in this tutorial:

terraform get
It is a light command to install modules.
In this case, Terraform will install the required 2 modules from registry, and install 1 local module by directly referring to its folder within your project.

Downloading registry.terraform.io/terraform-aws-modules/ec2-instance/aws 4.3.0 for ec2_instances...
- ec2_instances in .terraform\modules\ec2_instances
Downloading registry.terraform.io/terraform-aws-modules/vpc/aws 3.18.1 for vpc...
- vpc in .terraform\modules\vpc
- website_s3_bucket in modules\aws-s3-static-website-bucket

terraform output
The terraform output command is used to extract the value of an output variable from the state file.
https://developer.hashicorp.com/terraform/cli/commands/output
In this case we run terraform output to get the generated bucket website domain name so the developer knows easily what url to test with . E.g. https://.s3-us-west-2.amazonaws.com/index.html

Variables and Outputs
Tutorial
https://developer.hashicorp.com/terraform/tutorials/modules/module-use
The principal is to:

(Inputs)
Define variables in a standalone file variables.tf. E.g.

variable "vpc_name" {
  description = "Name of VPC"
  type        = string
  default     = "example-vpc"
}

Refer to the variables and assign them to the inputs of modules within main.tf:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  ...
  name = var.vpc_name

(Outputs)
Refer to the outputs of modules as outputs within main.tf:

output "ec2_instance_public_ips" {
  description = "Public IP addresses of EC2 instances"
  value       = module.ec2_instances[*].public_ip
}

Collection Operation

Tutorial:
https://developer.hashicorp.com/terraform/tutorials/configuration-language/troubleshooting-workflow#correct-a-for_each-error

In this case, the configuration defines a map collection of aws_instance:

resource "aws_instance" "web_app" {
   for_each               = local.security_groups
   ami                    = data.aws_ami.ubuntu.id
   instance_type          = "t2.micro"
   vpc_security_group_ids = [each.value]
   tags = {
    Name = "${var.name}-learn-${each.key}"
   }
 }
}

And the outputs refer to the collection of aws_instance

 output "instance_id" {
   description = "ID of the EC2 instance"
    value       = [for instance in aws_instance.web_app: instance.id]
 }

 output "instance_public_ip" {
   description = "Public IP address of the EC2 instance"
    value       = [for instance in aws_instance.web_app: instance.public_ip]
 }

output "instance_name" {
   description = "Tags of the EC2 instance"
   value        = [for instance in aws_instance.web_app: instance.tags.Name]
}

The outputs print:

Outputs:

instance_id = [
  "i-0a679d5acef2b8e01",
  "i-02bd8b86bfa93ae96",
]
instance_name = [
  "terraform-learn-sg_8080",
  "terraform-learn-sg_ping",
]
instance_public_ip = [
  "13.58.116.31",
  "3.131.82.166",
]

Github Workflow

JL — Sat, 25 Feb 2023 07:39:31 +0000

Events

Common filters:

on:
  push:
    paths-ignore:
      - 'docs/**'

Ref:
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#filter-pattern-cheat-sheet

Alternatively, developers can also add strings in the commit comments to skip manually:
[skip ci]
https://docs.github.com/en/actions/managing-workflow-runs/skipping-workflow-runs

Steps
can be shell scripts, or actions (using "with" to configure).

Jobs: In Parallel vs Sequential
Jobs can be sequential be specifying "needs: [job1, job2]" keywords.

Expressions & Context Objects
e.g.

run: echo "${{ toJSON(github) }}"

Example NodeJS project workflow

name: Deploy Project
on: [push, workflow_dispatch]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Get code
        uses: actions/checkout@v3
      - name: Install NodeJS
        uses: actions/setup-node@v3
        with:
          node-version: 18
      - name: Install dependencies
        run: npm ci
      - name: Run tests
        run: npm test
  deploy:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: Get code
        uses: actions/checkout@v3
      - name: Install NodeJS
        uses: actions/setup-node@v3
        with:
          node-version: 18
      - name: Install dependencies
        run: npm ci
      - name: Build project
        run: npm run build
      - name: Deploy
        run: echo "Deploying ..."

Useful actions:

Uploading Job Artifacts
https://github.com/actions/upload-artifact

Downloading Artifacts
https://github.com/actions/download-artifact


**Job Outputs**
find dist/assets/*.js -type f -execdir echo 'my-variable={}' >> $GITHUB_OUTPUT ';'
later on, this variable can be accessed via:

outputs:
    script-file: {{ steps.publis.outputs.my-variable }}

Sharing outputs across multiple jobs
needs
https://docs.github.com/en/actions/learn-github-actions/contexts#needs-context

Caching Dependencies for speeding up jobs
https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
e.g. use cache action in both build and test, with same hash value in the cache name (to make sure they are using the same cache in the central place):

(WIP) Useful Docker Commands

JL — Mon, 30 Jan 2023 01:32:19 +0000

Quickly Debug a Docker File

docker build . -t <name> 
docker image list
docker run -td <image>
docker ps
docker exec -it <container> /bin/bash

OS Arch (CPU) type
Checking the type of docker image/container

dpkg --print-architecture

Checking if your docker builder/node supports certain types

docker buildx ls

Clean up
To stop all Docker containers, simply run the following command in your terminal:

docker kill $(docker ps -q)
docker rm $(docker ps -a -q)
docker rmi $(docker images -q)

To clear the cache of builder, use following:

docker builder prune

Create multi-platform image
List all buildx on the machine
docker buildx ls

NAME/NODE       DRIVER/ENDPOINT STATUS  BUILDKIT PLATFORMS
default *       docker
  default       default         running 20.10.21 windows/amd64, linux/amd64
desktop-linux   docker
  desktop-linux desktop-linux   running 20.10.21 linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

run docker context use desktop-linux to switch to context desktop-linux

Use specified buildx
docker buildx use desktop-linux

Check what the current buildx is
docker buildx inspect --bootstrap

Name:   default
Driver: docker

Nodes:
Name:      default
Endpoint:  desktop-linux
Status:    running
Buildkit:  20.10.21
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Time to build an image
docker buildx build . -t <image_name> --platform linux/arm64

Inspect further on an instance of this image
docker run --rm <image_name> uname -m

WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
aarch64

Ref:
https://www.docker.com/blog/multi-arch-images/

*Conditional Logic in Docker Build *
By using ARG and ENV

FROM node:alpine

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

COPY package.json /usr/src/app/
RUN npm install
COPY . /usr/src/app
ENV PORT 3000
ARG DOCKER_ENV
ENV NODE_ENV=${DOCKER_ENV}
RUN if [ "$DOCKER_ENV" = "stag" ] ; then  echo   your NODE_ENV for stage is $NODE_ENV;  \
else  echo your NODE_ENV for dev is $NODE_ENV; \
fi

when you build this Dockerfile with this command

docker build --build-arg DOCKER_ENV=stag -t test-node .You will see at layer

---> Running in a6231eca4d0b your NODE_ENV for stage is stagWhen you run this docker container and run this command your output will be


/usr/src/app # echo $NODE_ENV

stag

Web Dev Suite Setup on Debian VM for Windows

JL — Sat, 28 Jan 2023 06:07:07 +0000

MacOS, Windows and Linux are the 3 players which come across in a developer's daily life. Virtual machine is always the solution. In this article I will go through the steps on how to quickly setup an Debian Linux VM with all required gears (VSCode, Chrome, Docker) for front-end devs, on a Windows laptop.

Download and install VMWare Workstation Player.
Download the ISO image of Debian installation (even you are on intel CPU)
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/
In VMWare, create a new machine from ISO installer. This will take about 20 min.
Once Debian is installed, log in and use terminal to install gdebi

sudo apt install gdebi

Now download Chrome installer on Linux (.deb) and use gdebi to load the package

Alternatively, you can install using wget via terminal:

wget --no-check-certificate https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
dpkg -i google-chrome-stable_current_amd64.deb

Download VS Code (https://code.visualstudio.com/) and install from deb installer.

Install Node

sudo apt update
sudo apt upgrade
sudo apt install nodejs npm -y

If you switch node versions often, install NVM as well

sudo apt install curl 
curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash 
source ~/.profile   
nvm install 12.18.3

Install docker desktop on Debian

sudo apt-get update
sudo apt-get install ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
Then run sudo apt-get install ./docker-desktop-<version>-<arch>.deb

Ref:
https://docs.docker.com/engine/install/debian/
https://docs.docker.com/compose/install/linux/

Note:
Update: snyk that comes with Docker Engine is not for arm64: you will get this error:
exec /user/local/bin/snyk exec format error
You will have to install the binary manually:

curl https://static.snyk.io/cli/latest/snyk-linux-arm64 -o snyk
chmod +x ./snyk
mv ./snyk /usr/local/bin/

~~If you are on arm64 Linux, your default installation will not contain Docker Scan plugin, and apt-get install docker-scan-plugin will not work. You will have to install manually:~~
(https://github.com/docker/scan-cli-plugin#on-linux)

mkdir -p ~/.docker/cli-plugins && \
curl https://github.com/docker/scan-cli-plugin/releases/latest/download/docker-scan_linux_arm64 -L -s -S -o ~/.docker/cli-plugins/docker-scan &&\
chmod +x ~/.docker/cli-plugins/docker-scan

OAuth 2.0 - Role-based Access Control

JL — Mon, 02 Jan 2023 10:33:17 +0000

Another way to control access is Role based, which enables "grouping" the privilege that makes management easier.

For example, the real life scenario is like:

Again, 3-party play:

Define user and roles in Auth Server

Resource Server: Securing endpoints to a specific role

@EnableWebSecurity
public class WebSecurity extends WebSecurityConfigurerAdapter {

    @Override
    protected void configure(HttpSecurity http) throws Exception {

        JwtAuthenticationConverter jwtAuthenticationConverter = new JwtAuthenticationConverter();
        jwtAuthenticationConverter.setJwtGrantedAuthoritiesConverter(new KeycloakRoleConverter());

        http
            .authorizeRequests()
                    .antMatchers(HttpMethod.GET, "/users/status/check")
                    .hasRole("developer")

                .anyRequest().authenticated()
                .and()
            .oauth2ResourceServer()
            .jwt()
            .jwtAuthenticationConverter(jwtAuthenticationConverter);
    }

}

Resource Server: Mapping the JWT role, into Autority class reckognised by Spring.

In the code, we will need to convert the role into spring-authorities
Here is a trick on how to inspect the role from the token (https://jwt.io/)

public class KeycloakRoleConverter implements Converter<Jwt, Collection<GrantedAuthority>> {

    @Override
    public Collection<GrantedAuthority> convert(Jwt jwt) {
        Map<String, Object> realmAccess = (Map<String, Object>) jwt.getClaims().get("realm_access");

        if (realmAccess == null || realmAccess.isEmpty()) {
            return new ArrayList<>();
        }

        Collection<GrantedAuthority> returnValue = ((List<String>) realmAccess.get("roles"))
                .stream().map(roleName -> "ROLE_" + roleName)  
                .map(SimpleGrantedAuthority::new)
                .collect(Collectors.toList());

        return returnValue;
    }

}

Spring framework has 2 methods to check:

hasAuthority("ROLE_developer"); //you have to add the prefix ROLE_ by yourself.
hasRole("developer"); // it is working the same as above, but without needing to add prefix ROLE_