Forem: ivan chiou

CI/CD pipeline using GitHub Actions to deploy to Google Cloud Platform (GCP)

ivan chiou — Wed, 30 Apr 2025 01:21:58 +0000

🚀 Why CI/CD?

CI/CD automates every stage of your software delivery process:

CI (Continuous Integration): Automatically builds and tests your code on each commit.
CD (Continuous Deployment): Automatically deploys your application to the production environment after passing tests.

Together, CI/CD reduces errors, improves productivity, and ensures reliable releases—critical in agile and DevOps workflows.

🏗️ Project Overview

You’ll learn how to:

Use GitHub Actions to automate Maven build and GCP deployment.
Securely manage SSH keys and secrets.
Use Docker for containerization (optional).
Deploy to a GCP Compute Engine VM running Ubuntu.

🔧 Prerequisites

Before you begin:

GCP project with Compute Engine VM set up (Ubuntu OS, Java installed).
GitHub repository for your Spring Boot project.
Domain name (e.g., from No-IP) and optional SSL certificate via Let’s Encrypt.
SSH key pair added to your GitHub and GCP metadata.
Application setup (e.g., application.properties, MySQL, Redis, etc.) already running on the VM.

📁 GitHub Repository Setup

Add Secrets to GitHub: Navigate to Settings → Secrets and Variables → Actions, and add:
- GCP_SSH_PRIVATE_KEY: Your private SSH key (no passphrase).
- GCP_VM_IP: Your VM’s external IP.
- GCP_VM_USER: The SSH username (usually your GCP email-based user).
- KEYSTORE_BASE64: Base64 encoded keystore.p12 SSL cert file.

To create KEYSTORE_BASE64:

   base64 cert/keystore.p12 > keystore_base64.txt

Public Key in GCP VM: Upload your public SSH key (gcp_ssh_key.pub) using GCP’s OS Login or manually via ~/.ssh/authorized_keys.

📜 GitHub Actions Workflow File (`.github/workflows/deploy.yml`)

name: Deploy Java App to GCP VM

on:
  push:
    branches: [main]

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up JDK 23
        uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '23'

      - name: Build with Maven
        run: mvn clean package -DskipTests

      - name: Set up SSH
        run: |
          mkdir -p ~/.ssh
          echo "${{ secrets.GCP_SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa
          chmod 600 ~/.ssh/id_rsa
          ssh-keyscan -H ${{ secrets.GCP_VM_IP }} >> ~/.ssh/known_hosts

      - name: Deploy JAR to GCP VM
        run: |
          scp -i ~/.ssh/id_rsa target/*.jar ${{ secrets.GCP_VM_USER }}@${{ secrets.GCP_VM_IP }}:/home/${{ secrets.GCP_VM_USER }}/app.jar
          ssh -i ~/.ssh/id_rsa ${{ secrets.GCP_VM_USER }}@${{ secrets.GCP_VM_IP }} << 'EOF'
            sudo pkill -f "java -jar" || true
            nohup java -jar /home/${{ secrets.GCP_VM_USER }}/app.jar --spring.config.location=/home/${{ secrets.GCP_VM_USER }}/application.properties > app.log 2>&1 &
          EOF

🔐 SSL and Secrets Management

Use Let’s Encrypt to generate a free SSL certificate:

sudo apt install certbot
sudo certbot certonly --standalone -d your-domain.ddns.net

Convert to keystore.p12 for Spring Boot:

openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out keystore.p12 -name mysslkey -password pass:yourpassword

Encode and add it to GitHub as KEYSTORE_BASE64, then restore it in your workflow:

- name: Restore keystore.p12
  run: |
    mkdir -p src/main/resources/cert
    echo "${{ secrets.KEYSTORE_BASE64 }}" | base64 --decode > src/main/resources/cert/keystore.p12

🐳 Bonus: Dockerizing Your Java App

To containerize your app:

Create a Dockerfile:

FROM openjdk:17-slim
COPY target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]

Add Docker build/push to your GitHub Actions:

- name: Build Docker Image
  run: docker build -t youruser/demo-spring-app .

- name: Push to Docker Hub
  run: |
    echo "${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}" | docker login -u "${{ secrets.DOCKER_HUB_USERNAME }}" --password-stdin
    docker push youruser/demo-spring-app

Pull and run the container on GCP VM:

docker pull youruser/demo-spring-app
docker run -d --name demo -p 8080:8080 youruser/demo-spring-app

✅ CI/CD Flow Summary

Developer pushes code to GitHub main branch.
GitHub Actions:
- Builds the project using Maven.
- Deploys .jar or Docker image to GCP VM.
App runs on VM, accessible via your domain and secured with SSL.

🎯 Conclusion

Setting up a CI/CD pipeline with GitHub Actions and GCP Compute Engine offers a powerful, scalable, and fully automated deployment workflow for your Java apps. By leveraging GitHub’s native integration with Secrets and Actions, plus the flexibility of GCP, you can deploy with confidence and consistency.

Implementing a Virtual-Physical Environment Manipulation System based on ROS using Python and Three.js

ivan chiou — Sun, 14 Jan 2024 06:19:46 +0000

Let's explore the architecture of the Virtual-Physical Environment system. On the frontend, we utilize the Vue framework and Three.js for visualizing our virtual world, encompassing a robotic arm and a 3D environment.

Moving to the middle layer, the backend employs Python to establish a connection between the frontend through WebSockets and the physical world via Rosbridge.

In the third section, which is the local side, we have the robotic arm and IP camera. The robotic arm, based on ROS, receives messages from Rosbridge, while the IP camera covers and monitors the physical environment. The captured data is then sent to the frontend for display on pages using WebRTC.

The demonstrations are presented as follows:

The JavaScript code for receiving ROS messages from both the physical and virtual robotic arms is as follows:

The messages are displayed on the console of the physical robotic arm.

The messages are printed on the console of the client webpages.

The entire interaction flow between the virtual (Unity and Three.js) and physical environments of the robotic arm based on ROS, facilitated by ROS TCP Connector to Unity and ROSBridge to Python, is as follows:

The topic will be presented at the upcoming Conf42.

Dockerizing a project using php Laravel, Composer, Artisan and Laradock

ivan chiou — Fri, 05 Jan 2024 10:33:11 +0000

Mastering how to dockerize everything is a good starting point for any project.

Initially, you must install the PHP executable environment. (My OS is windows using Chocolatey cli)
choco install php --version=7.3.0

Next, proceed to install Composer using the Chocolatey CLI.
choco install composer --version=2.1.14

Now, you can utilize the Composer command to create the Laravel project.
composer create-project --prefer-dist laravel/laravel laraveldock "7.*.*"

It will auo-create a laravel project with composer which is similar as npm in JS, and with artisan which is a command-based interface to develop your app easily. For example, you can launch server by
php artisan serve
and synchronize the database with the migration version.
php artisan migrate

The folder structure of the project created by Composer is as follows:

The next step is to create a dockerized app using docker-compose in Laradock.

Firstly, you should add the Laradock repository as a submodule in your Laravel project.
git submodule add https://github.com/Laradock/laradock.git laradock

Then, go into the Laradock folder.
cd laradock

Rename the default environment file to .env.
mv env-example .env

Execute the docker-compose command as follows:
docker-compose up -d --build nginx

If you encounter the below issue,

Failed to fetch http://deb.debian.org/debian/dists/bookworm/InRelease Could not connect to deb.debian.org:80 (127.0.0.1). - connect (111: Connection refused)

try to add this configuration into daemon.json of docker.

{ "dns": ["8.8.8.8"] }

Finally, it is done.

Type 127.0.0.1 into your browser, then you will be directed to the front page.

If you prefer not to use Laradock, you can create a Dockerfile based on the php:7.4-apache image by your own.

FROM php:7.4-apache
RUN docker-php-ext-install pdo_mysql
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y git
RUN apt-get install zip unzip
RUN apt-get install curl && \
  curl -sS https://getcomposer.org/installer | php \
  && chmod +x composer.phar && mv composer.phar /usr/local/bin/composer
RUN apt-get install -y libpng-dev
RUN apt-get install -y zlib1g-dev
RUN docker-php-ext-install mysqli
WORKDIR /var/www/html
CMD bash -c "composer install && chmod 777 storage && chmod 777 -R bootstrap/cache && php artisan key:generate && php artisan storage:link"

How to implement a microservices architecture for an AI learning platform using Python and Kubernetes

ivan chiou — Thu, 04 Jan 2024 09:44:32 +0000

The microservices architecture of the PAIA system, an AI learning platform that encourages student competition through AI game code writing, offers three key solutions:

Seamless integration and independent control of Python language with containerized apps in our AI training system.
Independent microservices that can be updated separately.
Rapid configuration of environment parameters based ondiverse development scenarios.

Our PAIA system is built on the Django framework using Python, where Django is primarily responsible for managing the Application Programming Interface (API), encompassing authentication, authorization, and accounting features connected to a PostgreSQL database. PAIA aims to provide an Artificial Intelligence (AI) learning and competitive environment based on gaming or edutainment methodology. Users can code their AI using Blockly, a block-based visual programming language, or Python, and then upload it to run their AI models and training data on our platform.

The entire system is divided into three components:

[Frontend (Vue)] <-> [Backend (Django)] <-> [MLGame (Python/Pygame)]

Both the backend and MLGame, excluding the frontend, are developed using Python and related skills. The details of their implementation and architecture are as follows:

Backend:

API design
Serializer and folder structure in Django
Database creation and migration
Redis and cache mechanism
Pika and WebSocket integration
AAA (authentication, authorization, accounting) design

MLGame (Game Core, Game, AI, and Game Data):

MLGame, authored by Kun-Yi Li and available on GitHub
RabbitMQ consumer (Pika) with WebSocket
Kubernetes client API in Python
Python multiple inheritance and design patterns in AI fields

The main purpose of MLGame is to run games created by creators through the Pygame framework. For a game to be executable within the MLGame framework, it needs to adhere to the structure specified by MLGame. In other words, MLGame can be considered as a game engine with a focus on machine learning.

It all starts with the program MLGame.py.

def __init__(self, propty: GameMLModeExecutorProperty):
    self._execution_cmd = propty.execution_cmd

    # Get the active ml names from the created ml processes
    self._active_ml_names = list(self._comm_manager.get_ml_names())
    self._recorder = get_recorder(self._execution_cmd, self._ml_names)
    self._progress_recorder = get_progress_recorder(self._execution_cmd, self._ml_names)

By executing the following command in the command prompt (cmd), a game can be launched.
python MLGame.py -i ml_play_template.py easy_game --score 10 --color FF9800 --time_to_play 1200 --total_point 15

Among them, ml_play_template.py is an AI program written by the player. The game to be executed is named "easy_game," followed by the configuration parameters for that game.

Before running the game, we use an external program to create the MLGame container. The program is responsible for using Pika to listen to commands from the API (Backend).

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host=MQ_HOST, credentials=MQ_CREDENTIAL),
)
channel = connection.channel()

channel.queue_declare(queue=queue_name, durable=True)
channel.basic_publish(exchange='', routing_key=queue_name, body=game_cmd,
            properties=pika.BasicProperties(delivery_mode=2))

After packaging the game to be executed as a Docker container, the container will be deleted upon completion of the execution.

import docker
container_client.from_env()
container = container_client.run(target_image, 
                                 ["tail","-f","/dev/null"], 
                                 name=container_name, 
                                 detach=True, 
                                 auto_remove=False,
                                 mem_limit=memory_limit,
                                 network="bridge",
                                 stdin_open=True,tty=True)

container.exec_run(mlgame_cmd,detach=True,stdin=True,tty=True)

However, it has been modified to use pods in Kubernetes to encapsulate containers with auto-deletion after execution.

from kubernetes import client, config as k8s_config, utils
from kubernetes.client import Configuration
from kubernetes.client.api import core_v1_api
api_response = api_instance.read_namespaced_pod_log(name=POD_NAME, namespace='default')

with open(os.path.join(os.path.dirname(__file__), "/root/pod.yaml")) as f:
    pod_body = yaml.safe_load(f)
    api_instance.create_namespaced_pod(namespace='default', body=pod_body)

api_instance.patch_namespaced_pod(name=POD_NAME, namespace='default', body={"metadata":{"labels":{"name": "mlgame-completed-pod“}}})

api_instance.delete_namespaced_pod(name=POD_NAME, namespace='default')

Following this, we proceed to download the relevant AI program, ml_play_template.py. After downloading the AI program, we use the subprocess to open another thread to execute the game.
process = subprocess.Popen(mlgame_cmd, stdout=subprocess.PIPE, universal_newlines=True)

We can monitor MLGame containers through RabbitMQ to ensure that their tasks have been completed.

ASGI in Django API service facilitates the handling of messages from MLGame through websocket.

"websocket": TokenAuthMiddlewareStack(
    URLRouter(
        [
          path("ws/game_room/<str:room_id>",GameRoomConsumer.as_asgi())
        ]
    )
),

from channels.generic.websocket import AsyncWebsocketConsumer

self.accept()
self.send(text_data=message)
async def receive(self, text_data)
await self.channel_layer.group_send(self.group_name, {'type': 'chat_message’, 'message': recv_data})

Within the GameMLModeExecutor, the game loop is executed. The send_game_progress function is employed to send data to the backend via WebSocket. This data includes the coordinates, dimensions, colors, status (player info), and time of the foreground, background, and objects within the game. The backend then forwards this information to the client, rendering it in the browser.

def _loop(self):
        game = self._game_cls(*self._execution_cmd.game_params)
        assert isinstance(game, PaiaGame), "Game " + str(game) + " should implement a abstract class : PaiaGame"
        scene_init_info_dict = game.get_scene_init_data()
        # game_view = PygameView(scene_init_info_dict)
        while not quit_or_esc():
            cmd_dict = game.get_keyboard_command()
            result = game.update(cmd_dict)
            view_data = game.get_scene_progress_data()
            # game_view.draw(view_data)
            if result == "QUIT":
                scene_info_dict = game.game_to_player_data()
                self._recorder.record(scene_info_dict, {})
                self._recorder.flush_to_file()
                game.reset()
                break

In the end, after deploying a Kubernetes Cluster using Jenkins for product deployment, we can play the game and observe its animation in the frontend.

Medusa storefront with multiple vendors

ivan chiou — Thu, 02 Jun 2022 02:18:47 +0000

Medusa is open-source project which provides many powerful e-commerce functions and extensions as Shopify. I follows the article for multi-vendor marketplaces shared from Shahed Nasser who is very outstanding engineer in Medusa Dev community. But it only shows for backend modification with medusa-extender, not include the change of frontend side.

So I start to try to figure out how to modify the part of store frontend, but first we need to customize the original APIs for all vendors.

At first, following all steps from the article for multi-vendor marketplaces.

Second, add the store.router.ts file as below. In where, I refined the client API '/store/products' for all vendors.

import { MedusaAuthenticatedRequest, Router } from 'medusa-extender';
import { Response, NextFunction } from "express";
import { User } from '../../user/entities/user.entity';

@Router({
    routes: [{
        requiredAuth: false,
        path: '/store/products',
        method: 'get',
        handlers: [
            async (req: MedusaAuthenticatedRequest, res: Response, next: NextFunction): Promise<Response<User[]>> => {
                const productService = req.scope.resolve("productService")
                const resProducts = await productService.list({},
                    {
                        relations: [
                            "variants",
                            "variants.prices",
                            "variants.options",
                            "options",
                            "options.values",
                            "images",
                            "tags",
                            "collection",
                            "type",
                        ],
                    });
                return res.send({
                    "products": resProducts,
                    "count": resProducts.length,
                    "offset": 0,
                    "limit": 100
                });
            }
        ]
    }] 
})
export class StoreRouter {}

Then add StoreRouter into your store.module.ts

@Module({
    imports: [Store, StoreRepository, StoreService, StoreRouter],
})

Easy way to skip the loggedInUser condition in product.service.ts then the product list will be changed to query for all vendors. It looks like tricky but I didn't get other options to make multi-vendors workable.

    prepareListQuery_(selector: object, config: object): object {
        /*const loggedInUser = this.container.loggedInUser
        if (loggedInUser) {
            selector['store_id'] = loggedInUser.store_id // this is for one store
        }*/

        return super.prepareListQuery_(selector, config);
    }

Rebuild and start backend in your local environment. Try to create a new vendor by postman.

authenticate to this vendor then create a new product.

In the end, all clients can query all products including the new product created by the new vendor, even the client didn't login yet.

The difficult part is for the modification of store frontend. I use Medusa Gatsby Starter to my store frontend. However, there are many incompatible with my above modification of backend.

When I deployed all modification on the production environment, the store frontend rebuilt as well. But it shows failure as below:

Finally, I found the create node function(in the build time) from gatsby-source-filesystem of gatsby does not support to pass the empty link address of product thumbnail. You must give it a valid url.

I manually updated it to imgur images I created in my previous post in postgresql using pgweb tool then it rebuilt successfully. But it is not the last mission. I still found an error in gatsby-starter-medusa if the product doesn't have any uploaded images.

So we need to add below condition to prevent from this error.

I think it is not efficient to debug simultaneously both on these open-source projects, gatsby and medusa. But I will keep fighting with open-sources:)

Medusa images uploading using Imgur API

ivan chiou — Mon, 02 May 2022 03:51:23 +0000

Medusa is very powerful, useful, open-source and easily-revised e-commerce project.

But in default, it doesn't provide any kind free online space to store products' pictures. The offical documents describe how to use digitalOcean and AWS S3 to put your uploaded images of products but those services are used with paid.

Eventually, I found one way to use Imgur API to store our images for free totally. Following are the steps you can go with.

create imgur account then go to its settings page. It will list client id and secrets key of apps as below capture.

If you don't have any apps yet, create one.
Declare the headers and create an promise image upload function to call imgur API(https://api.imgur.com/3/image). (request.js in src/services)

const headers = {
  "Content-Type": "multipart/form-data",
  Authorization: "Client-ID " + IMGUR_CLIENT_ID,
}

export const imgurUploadPromise = (formData) => {
  const apiUrl = "https://api.imgur.com/3/image"
  return axios
    .post(apiUrl, formData, {
      headers,
    })
    .then((res) => {
      console.log(res)
      return res.data.data.link
    })
    .catch((err) => console.log(err))
}

Make sure the link of response format should be res.data.data.link.

Using promise.all to upload multiple images in one operation. (api.js in src/services)

  uploads: {
    create(files) {
      const promiseArray = []
      for (const f of files) {
        const formData = new FormData()
        formData.append("image", f)
        promiseArray.push(imgurUploadPromise(formData))
      }

      return Promise.all(promiseArray).then((values) => {
        const uploads = values.map((value) => {
          return { url: value }
        })
        return {
          data: {
            uploads,
          },
        }
      })
    },

Remember the return data format must be { url: value } for each promise and return data: { uploads } for all.

Then it will be done successfully. (don't upload images each over than 5MB)