Forem: Winner Musole Masu

The Guide to Build and Dockerize a Go App

Winner Musole Masu — Tue, 10 May 2022 05:12:44 +0000

Hey guys, welcome to our tutorial. The verb Dockerize might sound unfamiliar for some, just to be in harmony, the verb simply means that we will pack, deploy and run our Go application using Docker containers.

Containers are units of software that package up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. In other words, all the necessary tools to build our Go Application will be piled together with the application itself to make one single portable and standard module; which could be accessible anywhere and run on various operating systems.

Prerequisites

The first prerequisite for this tutorial is to have a basic knowledge of Go programming and second, to have the following:

Docker installed on our computers,
Docker Hub account,
Go Runtime installed,
RapidAPI account and subscription to Alpha Vantage API

1. Go Application

This app is going to be an API that fetches stock market information of a number of listed companies. It will send a GET request to Alpha Vantage API which is the most effective way to receive stock data.

As designed in the scenario above, a client will make a GET request with a query parameter to the server running in the local machine. Then The server will take that query parameter to make another GET request now to the Alpha Vantage API, which will return stock data related to the query.

Note: Open this link to sign up with Rapid API and subscribe to Alpha Vantage API.

Now, let's dive into coding. First, create a directory named stock-list and move inside:

$ mkdir stock-list
$ cd stock-list/

Run this command to manage internal packages or dependencies:

$ go mod init stock-list

And run these commands to install the dependencies to be used during this tutorial:

$ go get github.com/gin-gonic/gin
$ go get github.com/rodaine/table

Create a file named stock.go, open it and put the code snippet below:

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"

    "github.com/rodaine/table"
)

type Quote struct {
    SYMBOL string `json:"01. symbol"`
    OPEN string `json:"02. open"`
    HIGH string `json:"03. high"`
    LOW string `json:"04. low"`
    PRICE string `json:"05. price"`
    VOLUME  string `json:"06. volume"`
    LATEST_TRADING_DAY string `json:"07. latest trading day"`
    PREVIOUS_CLOSE string `json:"08. previous close"`
    CHANGE string `json:"09. change"`
    CHANGE_PERCENT string `json:"10. change percent"`
}

type Stock struct {
    DETAIL Quote `json:"Global Quote"`
}

func getStock(s string, ch chan Stock) Stock{
    var stock Stock


    api := "https://alpha-vantage.p.rapidapi.com/query?function=GLOBAL_QUOTE&symbol="+s+"&datatype=json"

    req, _ := http.NewRequest("GET", api, nil);

    req.Header.Add("X-RapidAPI-Host", "alpha-vantage.p.rapidapi.com")
    req.Header.Add("X-RapidAPI-Key", "Here put your RapidAPI Key")

    res, err := http.DefaultClient.Do(req)

    if err != nil {
        fmt.Println("No response from request")
    }

    defer res.Body.Close()

    body, _ := ioutil.ReadAll(res.Body)


    if err := json.Unmarshal(body, &stock); err != nil {
        fmt.Println("Can not unmarshal JSON")
    }
    ch<- stock
    return stock
}

func genTable(watchList []Quote){

    tbl := table.New("SYMBOL", "OPEN", "HIGH", "LOW", "PRICE", "VOLUME", "PREVIOUS-CLOSE", "CHANGE")

    for _, stock := range watchList {
        tbl.AddRow(stock.SYMBOL, stock.OPEN, stock.HIGH, stock.LOW, stock.PRICE, stock.VOLUME, stock.PREVIOUS_CLOSE, stock.CHANGE)
    }

    tbl.Print()

}

1.1 Code overview 1

In the code above, we did the following:
1. Imported a couple of packages with

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"

    "github.com/rodaine/table"
)

This section of the file imports fmt that prints string to the console, github.com/rodaine/table which generates table from the stock data, net/http is used to make HTTP requests to Alpha Vantage API, io/ioutil is used to read stock data of the response body from the Alpha Vantage and encoding/json to convert the json stock data (which is in []byte) into our object struct.

2. Created structs for stock data

type Quote struct {
    SYMBOL string `json:"01. symbol"`
    OPEN string `json:"02. open"`
    HIGH string `json:"03. high"`
    LOW string `json:"04. low"`
    PRICE string `json:"05. price"`
    VOLUME  string `json:"06. volume"`
    LATEST_TRADING_DAY string `json:"07. latest trading day"`
    PREVIOUS_CLOSE string `json:"08. previous close"`
    CHANGE string `json:"09. change"`
    CHANGE_PERCENT string `json:"10. change percent"`
}

type Stock struct {
    DETAIL Quote `json:"Global Quote"`
}

The struct Quote basically represents the property value of the stock data contained in the response body returned by the Alpha Vantage. Stock struct is a nested struct that embodies the key and the value of the response.

3. Created the getStock() function

func getStock(s string, ch chan Stock) Stock{
    var stock Stock


    api := "https://alpha-vantage.p.rapidapi.com/query?function=GLOBAL_QUOTE&symbol="+s+"&datatype=json"

    req, _ := http.NewRequest("GET", api, nil);

    req.Header.Add("X-RapidAPI-Host", "alpha-vantage.p.rapidapi.com")
    req.Header.Add("X-RapidAPI-Key", "Here put your RapidAPI Key")

    res, err := http.DefaultClient.Do(req)

    if err != nil {
        fmt.Println("No response from request")
    }

    defer res.Body.Close()

    body, _ := ioutil.ReadAll(res.Body)


    if err := json.Unmarshal(body, &stock); err != nil {
        fmt.Println("Can not unmarshal JSON")
    }
    ch<- stock
    return stock
}

This function takes two parameters, a string which is just a stock symbol and a channel of Stock struct type. It then retrieves stock data of that stock symbol from the Alpha Vantage API.

Note: Make sure to add your RapidAPI Key where indicated.

Copy the key indicated in your side of the screen below:

And paste it here:

Now that you have pasted your Rapid API key to the appropriate place, let's carry on with this function explanation. In two sequences we did the following:

Sequence 1:
The function makes a GET request to the Alpha Vantage API using http package, it adds two header parameters to the request, which carries credentials containing the authentication information of our client for the stock data being requested. X-RapidAPI-Host and X-RapidAPI-Key.

Sequence 2:
The function reads the response body from the Alpha Vantage API with the io/ioutil package, converts the body from json of []byte to stock struct type with json.unmarshal and return the converted stock data via the channel to main thread ch<- stock.

4. Created the genTable() function:

func genTable(watchList []Quote){
    tbl := table.New("SYMBOL", "OPEN", "HIGH", "LOW", "PRICE", "VOLUME", "PREVIOUS-CLOSE", "CHANGE")

    for _, stock := range watchList {
        tbl.AddRow(stock.SYMBOL, stock.OPEN, stock.HIGH, stock.LOW, stock.PRICE, stock.VOLUME, stock.PREVIOUS_CLOSE, stock.CHANGE)
    }

    tbl.Print()

}

genTable() takes an array of Quote struct type as parameter and generates a table from it. It will generate a row for every item in the array.

In the same folder /stock-list, add a go file named main.go and put the following code:

package main

import (
    "strings"

    "github.com/gin-gonic/gin"
)

func main() {

    var watchList []Quote
    ch := make(chan Stock)

    router := gin.Default()

    router.GET("/stocks/", func (c *gin.Context){
        stockSymbol := c.Query("symbol")
        listOfSymbol := strings.Split(stockSymbol, ",")

        for _, symbol := range listOfSymbol{
            go getStock(symbol, ch)
        }

        for i := 0; i < len(listOfSymbol); i++ {
            result := <-ch

            watchList = append(watchList, Quote{
                SYMBOL: result.DETAIL.SYMBOL,
                OPEN: result.DETAIL.OPEN, 
                HIGH: result.DETAIL.HIGH, 
                LOW: result.DETAIL.LOW, 
                PRICE: result.DETAIL.PRICE, 
                VOLUME: result.DETAIL.VOLUME, 
                LATEST_TRADING_DAY: result.DETAIL.LATEST_TRADING_DAY, 
                PREVIOUS_CLOSE: result.DETAIL.PREVIOUS_CLOSE, 
                CHANGE: result.DETAIL.CHANGE, 
                CHANGE_PERCENT: result.DETAIL.CHANGE_PERCENT,
            })

        }


        genTable(watchList)
    })

    router.Run("localhost:8000")

}

1.2 Code overview 2

Let's break down the code above;
1. Imported packages:

package main

import (
    "strings"

    "github.com/gin-gonic/gin"
)

In this part of the code, strings package is imported to override string methods and github.com/gin-gonic/gin or Gin, a HTTP web framework to handle routing with simplicity.

2. main() function:

func main() {

    var watchList []Quote
    ch := make(chan Stock)

    router := gin.Default()

    router.GET("/stocks/", func (c *gin.Context){
        stockSymbol := c.Query("symbol")
        listOfSymbol := strings.Split(stockSymbol, ",")

        for _, symbol := range listOfSymbol{
            go getStock(symbol, ch)
        }

        for i := 0; i < len(listOfSymbol); i++ {
            result := <-ch

            watchList = append(watchList, Quote{
                SYMBOL: result.DETAIL.SYMBOL,
                OPEN: result.DETAIL.OPEN, 
                HIGH: result.DETAIL.HIGH, 
                LOW: result.DETAIL.LOW, 
                PRICE: result.DETAIL.PRICE, 
                VOLUME: result.DETAIL.VOLUME, 
                LATEST_TRADING_DAY: result.DETAIL.LATEST_TRADING_DAY, 
                PREVIOUS_CLOSE: result.DETAIL.PREVIOUS_CLOSE, 
                CHANGE: result.DETAIL.CHANGE, 
                CHANGE_PERCENT: result.DETAIL.CHANGE_PERCENT,
            })

        }


        genTable(watchList)
    })

    router.Run()

}

This is the entry point of the executable program. In there, we declared an array watchList of struct Quote type to which stock data will be appended. Next, we created a channel with ch := make(chan Stock) to make the communication between main thread and child threads.

router := gin.Default(), creates gin router which is used to make a handler with router.GET("/stocks/", HandlerFunction(){})

3. Handlerfunction(c *gin.Context):
This handler takes the query parameter with stockSymbol := c.Query("symbol") using the Gin package and since the query might contain more than one string parameter separated by a comma, we derived an array of strings out of it with listOfSymbol := strings.Split(stockSymbol, ",").

Then for each parameter(stock symbol), we concurrently make GET request to the Alpha Vantage API with:

for _, symbol := range listOfSymbol{
            go getStock(symbol, ch)
        }

Next for every GET request, Alpha Vantage sends back a response of stock data that go getStock(symbol, ch) as a child thread, passes through the channel to the main thread. In the main thread, we store stock data in the watchList array and generate a table with genTable(watchList).

4. Finally, we start the router using router.Run()

Our project directory should look like the project tree below:

.
├── go.mod
├── go.sum
├── main.go
└── stock.go

0 directories, 4 files

2. Dockerize Go App

In this part of the tutorial, we will dockerize the Go application newly built. To do so, stay in the working directory, create a file named Dockerfile where you put the code below:

FROM golang:1.18-alpine

WORKDIR /app

COPY go.mod ./
COPY go.sum ./

RUN go mod download


COPY *.go ./

RUN go build -o /stock-list

CMD [ "/stock-list" ]

FROM golang:1.18-alpine this line tells Docker to use goland:1.18-alpine as the base image of our Go app image. It comes with all tools and packages to compile and run our application.

WORKDIR /app this one will create a working direction inside the image that we are building.

COPY go.mod ./ and COPY go.sum ./ instructions copy go.mod and go.sum files to the working direction in our docker image /app.

RUN go mod download downloads our application dependencies inside the docker image working directory.

COPY *.go ./ this instruction copies every go file in the our Go app local machine directory to the working directory in the docker image.

RUN go build -o /stock-list this will compile our application and CMD [ "/stock-list" ] will execute our compiled code.

2.1 Build and Run docker image

After successfully creating the Dockerfile, let's build a docker image from it.

Note: Before doing that, make sure to have a Docker Hub account and with its credentials, sign in using docker login command in your local machine.

In the project directory /stock-list, use your Docker Hub ID in the command below to build the image:

$ docker build --tag <YOUR DOCKER HUB ID>/stock-list .

Visualize stock-list docker image by running this command:

$ docker image ls

Run the stock-list image:

$ docker run -p 8080:8080 <YOUR DOCKER HUB ID>/stock-list

In order to view our running image or container, let run the command below:

$ docker container ls

After running this command, we can see that the docker container of stock-list image is exposed to the 8080 on your local machine 0.0.0.0:8080->8080/tcp.

To see if the Go application is working correctly, we are going to make a GET request to the server using curl:

$ curl http://localhost:8080/stocks/?symbol=TSLA,TWTR,AAPL

Find more stock symbols here, to perform more test.

In addition, we can use different other commands to either stop the container in our case, the container running the stock-list image, restart or remove it.

To stop the container, let's take our container ID or container's name here;

Paste it to the command below:

$ docker stop <YOUR CONTAINER ID>

To restart, we can use the same container ID or container's name with this command:

$ docker restart <YOUR CONTAINER ID>

And to completely remove the container of stock-list image, still with the same container ID, let us use the following command:

$ docker rm <YOUR CONTAINER ID>

For more detailed docker commands, please check out the official documentation.

2.2 Deploy to Docker Hub

This is the last part of the tutorial where we will push our local stock-list image to Docker Hub. We successfully built our Go app and packed it with all necessary dependencies in a docker image; it is time to upload it to Docker Hub.

Let's push our image with this command:

$ docker push <YOUR DOCKER HUB ID>/stock-list

Doing this will make our stock-list image portable and accessible to other developers who can run it on their operating systems.

Our tutorial ends here guys, we built a Go application from scratch, we used some external go packages and made HTTP calls to a great API. To dockerize the application, we built a docker image from it, we ran it and pushed it to Docker Hub. Thanks for doing this together.

Find the project code here.

Data Streaming for Microservices using NATS Streaming - Part 2

Winner Musole Masu — Thu, 10 Mar 2022 02:20:54 +0000

Welcome to this tutorial, here we are going to finished what was started in Data Streaming for Microservices using Nats Streaming - Part 1. To recapitulate, we will complete the application designed in the figure below:

It's a basketball dunk contest application made of two services that communicate with each other using NATS Streaming. Service 1, Dunk Service is an Express Application that registers players for the contest, where players attempt dunk shots and publishes shots information through events on a NATS Streaming Server. This was successfully completed in Part 1, let's work on Service 2 in the next lines.

1. Statistic Service

This is an Express Application that will display statistics of the contest app in real-time. It will be ranking players as shown in table below:

Now open the terminal in the project directory dunk-contest/, run these commands:

$ mkdir statistic-service
$ cd statistic-service/
$ npm init -y
$ npm install ts-node-dev typescript express @types/express node-nats-streaming mongodb

Open the package.json file, replace the actual script section by the one below:

  "scripts": {
    "listen": "ts-node-dev --rs --notify false src/listener.ts"
  },

Save the file, and in the same directory create a folder named src/ where you add a typescript file called listener.ts.
Now in listener.ts put the following code:

import nats, { Message } from "node-nats-streaming";
import { MongoClient } from "mongodb";

const start = async () => {
  const stan = nats.connect("dunk-contest", "321", {
    url: "http://localhost:4222",
  });

  stan.on("connect", () => {
    console.log(
      "Statistic Service is connected to NATS Streaming Server \nWaiting for Events ..."
    );

    stan.on("close", () => {
      console.log("Nats connection closed!");
      process.exit();
    });

    const options = stan
      .subscriptionOptions()
      .setManualAckMode(true)
      .setDeliverAllAvailable()
      .setDurableName("Dunk-Shot-Queue-Group");

    const subscription = stan.subscribe(
      "Dunk-Shot",
      "Dunk-Shot-Queue-Group",
      options
    );

    subscription.on("message", async (msg: Message) => {
      const parsedData = JSON.parse(msg.getData().toString("utf-8"));
      console.log("EVENT RECEIVED WITH THE DATA BELOW :");
      console.table(parsedData);

      const mongoClient = await MongoClient.connect(
        "mongodb://localhost:27017/statistic-service"
      );

      const db = mongoClient.db();
      const dunkCollection = db.collection("dunks");
      await dunkCollection.insertOne(parsedData);

      const dunkStatistic = await dunkCollection
        .aggregate([
          {
            $group: {
              _id: "$PLAYER_NAME",
              TOTAL_DUNK: { $count: {} },
              TOTAL_POINT: { $sum: "$DUNK_POINT" },
            },
          },
          { $sort: { TOTAL_POINT: -1 } },
        ])
        .toArray();

      console.log("\x1b[36m%s\x1b[0m", "DUNK CONTEST STATISTIC :");
      console.table(dunkStatistic);
      mongoClient.close();

      msg.ack();
    });
  });
};

start();

In the code above;

We imported nats and Message from the node-nats-streaming library,
We imported MongoClient from the mongodb library,
And in the start function:

We defined a connection to the NATS-Streaming-Server that we started in part 1 on this tutorial series,
On connection, the app subscribes to Dunk-Shot channel, the channel which the Dunk-Service publishes events through. And apart from the channel, we added subscriptions options and a queue group,
On message event, which occurs when the Dunk Service publishes a dunk-shot info; the Statistic-Service connects to its MongoDB server using the mongodb library, takes the message data(dunk-shot info) and inserts it in the dunks collection,
With the dunks collection we made a query that selects and groups each player by the total number of dunk shots and total number of dunk points.

Now generate a tsconfig.json file to compile your TypeScript code in JavaScript code. Open your terminal, navigate back to statistic-service/ directory and run the command below:

$ tsc --init

The statistic-service/ directory should look like the tree below:

.
├── package.json
├── package-lock.json
├── src
│   └── listener.ts
└── tsconfig.json

1 directory, 4 files

Save all files and let's do the following tests:

Test 1: Access to NATS Streaming server Pod

Start the local Kubernetes Cluster by running this in the terminal:

$ minikube start

Then let's get the list of all running pods in the cluster by running:

$ kubectl get pods

Copy your pod's name, you will use it in the command that is coming.

Here we are going to make the NATS Streaming server pod running in the kubernetes cluster accessible in our local machine. Open the terminal, forward a local port on your machine to a port on your pod by running the following:

$ kubectl port-forward <YOUR POD NAME> 4222:4222

Note: Make sure to replace by your real pod's name.

Test 2: Access to MongoDB pod dedicated to the Dunk Service

Run kubectl get pods in the terminal to take the pod's name;

With this pod's name forward port 27017 on your local machine to port 27017 on the MongoDB pod inside the cluster:

$ kubectl port-forward <YOUR POD NAME> 27017:27017

Note: Make sure to replace by your real pod's name.

Test 3: Access to MongoDB pod dedicated to the Statistic Service

Here too, forward port 27016 on your local machine to port 27017 on the MongoDB pod dedicated to the statistic service in the cluster:

$ kubectl port-forward <YOUR POD NAME> 27016:27017

Note: Make sure to replace by your real pod's name.

Test 4: Start the Dunk and the Statistic Services(Express applications)

Note: All last three terminals we opened above must still be running.

We must be in the project directory dunk-contest/, where we open more terminals.
In the directory dunk-service/, open a terminal and run the following command:

$ npm start

And in the directory statistic-service/, open a terminal and run the following command:

$ npm run listen

By now, the Dunk Service and the Statistic Service should be connected to the NATS Streaming Server pod and to their respective MongoDB pods.

Test 5: Open your API Client and do the following requests

Request 1: Post request on http://localhost:4001/dunk-contest/register

Make 2 POST requests, with HEADERS Content-Type: application/json:
Post Request 1:

{
  "name": "LeBron",
  "height": 2.06,
  "weight": 113,
  "experience": 19
}

Post Request 2:

{
  "name": "Stephen",
  "height": 1.88,
  "weight": 84,
  "experience": 13
}

Note: Just here, we registered two players for the contest,in the next requests let's have them attempt shot.

Request 2:

Post request on http://localhost:4001/dunk-contest/attempt/LeBron
Post request on http://localhost:4001/dunk-contest/attempt/Stephen

Note: With the two post requests, we made our two players attempting each one dunk shot. Feel free to make a lot of these two last requests to see how the Statistic Service produces the real-time contest statistic in its console.

We are done with the tests, our two services are working as intended. Dunk Service registers players and have players attempt dunk shots while Statistic Service makes and updates the statistic of the contest at every dunk shot.

The tests were performed out of the Kubernetes Cluster so far, just for demonstration purposes. In next line, let's make all the services work inside the Cluster.

2. Dockerize Services

We reached the point of the tutorial where we are going to containerize our services with Docker. By doing this, docker images of our services will be uploaded on docker hub and later be used to create pods inside the Kubernetes Cluster.

2.1 Dunk Service Docker Image

In the dunk-service/ directory, create a file named Dockerfile and put the code below:

FROM node:alpine

WORKDIR /app
COPY package.json .
RUN npm install --only=prod
COPY . .

CMD ["npm", "start"]

Let's pull this apart piece by piece.
FROM
This command tells that the base image of this new docker is node:alpine.
WORKDIR
Here the working directory of the running image is defined.
COPY
COPY command just copies the selected file to the working directory.
RUN
RUN instruction allows you to install your application and packages required for it.
CMD
This allows us to set a command that will be executed when the container is running.

Still in the same direction, create another file named .dockerignore and paste the following code:

node_modules

.dockerignore will tell docker not to copy node_modules to the working directory.

Note: In part 1 of the tutorial, prerequisites stated that one should have docker installed in his/her computer, also have a docker hub account.

Now in same directory dunk-service/, open the terminal and do the commands below:

$ docker build -t <YOUR DOCKER HUB ID>/dunk-service .

Note: Make sure to replace by your real docker hub id.

With the command above, we built a docker image for the Dunk Service.
Next, push the image recently built to Docker Hub. The first step to pushing the image is to log in to the Docker Hub account:

$ docker login

Once logged in, push the image with the command below:

$ docker push <YOUR DOCKER HUB ID>/dunk-service

Note: Make sure to replace by your real docker hub id.

2.2 Statistic Service Docker Image

Here, we are going to go through the same steps we went through in 2.1 section of this tutorial but in the statistic-service/ directory.

Create a file named Dockerfile in statistic-service/ directory and put the code below:

FROM node:alpine

WORKDIR /app
COPY package.json .
RUN npm install --only=prod
COPY . .

CMD ["npm", "run", "listen"]

And in the same directory, create another file named .dockerignore and put the following:

node_modules

Build the Statistic Service image with

$ docker build -t <YOUR DOCKER HUB ID>/statistic-service .

Note: Make sure to replace by your real docker hub id.

Push the Statistic Service image to Docker Hub:

$ docker push <YOUR DOCKER HUB ID>/statistic-service

Note: Make sure to replace by your real docker hub id.

Your docker hub repositories should have these images as repositories by now:

3. Kubernetes Pods

In this section, we are going to deploy pods of our services images from Docker Hub in the Kubernetes Cluster.

3.1 Dunk-Service Pod

The figure shows how we shall proceed, we will first create a Deployment object, which will generate and monitor a Pod of the Dunk-Service image. After we will create a Service object that will make pod generated by the deployment object accessible.

Move to the kubernetes/ directory, create a yaml file named dunk-service-deployment.yaml and put the code below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dunk-service-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dunk-service
  template:
    metadata:
      labels:
        app: dunk-service
    spec:
      containers:
        - name: dunk-service
          image: <YOUR DOCKER HUB ID>/dunk-service

Note: Make sure to replace by your real docker hub id.

With these configurations, kubernetes will create a Deployment object to monitor a Pod running the Dunk-Service image.

And create another yaml file named dunk-service-svc.yaml and put the follow:

apiVersion: v1
kind: Service
metadata:
  name: dunk-service-svc
spec:
  selector:
    app: dunk-service
  type: ClusterIP
  ports:
    - name: dunk-service
      protocol: TCP
      port: 4001
      targetPort: 4001

Another config file that will create a kubernetes object of kind Service that will permit other pods in the cluster to access the Dunk-Service pod.

3.2 Statistic-Service Pod

Similar to 3.1, we will first create a Deployment object, which will generate and monitor a Pod of the Statistic-Service image.

In the same directory, create a yaml file named statistic-service-deployment.yaml and with the following code:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: statistic-service-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: statistic-service
  template:
    metadata:
      labels:
        app: statistic-service
    spec:
      containers:
        - name: statistic-service
          image: <YOUR DOCKER HUB ID>/statistic-service

Note: Make sure to replace by your real docker hub id.

Your kubernetes/ directory tree structure should look like this by now:

.
├── dunk-mongo-deployment.yaml
├── dunk-mongo-service.yaml
├── dunk-service-deployment.yaml
├── dunk-service-svc.yaml
├── nats-deployment.yaml
├── nats-service.yaml
├── statistic-service-deployment.yaml
├── stats-mongo-deployment.yaml
└── stats-mongo-service.yaml

0 directories, 9 files

Open the terminal in the same directory and run the command below to create kubernetes objects newly added:

kubectl apply -f .

At this stage of the tutorial we must have five pods running in the kubernetes cluster.

Early, we exposed our services outside of the cluster by forwarding ports in our local machines to different ports of pods running within the cluster. Optimally, let's use Ingress to do same thing.

3.3 Ingress

Ingress is a Kubernetes object which manages external accesses to services in a cluster.
Go ahead and create a yaml file in the kubernetes/ directory named ingress-service.yaml with the following code inside:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /?(.*)
spec:
  rules:
    - http:
        paths:
          - path: /?(.*)
            pathType: Prefix
            backend:
              service:
                name: dunk-service-svc
                port:
                  number: 4001

In this file, we defined the kind of kubernetes resource we want to create Ingress and in the Ingress spec we set http(s) rules which assure that all incoming requests are directed to the Dunk-Service pod via its dedicated kubernetes service resource.

Great, before moving further run this command in the terminal:

$ minikube addons enable ingress

This command will install ingress controller in the cluster, which enables ingress resource to work.
After, open the terminal in kubernetes/ directory and run:

$ kubectl apply -f ingress-service.yaml

Now that we can access pods running within our kubernetes cluster, let's see how this will work.

4. Test

In this test section, you are required to install Skaffold in your machine. It is required because it facilitates continuous development for kubernetes-native applications.
Basically, Skaffold will build a new docker image for each of our services anytime we make code changes, it will then push that image to docker hub and finally deploy it inside the kubernetes cluster.
Click Here to download Skaffold in your computer.

Create a new yaml file in the project directory dunk-contest/, name the file skaffold.yaml and put the code below:

apiVersion: skaffold/v2alpha4
kind: Config
deploy:
  kubectl:
    manifests:
      - ./kubernetes/*
build:
  local:
    push: false
  artifacts:
    - image: <YOUR DOCKER HUB ID>/dunk-service
      context: dunk-service
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: "src/**/*.ts"
            dest: .
    - image: <YOUR DOCKER HUB ID>/statistic-service
      context: statistic-service
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: "src/**/*.ts"
            dest: .

Up there, in the deploy section, we tell how images are to be deployed in the cluster. For this case it will be by running kubectl apply on all yaml files in location specified by the manifests.

Then with build section in the artifacts we define the name of the image to be built. In the context we define directory containing the artifact's sources. In docker we describes an artifact built from a Dockerfile, located in the workspace.
Finally with sync, we sync local files to pods instead of triggering an image build when changes occur. And manually we indicate the files sources and the destination.

Note: make sure to replace by Docker Hub id.

Save it, open the terminal in the project directory dunk-contest/ and run the command below:

$ skaffold dev

Note: The first attempt might fail, you can try the command even trice.

Let's make some changes in our services code. First changes will be done in the Dunk-Service code where instead of connecting the service to pods that were exposed to our local machines with port-forward, we will connect the service to those pods using their appropriate Kubernetes service object.
Open the project in the dunk-service/src/ directory and in the index.ts file, replace the old url by the new one:

In dunk-service/src/routes directory, open registerPlayerRoutes.ts to replace the old code by the new:

Still in dunk-service/src/routes directory/ open attemptDunkRoutes.ts to replace old code by new:

And the last changes will be done in the Statistic-Service code. Open the project in statistic-service/src/ directory and in listener.ts replace old code by new ones:

Make sure to save all changes we did so far and if skaffold has stopped running, run skaffold dev command again in the project directory. Now proceed with final test:

Here take your cluster IP with

$ minikube ip

It's going to be part of the URL for the test.

Open your API client, do these post requests:

2 Post requests on http://192.168.49.2/dunk-contest/register with HEADERS Content-Type: application/json and each request for one body:

Body 1

{
  "name": "LeBron",
  "height": 2.06,
  "weight": 113,
  "experience": 19
}

Body 2:

{
  "name": "Stephen",
  "height": 1.88,
  "weight": 84,
  "experience": 13
}

Note: Just here, we registered two players for the contest,in the next requests let's have them attempt shot.

Post request on http://192.168.49.2/dunk-contest/attempt/LeBron
Post request on http://192.168.49.2/dunk-contest/attempt/Stephen

Note: With the two post requests, we made our two players attempting each one dunk shot. Feel free to make a lot of these two last requests to see how the Statistic Service produces the real-time contest statistic in its console.

5. Conclusion

Hopefully through this tutorial series, you have seen and understood how to stream data between services in a Kubernetes Cluster with NATS Streaming server. The all topic is complex, so there is a lot more to learn and to apply out there.

For your questions, comments, feel free to reach out here or on my Twitter; I will be more than happy to answer and Here, you may find the project code.

See you soon.

Data Streaming for Microservices using NATS Streaming - Part 1

Winner Musole Masu — Thu, 16 Dec 2021 21:21:43 +0000

The goal behind Data streaming is to process and analyze in real-time, data that move from data sources to destinations.

It is mostly used in microservices to ensure inter-services communication. In a microservice architecture, the recommendation is to build independent services that can be altered, updated or taken down without affecting the rest of the architecture.

In this tutorial, we are going to learn how to use NATS Streaming in a Kubernetes Cluster. NATS Streaming is a data streaming system powered by NATS.

We will build a Basketball Dunk Contest App with two services, a Dunk Service that will handle players registration and dunk shot attempts for registered players. And a Statistic Service which will be displaying the Dunk Contest statistic in real-time from data accompanying events messages. NATS Streaming here will be the events transporter between our two services.

Before we dive into the code, make sure you have the following in order to follow along with this tutorial:

Working knowledge of Node.js / typescript, Docker, Kubernetes Objects,
Node.js (preferably the latest LTS version), Docker How to install docker on Ubuntu? and local Kubernetes Cluster installed via Minikube, click HERE to install minikube,
Docker Hub account, click HERE to sign up,

I will be coding in a Linux machine for the tutorial.

1. Project Structure

Let's set up our project, we will first work on a number of kubernetes objects related to the project.
Run the following:

$ mkdir dunk-contest
$ cd dunk-contest/
$ mkdir kubernetes

These commands create the project directory dunk-contest/ then navigate inside the directory to create another directory named kubernetes/.

In kubernetes/ directory, we are going to add new files with required configurations to build the following Kubernetes objects:

Deployment objects for the NATS Streaming Server, for the Dunk Service MongoDB database and for the Statistic Service MongoDB database,
Service objects for Pods running containers of NATS Streaming image, Mongo image for Dunk Service and for Statistic Service.

1.1 Deployment and Service objects

Make sure to work this part of the tutorial in the kubernetes/ directory!

1.1.1 NATS Deployment and Service

Add a new YAML file named nats-deployment.yaml and put the configuration below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nats-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nats
  template:
    metadata:
      labels:
        app: nats
    spec:
      containers:
        - name: nats
          image: nats-streaming:0.22.1
          args:
            [
              "-p",
              "4222",
              "-hbi",
              "5s",
              "-hbt",
              "5s",
              "-hbf",
              "2",
              "-SD",
              "-cid",
              "dunk-contest",
            ]

This config file will create a Pod running a container of nats-streaming:0.22.1 docker image and a Deployment to monitor the Pod. Practically, this Pod will act as the project NATS Streaming Server exposing port 4222 to clients(Dunk Service and Statistic Service).

Add a new YAML file named nats-service.yaml and put the configuration below:

apiVersion: v1
kind: Service
metadata:
  name: nats-service
spec:
  selector:
    app: nats
  ports:
    - name: client
      protocol: TCP
      port: 4222
      targetPort: 4222

This config file will create a kubernetes object of kind Service, that other pods inside the kubernetes cluster will use to access the NATS streaming server Pod on port 4222.

1.1.2 MongoDB Deployment and Service

Here we are going to add 4 new config files:

1 dunk-mongo-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dunk-mongo-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dunk-mongo
  template:
    metadata:
      labels:
        app: dunk-mongo
    spec:
      containers:
        - name: dunk-mongo
          image: mongo

With these configurations, kubernetes will create a Deployment object to monitor a MongoDB Pod of mongo docker image. This database will be dedicated to the Dunk Service.

2 dunk-mongo-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: dunk-mongo-service
spec:
  selector:
    app: dunk-mongo
  ports:
    - name: db
      protocol: TCP
      port: 27017
      targetPort: 27017

Another config file that will create a kubernetes object of kind Service that will permit other pods in the cluster to access the mongo pod of the Dunk Service.

3 stats-mongo-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: stats-mongo-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: stats-mongo
  template:
    metadata:
      labels:
        app: stats-mongo
    spec:
      containers:
        - name: stats-mongo
          image: mongo

Kubernetes will use this set of configurations to create a Deployment object to manage the MongoDB Pod of mongo docker image. This mongo database will be dedicated to the Statistic Service only.

4 stats-mongo-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: stats-mongo-service
spec:
  selector:
    app: stats-mongo
  ports:
    - name: db
      protocol: TCP
      port: 27017
      targetPort: 27017

Finally we have this config file to create a Service object that will expose the MongoDB Pod of the Statistic Service to other pods in the kubernetes cluster.

Your kubernetes/ directory tree structure should look like this by now:

.
├── dunk-mongo-deployment.yaml
├── dunk-mongo-service.yaml
├── nats-deployment.yaml
├── nats-service.yaml
├── stats-mongo-deployment.yaml
└── stats-mongo-service.yaml

0 directories, 6 files

Save all created files in the kubernetes/ directory and make sure that your Kubernetes cluster is up and running. Open the terminal and run the following:

$ minikube start

Minikube quickly sets up a local Kubernetes cluster on macOS, Linux, and Windows.

Now, let's tell Kubernetes to create objects using our configuration files. Run the command below in the kubernetes/ directory:

$ kubectl apply -f .

At this moment, we must have 3 running pods, one for nats-streaming, for mongodb of the Dunk Service and one for mongodb of the Statistic Service. Verify it with this command:

$ kubectl get pods

We reached the point in the tutorial where we are going to build our two services and connect them to the Pods created in this part of the tutorial. Let's do it in the next points.

1.2 Dunk Service

Here we are going to build an express application, listening on port 4001 for connections. It will have two API endpoints, one http://localhost:4001/dunk-contest/register to handle POST request for players registration and the other http://localhost:4001/dunk-contest/attempt/:playerName to handle POST request for players dunk shots attempts.

Navigate back to the project directory dunk-contest/ and create a new directory named dunk-service/. In the dunk-service/ directory, generate a package.json and install ts-node-dev, typescript, express, @types/express, node-nats-streaming and mongodb as dependencies:

$ cd dunk-service/
$ npm init -y
$ npm install ts-node-dev typescript express @types/express node-nats-streaming mongodb

Open the package.json file, replace the actual script section by the one below:

 "scripts": {
    "start": "ts-node-dev src/index.ts"
  }

Save the file. In the same directory create a directory named src/ ,in src/ add a typescript file named nats-connector.ts and paste the following:

import nats, { Stan } from "node-nats-streaming";

class NatsConnector {
  private _client?: Stan;

  get client() {
    if (!this._client) {
      throw new Error("Cannot access NATS Client before connecting!");
    }
    return this._client;
  }

  connectToNats(clusterId: string, clientId: string, url: string) {
    this._client = nats.connect(clusterId, clientId, { url });

    return new Promise<void>((resolve, reject) => {
      this.client.on("connect", () => {
        console.log(`DUNK SERVICE IS CONNECTED TO NATS STREAMING SERVER`);
        resolve();
      });
      this.client.on("error", (err) => {
        reject(err);
      });
    });
  }
}

export const natsConnector = new NatsConnector();

Inside this file:

We define a variable _client of Stan type, a type imported from node-nats-streaming library,
We export an instance of NatsConnector class that has a method called connectToNats() - connectToNats() takes three parameters, the clusterId, the clientId and the url:

clusterId: This was set early in the NATS streaming server deployment configuration file. Dunk Service, here being a client will use it to connect to the NATS server,
clientId: An identifier for the Dunk Service as client to the NATS server,
url: The NATS Streaming server endpoint, that the Dunk Service will use to access resources in the NATS running pod.

In connectToNats(), to _client we assign a function imported from node-nats-streaming called connect() on which we passe our three parameters as arguments.
And connectToNats() returns a promise that resolve if _client get successfully connected to NATS server and reject if otherwise.

Next, add another typescript file named event-publisher.ts and put the following:

import { Stan } from "node-nats-streaming";

export class EventPublisher {
  private client: Stan;

  constructor(client: Stan) {
    this.client = client;
  }

  publishEvent(subject: string, data: any): Promise<void> {
    return new Promise((resolve, reject) => {
      this.client.publish(subject, JSON.stringify(data), (err) => {
        if (err) {
          return reject(err);
        }
        console.log("\x1b[36m%s\x1b[0m", `EVENT ${subject} PUBLISHED!`);
        resolve();
      });
    });
  }
}

In this one, we export the class EventPublisher that has a variable named client of type Stan just like in the NatsConnetor class. We have a method in this class called publishEvent() of two parameters:

subject: This is the name of the channel which events passe through and reach clients that had subscribed to,
data: the data or message accompanying published events.

And publishEvent() returns a promise that resolve when events are successfully published and reject when there is a failure.

After this, in src/ directory, create a directory named routes/; add two new typescript files:

1 registerPlayerRoutes.ts a middleware and put the code below:

import { Router, Request, Response } from "express";
import { MongoClient } from "mongodb";

interface Players {
  NAME: string;
  HEIGHT: number;
  WEIGHT: number;
  EXPERIENCE: number;
}

const registerPlayerRouter = Router();
registerPlayerRouter.post(
  "/dunk-contest/register",
  async (req: Request, res: Response) => {
    const player: Players = {
      NAME: req.body.name,
      HEIGHT: req.body.height,
      WEIGHT: req.body.weight,
      EXPERIENCE: req.body.experience,
    };

    const mongoClient = await MongoClient.connect(
      "mongodb://localhost:27017/dunk-service"
    );

    const db = mongoClient.db();
    const playerCollection = db.collection("players");
    await playerCollection.insertOne(player);

    console.log("\x1b[36m%s\x1b[0m", "PLAYER REGISTERED WITH SUCCESS");
    const newPlayer = await playerCollection.findOne({
      NAME: req.body.name,
    });
    console.table(newPlayer);
    res.send({});

    mongoClient.close();
  }
);

export { registerPlayerRouter };

In the registerPlayerRoutes.ts file above we did the following:

Imported Router, Request and Response from express;
Imported MongoClient from mongodb;
Implemented a POST request on registerPlayerRouter.post("/dunk-contest/register") to register players to the players collection in dunk-service MongoDB database and fetch the registered player. MongoClient is used here to connect this process to the appropriate MongoDB Pod.
2 attemptDunkRoutes.ts and put the code below:

import { Router, Request, Response } from "express";
import { MongoClient } from "mongodb";
import { natsConnector } from "./../nats-connector";
import { EventPublisher } from "./../event-publisher";

const attemptDunkRouter = Router();
attemptDunkRouter.post(
  "/dunk-contest/attempt/:playerName",
  async (req: Request, res: Response) => {
    const mongoClient = await MongoClient.connect(
      "mongodb://localhost:27017/dunk-service"
    );

    const db = mongoClient.db();
    const playerCollection = db.collection("players");

    const playerFound = await playerCollection.findOne({
      NAME: req.params.playerName,
    });

    const dunkPoint: number =
      (playerFound?.HEIGHT *
        playerFound?.WEIGHT *
        playerFound?.EXPERIENCE *
        Math.random()) /
      100;

    await new EventPublisher(natsConnector.client).publishEvent("Dunk-Shot", {
      PLAYER_NAME: playerFound?.NAME,
      DUNK_POINT: dunkPoint,
    });
    res.send({});

    mongoClient.close();
  }
);

export { attemptDunkRouter };

With attemptDunkRoutes.ts we worked on a middleware, we did the following:

Imported Router, Request and Response from express;
Imported MongoClient from mongodb;
Imported natsConnector, an NatsConnector instance;
Imported the class EventPublisher;
Implemented a POST request on attemptDunkRouter.post( "/dunk-contest/attempt/:playerName") to attempt a dunk shots for a player found in the players collection by player's name got with req.params.playerName;
MongoClient is used here to connect this process to the appropriate MongoDB pod;
With EventPlubilsher class, we created a new instance that passes natsconnector.client as argument and calls publishEvent function to publish an event through the Dunk-Shot channel with PLAYER_NAME and DUNK_POINT as event message;
DUNK_POINT is number calculated with the player's HEIGHT, WEIGHT, EXPERIENCE and a random number.

To wrap with service up, go ahead, move back to src/ directory, add a typescript file named index.ts and paste the code below:

import express from "express";
import { registerPlayerRouter } from "./routes/registerPlayerRoutes";
import { attemptDunkRouter } from "./routes/attemptDunkRoutes";
import { natsConnector } from "./nats-connector";

const app = express();
app.use(express.json());

app.use(registerPlayerRouter);
app.use(attemptDunkRouter);

const start = async () => {
  try {
    await natsConnector.connectToNats(
      "dunk-contest",
      "123",
      "http://localhost:4222"
    );

    natsConnector.client.on("close", () => {
      process.exit();
    });
  } catch (error) {
    console.error(error);
  }
  app.listen(4001, () => {
    console.log("\x1b[36m%s\x1b[0m", "DUNK SERVICE LISTENING ON 4001");
  });
};

start();

In the index.ts file above we did the following:

Imported express, Request and Response from express;
Imported registerPlayerRouter and attemptDunkRouter, two middlewares;
Imported natsConnector, an instance of class NatsConnector that was created early;
Called the express function express() and puts new Express application inside the app variable (to start a new Express application);
Used the middlewares with app.use(registerPlayerRouter) and app.use(attemptDunkRouter);
Wrote the start function to connect the Express application to NATS Streaming server and have it listens for connection on port 4001.

Now generate a tsconfig.json file to compile your TypeScript code in JavaScript code. Open your terminal, navigate back to dunk-service/ directory and run the command below:

$ tsc --init

Great, we are almost done with the Dunk Service, we shall come back later to fix some little things.

The dunk-service/ directory should look like the tree below:

.
├── package.json
├── package-lock.json
├── src
│   ├── event-publisher.ts
│   ├── index.ts
│   ├── nats-connector.ts
│   └── routes
│       ├── attemptDunkRoutes.ts
│       └── registerPlayerRoutes.ts
└── tsconfig.json

2 directories, 8 files

Let's perform a simple test to check the following:

Dunk Service connection to its dedicated MongoDB running pod;
Dunk Service connection to the Nats Streaming Server;

In the steps below, do the test:
Step - 1: Access to NATS Streaming server Pod

Take your pod's name by running:

$ kubectl get pods

Copy your pod's name, you will use it in the command that is coming.

$ kubectl port-forward <YOUR POD NAME> 4222:4222

Make sure to paste your pod's name where indicated in the command above.

Step - 2: Access to MongoDB pod dedicated to the Dunk Service
Take your pod's name by running:

$ kubectl get pods

Copy your pod's name, you will use it in the command that is coming.
Here we are going to make the MongoDB pod of Dunk Service running in the kubernetes cluster accessible in our local machine. Open another terminal and forward a local port on your machine to a port on your pod by running the following:

$ kubectl port-forward <YOUR POD NAME> 27017:27017

Step - 3: Start the Dunk Service (Express application)

Open a third terminal in the dunk-service/ directory and run this command:

$ npm start

By now, the Dunk Service must be connected to the NATS Streaming server pod and to its MongoDB pod.

Step - 4: Open your API Client and do these tests

Test - 1. POST request on http://localhost:4001/dunk-contest/register

Make a POST request, with HEADERS Content-Type: application/json and a BODY of :

{
  "name": "LeBron",
  "height": 2.18,
  "weight": 105,
  "experience": 5
}

Test - 2. POST request on http://localhost:4001/dunk-contest/attempt/LeBron

Hopefully, you will have a similar output in your terminals as below:

2. Conclusion

In this part of the tutorial, we started to build our Dunk Contest Application with the purpose of learning how to use NATS Streaming in a microservice architecture in a Kubernetes Cluster set and running in our local machines.

The completed Application should have featured two services, the Dunk Service and Statistic Service streaming data using NATS Streaming.

In the process, we started a NATS Streaming Server running in the kubernetes cluster and two MongoDB Pods, each dedicated to a specific service. We also started the Dunk Service, which successfully registered players in its MongoDB Pod and successfully published an event to the NATS Streaming Server; event consisting of a dunk shot of a registered player.

Please stay tuned for more important NATS Streaming practices that we are going to implement in part 2 of this tutorial as we will be completing our Dunk Contest Application.

If you have questions, comments, feel free to reach out here or on my Twitter; I will be more than happy to answer and Here, you may find the project code.

See you soon.

Deploy a MySQL database server in Kubernetes - Static

Winner Musole Masu — Tue, 23 Nov 2021 20:39:47 +0000

In this tutorial, you are going to learn how to deploy a MySQL database server in a Kubernetes Cluster set up in a local machine, this is one of many ways to persist data in Kubernetes.

Kubernetes is a tool for automating deployment, scaling, and management of containerized applications.

Get familiar with some terminologies and kubernetes objects that will be used through this tutorial:

Docker Image: A collection of files that packs together all the necessities needed to set up a completely functional container,
Container: An instance of an image, a running image,
Node: A Kubernetes Object, a virtual machine that runs a container and provide resources,
Kubernetes Cluster: A collection of nodes and configurations to manage them,
Pod: A Kubernetes object, a running container, the smallest deployable units of computing that can be created and managed in kubernetes,
Deployment: A Kubernetes Object, that monitors set of pods, it make sure that those pods are running and make sure to restart pods if they are down,
Service: A Kubernetes Object that provides a way to access a running container(pod),
Persistent Volume: A Kubernete object, is a piece of storage in the cluster,
Persistent Volume Claim: A request for the Persistent Volume storage,
Kubernetes Config file: A file that tells Kubernetes about the different Objects to be created. It's written in YAML syntax.

Technically, you will create a Deployment that will manage a Pod running a container of a MySQL docker image, then you will create a Service that will permit access to the pod. This pod will request for storage (using Persistent Volume Claim) to a storage resource (Persistent Volume).

A Persistent Volume can be created statically or dynamically. In the next phase of this tutorial, you learn how to do it statically.

For this tutorial to go smooth, you will need to install:

Docker in your machine, click HERE to install;
Local Kubernetes Cluster via Minikube, click HERE to install minikube,
Optionally, you can create a Docker Hub account HERE

1. Build a Persistent Volume (PV)

First, create a working directory and navigate in:



$ mkdir mysql-kube
$ cd mysql-kube/

Create a yaml file named mysql-pv.yaml, put in the following:



apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-pv
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/data"

Save and close the file.

This yaml file once applied in kubernetes, will provision a Persistent Volume, for the MySQL database server Pod. The persistent volume will not depend on the pod's lifecycle. This means that anytime the pod restarts due to a crash or a malfunction, the provisioned storage will survive.

2. Build a Persistent Volume Claim (PVC)

In the working directory mysql-kube/, create a file named mysql-pvc.yaml, put the following:



apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Up there, you created the file that will provision a storage when applied, this file on the other hand will create a Persistent Volume Claim that will be used by the MySQL Pod to request for that provisioned storage.

3. MySQL pod's deployment

Here you are going to create a file named mysql-deployment.yaml in the same directory, mysql-kube/. Create the file and put the code below:



apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - image: mysql:8.0
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: password
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-pv-claim

This file will create a deployment object to manage a Pod running a container of MySQL docker image and in its specifications, there is a reference to the Persistent Volume Claim that the pod will use to request for the Persistent Volume.

Before applying this deployment file, create a service object that will permit other pods to access the MySQL database pod that will be created.
Still in the mysql-kube/ directory, create a yaml file named mysql-service.yaml and put the code below:



apiVersion: v1
kind: Service
metadata:
  name: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql
  clusterIP: None

Save the file, make sure that your Kubernetes cluster is up and running. Open the terminal and navigate to mysql-kube/ run the following:



$ minikube start

Minikube quickly sets up a local Kubernetes cluster on macOS, Linux, and Windows.
Now, let's tell Kubernetes that we want to use all config files created. Run these commands in the mysql-kube/ in the sequential order below:

Step 1: Create the Persistent Volume ```

$ kubectl apply -f mysql-pv.yaml

![Persistent Volume](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sqp416n0ee3cdt0td23g.png)

- **Step 2**: Create the Persistent Volume Claim

$ kubectl apply -f mysql-pvc.yaml

![Persistent Volume Claim](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5rz3f3st95fjpczbwo00.png)

- **Step 3**: Create the Deployment

$ kubectl apply -f mysql-deployment.yaml

![Deployment](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ysr6h2ix2jvwaw82crf6.png)

- **Step 4**: Create the Service

$ kubectl apply -f mysql-service.yaml

![Service](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bhwaz0nlhkmehyc6y1td.png)

This sequence of commands created a `Persistent Volume`, a `Persistent Volume Claim`, a `Deployment` that manages a `Pod` running a `container` of a mysql docker image and a `Service` that permits access to that Pod.

Check if your kubernetes objects were successfully created with:
- **Deployment**

$ kubectl get deployments

![Kubectl Depl](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yi76xgwwusbp8rn42ml6.png)

- **Pod**

$ kubectl get pods

![get-pods](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tmjxwauwbr7mzfdd4c4m.png)
 - **Service**

$ kubectl get services

![Pod service](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wybzhaspafrxoxu8upzj.png)

You did great so far, now run a test to create a Pod running a MySQL container that connects to the MySQL database server Pod as a client;

$ kubectl run -it --rm --image=mysql:8.0 --restart=Never mysql-client -- mysql -h mysql -password="password"

This command runs the MySQL container in an interactive mode, which allows you to execute commands at the time of running the container.
A MySQL  shell will open and you could create new databases, new tables, insert data to tables and do more SQL commands.

![sql SHELL](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rhd5rizi2uyd24prbcgh.png)


##4. Conclusion
With this, you learnt through kubernetes Objects how to deploy a MySQL database server in a Kubernetes Cluster using a static method of provisioning storage.

Also you tested how to connect a client to that deployed server, by executing SQL commands when running the container in interactive mode. 

If you have questions, comments, please feel free to reach out here or on my [Twitter](https://twitter.com/MusoleMasu); I will be more than happy to answer.

See you soon..

Let’s build and deploy a Full Stack Next.js App: Part 3

Winner Musole Masu — Mon, 15 Nov 2021 10:32:04 +0000

Here is the last part of this tutorial series. In this part we are going to fetch data from the Mongo DB and to deploy the application on Vercel.

In case you missed part 2 of this tutorial series, click HERE to catch up.

1. Let's fetch from Mongo

Note: Now, You might have a different IP address than what you put on the Network Access section in your mongo cluster. So make sure to update.

Previously, we successfully connected our App to a Mongo DB database and successfully stored data in. Open the terminal, in the project directory run $ npm run dev and let's proceed with the fetch.

1.1 Fetch at build time

Next.js provides two methods of pre-rendering. You can preload elements on pages by exporting the getStaticProps or the getServerSideProps functions on your pages. For this tutorial we will go with getStaticProps, since we will not make a lot of changes with the data and also we will not have too many requests.

This should be your project directory. On "/pages", open the "index.js" file and replace the existing code with the code below:

import MealList from "../components/MealList";
import { MongoClient } from "mongodb";
const HomePage = (props) => {
  return <MealList meals={props.mealList} />;
};
export async function getStaticProps() {
  const DATABASE_NAME = "FoodApp";
  const DATABASE_PASSWORD = "1234567H";

  const client = await MongoClient.connect(
    `mongodb+srv://masu:${DATABASE_PASSWORD}@foodapp.0tpop.mongodb.net/${DATABASE_NAME}?retryWrites=true&w=majority`
  );
  const db = client.db();
  const mealsCollection = db.collection("meals");
  const meals = await mealsCollection.find().toArray();

  client.close();

  return {
    props: {
      mealList: meals.map((meal) => ({
        id: meal._id.toString(),
        name: meal.name,
        image: meal.image_path,
        dish: meal.dishes,
        chef: meal.chef,
      })),
    },
  };
}
export default HomePage;

Warning: Replace const DATABASE_NAME and const DATABASE_PASSWORD with the value you put early in part 2 of the tutorial.

In the code above, we did the following:

Imported MongoClient,
Imported MealList react component,
Created a function component "HomePage" that returns the MealList component imported earlier,
Exported async getStaticProps function, where we connected the app to the Mongo DB database using the MongoClient and fetched meals from the database.
async getStaticProps function returns a props which the fetched meals values are assigned to, then passed to the "HomePage" component as a props,
The "HomePage" function component passes its props to the "MealList" component.

1.2 Let's make dynamic components

We are still displaying a static page, let's use the data passed with the props in the " MealList" component; to have a dynamic page.
In the "/components" directory, open the "MealList.js" file and replace all with the below:

import MealItem from "./MealItem";
const MealList = (props) => {
  return (
    <div className="flex flex-wrap">
      {props.meals.map((meal) => (
        <MealItem meal={meal} key={meal.id} />
      ))}
    </div>
  );
};
export default MealList;

Still in the same directory, open the "MealItem.js" file where you put these:

const MealItem = ({ meal }) => {
  return (
    <div className="w-1/4 p-8 shadow-lg rounded-lg bg-yellow-50">
      <img src={meal.image} className="w-auto h-auto" alt="Chicken Salad" />
      <div className="text-center py-2">
        <h3 className="text-xl font-normal">
          {meal.name}
          <span className="px-3 font-light text-yellow-500">
            ({meal.dish} dishes)
          </span>
        </h3>
        <p className="text-gray-500 text-base">{meal.chef}</p>
        <button className="bg-yellow-500 px-4 py-2 rounded-lg text-gray-50 font-medium mt-2">
          Details
        </button>
      </div>
    </div>
  );
};
export default MealItem;

By this moment, if your command $ npm run dev is still running, the home page will be served with data from the database; reason why, you will notice a heavy loading.

2. Let's deploy on Vercel

Vercel is a platform for front-end frameworks and static sites, built to integrate with your headless content, commerce, or database.

It provides a friction-less developer experience to take care of the hard things: deploying instantly, scaling automatically, and serving personalized content around the globe.

- Step 1: Create a git repository, we created ours with GitHub and named it "next-food-app", and push your project code in there.

With Vercel, you don't need to run the Build and Start commands on your own. Just import your project repository and Vercel takes care of all during the deployment.

- Step 2: Use your GitHub Account to sign up with Vercel, HERE

Click on "Continue with GitHub" button:

After a successful sign up, you will be redirected here:

- Step 3: Click on the "New Project" button
When you click on "New Project", Vercel will directly link all your GitHub repositories to the page you are redirected to.

Import the repository you have just created, for us it will be the "next-food-app" repository,

Click on Skip in the "Create a Team" section,
In the "Configure Project" section click on Deploy and wait for deployment.

During the deployment, Vercel runs $ npm run build and $ npm start in the project directory for the imported repository. And at every $ git push command, Vercel redeploys your project.

Congratulations! Now your the project is deployed, navigate to https://next-food-app.vercel.app/

3. Conclusion

Here we are at the end of this tutorial series. I hope that you like and use it and that you will go further with Next.js.

Feel free to reach out for questions on my Twitter.

Find the completed project HERE.

À bientôt.

Let’s build and deploy a Full Stack Next.js App: Part 2

Winner Musole Masu — Mon, 01 Nov 2021 16:09:23 +0000

In this part, we are going to connect our Next.js app to a remote MongoDB database.

In case you missed part 1 of this tutorial series, click HERE to catch up.

As called in the title, we are going to add more features to the food Next.js App, to make it FullStack; it will be blending client-side and server-side. This one app will store and fetch data straight to and from the server and render finished pages.

1. Let’s connect to Mongo

First, run $ npm run dev in the terminal on the project directory. Have a look at the Home page on http://localhost:3000;

It displays a card of a meal item that contains the image of a delicious meal, the name of the meal, the number of cooked dishes and the name of the chef. At the moment, the information inside the meal item is hard coded and static. Let’s go ahead and store it in a real database.

The project directory should look like the directory below:

1.1 Build the back-end Api:

Here, we are going to build a back-end API together with the react front-end in this same project. For this, we will use special pages called “API Routes”.

Go ahead, in the “/pages” directory, create a directory named “/api”, in “/api” add a new file called “new-meal.js”.

Note: “any file added in “/pages/api” directory will be treated as an API endpoint and will be mapped as “/api/the_name_of_the_file.js”

For this project case, the API endpoint is => “/api/new-meal”.

1.2 Before adding any code in the "new-meal.js" file, the API endpoint; if you don’t have a MongoDB account, create one HERE.

1.3 After a successful login/sign up, you will be redirected here:

You are asked to create a new organization, so click on the green button “Create an Organization” and follow these steps:

Click on "Create an Organization",
Next, Name your organization,
Click on the "Next" button at the bottom,
And Click on "Create Organization" to give yourself access permissions.

Great, now that you created an organization, go ahead, create a new project;

1.4 Click on the green button “New Project” at the top-right, as shown in the page above:

Then follow these steps:

Click on "New Project",
Next, Name your project,
Click on the "Next" button,
Final, click on "Create Project" to give yourself access permissions as a Project Member.

1.5 Good job, now let’s build this project database:

Click on the green button “Build a Database” below:

Follow these steps to build the database:

After the click on "Buid a Database", choose the Free and Hobby deployment option,
Click on "Create",
On the next page, scroll-down, give a name to your Cluster and click on "Create Cluster",
Wait about 3 minutes for the deployment.

This is what waiting for 3 minutes deployment looks like:

So be patient ...

Hope that this process was not long. Now, let’s connect the application to the newly created Database. You will need to add a database user and add your computer IP address, to let it have access to your cluster.

1.6 Add a Database User:

In the SECURITY section, click on "Database Access" and follow the steps below:

Click on "Add New Database User",
Insert the database username and the password, better write it down somewhere,
And at the bottom,click on "Add User".

1.7 Add your computer IP address:

Still in the SECURITY section, click on "Network Access" and follow the steps:

Click on "Add IP Address",
Add your IP address by clicking on "ADD CURRENT IP ADDRESS" button in modal box,
And click on "Confirm".

Super guys, now let's go back to the API endpoint, to connect the App to mongo.

1.8 Install mongodb in the project:

Open the terminal, on the project directory put this:

$ npm install  mongodb

1.9 Import mongodb in the API endpoint js file and connect to mongo:

But before that, let's go to grab the Application Connection String that will connect the app to the cluster created earlier.
Go back to MongoDB, in DEPLOYMENT section:

Click on "databases",
Then click on "Connect",
Choose the second connection method labelled "Connect your application",
Copy the application connection string, that will be pasted into the project code.

Back to the project, open "new-meal.js" in "/pages/api/" directory and put the code below it, but replace the value of const DATABASE_PASSWORD by the password you put while adding the database user in sub-point 1.6:

import { MongoClient } from "mongodb";

const handler = async (req, res) => {
  const DATABASE_NAME = "FoodApp";
  const DATABASE_PASSWORD = "1234567H";

  if (req.method === "POST") {
    const client = await MongoClient.connect(
      `mongodb+srv://masu:${DATABASE_PASSWORD}@foodapp.0tpop.mongodb.net/${DATABASE_NAME}?retryWrites=true&w=majority`
    );
    const db = client.db();
    const mealsCollection = db.collection("meals");
    await mealsCollection.insertOne(req.body);

    client.close();

    res.status(201).send({ Message: "Meal inserted" });
  }
};

export default handler;

In code above, There is some modification with the application connection string that you copied early. So just paste and modify it as it's done in the GIF image below:

Basically, the application connection string is pasted in the file where, const DATABASE_NAME replaced "myFirstDatabase" and const DATABASE_PASSWORD replaced "<password>" in the string connection.

Again Good job, now the API endpoint is ready to be used.

1.10 Add a component page named “new-meal.js” in the “/pages” directory:

This is the form that we will use to add new meal. Put the code below in the page :

import { useRef } from "react";
import { useRouter } from "next/router";
const NewMealForm = () => {
  // use of useRef to capture input value
  const mealNameInputRef = useRef();
  const mealImagePathInputRef = useRef();
  const mealNumberOfDishInputRef = useRef();
  const chefInputRef = useRef();

  // use of useRouter from next/router to redirect this page to the Homepage
  const router = useRouter();

  // implementation of newMealHandler function
  const newMealHandler = async (event) => {
    event.preventDefault();

    // store meal data in an object
    const mealData = {
      name: mealNameInputRef.current.value,
      image_path: mealImagePathInputRef.current.value,
      dishes: mealNumberOfDishInputRef.current.value,
      chef: chefInputRef.current.value,
    };

    // use of Fetch API to make a request to the new-meal api and get back a response
    const response = await fetch("/api/new-meal", {
      method: "POST",
      body: JSON.stringify(mealData),
      headers: {
        "content-Type": "application/json",
      },
    });

    // parses JSON response into native JavaScript objects
    const data = await response.json();

    console.log(data);

    // redirects this page to the Homepage
    router.replace("/");
  };

  const INPUT_STYLE =
    "my-2 p-2 border border-gray-300 focus:ring-2 focus:ring-indigo-300 focus:ring-opacity-50 focus:outline-none w-full h-10 rounded-md";

  return (
    <div className="flex-col px-12 py-12 max-w-3xl mx-auto shadow-xl rounded-2xl">
      <h1 className="font-light text-4xl">Add a new Meal </h1>
      <form onSubmit={newMealHandler}>
        <input
          type="text"
          placeholder="Meal Name"
          required
          ref={mealNameInputRef}
          className={INPUT_STYLE}
        />
        <input
          type="text"
          placeholder="Image Path"
          required
          ref={mealImagePathInputRef}
          className={INPUT_STYLE}
        />
        <input
          type="number"
          max="5"
          min="1"
          placeholder="Number of Dishes"
          required
          ref={mealNumberOfDishInputRef}
          className={INPUT_STYLE}
        />
        <input
          type="text"
          placeholder="Name of Chef"
          required
          ref={chefInputRef}
          className={INPUT_STYLE}
        />
        <button
          type="submit"
          className="bg-yellow-500 text-gray-800 font-medium text-xl inline-flex  w-full items-center px-4 py-4 rounded-xl"
        >
          Add Now
        </button>
      </form>
    </div>
  );
};

export default NewMealForm;

Save it, open the browser on http://localhost:3000/new-meal and add a new meal.

Big achievement here with the project, we can store data to a MongoDB database :)

2. Conclusion

The Part 2 of our tutorial ends here. We basically worked on MongoDB database connection to our Food App, using the API Routes feature that comes with Next.js.

Warning: While practicing this tutorial, you might get this error => error - MongoNetworkError: connection 1 to 54.145.177.180:27017 closed. To avoid it, make sure that your computer IP address always match with the one you put on the MongoDB cluster in sub point 1.7

The final Part, part 3 will be published very soon, so again stay tuned and follow me on Twitter for updates.

Find the completed project here.

À bientôt

Let’s build and deploy a Full Stack Next.js App: Part 1

Winner Musole Masu — Mon, 25 Oct 2021 10:31:46 +0000

Next.js is referred to as the React framework for production, which comes with more features than simple React as a library. It gives us rules and guidelines on how we should structure our projects and it improves React developers' life, with built-in features like file-system based routing or “head” metadata for SEO.

In this article, we will build a simple food app that displays menu. It will be styled with Tailwind Css in the first part, and in the coming parts, we will connect the app to a MongoDB database and deploy it on Vercel.

Prerequisites

For the tutorial to go well, you must have basic knowledge of Node js and React and have either a GitHub / Bitbucket / Gitlab account. You need also to have at least Node js 14.18.1 LTS and Git installed on your computers.

1. Let's start a new project

All the prerequisites are met, open your terminal and create a new project;

Run the three commands below, to create a new project directory and set up a new "package.json" file :

$ mkdir next-food-app
$ cd next-food-app
$ npm init -y

Next step, in the project directory, install react and react-dom as dependencies, Next.js requires them as peer dependencies:

$ npm install react react-dom

Free from npm warnings, you can now install Next.js, run:

$ npm install next

Open the "package.json" file, in the scripts section, replace the actual code by the one below :

 "scripts": {
   "dev": "next",
   "build": "next build",
   "start": "next start"
 },

Back to the project directory, create a folder named “/pages” and inside “/pages” create a file named “index.js” :

Note: Next.js comes with the file-system based routing feature, which makes all files added to the “/pages” directory available as routes. All files except “index.js”, which the Router automatically routes to the root of the directory.

/pages/index.js => /
/pages/meals/index.js => /meals
/pages/meals.js => /meals

Great, in the "/pages/index.js" create the application HomePage with the following code :

const HomePage = () => {
  return <h1>Welcome to our Food Next.js App &#x1F60a;</h1>;
};
export default HomePage;

Now, go to your the terminal, in the project directory, run the command below and on your browser open http://localhost:3000 :

$ npm run dev

You will get the same page on your browser if nothing was omitted.

2. Let's add Tailwind css

First step here is to install tailwindcss in the project, you will do so by running the following command in the project directory:

$ npm install tailwindcss postcss autoprefixer

Next, generate “tailwind.config.js” and “postcss.config.js” files with this command in the project directory:

$ npx tailwindcss init -p

Open the “tailwind.config.js” file and replace all code in there by these:

module.exports = {
 purge: ["./pages/**/*.{js,ts,jsx,tsx}", "./components/**/*.{js,ts,jsx,tsx}"],
 darkMode: false, // or 'media' or 'class'
 theme: {
   extend: {},
 },
 variants: {
   extend: {},
 },
 plugins: [],
};

In the project directory, create a folder named "/public", inside "/public", create another folder named "/style" where you will add a file named "global.css" and put the code below in it:

@tailwind base;
@tailwind components;
@tailwind utilities;

After, import your style in your Javascript: To do so, create inside the “/pages” directory, a file named “_app.js”.

Note: In Next.js the file "_app.js" in the "/pages" directory overrides the default “App” component that allows us to add global css to the whole application.

import "../public/style/global.css";

const MyApp = ({ Component, pageProps }) => {
 return <Component {...pageProps} />;
};
export default MyApp;

Put the code above in your "_app.js" file, run $ npm run dev and open your browser on http://localhost:3000 .
If the procedure was followed correctly, you will notice a difference with the text font between the project before and after tailwind css. Just like in the images below;

Before

After

3. Finally, let’s add React components

Note: Next.js is built on top on react, therefore react components are part of it.

In the project directory, create a folder named "/components", and add two js files respectively named “MealList.js” and “MealItem.js”.
Drop the code below in “MealItem.js”:

const MealItem = () => {
  return (
    <div className="w-1/4 p-8 shadow-lg rounded-lg bg-yellow-50">
      <img src="/img/meal1.PNG" className="w-auto h-auto" alt="Chicken Salad" />
      <div className="text-center py-2">
        <h3 className="text-xl font-normal">
          Chicken Salad
          <span className="px-3 font-light text-yellow-500">(4 dishes)</span>
        </h3>
        <p className="text-gray-500 text-base">Herico de Porto</p>
        <button className="bg-yellow-500 px-4 py-2 rounded-lg text-gray-50 font-medium mt-2">
          Details
        </button>
      </div>
    </div>
  );
};
export default MealItem;

And drop this in “MealList.js”:

import MealItem from "./MealItem";
const MealList = () => {
  return (
    <div className="flex flex-wrap">
      <MealItem />
    </div>
  );
};
export default MealList;

Import your react components in the Homepage “/pages/index.js” : Delete all code in there and put the code below:

import MealList from "../components/MealList";
const HomePage = () => {
  return <MealList />;
};
export default HomePage;

Download the image found on this link -> https://github.com/musole-masu/nextjs-food-app/blob/main/public/img/meal1.PNG
Create in the "/public" directory a folder named "/img" and put the downloaded image in there.
Now, wrap your pages with the style added in the "_app.js" just like it's done below:

import "../public/style/global.css";

const MyApp = ({ Component, pageProps }) => {
  return (
    <div className="max-w-7xl mx-auto py-40">
      <Component {...pageProps} />
    </div>
  );
};
export default MyApp;

In your terminal, in the project directory, run $ npm run dev, then go to http://localhost:300

If you have this exact page on the browser, it means that you have completed each step of this tutorial correctly. Congrats :)

4. Conclusion

The Part 1 of our tutorial ends here. To summarize, it was focused on the project creation, with the installation of dependencies and also focused on styling.
Part 2 will be published very soon, so stay tuned and follow me on Twitter for updates.

Find the completed project here.

À bientôt