Forem: murugan

Kubernets Helm Chart

murugan — Tue, 13 Feb 2024 04:50:08 +0000

Definition:

Helm is a package manager for Kubernetes. Similar to yum but for Kubernetes. It bundles all related manifests(such as deployment, service, etc) into a chart. When installing chart, helm creates a release. Benefits of helm is it provide templating, repeatability, reliability, multiple environment and ease of collaboration.

Helm uses a packaging format called Charts. A Helm Chart in Kubernetes is a collection of files that describe a set of Kubernetes resources. The Helm Charts can be sent to a Helm Chart Repository. The details specified in the Helm Chart are used to enable a more consistent Kubernetes deployment.

Components of a Helm Chart!

Chart: A Helm chart is a collection of files that define a specific application deployment on Kubernetes. It includes templates, which are parameterized YAML files that generate Kubernetes manifests. Charts also contain a Chart.yaml file that describes the chart, and a values.yaml file that holds configurable parameters.
Release: It is a specific instance of a chart which has been deployed to the Kubernetes cluster using Helm.
Repository:Helm repositories are collections of charts hosted on a server or a remote location. Charts can be published to a public repository like the official Helm Hub or to private repositories.
Helm Client: The Helm client is the command-line interface (CLI) tool used to interact with Helm. It allows you to create, package, and manage charts, as well as install and upgrade releases

Installing Helm

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 |bash

Type	#Verify helm version	#To check the current state of helm locally	#helm use the Kubernetes cluster/host via the config file
Comments	helm version	helm env	~/.kube/config

Advantage of using Helm

Traditional deployments in Kubernetes is done with Kubectl across files into separately managed items. Helm deploys units called charts as managed releases.
We can search for charts https://helm.sh/ . Charts can be pulled(downloaded) and optionally unpacked(untar).

Chart Repo:

#To show all the repos	#To add a repo	#Search a repo	#Search for hub
helm repo list	helm repo add url	helm search repo mysql	helm search hub

Creating your first chart

The easiest way to get started with chart is by using helm create command. The below command create a new chart named firsthelmchart

helm create myhelmchart

If you see the firsthelmchart directory you will see the structure like below

firsthelmchart
├── charts
├── Chart.yaml
├── templates
│ ├── deployment.yaml
│ ├── _helpers.tpl
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── NOTES.txt
│ ├── serviceaccount.yaml
│ ├── service.yaml
│ └── tests
│ └── test-connection.yaml
└── values.yaml

Chart.yaml Example

apiVersion: v2
name: my-web-app
description: A Helm chart for a simple web application
version: 1.0.0

values.yaml

replicaCount: 3
image:
  repository: my-web-app
  tag: latest
  pullPolicy: IfNotPresent
service:
  name: my-web-app
  type: ClusterIP
  port: 80
ingress:
  enabled: false

Create a new directory called templates, and create a file called deployment.yaml inside it with the following contents:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Chart.Name }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Chart.Name }}
  template:
    metadata:
      labels:
        app: {{ .Chart.Name }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 80

Create another file called service.yaml inside the templates directory with the following contents:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.service.name }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: http
  selector:
    app: {{ .Chart.Name }}

Finally, package your chart by running the following command:

helm package .

templates

Templates is one of the most important directory and this is where helm looks for your kubernetes object yaml definitions(Deployment,Services,etc).

values.yaml

If templates holds the resource definition, then values provides the way to parameterize it.

_helpers.tpl

Helm allows the use of Go templating in the resources files for kubernetes. The _helpers.tpl is used to define Go template helpers.

NOTES.txt

This is the plain text file that get printed out when the chart is successfully deployed. Usually this contains the next step for using the chart.

Chart.yaml

This file contain the metadata such as chart version, application version or constraints like minimum version of kubernetes/Helm which is required to manage this chart. Some required fields in Chart.yaml

Now you understand all the chart structure and all the corresponding file, it’s time to create your first chart. For the purpose of that I will start with a simple nginx application and later on we will parameterize it using values.yaml.

Step1: Cleanup all the files inside templates directory so that we will start from scratch

rm -rf templates/*

Step2: Create deployment file

kubectl create deployment nginx --image=nginx --dry-run=client -o yaml >> templates/deployment.yaml

Step3: Expose your deployment

kubectl expose deploy nginx --port 80 --type NodePort --dry-run=client -o yaml > /tmp/service.yaml

create deployment temporarily(as we want deploy it using helm).

kubectl create deployment nginx --image=nginx
kubectl expose deploy nginx --port 80 --type NodePort --dry-run=client -o yaml > templates/service.yaml

As this my first chart, I want to keep it bare bone and define only the required values inside Chart.yaml

cat Chart.yaml               
apiVersion: v2
name: myhelmchart
description: A Helm chart for Kubernetes
version: 0.1.0

One more additional file, I need to create is NOTES.txt inside templates directory

echo "This is first helm chart and it will deploy nginx application" >>templates/NOTES.txt

I also cleanup rest of the files and directory to draw the clean slate

rm -rf values.yaml charts

Deploying your first chart

With all the config in place it’s time to deploy first helm chart. We can also set the name of the release so that we can refer it back at the later stages.

It always a good idea to run linter before deploying your chart to make sure there is no syntax error or your are following all the best practices

helm lint ./firsthelmchart

Let’s now start with a dry-run to make sure everything looks good

helm install demochart ./firsthelmchart --dry-run

If things good deploy your helm chart without dry-run option

helm install demochart ./firsthelmchart

To verify it

helm list

Or you can access your application via hitting any of the kubernetes node

curl http://175.12.0.1:32543/

Singlestore, MemSQL Basic understanding

murugan — Wed, 07 Feb 2024 12:18:34 +0000

Single Store

SingleStore DB is a distributed, relational database that handles both transactions and real-time analytics at scale. Querying is done through standard SQL drivers and syntax, leveraging a broad ecosystem of drivers and applications. Read the links below to get familiar with SingleStore DB:

Reference

How SingleStore DB Works

SingleStore DB is a distributed, relational database that handles both transactions and real-time analytics at scale. It is accessible through standard SQL drivers and supports ANSI SQL syntax including joins, filters, and analytical capabilities (e.g. aggregates, group by, and windowing functions).

Data Ingestion

SingleStore DB can load data continuously or in bulk from a variety of sources. Popular loading sources include: files, a Kafka cluster, cloud repositories like Amazon S3, HDFS, or from other databases. As a distributed system, SingleStore DB ingests data streams using parallel loading to maximize throughput.

SingleStore Pipelines is an easy-to-use built-in capability that extracts, transforms, and loads external data using sources such as Kafka, S3, Azure Blob, and filesystems.

Deploy

SingleStore DB can be deployed on bare metal, on virtual machines, or in the cloud by using SingleStore Tools

Reference

Managed Service
- The scalable cloud database for data-intensive applications, deployed on AWS, Azure, or GCP
DB Software
- Manually deploy and manage a SingleStore database cluster on your own hardware.

DB Creation

DROP TABLE IF EXISTS play_game;
CREATE DATABASE play_game;
USE play_game;

CREATE TABLE play_game (
    msgId INT,
    msgDateTime DATETIME NOT NULL,
    gameName  varchar(60),
    SORT KEY (timeSince),
    SHARD KEY (msgId)
);

Pipelines

MemSQL Pipelines is a MemSQL Database feature that natively ingests real-time data from external sources. As a built-in component of the database, Pipelines can extract, transform, and load external data without the need for third-party tools or middleware. Pipelines is robust, scalable, highly performant, and supports fully distributed workloads.

Alter pipeline
Create Pipeline
Create Pipeline with Transform
Create Pipeline into Procedure
Extract Pipeline into outfile

Reference

Create Pipeline

CREATE PIPELINE play_game
AS LOAD DATA KAFKA '127.0.0.1:9092/play-games-1'
SKIP DUPLICATE KEY ERRORS
INTO TABLE play_game
FORMAT JSON (
    msgId <- msgId,
    msgDateTime <- msgDateTime,
    gameName <- gameName,
    timeSince <- timeSince
)

START PIPELINE play_game;

Basic Comments

Syntax	Comments
Delete Pipeline	DELETE PIPELINE play_games;
Delete Database	DROP DATABASE IF EXISTS play_game;
Delete Table	DROP TABLE IF EXISTS play_game;
Stop Pipeline	STOP PIPELINE play_game;
Test Pipeline	Test PIPELINE play_game;
Extract Pipeline	EXTRACT PIPELINE play_games INTO OUTFILE 'file_name.json'

Reference

Load Data

The easiest way to load data is to first upload it to Amazon S3 or Azure Blob Storage. Then, use SingleStore Pipelines to extract your data.

Load Data from Amazon Web Services (AWS)
Load Data from Microsoft Azure
Load Data from the Filesystem using a Pipeline
Load Data from Kafka
Load Data from MySQL etc

Load Data from KAFA

bin/kafka-topics.sh --create --topic play-games-1 --partitions 1 --bootstrap-server localhost:9092 --replication-factor 1


bin/kafka-topics.sh --describe --topic play-games-1 --bootstrap-server localhost:9092

bin/kafka-console-producer.sh --topic play-games-1 --bootstrap-server localhost:9092

Insert into the data from Kafka usimng JSON

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic play-games-1 < PlayActivity.json

bin/kafka-console-consumer.sh --topic play-games-1 --from-beginning --bootstrap-server localhost:9092

List Topic

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Delete Topic

bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic DummyTopic

Single Store Stored Procedure Example

USE quickstart_kafka;
CREATE TABLE test2 (id int, fname varchar(50), lname varchar(50), addr varchar(50));

DELIMITER //
CREATE OR REPLACE PROCEDURE process_users(GENERIC_BATCH query(GENERIC_JSON json)) AS
BEGIN
INSERT INTO test3(id,fname,lname,addr)
SELECT GENERIC_JSON::id, GENERIC_JSON::fname,GENERIC_JSON::lname,GENERIC_JSON::addr
FROM GENERIC_BATCH;
END //
DELIMITER ;

CREATE or replace PIPELINE jsonproce1 AS LOAD DATA KAFKA '127.0.0.1:9092/test2'
INTO PROCEDURE process_users (GENERIC_JSON <- %)FORMAT JSON ;

test pipeline jsonproce1;

start pipeline jsonproce1;

Manage Data

Local and Unlimited Database Storage Concepts
Unlimited Data Storage
Backing Up and Restoring Data
Exporting Particular Database >> mysqldump -h 127.0.0.1 -u root -p -P 3306 foo > foo.sql
Export All Database >> mysqldump -h 127.0.0.1 -u root -p -P 3306 --all-databases > full_backup.sql

Aditional Concepts:

Open MemSQL Studio

https://aws.amazon.com/quickstart/architecture/memsql/
https://aws-quickstart.s3.amazonaws.com/quickstart-memsql/doc/memsql-on-the-aws-cloud.pdf

--> SQL Editor
--> Master Aggregator Node (1 Master) (Aggregator we will share the meta data)
--> Child Aggregator Node (2 Nodes)
--> Leaf Node (4 Nodes) - Leaf node we will share the data

Every cluster atleast 1 master aggregater and 1 leaf node

Cluster

A cluster encompasses all of the nodes that are included in a complete SingleStore DB installation. A cluster contains aggregator nodes and leaf nodes.

Connect to your Cluster

You have three ways to connect to your SingleStore DB cluster: SingleStore DB Studio, the singlestore client application, or through any compatible third-party applications.

Node

A node is a server that has an installation of a SingleStore DB instance.

Leaf

A leaf is a node that stores a subset of a cluster’s data. A cluster typically contains many leaves.

Partition

A partition contains a subset (a shard) of a database’s data. A leaf contains multiple partitions. When you run CREATE DATABASE, SingleStore DB splits the database into partitions,
which are distributed evenly among available leaves. With CREATE DATABASE, you can specify the number of partitions with the PARTITIONS=X option.

If you don’t specify the number of partitions explicitly, the default is used (the number of leaves times the value of the default_partitions_per_leaf engine variable.

Aggregator

An aggregator is a node that routes queries to the leaves, aggregates intermediate the results, and sends the results back to the client.
There are two types of aggregators: master and child. A cluster contains exactly one master aggregator, a specialized aggregator responsible for cluster monitoring and failover.
A cluster may also contain zero or more child aggregators (depending on query volume).

Time Series Functions:

For storing and manipulating time series data, SingleStore supports the following functions:

FIRST
LAST
TIME_BUCKET

FIRST
An aggregate function that returns the first value of a set of input values, defined as the value associated with the minimum time.

LAST
An aggregate function that returns the last value of a set of input values, defined as the value associated with the maximum time.

SELECT TIME_BUCKET('4d'), gameName
FROM play_activity ORDER BY 2, 1;

SELECT TIME_BUCKET('1d'), gameName
FROM play WHERE timeSince BETWEEN '2021-10-06T00:00:10.530+00:00' AND '2021-10-07T23:00:10.530+00:00' ORDER BY 2, 1;

SELECT TIME_BUCKET('6h', timeSince) as timeSince, gameName
FROM play
WHERE timeSince > now() - INTERVAL 1 day ORDER BY 2, 1;

Additional comments

Restart Studio

sudo systemctl restart singlestoredb-studio

Installation Steps

sudo apt-get update

http://localhost:9000/cluster
http://localhost:8080/cluster/localhost/dashboard

References

timeSince

Golang Design Patterns

murugan — Wed, 07 Feb 2024 12:18:16 +0000

Design Patterns in go

Creational design pattern

- Abstract design Pattern


Singleton Pattern
Builder
Factory
Object Pool
Prototype

Abstract Factory Design Pattern in Go

Definition:

Abstract Factory Design Pattern is a creational design pattern that lets you create a family of related objects. It is an abstraction over the factory pattern.

Example use cases:

Imagine you need to buy a sports kit which has a shoe and short. Preferably most of the time you would want to buy a full sports kit of a similar factory i.e either nike or adidas. This is where the abstract factory comes into the picture as concrete products that you want is shoe and a short and these products will be created by the abstract factory of nike and adidas.
Both these two factories – nike and adidas implement iSportsFactory interface.
We have two product interfaces.

iShoe – this interface is implemented by nikeShoe and adidasShoe concrete product.
iShort – this interface is implemented by nikeShort and adidasShort concrete product.

Note

Builder Pattern in GoLang

Builder Pattern is a creational design pattern used for constructing complex objects.

When To Use:

Use Builder pattern when the object constructed is big and requires multiple steps. It helps in less size of the constructor. The construction of the house becomes simple and it does not require a large constructor
When a different version of the same product needs to be created. For example, in the below code we see a different version of house ie. igloo and the normal house being constructed by iglooBuilder and normalBuilder
When half constructed final object should not exist. Again referring to below code the house created will either be created fully or not created at all. The Concrete Builder struct holds the temporary state of house object being created

Factory Design Pattern in Go

Factory design pattern is a creational design pattern and it is also one of the most commonly used pattern. This pattern provides a way to hide the creation logic of the instances being created.
The client only interacts with a factory struct and tells the kind of instances that needs to be created. The factory class interacts with the corresponding concrete structs and returns the correct instance back.

When To Use:

We have iGun interface which defines all methods a gun should have
There is gun struct that implements the iGun interface.
Two concrete guns ak47 and maverick. Both embed gun struct and hence also indirectly implement all methods of iGun and hence are of iGun type
We have a gunFactory struct which creates the gun of type ak47 or maverick.
The main.go acts as a client and instead of directly interacting with ak47 or maverick, it relies on gunFactory to create instances of ak47 and maverick

Object Pool Design Pattern in Go

The Object Pool Design Pattern is a creational design pattern in which a pool of objects is initialized and created beforehand and kept in a pool. As and when needed, a client can request an object from the pool, use it, and return it to the pool. The object in the pool is never destroyed.

When to Use:

When the cost to create the object of the class is high and the number of such objects that will be needed at a particular time is not much.
- Let’s take the example of DB connections. Each of the connection object creation is cost is high as there is network calls involved and also at a time not more than a certain connection might be needed. The object pool design pattern is perfectly suitable for such cases.
When the pool object is the immutable object
- Again take the example of DB connection again. A DB connection is an immutable object. Almost none of its property needs to be changed
For performance reasons. It will boost the application performance significantly since the pool is already created

Prototype Pattern in Go

It is a creational design pattern that lets you create copies of objects. In this pattern, the responsibility of creating the clone objects is delegated to the actual object to clone.

The object to be cloned exposes a clone method which returns a clone copy of the object

When to Use

We use prototype pattern when the object to be cloned creation process is complex i.e the cloning may involve vases of handling deep copies, hierarchical copies, etc. Moreover, there may be some private members too which cannot be directly accessed.
A copy of the object is created instead of creating a new instance from scratch. This prevents costly operations involved while creating a new object such as database operation.
When you want to create a copy of a new object, but it is only available to you as an interface. Hence you cannot directly create copies of that object.

Singleton Pattern in Go

Singleton Design Pattern is a creational design pattern and also one of the most commonly used design pattern. This pattern is used when only a single instance of the struct should exist. This single instance is called a singleton object. Some of the cases where the singleton object is applicable:

DB instance – we only want to create only one instance of DB object and that instance will be used throughout the application.
Logger instance – again only one instance of the logger should be created and it should be used throughout the application.

Behavioural Design Pattern

Observer Design Pattern in Go

Observer Design Pattern is a behavioral design pattern. This pattern allows an instance (called subject) to publish events to other multiple instances (called observers). These observers subscribe to the subject and hence get notified by events in case of any change happening in the subject.

Let’s take an example. In the E-Commerce website, many items go out of stock. There can be customers, who are interested in a particular item that went out of stock. There are three solutions to this problem

The customer keeps checking the availability of the item at some frequency.
E-Commerce bombard customers with all new items available which are in stock
The customer subscribes only to the particular item he is interested in and gets notified in the case that item is available. Also, multiple customers can subscribe to the same product

Strategy pattern

Strategy pattern is a behavioral design pattern. Strategy pattern allows changing the behavior of an object at the runtime which is useful in certain cases.

https://golangbyexample.com/facade-design-pattern-in-golang/

Facade Design Pattern in Go:

Facade Pattern is classified as a structural design pattern. This design pattern is meant to hide the complexities of the underlying system and provide a simple interface to the client. It provides a unified interface to underlying many interfaces in the system so that from the client perspective it is easier to use. Basically it provides a higher level abstraction over a complicated system.

Check Account
Check Security Pin
Credit/Debit Balance
Make Ledger Entry
Send Notification

As can be noticed, there are a lot of things that happen for a single debit/credit operation. This is where the Facade pattern comes into picture. As a client one only needs to enter the Wallet Number, Security Pin, Amount and specify the type of operation. The rest of the things are taken care of in the background. Here we create a WalletFacade which provides a simple interface to the client and which takes care of dealing with all underlying operations

Structural Design Pattern:

Adapter Patern:

Adapter is a structural design pattern, which allows incompatible objects to collaborate.

The adapter pattern is a design pattern that allows the interface of a class to be used as another interface.

This is a frequently used pattern in ORM libraries (ActiveRecord, GORM, etc), because it allows for connections and queries to different data backends (databases) while keeping the same client interface.

Another common occurrence of this pattern is in hardware drivers. Take printers for instance. Most printers can be used either via USB or serial connections or over the network

Example:

We have a client code that expects some features of an object (Lightning port), but we have another object called adaptee (Windows laptop) which offers the same functionality but through a different interface (USB port)

Bridge in Go

Bridge is a structural design pattern that divides business logic or huge class into separate class hierarchies that can be developed independently.

One of these hierarchies (often called the Abstraction) will get a reference to an object of the second hierarchy (Implementation). The abstraction will be able to delegate some (sometimes, most) of its calls to the implementations object. Since all implementations will have a common interface, they’d be interchangeable inside the abstraction.

Example:

Say, you have two types of computers: Mac and Windows. Also, two types of printers: Epson and HP. Both computers and printers need to work with each other in any combination. The client doesn’t want to worry about the details of connecting printers to computers.

If we introduce new printers, we don’t want our code to grow exponentially. Instead of creating four structs for the 2*2 combination, we create two hierarchies:

Abstraction hierarchy: this will be our computers
Implementation hierarchy: this will be our printers
These two hierarchies communicate with each other via a Bridge, where the Abstraction (computer) contains a reference to the Implementation (printer). Both the abstraction and implementation can be developed independently without affecting each other.

See the example code https://refactoring.guru/design-patterns/bridge/go/example

Composite in Go

Composite is a structural design pattern that allows composing objects into a tree-like structure and work with the it as if it was a singular object.

Composite became a pretty popular solution for the most problems that require building a tree structure. Composite’s great feature is the ability to run methods recursively over the whole tree structure and sum up the results.

Example
Let’s try to understand the Composite pattern with an example of an operating system’s file system. In the file system, there are two types of objects: files and folders. There are cases when files and folders should be treated to be the same way. This is where the Composite pattern comes in handy.

Imagine that you need to run a search for a particular keyword in your file system. This search operation applies to both files and folders. For a file, it will just look into the contents of the file; for a folder, it will go through all files of that folder to find that keyword.

GraphQL with Golang

murugan — Thu, 13 Apr 2023 13:00:25 +0000

GraphQL with Golang

GraphQL is an open-source data query and manipulation language for APIs, and a runtime for fulfilling queries with existing data. GraphQL was developed internally by Facebook in 2012 before being publicly released in 2015

GraphQL is designed to make APIs fast, flexible, and developer-friendly. As an alternative to REST, GraphQL lets developers construct requests that pull data from multiple data sources in a single API call. GraphQL uses HTTP POST method to submit queries. We have a single service endpoint with variations in HTTP body, driven by schema and query patterns.

API developers will create a schema to define queries, data types, mutations that clients can query through

type Todo {
  id: ID!
  text: String!
  done: bool
}
type Query {
  todos: [Todo!]!
  findTodor(id: String!): Todo
}

Consider a the above definition. We define Todo type, and what queries are exposed as a part of this API. When a query request comes-in, GraphQL resolver will validate the request against pre-defined schema and invokes connected methods for resolution. Another benefit to this approach is flexibility for clients to request fields they are interested in - It’s better for network, security and process. If we don’t want “lastName”, we don’t need to fetch it.

query {
  todos {
    id
    text
    done
  }
}

Golang & GraphQL

Let's start with the bootstrap process.

mkdir gographql
cd gographql
go mod init gographql
go get github.com/99designs/gqlgen
mkdir tools && cd tools
touch tools.go
cd ..
code .

Open file tools.go and add the following content to it

package tools
import _ "github.com/99designs/gqlgen"

Save your changes in VSCode and back to terminal. Let's now generate sample graphql schema and golang connectors to execute

go run github.com/99designs/gqlgen init
Creating gqlgen.yml
Creating graph/schema.graphqls
Creating server.go
Generating…
go: downloading gopkg.in/check.v1 v1.0.0–20190902080502–41f04d3bba15
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/kr/text v0.1.0
Exec "go run ./server.go" to start GraphQL server

You should now see a bunch of files added.

model/model_gen.go golang data types for graphql types created in schema.graphqls
generated/generated.go this is a file with generated code that injects context and middleware for each query and mutation
schema.graphqls a GraphQL schema file where types, queries, and mutations are defined.
schema.resolvers.go a go file with wrapper code for queries and mutations defined in schema.graphqls

If you want make any changes in the schema.graphqls do the changes and delete the schema.resolvers.go and run the following command

go run github.com/99designs/gqlgen generate

Back to terminal and run the following comments

go run ./server.go
2022/06/22 18:20:03 connect to http://localhost:8080/ for GraphQL playground

Head over to http://localhost:8080. You’ll see the playground.

Let’s introduce a custom schema and see how it works. Let’s say we want to have a simple CRUD ops on Todo model. Nothing complicated in Todo model — Id, Text and Done should be good for us to start.

Head over to “schema.graphqls” replace file content with the following

type Todo {
  id: ID!
  text: String!
  done: Boolean!
  todo: Todo!
}

type Todo {
  id: ID!
  text: String!
}

type Query {
  todos: [Todo!]!
  findTodo(id: String!): Todo
}

input NewTodo {
  text: String!
  Id: String!
}

input DeleteTodo {
 Id: String!
}

type Mutation {
  createTodo(input: NewTodo!): Todo!
  removeTodo(input: DeleteTodo!): Todo!
}

We have defined a Todo type
Couple of queries — One to list all todos and other to search for todo
Input types — Types for mutation requests.
Mutation — Capability to modify data/model

To proceed, delete "schema.resolvers.go" and head back to terminal

go run github.com/99designs/gqlgen generate

You should not see any error(s). New “schema.resolvers.go” file should be generated.

// CreateTodo is the resolver for the createTodo field.
func (r *mutationResolver) CreateTodo(ctx context.Context, input model.NewTodo) (*model.Todo, error) {
    log.Println("Create a new Todo")
    uuidValue := uuid.NewString()
    todo := &model.Todo{ID: uuidValue, Text: input.Text, Done: true}
    todos = append(todos, todo)
    return todo, nil
}

// RemoveTodo is the resolver for the removeTodo field.
func (r *mutationResolver) RemoveTodo(ctx context.Context, input model.DeleteTodo) (*model.Todo, error) {
    index := -1
    for i, todo := range todos {
        if todo.ID == input.ID {
            index = i
        }
    }
    if index == -1 {
        return nil, errors.New("Cannot find todo you are looking for!")
    }
    todo := todos[index]
    todos = append(todos[:index], todos[index+1:]...)

    return todo, nil
}

// Todos is the resolver for the todos field.
func (r *queryResolver) Todos(ctx context.Context) ([]*model.Todo, error) {
    return todos, nil
}

// FindTodo is the resolver for the findTodo field.
func (r *queryResolver) FindTodo(ctx context.Context, id string) (*model.Todo, error) {
    panic(fmt.Errorf("not implemented: FindTodo - findTodo"))
}

// Mutation returns MutationResolver implementation.
func (r *Resolver) Mutation() MutationResolver { return &mutationResolver{r} }

// Query returns QueryResolver implementation.
func (r *Resolver) Query() QueryResolver { return &queryResolver{r} }

type mutationResolver struct{ *Resolver }
type queryResolver struct{ *Resolver }

// !!! WARNING !!!
// The code below was going to be deleted when updating resolvers. It has been copied here so you have
// one last chance to move it out of harms way if you want. There are two reasons this happens:
//   - When renaming or deleting a resolver the old code will be put in here. You can safely delete
//     it when you're done.
//   - You have helper methods in this file. Move them out to keep these resolver files clean.
func (r *queryResolver) Todo(ctx context.Context, id string) (*model.Todo, error) {
    panic(fmt.Errorf("not implemented: Todo - todo"))
}

var todos []*model.Todo

func init() {
    log.Println("Init - Todo array to be created")
    todos = make([]*model.Todo, 0)
    todos = append(todos, &model.Todo{ID: "1", Text: "Hello One", Done: true, Todo: &model.Todo{Text: "Text1"}})
    todos = append(todos, &model.Todo{ID: "2", Text: "Hello Two", Done: true, Todo: &model.Todo{Text: "Text2"}})
    todos = aid:ppend(todos, &model.Todo{ID: "3", Text: "Hello Three", Done: true, Todo: &model.Todo{Text: "Text3"}})
    log.Println("Init - Todo array has been created")
}
func (r *queryResolver) FindTodos(ctx context.Context, id string) (*model.Todo, error) {
    for _, todo := range todos {
        if todo.ID == id {
            return todo, nil
        }
    }
    return nil, errors.New("Cannot find todo you are looking for!")
}

Run the server.go and and see how the application works using below queries

query{
    todos{
        id
        text
        done
    }
}

Find Todo

query{
    findTodo(id:"1"){
        id
        text
        done
    }
}

Create Todo

mutation createTodo($todo: NewTodo!){
    createTodo(input:$todo){
        id
        text
        done
    }
}

Query Variable:

{
    "todo": {
        "id":4,
        "text": "Hello 4",
        "done": true
    }
}

NATS WITH GOLANG USING DOCKER

murugan — Thu, 21 Jul 2022 09:52:00 +0000

NATS

NATs is an open source, lightweight and high performance native message system developed by go language. It provides an abstraction layer between an application or service and the underlying physical network. The data is encoded and sent as a message by the publisher. Messages are received, decoded, and processed by one or more subscribers.
NATs makes it easy for programs to communicate across different environments, languages, cloud providers, and internal systems

Advantages

Ease of use
Highly performant
Zero downtime scaling
Supports edge, cloud or hybrid deployments

Install NATS in windows Locally

choco install nats-server

To start a simple demonstration server locally, simply run:

nats-server

nats-server -m 8222 (if you want to enable the HTTP monitoring functionality)
You can also test the monitoring endpoint, viewing http://localhost:8222 with a browser.

Installing via Docker

docker pull nats:latest

To run NATS on docker

docker run -p 4222:4222 -ti nats:latest

NATS Server Containerization

docker pull nats
docker run nats

By default the NATS server exposes multiple ports:
4222 is for clients.
8222 is an HTTP management port for information reporting.
6222 is a routing port for clustering.
Use -p or -P to customize.

docker run -it nats:alpine sh
docker run -itd nats:alpine

To run a server with the ports exposed on a docker network:

docker network create nats

docker run --name nats --network nats --rm -p 4222:4222 -p 8222:8222 nats --http_port 8222

or 

docker run --itd --restart always --name nats-alpine --network nats --rm -p 4222:4222 -p 8222:8222 nats:alpine --http_port 8222

NATS CLI

Downloading Release Build

You can find the latest release of nats-server https://github.com/nats-io/nats-server/releases.
You could manually download the zip file matching your systems architecture, and unzip it. You could also use curl to download a specific version

Download this build (https://github.com/nats-io/nats-server/releases) "nats-0.0.33-windows-amd64" from this URL and unzip in you machine and navigating into the given path and run the nats account info command which show the accout information. Run the below commands to perform publish or subscribe the given messages

nats-server -js
nats account info

Publish Message

nats publish hello my-data

nats subscribe hello

Request-Reply

nats request hello

nats reply hello data

Benchmark

nats bench --msgs 1000 --pub 2 --sub 2 test

Subject based Message

Basically, NATs is about publishing and listening to messages. Both depend heavily on the subject of the message. Simply put, subject is a string of characters that publishers and subscribers can use to find each other’s names.

Publish Subscribe

NATS implements a publish-subscribe message distribution model for one-to-many communication. A publisher sends a message on a subject and any active subscriber listening on that subject receives the message. Subscribers can also register interest in wildcard subjects that work a bit like a regular expression (but only a bit). This one-to-many pattern is sometimes called a fan-out.

Example

Pub

nc, _ := nats.Connect(nats.DefaultURL)
defer nc.Close()

nc.Publish("foo", []byte("Hello World!"))

Sub

nc, _ := nats.Connect(nats.DefaultURL)
defer nc.Close()

nc.Subscribe("foo", func(m *nats.Msg) {
    fmt.Printf("Received a message: %s\n", string(m.Data))
})

Request Reply

Request response is a common pattern in modern distributed system. When sending a request, the application either waits for a response with a specific timeout or receives the response asynchronously. With the increasing complexity requirements of modern systems, many technologies need additional components to complete the complete feature set.
NATs supports this pattern through its core communication mechanisms (publish and subscribe). The request is published together with the answer topic on a given topic, and the responder listens for the topic and sends the response to the answer topic. The reply topic is usually a topic called “inbox”, which will be dynamically directed back to the requester, regardless of the location of either party.
The ability of NATs even allows multiple responses, the first of which is utilized and the system effectively discards the additional responses. This allows a complex pattern to have multiple responders to reduce response latency and jitter.

Go to the /nats-0.0.33-windows-amd64 path and run below commands

1.Start the first member of the queue group (In new window)
nats reply hello "service instance A Reply# {{Count}}"

2.Start the second member of the queue group [In new window]
> nats reply hello "service instance B Reply# {{Count}}"

3.Start the thrid member of the queue group [In new window]
> nats reply hello "service instance C Reply# {{Count}}"

4.Publish NATS Message [In new window]
> nats request foo "Simple request"
5.Publish another Message [In new window]
> nats pub foo "Another simple request"

You should see that only one of the hello group subscribers receives the message and replies it, and you can also see which one of the available hello subscribers processed the request from the reply message received (i.e. service instance A, B or C)

Example

nc, _ := nats.Connect(nats.DefaultURL)
defer nc.Close()

nc.Subscribe("foo", func(m *nats.Msg) {
    nc.Publish(m.Reply, []byte("I will help you"))
})

reply, _ := nc.Request("foo", []byte("help"), 50*time.Millisecond)

fmt.Println(string(reply.Data))

Queue Group

NATs provides a built-in load balancing feature called distributed queues. Using queue subscribers balances messaging between a set of subscribers that can be used to provide application fault tolerance and large-scale workload processing.
To create a queue subscription, you only need the subscriber to register the queue name. All subscribers with the same queue name form a queue group. No configuration is required. When a message is published on a registered topic, a member of the group is randomly selected to receive the message. Although the queue group has multiple subscribers, each message is used by only one subscriber.
An important feature of NATs is that queue groups are defined by the application and its queue subscribers, not on the server configuration.
Queue subscribers are ideal for extended services. Scaling up is as simple as running another application, and scaling down is to terminate the application with a signal that exhausts the running request. This flexibility and lack of any configuration changes make NATs an excellent service communication technology that can work with all platform technologies

Example:

nc, _ := nats.Connect(nats.DefaultURL)
defer nc.Close()

received := 0

nc.QueueSubscribe("foo", "worker_group", func(_ *nats.Msg) {
    received++
})

Acknowledgements

In systems with up to one semantics, messages are sometimes lost. If your application is performing a request answer, it should use a timeout to handle any network or application failures. It’s always a good idea to set a timeout on a request and use code that handles the timeout. When publishing an event or data flow, one way to ensure message delivery is to convert it to a request reply with an ACK concept. In NATs, an ACK can be an empty message, a message without a payload

Example:

nc, _ := nats.Connect(nats.DefaultURL)
defer nc.Close()

nc.Subscribe("foo", func(m *nats.Msg) {
    //nc.Publish(m.Reply, []byte("I will help you"))
    m.Respond([]byte(""))
})

reply, _ := nc.Request("foo", []byte("help"), 50*time.Millisecond)

fmt.Println("ack:", string(reply.Data))

NATS-TOP

nats-top is a top-like tool for monitoring nats-server servers.
The nats-top tool provides a dynamic real-time view of a NATS server. nats-top can display a variety of system summary information about the NATS server, such as subscription, pending bytes, number of messages, and more, in real time

Installation in GO

go install github.com/nats-io/nats-top

If you want to start the nats server monitoring enabled we can use the below commands

nats-server -m 8222

Start Nats-top

nats-top

References

Containerizing your Go Applications with Docker

murugan — Thu, 03 Mar 2022 14:52:23 +0000

Containerizing your Go Applications with Docker

Create docker-go folder and Build a simple Web Server in Go. For create the main.go file and add the below code

package main

import (
    "fmt"
    "html"
    "log"
    "net/http"
)

func main() {

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
    })

    http.HandleFunc("/hi", func(w http.ResponseWriter, r *http.Request){
        fmt.Fprintf(w, "Hi")
    })

    log.Fatal(http.ListenAndServe(":8083", nil))

}

if we want to run this go run main.go which will kick of a server on http://localhost:8083.

Write Dockerfile

## We specify the base image we need for our
## go application
FROM golang:1.16-alpine
## We create an /app directory within our
## image that will hold our application source
## files
RUN mkdir /app
## We copy everything in the root directory
## into our /app directory
ADD . /app
## We specify that we now wish to execute 
## any further commands inside our /app
## directory
WORKDIR /app
## we run go build to compile the binary
## executable of our Go program
RUN go build -o main .
## Our start command which kicks off
## our newly created binary executable
CMD ["/app/main"]

Run the below command to create the docker image

docker build -t docker-go .

It will build the code and showing the below output in your terminal

C:\work\latest\docker-go>docker build -t docker-go . 
[+] Building 38.4s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                                    0.2s 
 => => transferring dockerfile: 643B                                                                                    0.0s 
 => [internal] load .dockerignore                                                                                       0.2s 
 => => transferring context: 2B                                                                                         0.0s 
 => [internal] load metadata for docker.io/library/golang:1.12.0-alpine3.9                                              3.2s 
 => [1/5] FROM docker.io/library/golang:1.12.0-alpine3.9@sha256:6c143f415448f883ed034529162b3dc1c85bb2967fdd1579a8735  29.4s 
 => => resolve docker.io/library/golang:1.12.0-alpine3.9@sha256:6c143f415448f883ed034529162b3dc1c85bb2967fdd1579a87356  0.1s 
 => => sha256:69371d496b2b4e99120216fa3c5057b0c5468411370ab24ea99cd87d7b1d9203 1.36kB / 1.36kB                          0.0s 
 => => sha256:2205a315f9c751a8c205aa42f29ad0ff29918c40d85c8ddaabac99e0cb46b5d8 3.80kB / 3.80kB                          0.0s 
 => => sha256:6c143f415448f883ed034529162b3dc1c85bb2967fdd1579a873567b22bcb790 2.37kB / 2.37kB                          0.0s 
 => => sha256:8e402f1a9c577ded051c1ef10e9fe4492890459522089959988a4852dee8ab2c 2.75MB / 2.75MB                          0.7s 
 => => sha256:de1a1e452942df2228b914d2ce9be43f18b137f4ebc3dce9491bc08c2630a2ea 154B / 154B                              0.7s 
 => => sha256:ce7779d8bfe3415e27ec3bf5950b2ab67a854c608595f0f2e49066fb5806fd12 301.88kB / 301.88kB                      0.8s 
 => => extracting sha256:8e402f1a9c577ded051c1ef10e9fe4492890459522089959988a4852dee8ab2c                               0.7s 
 => => sha256:a8c461e224a623234c9f2ff60e4249678c9e6847add7152ac80239b13d14df4c 126B / 126B                              1.0s 
 => => sha256:1bdc943bc000449a960c5121688afe0a9b51837407bf0478391b6bad6642d36f 124.28MB / 124.28MB                     15.7s 
 => => extracting sha256:ce7779d8bfe3415e27ec3bf5950b2ab67a854c608595f0f2e49066fb5806fd12                               0.5s 
 => => extracting sha256:de1a1e452942df2228b914d2ce9be43f18b137f4ebc3dce9491bc08c2630a2ea                               0.0s 
 => => extracting sha256:1bdc943bc000449a960c5121688afe0a9b51837407bf0478391b6bad6642d36f                              11.8s 
 => => extracting sha256:a8c461e224a623234c9f2ff60e4249678c9e6847add7152ac80239b13d14df4c                               0.0s 
 => [internal] load build context                                                                                       0.2s 
 => => transferring context: 1.06kB                                                                                     0.1s 
 => [2/5] RUN mkdir /app                                                                                                1.7s
 => [3/5] ADD . /app                                                                                                    0.3s 
 => [4/5] WORKDIR /app                                                                                                  0.2s 
 => [5/5] RUN go build -o main .                                                                                        2.1s 
 => exporting to image                                                                                                  0.6s 
 => => exporting layers                                                                                                 0.5s 
 => => writing image sha256:9bbcb2070c03ab1affa9f2dc62292f1cea589a60c05d4796c4490c0fa31afedb                            0.0s 
 => => naming to docker.io/library/docker-go

We can now verify that our image exists on our machine by typing docker images:

docker images

which will show the list of images which contain docker-go docker images.

In order to run this newly created image, we can use the docker run command and pass in the ports we want to map to and the image we wish to run.

docker run -p 8083:8083 -it docker-go

p 8083:8083 - This exposes our application which is running on port 8083 within our container on http://localhost:8083 on our local machine.
it - This flag specifies that we want to run this image in interactive mode with a tty for this container process.
docker-go - This is the name of the image that we want to run in a container.

if we open up http://localhost:8083 within our browser, we should see that our application is successfully responding with Hello, "/".

When i see the docker image size it roughly take 800MB in size. This is absolutely massive for smaller go application. If you want to reduce this do the following steps

A Simple Multi-Stage Dockerfile

In order to see why multi-stage Dockerfiles are useful, we'll be creating a simple Dockerfile that features one stage to both build and run our application, and a second Dockerfile which features both a builder stage and a production stage.

Once we've created these two distinct Dockerfiles, we should be able to compare them and hopefully see for ourselves just how multi-stage Dockerfiles are preferred over their simpler counterparts!

So, The above dockerfile, we created a really simple Docker image in which our Go application was both built and run from.

You should hopefully notice that last column states that the size of this image is 800MBs in size. This is absolutely massive for something that builds and runs a very simple Go application.

Within this image will be all the packages and dependencies that are needed to both compile and run our Go applications. With multi-stage dockerfiles, we can actually reduce the size of these images dramatically by splitting things up into two distinct stages.

let's take a look at how we could define a real multi-stage Dockerfile that will first compile our application and subsequently run our application in a lightweight Docker alpine image.

Next, we'll create a Dockerfile in the same directory as our main.go file above. This will feature a builder stage and a production stage which will be built from two distinct base images:

## We'll choose the incredibly lightweight
## Go alpine image to work with
FROM golang:1.16-alpine AS builder

## We create an /app directory in which
## we'll put all of our project code
RUN mkdir /app
ADD . /app
WORKDIR /app
## We want to build our application's binary executable
RUN CGO_ENABLED=0 GOOS=linux go build -o main ./...

## the lightweight scratch image we'll
## run our application within
FROM alpine:latest AS production
## We have to copy the output from our
## builder stage to our production stage
COPY --from=builder /app .
## we can then kick off our newly compiled
## binary exectuable!!
CMD ["./main"]

Now that we've defined this multi-stage Dockerfile, we can proceed to build it using the standard docker build command:

docker build -t go-multi-stage .

Now, when we compare the sizes of our simple image against our multi-stage image, we should see a dramatic difference in sizes. Our previous, docker-go image was roughly 800MB in size, whereas this multi-stage image is about 1/80th the size.

If we want to try running this to verify it all works, we can do so using the following docker run command:

docker run -d -p 8083:8083 go-multi-stage

if we open up http://localhost:8083 within our browser, we should see that our application is successfully responding with Hello, "/".

# Golang Profiling

murugan — Tue, 22 Feb 2022 05:48:04 +0000

Golang Profiling

Profiling is a form of analyzing the program for optimizable code or functions. In software engineering, it is an essential task since optimization is a key factor when developing an application.
Avoiding memory leaks and optimizing for better performance is almost always a target for enterprise-level software.

Profiling is an important task that cannot be avoided for larger applications. Profiling helps us understand CPU and memory intensive code and helps us write better code for optimization

To create any profile first we need to have a test file. Here we are going to use the Fibonacci function to see profiles of it

// main.go
package main

func Fib2(n int) uint64 {
    if n == 0 {
        return 0
    } else if n == 1 {
        return 1
    } else {
        return Fib2(n-1) + Fib2(n-2)
    }
}

func main() {
    // fmt.Println(Fib2(30)) // 832040
}

Now, the test file is:

// main_test.go
package main

import "testing"

func TestGetVal(t *testing.T) {
    for i := 0; i < 1000; i++ {             // running it a 1000 times
        if Fib2(30) != 832040 {
            t.Error("Incorrect!")
        }
    }
}

Now when we run the go test we got the below output

output:

go test
Pass
Ok

It took almost 7.25s to complete. Now let’s create a CPU profile. We will use this command shown below to generate a profile file.

go test -cpuprofile cpu.prof -bench .

which will return the below output

PS C:\work\latest\go-tutorial\profile> go test -cpuprofile cpu.prof -bench .
PASS
ok      profile 8.052s

Now, we will view it using the pprof tool. The command will be:

go tool pprof cpu.prof

When we run the above commands we got the below output

PS C:\work\latest\go-tutorial\profile> go tool pprof cpu.prof
Type: cpu
Time: Feb 22, 2022 at 10:53am (IST)
Duration: 7.64s, Total samples = 4.86s (63.60%)
Entering interactive mode (type "help" for commands, "o" for options")

Typing help will show all commands available. We will run the following command

top5 -cum

The topN function shown top N entries and the -cum flag shows the cumulative time taken

(pprof) top5 -cum
Showing nodes accounting for 4700ms, 96.71% of 4860ms total
Dropped 33 nodes (cum <= 24.30ms)
Showing top 5 nodes out of 15
      flat  flat%   sum%        cum   cum%
    4700ms 96.71% 96.71%     4700ms 96.71%  profile.Fib2
         0     0% 96.71%     4700ms 96.71%  profile.TestGetVal
         0     0% 96.71%     4700ms 96.71%  testing.tRunner
         0     0% 96.71%       90ms  1.85%  runtime.mcall
         0     0% 96.71%       90ms  1.85%  runtime.park_m

based on the above results we will optimize our code.

To create a memory profile we simply use this command:

go test -memprofile mem.prof -bench .

which will return the below output

PS C:\work\latest\go-tutorial\profile> go test -memprofile mem.prof -bench .
PASS
ok      profile 0.644s

We can generate both profiles (main, test) using the following command

go test -cpuprofile cpu.prof -memprofile mem.prof -bench .

which will return the below output

PS C:\work\latest\go-tutorial\profile> go test -cpuprofile cpu.prof -memprofile mem.prof -bench .
PASS
ok      profile 0.655s

Basic Linux commands for developers

murugan — Thu, 10 Feb 2022 09:25:00 +0000

Basic Linux Commands for developers

FILE AND DIRECTORY COMMANDS

S.No	Description	Commands
1	List all files in a long listing (detailed) format	ls -al
2	Display the present working directory	pwd
3	Create a directory	mkdir
4	Remove (delete) file	rm file
5	Remove the directory and its contents recursively	rm -r directory
6	Force removal of file without prompting for confirmation	rm -f file
7	Copy file1 to file2	cp file1 file2
8	Copy source_directory recursively to destination. If destination exists, copy source_directory into destination, otherwise create destination with the contents of source_directory	cp -r source_directory destination
9	Rename or move file1 to file2. If file2 is an existing directory, move file1 into directory file2	mv file1 file2
10	View the contents of file	cat file
11	Browse through a text file	less file
12	Display the first 10 lines of file	head file
12	Display the last 10 lines of file	tail file

ARCHIVES (TAR FILES)

S.No	Description	Commands
1	Create tar named archive.tar containing directory	tar cf archive.tar directory
2	Extract the contents from archive.tar	tar xf archive.tar
3	Create a gzip compressed tar file name archive.tar.gz.	tar czf archive.tar.gz directory
4	Extract a gzip compressed tar file.	tar xjf archive.tar.bz2
5	Create a tar file with bzip2 compression	tar cjf archive.tar.bz2 directory
6	Extract a bzip2 compressed tar file	tar xjf archive.tar.bz2

FILE PERMISSIONS

S.No	Description	Commands
1	chmod 777 filename	rwx rwx rwx
2	chmod 775 filename	rwx rwx r-x
3	chmod 755 filename	rwx r-x r-x
4	chmod 664 filename	rw- rw- r--
5	chmod 644 filename	rw- r-- r--

NETWORKING

S.No	Description	Commands
1	Display DNS information for domain	dig domain
2	Display DNS IP address for domain	host domain
4	Display all local IP addresses of the host.	hostname -I
5	Display listening tcp and udp ports and corresponding programs	netstat -nutlp

INSTALLING PACKAGES

S.No	Description	Commands
1	Search for a package by keyword	yum search keyword
2	Install package	yum install package
3	Display description and summary information about package	yum info package
4	Install package from local file named package.rpm	rpm -i package.rpm
5	Remove/uninstall package	yum remove package

SEARCH

S.No	Description	Commands
1	Search for pattern in file	grep pattern file
2	Search recursively for pattern in directory	grep -r pattern directory
3	Find files and directories by name	locate name
4	Find files in /home/john that start with "prefix".	find /home/john -name 'prefix*'

VIM Exiting

S.No	Description	Commands
1	write (save) the file, but don’t exit	:w
2	write out the current file using sudo	:w !sudo tee %
3	write (save) and quit	:wq or :x or ZZ
4	quit (fails if there are unsaved changes)	:q – quit
5	quit and throw away unsaved changes	:q! or ZQ
6	search for pattern	/pattern –

Docker-compose basic tutorial

murugan — Mon, 07 Feb 2022 12:18:51 +0000

Docker Compose Tutorial

Docker simplifies the process of managing application processes in containers. While containers are similar to virtual machines in certain ways, they are more lightweight and resource-friendly. This allows developers to break down an application environment into multiple isolated services.

Docker Compose

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.
Compose works in all environments: production, staging, development, testing, as well as CI workflow etc

Setting Up a docker-compose.yml File

mkdir ~/compose-demo

cd ~/compose-demo

mkdir app

nano app/index.html

<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>Docker Compose Demo</title>
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/kognise/water.css@latest/dist/dark.min.css">
</head>
<body>

    <h1>This is a Docker Compose Demo Page.</h1>
    <p>This content is being served by an Nginx container.</p>

</body>
</html>

nano docker-compose.yml

version: '3.7'
services:
  web:
    image: nginx:alpine
    ports:
      - "8000:80"
    volumes:
      - ./app:/usr/share/nginx/html

We then have the services block, where we set up the services that are part of this environment. In our case, we have a single service called web. This service uses the nginx:alpine image and sets up a port redirection with the ports directive. All requests on port 8000 of the host machine (the system from where you’re running Docker Compose) will be redirected to the web container on port 80, where Nginx will be running.

The volumes directive will create a shared volume between the host machine and the container. This will share the local app folder with the container, and the volume will be located at /usr/share/nginx/html inside the container, which will then overwrite the default document root for Nginx.

docker-compose up -d

This command will show you information about the running containers and their state, as well as any port redirections currently in place:

docker-compose ps

To check the logs produced by your Nginx container, you can use the logs command:

docker-compose logs

If you want to pause the environment execution without changing the current state of your containers, you can use:

docker-compose pause

To resume execution after issuing a pause:

docker-compose unpause

The stop command will terminate the container execution, but it won’t destroy any data associated with your containers:

docker-compose stop

If you want to remove the containers, networks, and volumes associated with this containerized environment, use the down command:

Notice that this won’t remove the base image used by Docker Compose to spin up your environment (in our case, nginx:alpine). This way, whenever you bring your environment up again with a docker-compose up, the process will be much faster since the image is already on your system.

docker-compose down

In case you want to also remove the base image from your system, you can use:

docker image rm nginx:alpine

Building Web Applications using Beego + Mysql+ORM

murugan — Thu, 29 Apr 2021 09:53:47 +0000

Beego:

Beego is a RESTful HTTP framework for the rapid development of Go applications including APIs, web apps and backend services with integrated Go-specific features such as interfaces and struct embedding.

M (models), V (views), C (controllers) each have top level folders. main.go is the entry point.

Installing Beego

Run the below command in your gopath

go get -u github.com/beego/beego/v2
go get -u github.com/beego/bee/v2

Beego has in-built scaffolding support, via the command line tool bee

Create new applications
Run an application
Test the application
Create routes and more

Create The Core Project

Once installed, from your $GOPATH directory, run the following command, which will scaffold the application, called sitepointgoapp:

bee new sitepointgoapp

Looking at the files, we have:
1.Our bootstrap file main.go
2.The core configuration file conf/app.conf
3.A default controller controllers/default.go
4.A default set of tests in tests/default_test.go
5.A default view template in views/index.tpl

Right, the basic app is ready, now run the below command

bee run

This loads our new application.

Router settings:

beego.Router("/user/home", &controllers.UserController{}, "*:User")
beego.Router("/user/add", &controllers.UserController{}, "get,post:Add")

Here i've used the mysql with ORM. First we need to install the MySQL driver using golang by using the below command

go get -u github.com/go-sql-driver/mysql

The first one imports Beego’s ORM library "github.com/astaxie/beego/orm", the second one provides support for mysql, required because we’re using an mysql database. The third one imports the models we just created, giving them an alias of models.

create the table in MySQL for orm_test and run the below query to create the users table.

CREATE TABLE `users` (
  `id` int(11) NOT NULL,
  `name` varchar(200) NOT NULL,
  `client` varchar(200) NOT NULL,
  `url` varchar(200) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

In the main.go we need to register the driver along with database settings like below for mysql

orm.RegisterDriver("mysql", orm.DRMySQL)
orm.RegisterDataBase("default", "mysql", "root:@/orm_test?charset=utf8")
orm.RegisterModel(new(models.User))

Install ORM using

go get github.com/beego/beego/v2/client/orm

command. By using this line we need to insert the record in db using ORM

id, err := o.Insert(&user)
if err == nil {
    msg := fmt.Sprintf("User inserted with id:", id)
    beego.Debug(msg)
    flash.Notice(msg)
    flash.Store(&manage.Controller)
} else {
    msg := fmt.Sprintf("Couldn't insert new User. Reason: ", err)
    beego.Debug(msg)
    flash.Error(msg)
    flash.Store(&manage.Controller)
}

If we using the sqlite we need to install the GCC and then we need to setup the database setting same as like below in main.go file

orm.RegisterDriver("sqlite", orm.DR_Sqlite)
orm.RegisterDataBase("default", "sqlite3", "database/orm_test.db")
orm.RegisterModel(new(models.User))

Check the http://localhost:8086/user/home we have displayed the home page which we have done.

Beego commands:

1.bee pack The pack command is used to compress the project into a single file. The compressed file can be deployed by uploading and extracting the zip file to the server

______
| ___ \
| |_/ /  ___   ___   
| ___ \ / _ \ / _ \  
| |_/ /|  __/|  __/  
\____/  \___| \___| v1.12.0
2021/04/29 15:15:17 INFO     ▶ 0001 Packaging application on 'D:\Go-work\src\github.com\aws-learning\bee-go\sitepointgoapp'...
2021/04/29 15:15:17 INFO     ▶ 0002 Building application (sitepointgoapp)...
2021/04/29 15:15:17 INFO     ▶ 0003 Using: GOOS=windows GOARCH=amd64
2021/04/29 15:15:21 SUCCESS  ▶ 0004 Build Successful!
2021/04/29 15:15:21 INFO     ▶ 0005 Writing to output: D:\Go-work\src\github.com\aws-learning\bee-go\sitepointgoapp\sitepointgoapp.tar.gz
2021/04/29 15:15:21 INFO     ▶ 0006 Excluding relpath prefix: .
2021/04/29 15:15:21 INFO     ▶ 0007 Excluding relpath suffix: .go:.DS_Store:.tmp:go.mod:go.sum
2021/04/29 15:15:24 SUCCESS  ▶ 0008 Application packed!

2.bee bale - This command is currently only available to the developer team. It is used to compress all static files in to a single binary file so that they do not need to carry static files including js, css, images and views when publishing the project. Those files will be self-extracting with non-overwrite when the program starts

3.bee version - This command displays the version of bee, beego, and go

4.bee generate - generate the particular scaffoldname

References

Docker Basic

murugan — Tue, 13 Apr 2021 09:25:25 +0000

Docker

Docker is a platform used to containerize our software, using which we can easily build our applications and package them, with the dependencies required, into containers, and further these containers are easily shipped to run on other machines.

In simple words, Docker is a software containerization platform, meaning you can build your application, package them along with their dependencies into a container and then these containers can be easily shipped to run on other machines.

For example: Lets consider a linux based application which has been written both in Ruby and Python. This application requires a specific version of linux, Ruby and Python. In order to avoid any version conflicts on user’s end, a linux docker container can be created with the required versions of Ruby and Python installed along with the application. Now the end users can use the application easily by running this container without worrying about the dependencies or any version conflicts.

Docker is an open-source centralized platform designed to create, deploy, and run applications. Docker uses container on the host's operating system to run applications. It allows applications to use the same Linux kernel as a system on the host computer, rather than creating a whole virtual operating system. Containers ensure that our application works in any environment like development, test, or production.

Docker includes components such as Docker client, Docker server, Docker machine, Docker hub, Docker composes, etc.

Docker simplifies the DevOps Methodology by allowing developers to create templates called ‘images’ using which we can create lightweight virtual machines called ‘containers.’ Docker makes things easier for software developers giving them the capability to automate infrastructure, isolate applications, maintain consistency, and improve resource utilization.

Docker is a containerization platform that packages your application and all its dependencies together in the form of a docker container to ensure that your application works seamlessly in any environment.

Virtualization

Virtualization is the technique of importing a Guest operating system on top of a Host operating system. This technique was a revelation at the beginning because it allowed developers to run multiple operating systems in different virtual machines all running on the same host. This eliminated the need for extra hardware resource. The advantages of Virtual Machines or Virtualization are:
1.Multiple operating systems can run on the same machine
2.Maintenance and Recovery were easy in case of failure conditions
3.Total cost of ownership was also less due to the reduced need for infrastructure

Containerization

Containerization is the technique of bringing virtualization to the operating system level. While Virtualization brings abstraction to the hardware, Containerization brings abstraction to the operating system. Do note that Containerization is also a type of Virtualization. Containerization is however more efficient because there is no guest OS here and utilizes a host’s operating system, share relevant libraries & resources as and when needed unlike virtual machines. Application specific binaries and libraries of containers run on the host kernel, which makes processing and execution very fast. Even booting-up a container takes only a fraction of a second. Because all the containers share, host operating system and holds only the application related binaries & libraries. They are lightweight and faster than Virtual Machines.

Advantages of Containerization over Virtualization:

1.Containers on the same OS kernel are lighter and smaller
2.Better resource utilization compared to VMs
3.Boot-up process is short and takes few seconds

Docker Containers

Docker containers are the lightweight alternatives of the virtual machine. It allows developers to package up the application with all its libraries and dependencies, and ship it as a single package. The advantage of using a docker container is that you don't need to allocate any RAM and disk space for the applications. It automatically generates storage and space according to the application requirement.

It is basically the instance of an image. Multiple containers can exist for a single image.

Docker Containers are the ready applications created from Docker Images. Or you can say they are running instances of the Images and they hold the entire package needed to run the application.

Docker Images

Images are nothing but a read-only binary template that can build containers.
Docker Image can be compared to a template which is used to create Docker Containers.They are the building blocks of a Docker Container. These Docker Images are created using the build command. These Read only templates are used for creating containers by using the run command.

Virtual Machine

A virtual machine is a software that allows us to install and use other operating systems (Windows, Linux, and Debian) simultaneously on our machine. The operating system in which virtual machine runs are called virtualized operating systems. These virtualized operating systems can run programs and preforms tasks that we perform in a real operating system.

Containers Vs. Virtual Machine

Containers	Virtual Machine
Integration in a container is faster and cheap.	Integration in virtual is slow and costly.
No wastage of memory.	Wastage of memory.
It uses the same kernel, but different distribution.	It uses multiple independent operating systems.

Why Docker:

1.Docker allows us to easily install and run software without worrying about setup or dependencies.
2.Developers use Docker to eliminate machine problems, i.e. "but code is worked on my laptop." when working on code together with co-workers.
3.Operators use Docker to run and manage apps in isolated containers for better compute density.
4.Enterprises use Docker to securely built agile software delivery pipelines to ship new application features faster and more securely.
5.Since docker is not only used for the deployment, but it is also a great platform for development, that's why we can efficiently increase our customer's satisfaction.

Dockerfile, Docker Image And Docker Container:

1.A Docker Image is created by the sequence of commands written in a file called as Dockerfile.
2.When this Dockerfile is executed using a docker command it results into a Docker Image with a name.
3.When this Image is executed by “docker run” command it will by itself start whatever application or service it must start on its execution.

Docker Compose:

Docker Compose is basically used to run multiple Docker Containers as a single server. Let me give you an example:

Suppose if I have an application which requires WordPress, Maria DB and PHP MyAdmin. I can create one file which would start both the containers as a service without the need to start each one separately. It is really useful especially if you have a microservice architecture.

Docker daemon:

A daemon creates, runs, and monitors containers, along with building and storing images.

Docker Registry

Docker Registry is where the Docker Images are stored. The Registry can be either a user’s local repository or a public repository like a Docker Hub allowing multiple users to collaborate in building an application. Even with multiple teams within the same organization can exchange or share containers by uploading them to the Docker Hub, which is a cloud repository similar to GitHub.

References

Docker commands with simple PHP example

murugan — Tue, 13 Apr 2021 09:19:53 +0000

Docker Commands

1.docker –version

This command is used to get the currently installed version of docker

2.docker pull

docker pull - This command is used to pull images from the docker repository(hub.docker.com)

3.docker run

Usage: docker run -it -d - This command is used to create a container from an image

4.docker ps

This command is used to list the running containers

5.docker ps -a

This command is used to show all the running and exited containers

6.docker exec

docker exec -it bash - This command is used to access the running container

7.docker stop

docker stop - This command stops a running container

8.docker kill

docker kill - This command kills the container by stopping its execution immediately. The difference between ‘docker kill’ and ‘docker stop’ is that ‘docker stop’ gives the container time to shutdown gracefully, in situations when it is taking too much time for getting the container to stop, one can opt to kill it

9.docker commit

docker commit - This command creates a new image of an edited container on the local system

10.docker login

This command is used to login to the docker hub repository

11.docker push

docker push - This command is used to push an image to the docker hub repository

12.docker images

This command lists all the locally stored docker images

13.docker rm

docker rm - This command is used to delete a stopped container

14.docker rmi

docker rmi - This command is used to delete an image from local storage

15.docker build

docker build - This command is used to build an image from a specified docker file

Docker Php Application Example

A Dockerfile is a text document that contains commands that are used to assemble an image

1.Create a directory

mkdir php-docker-app

2.Create a Php File

<?php  
    echo ?Hello, Php?;  
?>

3.Create a DockerFile

FROM php:7.0-apache  
COPY . /var/www/php

4.Create Docker Image

docker build -t php-docker-app .

Which will create the docker image

5.Run the Docker image

docker run php-docker-app

We can see that our docker image is running and output is shown to the browser. This image is running on the local host.