Forem: Yuen Ying Kit

Setting the time interval of metrics collection in Boatswain

Yuen Ying Kit — Thu, 13 Feb 2020 14:20:18 +0000

Originally posted on Boatswain Blog.

Boatswain Agent 0.2.12 is released!

Earlier this week we have released a new version of Boatswain (Agent version 0.2.12). Please follow this update guide for different Linux distros. If you are running Boatswain as a container, just use the 0.2.12 image available from Docker Hub.

The new Period attribute

In this release, we have added a new configurable attribute called Period which is the collection time interval of the Server Host and the Docker metrics. Here is the new /etc/boatswain.yml looks like:

# Please register and get the token from https://app.boatswain.io/
token: token_here
# Period is the collection time interval of the host and docker metrics.
# CPU load could be reduced by using longer period.
#   - short   = 10s
#   - default = 30s
#   - long    = 60s
period: default

Prior to agent version 0.2.12, the collection time interval is 10s and there is not way to control it.

Now, as stated in the above config file, there are 3 available options for the new Period attribute and default would be used if this attribute is missing.

The command flag for config file is updated too

When running Boatswain directly through the binary as a foreground process, we could specify the location of the config file using the -f flag (instead of -t prior to agent version 0.2.12).

ykyuen@camus:~# boatswain --help

Usage: boatswain [options]
Options:
  -f string
      boatswain configuration file.
  -v  Show boatswain version and exit.

Server resources consumption VS Period

Setting a longer period means the statistics shown on the Boatswain web portal would have a less sample size but in return, it could reduce system load significantly. Moreover, it could free more data space for monitoring more Docker servers without hitting our BETA 2GB data limit.

Let's run Boatswain to monitor itself with different Period settings. Here are the three runs on a Digital Ocean droplet (1vCPU, 1GB Memory) and it hosts a Drupal 7 application. Each run lasts for 30 mins.

Note: The data size below only includes the metrics which depend on the Period setting.

Period: short (10s)

# CPU consumption:    1.39%
# Memory consumption: 10.06% (104,647,789 Bytes)
# Data size:          8.8MB

Period: default (30s)

# CPU consumption:    0.70%
# Memory consumption: 11.00% (114,425,856 Bytes)
# Data size:          4.3MB

Period: long (60s)

# CPU consumption:    0.57%
# Memory consumption: 6,61% (68,772,546 Bytes)
# Data size:          1.9MB

Result

CPU consumption dropped significantly.
- 🔻 49.64% from 1.39% to 0.70% (Period: 10s ➡️ 30s)
- 🔻 58.99% from 1.39% to 0.57% (Period: 10s ➡️ 60s)
Memory consumption was more or less the same.
- Although the value dropped a lot (Period: 10s ➡️ 60s), it raised to 11.90% (123,901,542 Bytes) after running 6 hours.
Data size dropped significantly.
- 🔻 51.14% from 8.8MB to 4.3MB (Period: 10s ➡️ 30s)
- 🔻 78.41% from 8.8MB to 1.9MB (Period: 10s ➡️ 60s)

Other metrics

Boatswain also collects other metrics including

Container Status
Container Logs
Web Analytics

They are independent of the new Period attribute as the logs and web analytics metrics are received in real time while the container status is monitored in a fixed 10s time interval.

What's next?

It has been more than one year since our BETA-I release and Boatswain looks more complete after the recent updates. Probably it's time to move on. 🚢

Stay tuned with us!

Real time notification on Docker container status

Yuen Ying Kit — Tue, 21 Jan 2020 09:40:11 +0000

Originally posted on Boatswain Blog.

In the recent release of the Boatswain web portal (version 0.1.6), we have implemented a simple real time notification mechanism on the Docker container status. We hope this is helpful to engineer responding quickly to any unexpected issue happen on their container.

Setup the real time notification

Now, Boatswain user could setup the real time notification on the Notification Settings page which in under the Settings menu on the top right hand corner.

Enable the real time notification

Sometimes, not all the containers running on the Docker server are our concerns. So we allow user to enable the notification feature selectively by using regular expression. Any containers with name matched with the regular expressions would have notification enabled.

Here are the two rules when setting up the regular expression:

Include the / before and after.
Only accepts the i flag.

Check out which container have notification enabled

Below the regular expression form, there is a list showing the containers which have notification enabled. Any detected containers in the recent 24 hours will be listed.

Please note that the above list is just for reference and any newly added container would also have notification enabled as long as its name matched any regular expression.

Receive the notification email

Whenever the container status is changed, the email address associated with the corresponding Boatswain account would receive an notification email. Here is an example:

Hi ykyuen,

We have detected a status change of the following container:
---
Host:            camus
Container ID:    1d2ce3c96b372fc61622c0c0133efda6de24cd2fad2e8faab810b8539d3ce4a1
Container name:  nginx-test
Container image: nginx:latest
Command:         nginx -g 'daemon off;'
Status change:   running -> exited
Exit code:       0
Error:           
Is OOM killed:   No
---

- Boatswain Team
 ____              _                     _
|  _ \            | |                   (_)
| |_) | ___   __ _| |_ _____      ____ _ _ _ __
|  _ < / _ \ / _` | __/ __\ \ /\ / / _` | | '_ \
| |_) | (_) | (_| | |_\__ \\ V  V / (_| | | | | |
|____/ \___/ \__,_|\__|___/ \_/\_/ \__,_|_|_| |_|

What's next?

We notice that Boatswain consumes certain amount of server resources and user might hang back especially when his/her application is running on a small instance. So we are working on the Boatswain configuration and let user decide the frequency of the metrics and log collections.

Lastly, happy coding 👨‍💻 and wishing all the best to all of you 🎊 🐀 🎉.

Monitor Kubernetes cluster by running Boatswain as a DaemonSet

Yuen Ying Kit — Thu, 09 Jan 2020 09:04:45 +0000

Originally posted on Boatswain Blog.

In a Kubernetes cluster, each node has a container runtime for running containers in pods. Kubernetes supports various container runtimes including:

So if the Kubernetes cluster is using Docker as container runtime, we could install Boatswain on the nodes and monitoring its status as well as collecting the container logs.

NOTE: The example in this article is for experimental purpose and we do not suggest using Boatswain in any production Kubernetes cluster.

The DaemonSet in Kubernetes

In Kubernetes, we could deploy the Boatswain pod as a DaemonSet. Unlike a Deployment which we could replicates multiple pods across different nodes, a DaemonSet makes sure that a single pod runs on the selected nodes even if the node is newly added to the cluster. It is particular useful for storage daemon, log collection and monitoring on the node.

Running Boatswain as a DaemonSet

Here are the steps about setting up the Boatswain DaemonSet in a Kubernetes cluster.

Create the Boatswain namespace
Setup the Boatswain token as a secret
Setup the config file of the Boatswain DaemonSet
Start the Boatswain DaemonSet

Create the Boatswain namespace

Let's create a new namespace to separate the Boatswain setup from those already exist.

kubectl create namespace boatswain

Setup the Boatswain token as a secret

This token is required for running Boatswain. We store it as a secret called boatswain and later use it in the DaemonSet configuration.

kubectl create secret generic boatswain --from-literal=token=<TO_BE_REPLACED> -n boatswain

Setup the config file of the Boatswain DaemonSet

Here comes the DaemonSet configuration.

daemon-set.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: boatswain
  namespace: boatswain
spec:
  selector:
    matchLabels:
      name: boatswain
  template:
    metadata:
      labels:
        name: boatswain
    spec:
      containers:
      - name: boatswain
        image: boatswainio/boatswain:<latest or tag>
        env:
        - name: BOATSWAIN_TOKEN
          valueFrom:
            secretKeyRef:
              name: boatswain
              key: token
        securityContext:
          capabilities:
            add: ["NET_ADMIN"]
        volumeMounts:
          - name: dockersock
            mountPath: "/var/run/docker.sock"
      volumes:
      - name: dockersock
        hostPath:
          path: /var/run/docker.sock
      hostNetwork: true
      hostPID: true
      restartPolicy: Always
      nodeSelector:
        kubernetes.io/hostname: <hostname of the selected node>

Before saving the yaml file:

Replace the version of Boatswain at line 17.
Set the hostname of the target node which you want to install Boatswain or remove line 37 and 38 for installation on all nodes.

Start the Boatswain DaemonSet

Execute the following command to start the Boatswain DaemonSet.

kubectl create -f daemon-set.yaml

Check the status of the DaemonSet.

[ykyuen@camus ~]$ kubectl get ds -n boatswain
NAME        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                                                        AGE
boatswain   1         1         1       1            1           kubernetes.io/hostname=gke-boatswain-default-pool-v2-1234abcd-5678   52s

And we could read the metrics and logs on Boatswain.

Summary

The example above shows that it is possible to use Boatswain to monitor the Kuberenetes cluster. But currently it might not able to show the network metrics (confirmed on GKE cluster above). It might be related to the security settings and so far we haven't investigated further about this issue. Moreover, running Boatswain on all nodes does not work well as the usage will hit the 2GB data limit of the beta trial in a few hours. If you have any suggestions or thoughts, please let us know. 😀

Monitor your Docker container status in real time

Yuen Ying Kit — Fri, 20 Dec 2019 09:00:09 +0000

Originally posted on Boatswain Blog.

The new Container Status dashboard!

In the recent update of Boatswain (agent version 0.2.11), it keeps track of all the container statuses, no matter running or stopped, and display them on the new Container Status dashboard implemented on the Boatswain.io.

The Container Status Timeline

With the timeline above, we could get a rough idea on the recent lifecycle of any container. There are all together 8 possible statuses:

created
running
paused
restarting
exited
dead
removing
unknown

Please note that Boatswain makes his best effort to collect the container status and may not be able to acknowledge a change if a status duration is too short.

The Container Info

This pane displays the current container status which is equivalent to the docker ps command.

The Container Status History

This table list out every change in status detected and it is just the textual format representing the timeline graph above. The first row with highlighted background is the current container status.

Check out the latest status of all containers

Other than the new Container Status dashboard, we also added a new pane in the Docker Status dashboard listing the current status of all the containers.

What's next?

We hope the above new feature could help Docker users getting up-to-date information of their Docker servers but dashboard is a just passive solution which means it is futile if you are not looking at it. Real time notification could fill this gap and that's our final stage of development before the official launch. Stay tuned and follow us on twitter!

Handling HTTP request in Go Echo framework (2)

Yuen Ying Kit — Thu, 18 Apr 2019 09:24:54 +0000

Originally posted on Boatswain Blog.

This is the second post about handling HTTP request in Echo framework and we will continue to work on the example we made previously.

Handling HTTP request in Go Echo framework (1)

This time we will create a simple HTML form and submit it as a POST request.

Create a POST request endpoint

Checkout this repo and create the api/post_full_name.go to handle the POST request. This new file is almost the same as api/get_full_name.go except the JSON is prettified before returning to the client.

api/post_full_name.go



package api

import (
  "bytes"
  "encoding/json"
  "fmt"
  "net/http"

  "github.com/labstack/echo"

  "gitlab.com/ykyuen/handling-http-request-in-go-echo-example-1/model"
)

func PostFullName(c echo.Context) error {
  // Bind the input data to ExampleRequest
  exampleRequest := new(model.ExampleRequest)
  if err := c.Bind(exampleRequest); err != nil {
    return err
  }

  // Manipulate the input data
  greeting := exampleRequest.FirstName + " " + exampleRequest.LastName

  // Pretty print the json []byte
  var resp bytes.Buffer
  var b = []byte(
    fmt.Sprintf(`{
      "first_name": %q,
      "last_name": %q,
      "msg": "Hello %s"
    }`, exampleRequest.FirstName, exampleRequest.LastName, greeting),
  )
  err := json.Indent(&resp, b, "", "  ")
  if err != nil {
    return err
  }

  // Return the json to the client
  return c.JSONBlob(
    http.StatusOK,
    []byte(
      fmt.Sprintf("%s", resp.Bytes()),
    ),
  )
}

Add the new route in main.go.



...
func main() {
  // Echo instance
  e := echo.New()

  // Instantiate a template registry with an array of template set
  // Ref: https://gist.github.com/rand99/808e6e9702c00ce64803d94abff65678
  templates := make(map[string]*template.Template)
  templates["home.html"] = template.Must(template.ParseFiles("view/home.html", "view/base.html"))
  templates["about.html"] = template.Must(template.ParseFiles("view/about.html", "view/base.html"))
  e.Renderer = &TemplateRegistry{
    templates: templates,
  }

  // Route => handler
  e.GET("/", handler.HomeHandler)
  e.GET("/about", handler.AboutHandler)

  // Route => api
  e.GET("/api/get-full-name", api.GetFullName)
  e.POST("/api/post-full-name", api.PostFullName)

  // Start the Echo server
  e.Logger.Fatal(e.Start(":1323"))
}

Test the new endpoint using curl

Run the Echo server as send a POST request using curl.



curl http://localhost:1323/api/post-full-name -H "Content-Type: application/json" -d '{ "first_name": "Kit", "last_name": "Yuen"  }'

Here is the example result.

Add an HTML form

The POST request endpoint is ready and now we could add an HTML form and complete the whole setup. Let's add the form in the about page.

view/about.html



{{define "title"}}
  Boatswain Blog | {{index . "name"}}
{{end}}

{{define "body"}}
  <h1>{{index . "msg"}}</h1>
  <h2>This is the about page.</h2>

  <!-- HTML form -->
  <form id="post-full-name-form">
    <input type="text" id="first-name" placeholder="First Name">
    <input type="text" id="last-name" placeholder="Last Name">
    <button type="button" id="submit-btn">Submit</button>
  </form>

  <!-- Print the response after the jQuery ajax request -->
  <pre id="form-result"></pre>

  <script type="text/javascript">
    // Send a ajax request when the submit button is clicked
    $("#submit-btn").on("click", function(){
      var firstName = $("#first-name").val();
      var lastName = $("#last-name").val();

      $.ajax({
        type: "POST",
        url: "/api/post-full-name",
        data: JSON.stringify({
          first_name: firstName,
          last_name: lastName
        }),
        processData: false,
        contentType: "application/json",
        dataType: "json"
      }).done(
        function(data){
          $("#form-result").text(JSON.stringify(data, null, 2));
        }
      ).fail(
        function(data){
          $("#form-result").text("POST request failed!");
        }
      )
    });
  </script>
{{end}}

As the form submission required jQuery, let's include it in the view/base.html.

view/base.html



{{define "base.html"}}

  <!DOCTYPE html>

  <html>

    <head>

      <title>{{template "title" .}}</title>

      <!-- jQuery -->

      <script

        src="https://code.jquery.com/jquery-3.3.1.min.js"

        integrity="sha256-FgpCb/KJQlLNfOu91ta32o/NMZxltwRo8QtmkMRdAu8="

        crossorigin="anonymous"></script>

    </head>

    <body>

      {{template "body" .}}

    </body>

  </html>

{{end}}

Test the HTML form in the about page

Start the Echo server and browse the about page. Fill in the first and last name and then click the submit button. It should show the response JSON as follow.

The project structure

Here is our new project structure. (The project is renamed to handling-http-request-in-go-echo-example-2, make sure you set the import statement in the Golang source files correctly if you want to change the project name.)



handling-http-request-in-go ├── api/ │   ├── get_full_name.go │   ├── post_full_name.go ├── handler/ │   ├── home_handler.go │   └── about_handler.go ├── model/ │   ├── example_request.go │   └── example_response.go ├── vendor/ │   ├── │   └── ├── view/ │   ├── base.html │   ├── home.html │   └── about.html ├── Gopkg.lock ├── Gopkg.toml └── main.go

-echo-example-2/
 # folder of api endpoints
 # api for get full name
 # api for post full name
 # folder of request handlers
 # handler for home page
 # handler for about page
 # folder of custom struct types
 # struct type of get_full_name request
 # hstruct type of get_full_name response
 # dependencies managed by dep
 github.com/
 golang.org/
 # folder of html templates
 # base layout template
 # home page template
 # about page template
 # dep config file
 # dep config file
 # programme entrypoint

Summary

This is the second episode of Handling HTTP request in Go Echo framework (1) and we have added the following:

Setup a POST request endpoint in the Echo framework.
Prettify the JSON byte array
Add an HTML form and bind it to the new POST request using AJAX in jQuery/javascript.

The complete example could be found on gitlab.com.

Handling HTTP request in Go Echo framework (1)

Yuen Ying Kit — Mon, 08 Apr 2019 17:31:20 +0000

Originally posted on Boatswain Blog.

In the last post, we have discussed about how we could setup nested templates in Echo framework.

Setup nested HTML template in Go Echo web framework

In the following sections, we will move on and see how we could make HTTP request in Echo server and render the result back to the client. Let's start with the example we did in the last article.

Setup an API endpoint

Firstly, create a new API endpoint and return a simple JSON which is later called by the Echo server itself.

Create new endpoint in the api folder

Unlike a hardcoded JSON request, our new API endpoint takes the request input from JSON data, form data or query parameters as defined in the model/example_request.go and then return it the the client.

model/example_request.go



package model

type ExampleRequest struct {
  FirstName string `json:"first_name" form:"first_name" query:"first_name"`
  LastName  string `json:"last_name" form:"last_name" query:"last_name"`
}

api/get_full_name.go



package api

import (

  "fmt"

  "net/http"

"github.com/labstack/echo"

"gitlab.com/ykyuen/golang-echo-template-example/model"

)

func GetFullName(c echo.Context) error {

  // Bind the input data to ExampleRequest

  exampleRequest := new(model.ExampleRequest)

  if err := c.Bind(exampleRequest); err != nil {

    return err

  }

// Manipulate the input data

  greeting := exampleRequest.FirstName + " " + exampleRequest.LastName

return c.JSONBlob(

    http.StatusOK,

    []byte(

      fmt.Sprintf({

        "first_name": %q,

        "last_name": %q,

        "msg": "Hello %s"

      }, exampleRequest.FirstName, exampleRequest.LastName, greeting),

    ),

  )

}

Setup the route for the above API endpoint

Add our new API endpoint route in _main.go*.



...

func main() {

  // Echo instance

  e := echo.New()

// Instantiate a template registry with an array of template set

  // Ref: https://gist.github.com/rand99/808e6e9702c00ce64803d94abff65678

  templates := make(map[string]*template.Template)

  templates["home.html"] = template.Must(template.ParseFiles("view/home.html", "view/base.html"))

  templates["about.html"] = template.Must(template.ParseFiles("view/about.html", "view/base.html"))

  e.Renderer = &TemplateRegistry{

    templates: templates,

  }

// Route => handler

  e.GET("/", handler.HomeHandler)

  e.GET("/about", handler.AboutHandler)

// Route => api

  e.GET("/api/get-full-name", api.GetFullName)

// Start the Echo server

  e.Logger.Fatal(e.Start(":1323"))

}

Test the new API endpoint

Run the Echo server as see if it works.

This new endpoint could serve the browser client directly or it could be called by the Echo server itself.

Call the API from Echo server

Create the response struct type

Similar to the model/example_request.go, we need to define another struct type for the HTTP response.

model/example_response.go



package model

type ExampleResponse struct {

  FirstName string json:"first_name"

  LastName  string json:"last_name"

  Msg       string json:"msg"

}

Show the new greeting message in the about page

Let's modify the about page handler such that it will show the greeting message.

handler/about_handler.go



package handler

import (

  "encoding/json"

  "io/ioutil"

  "net/http"

"github.com/labstack/echo"

"gitlab.com/ykyuen/golang-echo-template-example/model"

)

func AboutHandler(c echo.Context) error {

tr := &http.Transport{}

  client := &http.Client{Transport: tr}

// Call the api

  resp, err := client.Get(

    "http://localhost:1323/api/get-full-name?first_name=Kit&amp;last_name=Yuen",

  )

if err != nil {

    return err

  }

defer resp.Body.Close()

  body, err := ioutil.ReadAll(resp.Body)

// Unmarshal the response into a ExampleResponse struct

  var exampleResponse model.ExampleResponse

  if err = json.Unmarshal(body, &exampleResponse); err != nil {

    return err

  }

// Please note the the second parameter "about.html" is the template name and should

  // be equal to one of the keys in the TemplateRegistry array defined in main.go

  return c.Render(http.StatusOK, "about.html", map[string]interface{}{

    "name": "About",

    "msg": exampleResponse.Msg,

  })

}

Test the about page

Start the Echo server and browse the about page. We could see the new greeting message.

The project structure

After making the above changes, the project structure should look like this. (The project is renamed to handling-http-request-in-go-echo-example-1, make sure you set the import statement in the Golang source files correctly if you want to change the project name.)



handling-http-request-in-go-echo-example-1/

  ├── api/                     # folder of api endpoints

  │   ├── get_full_name.go     # api for get full name

  ├── handler/                 # folder of request handlers

  │   ├── home_handler.go      # handler for home page

  │   └── about_handler.go     # handler for about page

  ├── model/                   # folder of custom struct types

  │   ├── example_request.go   # struct type of get_full_name request

  │   └── example_response.go  # hstruct type of get_full_name response

  ├── vendor/                  # dependencies managed by dep

  │   ├── github.com/

  │   └── golang.org/

  ├── view/                    # folder of html templates

  │   ├── base.html            # base layout template

  │   ├── home.html            # home page template

  │   └── about.html           # about page template

  ├── Gopkg.lock               # dep config file

  ├── Gopkg.toml               # dep config file

  └── main.go                  # programme entrypoint

Summary

This article shows you how to setup a simple API endpoint in Echo framework. Since Go is a strongly typed language, we have to convert the JSON into well defined struct type before for we could manipulate its content. For example:

Binding the GET request input to exampleRequest in api/get_full_name.go.
Unmarshal the GET response into exampleResponse in handler/about_handler.go.

To invoke the API call, we could simply make use of the Go standard net/http library.

The complete example could be found on gitlab.com.

Setup nested HTML template in Go Echo web framework

Yuen Ying Kit — Tue, 20 Nov 2018 16:49:41 +0000

Originally posted on Boatswain Blog.

Echo is a lightweight but complete web framework in Go for building RESTful API. It is fast and includes a bunch of middleware for handling the whole HTTP request-response cycle. For the rendering part, it works with any template engine but i pick the standard html/template package for the purpose of simplicity. And at the end of this article, a nested template Echo project setup is demonstrated.

If you already have an idea on how Echo works, jump to the Using nested template section.

A basic Echo project setup

Create the project folder under proper $GOPATH

The complete project code is hosted on GitLab so we first create the project folder $GOPATH/src/gitlab.com/ykyuen/golang-echo-template-example.

Create the main.go

Inside the newly created folder, let's just copy the hello world example from the Echo official site and create the main.go.

main.go

package main

import (
  "net/http"

  "github.com/labstack/echo"
)

func main() {
  e := echo.New()
  e.GET("/", func(c echo.Context) error {
    return c.String(http.StatusOK, "Hello, World!")
  })
  e.Logger.Fatal(e.Start(":1323"))
}

Download the Echo package using dep

Simply run dep init if dep is installed. You can refer to this post for more information about dep.

Manage Go dependencies using dep

Or run go get github.com/labstack/echo to download the Echo package in $GOPATH.

Run the hello world

Start the application by go run main.go and then visit http://localhost:1323 thru browser or the curl command.

Return a JSON response

When building a RESTful API, it is more likely that the client wants to receive and JSON response instead of a string. Let's write some Go code in main.go.

main.go

package main

import (
  "net/http"

  "github.com/labstack/echo"
)

func main() {
  e := echo.New()

  e.GET("/", func(c echo.Context) error {
    return c.String(http.StatusOK, "Hello, World!")
  })

  e.GET("/json", func(c echo.Context) error {
    return c.JSONBlob(
      http.StatusOK,
      []byte(`{ "id": "1", "msg": "Hello, Boatswain!" }`),
    )
  })

  e.Logger.Fatal(e.Start(":1323"))
}

Return an HTML

Similar to returning a JSON object, we just need to call another method in the return statement.

main.go

package main

import (
  "net/http"

  "github.com/labstack/echo"
)

func main() {
  e := echo.New()

  e.GET("/", func(c echo.Context) error {
    return c.String(http.StatusOK, "Hello, World!")
  })

  e.GET("/json", func(c echo.Context) error {
    return c.JSONBlob(
      http.StatusOK,
      []byte(`{ "id": "1", "msg": "Hello, Boatswain!" }`),
    )
  })

  e.GET("/html", func(c echo.Context) error {
    return c.HTML(
      http.StatusOK,
      "<h1>Hello, Boatswain!</h1>",
    )
  })

  e.Logger.Fatal(e.Start(":1323"))
}

The above are just two simple examples, Echo has a few more convenient ways to return JSON and HTML. For details please refer to the documentation.

Render HTML using template engine

As mentioned at the very beginning, we could implement template engine when returning the HTTP response but before that, let's restructure the project as follow.

golang-echo-template-example/
  ├── handler/             # folder of request handlers
  │   └── home_handler.go
  ├── vendor/              # dependencies managed by dep
  │   ├── github.com/*
  │   └── golang.org/*
  ├── view/                # folder of html templates
  │   └── home.html
  ├── Gopkg.lock           # dep config file
  ├── Gopkg.toml           # dep config file
  └── main.go              # programme entrypoint

main.go

package main

import (
  "html/template"
  "io"

  "github.com/labstack/echo"

  "gitlab.com/ykyuen/golang-echo-template-example/handler"
)

// Define the template registry struct
type TemplateRegistry struct {
  templates *template.Template
}

// Implement e.Renderer interface
func (t *TemplateRegistry) Render(w io.Writer, name string, data interface{}, c echo.Context) error {
  return t.templates.ExecuteTemplate(w, name, data)
}

func main() {
  // Echo instance
  e := echo.New()

  // Instantiate a template registry and register all html files inside the view folder
  e.Renderer = &TemplateRegistry{
    templates: template.Must(template.ParseGlob("view/*.html")),
  }

  // Route => handler
  e.GET("/", handler.HomeHandler)

  // Start the Echo server
  e.Logger.Fatal(e.Start(":1323"))
}

In this main.go, we define a type called TemplateRegistry and implement the Renderer interface. A Renderer is a simple interface which wraps the Render() function. Inside a TemplateRegistry instance, it has a templates field containing all the templates needed for the Echo server to render html response and this is configured in the main() flow.

On the other hand, we define the HomeHandler in order to keep the logic in separate file.

handler/home_handler.go

package handler

import (
  "net/http"

  "github.com/labstack/echo"
)

func HomeHandler(c echo.Context) error {
  // Please note the the second parameter "home.html" is the template name and should
  // be equal to the value stated in the {{ define }} statement in "view/home.html"
  return c.Render(http.StatusOK, "home.html", map[string]interface{}{
    "name": "HOME",
    "msg": "Hello, Boatswain!",
  })
}

When the c.Render() is invoked, it executes the template which is already set in our TemplateRegistry instance as stated in main.go. The three paramaters are

HTTP response code
The template name
The data object which could be used in the template

view/home.html

{{define "home.html"}}
  <!DOCTYPE html>
  <html>
    <head>
      <title>Boatswain Blog | {{index . "name"}}</title>
    </head>
    <body>
      <h1>{{index . "msg"}}</h1>
    </body>
  </html>
{{end}}

This above template is named as home.html as stated in the define statement and it could read the name and msg strings from c.Render() for the <title> and <h1> tags.

Using nested template

In the above setup, every HTML template has a complete set of HTML code and many of them are duplicated. Using nested template makes it easier to maintain the project. Originally the templates field in the TemplateRegistry contains all the templates files. In the new setup, we make it into an map field and each item is a single set of template files for a particular HTML page.

We add a few files to the project and it should look like this.

golang-echo-template-example/
  ├── handler/              # folder of request handlers
  │   ├── home_handler.go   # handler for home page
  │   └── about_handler.go  # handler for about page
  ├── vendor/               # dependencies managed by dep
  │   ├── github.com/*
  │   └── golang.org/*
  ├── view/                 # folder of html templates
  │   ├── base.html         # base layout template
  │   ├── home.html         # home page template
  │   └── about.html        # about page template
  ├── Gopkg.lock            # dep config file
  ├── Gopkg.toml            # dep config file
  └── main.go               # programme entrypoint

The codes below are based on this gist created by rand99.

main.go

package main

import (
  "errors"
  "html/template"
  "io"

  "github.com/labstack/echo"

  "gitlab.com/ykyuen/golang-echo-template-example/handler"
)

// Define the template registry struct
type TemplateRegistry struct {
  templates map[string]*template.Template
}

// Implement e.Renderer interface
func (t *TemplateRegistry) Render(w io.Writer, name string, data interface{}, c echo.Context) error {
  tmpl, ok := t.templates[name]
  if !ok {
    err := errors.New("Template not found -> " + name)
    return err
  }
  return tmpl.ExecuteTemplate(w, "base.html", data)
}

func main() {
  // Echo instance
  e := echo.New()

  // Instantiate a template registry with an array of template set
  // Ref: https://gist.github.com/rand99/808e6e9702c00ce64803d94abff65678
  templates := make(map[string]*template.Template)
  templates["home.html"] = template.Must(template.ParseFiles("view/home.html", "view/base.html"))
  templates["about.html"] = template.Must(template.ParseFiles("view/about.html", "view/base.html"))
  e.Renderer = &TemplateRegistry{
    templates: templates,
  }

  // Route => handler
  e.GET("/", handler.HomeHandler)
  e.GET("/about", handler.AboutHandler)

  // Start the Echo server
  e.Logger.Fatal(e.Start(":1323"))
}

We add a new route /about which is handled by a AboutHandler and as you can see from the above highlighted lines, the templates map contains different set of template files for different HTML pages and the Render() takes the name parameter as the templates map key so it could execute the correct template set.

view/base.html

{{define "base.html"}}
  <!DOCTYPE html>
  <html>
    <head>
      <title>{{template "title" .}}</title>
    </head>
    <body>
      {{template "body" .}}
    </body>
  </html>
{{end}}

The template statement tells the template engine that it should look for the {{title}} and {{body}} definitions in the template set and they are defined in the home.html and about.html.

view/about.html

{{define "title"}}
  Boatswain Blog | {{index . "name"}}
{{end}}

{{define "body"}}
  <h1>{{index . "msg"}}</h1>
  <h2>This is the about page.</h2>
{{end}}

And here is the AboutHanlder which has no big difference from the HomeHandler.

handler/about_handler.go

package handler

import (
  "net/http"

  "github.com/labstack/echo"
)

func AboutHandler(c echo.Context) error {
  // Please note the the second parameter "about.html" is the template name and should
  // be equal to one of the keys in the TemplateRegistry array defined in main.go
  return c.Render(http.StatusOK, "about.html", map[string]interface{}{
    "name": "About",
    "msg": "All about Boatswain!",
  })
}

Summary

This is just a basic example implementing nested template using the Go standard html/template library in Echo. With proper setup, we could develop a more customized and convenient pattern for Echo or even make it works with any other template engine.

The complete example could be found on gitlab.com.

How to setup log rotation for Docker container

Yuen Ying Kit — Thu, 14 Jun 2018 03:45:41 +0000

Originally posted on Boatswain Blog.

We all need logs!

Sometimes working with Docker makes me feel like working with a black box especially when playing with the Docker image from by the community and it doesn't go the way as expected. In many cases, reading logs takes up a large portion of time for debugging.

This article is about setting up log rotation for Docker containers.

The default logging driver

We could configure different logging driver for containers and by default the stdout and stderr of the container are written in a json file located in /var/lib/docker/containers/[container-id]/[container-id]-json.log. If you leave it unattended, it could takes up a large amount of disk space as shown below.

Purge the log manually

In case this json log file takes up a significant amount of the disk, we could purge it using the following command.

truncate -s 0 <logfile>

We could setup a cronjob to purge these json log files regularly but for long term, it would be better to setup log rotation.

Setup the log rotation

Configure the default logging driver

This could be done by adding the following values in /etc/docker/daemon.json. Create this file if it doesn't exist.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "10"
  }
}

The json-file logging driver has a few more options and we could even change to other logging drivers such as syslog. For more information, please refer to the Docker Docs - Configure logging drivers.

Execute the following commands to reload the updated daemon.json. The new configuration will apply to all newly created container after restart.

$ systemctl daemon-reload

$ systemctl restart docker

Configure the logging driver for a container

The configuration could also be done on container level if you do not want to apply it globally.

The docker run command

We could specify the logging driver and options in the docker run command. For example:

$ docker run \
    --log-driver json-file \
    --log-opt max-size=10m \
    --log-opt max-file=10 \
    alpine echo hello world

Using docker-compose

The logging driver and options could also be configured using docker-compose. For example:

version: '3.2'
services:
  nginx:
    image: 'nginx:latest'
    ports:
      - '80:80'
    logging:
      driver: "json-file"
      options:
        max-size: "1k"
        max-file: "3"

Verify if the setup is working.

Summary

Although the default setting works fine, you never know when the container logs take up all the disk space and that could be avoided by the above few steps. Other than that, logs are important asset which is not only useful when something goes wrong but also contains a lot of hidden values. So never never let the logs go.

If you are looking for a log management SAAS solution, consider using Boatswain and we help you managing all the logs as well as monitoring your Docker servers. 💫

Insufficient facts always invite danger.
-- Spock @ Star Trek

Driving web browser in Go

Yuen Ying Kit — Fri, 09 Mar 2018 02:02:45 +0000

Originally posted on Boatswain Blog.

This article is about driving web browser in Golang.

Agouti

Agouti is an acceptance and testing framework. It could be used together with Ginkgo which is a BDD testing framework and Gomega matcher/assertion library if you are looking for a complete testing solution for your Go project.

The following example only make use of Agouti to drive a browser.

Create the project inside $GOPATH

Create the main.go under $GOPATH/src/gitlab.com/ykyuen/driving-web-browser-in-go-example.

main.go

package main

import (
  "log"

  "github.com/sclevine/agouti"
)

func main() {
  // driver := agouti.PhantomJS()
  driver := agouti.ChromeDriver()
  // driver := agouti.ChromeDriver(
  //   agouti.ChromeOptions("args", []string{"--headless", "--disable-gpu", "--no-sandbox"}),
  // )

  if err := driver.Start(); err != nil {
    log.Fatal("Failed to start driver:", err)
  }

  page, err := driver.NewPage()
  if err != nil {
    log.Fatal("Failed to open page:", err)
  }

  if err := page.Navigate("https://agouti.org/"); err != nil {
    log.Fatal("Failed to navigate:", err)
  }

  sectionTitle, err := page.FindByID(`getting-agouti`).Text()
  log.Println(sectionTitle)

  if err := driver.Stop(); err != nil {
    log.Fatal("Failed to close pages and stop WebDriver:", err)
  }
}

Agouti supports web drivers such as PhantomJS, Selenium and Chrome.

Download the Go dependency

Let's use dep to manage the Go dependency. Simply run the dep init command.

[ykyuen@camus driving-web-browser-in-go-example]$ dep init
  Using ^2.0.0 as constraint for direct dep github.com/sclevine/agouti
  Locking in v2.0 (b920a9c) for direct dep github.com/sclevine/agouti

At the moment when i am writing this article. The default downloaded version of Agouti by dep is too old. Let's update it to the latest master manually.

Gopkg.toml

[[constraint]]
  name = "github.com/sclevine/agouti"
  branch = "master"

Then run dep ensure to update Agouti version.

Run the code

[ykyuen@camus driving-web-browser-in-go-example]$ go run main.go
2018/01/21 19:02:42 Getting Agouti

If you are using ChromeDriver, you should be able to see the Chrome browser will be started automatically and execute the tasks as stated in main.go.

Running in headless browser

We could also run the code in headless browsers like PhantomJS. Chrome also supports headless mode. Let's update the code as follow.

main.go

package main

import (
  "log"

  "github.com/sclevine/agouti"
)

func main() {
  // driver := agouti.PhantomJS()
  // driver := agouti.ChromeDriver()
  driver := agouti.ChromeDriver(
    agouti.ChromeOptions("args", []string{"--headless", "--disable-gpu", "--no-sandbox"}),
  )
  ...

We could still get the title from the Agouti website without starting the Chrome window.

Summary

This example is not about writing test case but just showing how to drive a web browser.
For testing, we probably need Ginkgo and Gomega.
I also tried another Go package called chromedp but it doesn't work well.
Complete example is available at gitlab.com

Web UI Testing in NodeJS

Yuen Ying Kit — Wed, 28 Feb 2018 16:37:20 +0000

Originally posted on Boatswain Blog.

In this article i would like to demonstrate some tools in NodeJS for testing web UI. They are:

Mocha

Mocha is a Javascript framework for testing. It has a BDD-style syntax (BDD: Behavior-driven development) and allows asychronous call testing. It also supports different assertion libraries and in our exampe we will use chai for assertion.

Chai

Chai is a BDD/TDD assertion library for NodeJS and the browser that can be delightfully paired with any Javascript testing framework.

CasperJS

CasperJS is a helper library for building navigation scenarios. It is often used with PhantomJS but actually it also supports SlimerJS which is another headless browser with the Firefox rendering engine called Gecko.

PhantomJS

PhantomJS is a headless browser with the WebKit rendering engine. It allows running browser-based tests in a headless system.

Check if the Google search is working

Let's get started with a simple example. Assume we would like to test the following:

Is the Google search page accessible?
Is the search function able to return a list of result?

For the first question, we could simple make a HTTP request to the Google URL and see if it returns a HTTP 200 response code and this could be done easily by CasperJS.

The second question is a bit more complicated. It could be broken down into the following steps.

Wait for the search form
Fill in the form and submit
Check if the result set contains the search text

Let's go through the code.

Initialize a new NodeJS project

npm init

Install the following node modules

npm install casperjs chai mocha phantomjs --save-dev
# In addition to the above libraries, we also need the following extensions.
npm install casper-chai mocha-casperjs --save-dev

Setup the tests

Create the test/google-search.js

describe('Google Search', function() {
  // Before script for each test
  before(function() {
    casper.start('https://www.google.com.hk/');
  });

  // *** Test 1 ***
  // Is the Google search page accessible?
  it('should have return HTTP 200', function() {
    expect(casper.currentHTTPStatus).to.equal(200);
  });

  // *** Test 2 ***
  // Is the search function able to return a list of result?
  it('should be able to search', function() {
    // Wait for the search form
    casper.waitForSelector('form[action="/search"]', function() {
      'form[action="/search"]'.should.be.inDOM;
    });

    // Fill in the form and submit
    casper.then(function() {
      this.fill('form[action="/search"]', { q: 'Boatswain' }, true);
    });

    // Check if the result set contains text "Boatswain"
    casper.waitForSelector('h3.r a', function() {
      'h3.r a'.should.be.inDOM;
      expect('h3.r a').to.contain.text(/Boatswain/);
    });
  });
});

Add a npm script for running the test

Edit the package.json as follow.

{
  "name": "ui-test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "mocha-casperjs test/google-search.js"
  },
  "author": "ykyuen",
  "license": "ISC",
  "devDependencies": {
    "casper-chai": "^0.3.0",
    "casperjs": "^1.1.4",
    "chai": "^4.1.2",
    "mocha": "^4.1.0",
    "mocha-casperjs": "^0.6.0",
    "phantomjs": "^2.1.7"
  }
}

Run the test

npm test

Test passed! ✅

A brief test report will be shown after the test run has finished.

Let's try to fail the test

// Check if the result set contains text "Boatswain"
casper.waitForSelector('h3.r a', function() {
  'h3.r a'.should.be.inDOM;
  expect('h3.r a').to.contain.text(/nosuchtext/);
});

Test failed! ❌

Summary

This example shows how to create a simple web UI test in NodeJS and execute the test in command line prompt. It could be used in smoke testing for staging environment. Please also note that CasperJS is NOT for unit testing but rather web UI testing. In addition, test runner like Karma does not support CasperJS. If you are looking for unit testing solution, you should probably rule out CasperJS.

Complete example is available on gitlab.com.

Manage Go dependencies using dep

Yuen Ying Kit — Tue, 23 Jan 2018 01:49:38 +0000

Update @ 2018-11-26: Technology is not just moving at a breakneck speed but also changing rapidly. Within a year, this article is OUTDATED!

// Detect dark theme var iframe = document.getElementById('tweet-1066473266020261888-542'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1066473266020261888&theme=dark" }

And according to the dep project page:

dep was the "official experiment." The Go toolchain, as of 1.11, has (experimentally) adopted an approach that sharply diverges from dep. As a result, we are continuing development of dep, but gearing work primarily towards the development of an alternative prototype for versioning behavior in the toolchain.

For more information about the new Go build-in management, please refer to the official GitHub Wiki - Go 1.11 Modules.

Thanks John Arundel @bitfield and Erhan Yakut @yakuter for revealing the problem. 🙇🏽‍♂️

Update @ 2018-02-03: Sam Boyer from the godep team has clarified some incorrect information in this article. I apologize to Sam Boyer and the readers for any inconvenience. 😢

Originally posted on Boatswain Blog.

Previously i posted an article about dependency management in Go using Glide and i got a feedback about Glide will become obsolete and the Glide team is suggesting users moving to a another dependency management tool called dep written by the golang team.

The Go community now has the dep project to manage dependencies. Please consider trying to migrate from Glide to dep... Glide will continue to be supported for some time but is considered to be in a state of support rather than active feature development.

There is a plan about integrating dep into the toolchain in Go 1.10 release but seems it still has a way to go.

Update @ 2018-02-03:

dep is officially released.
dep is not moving into the toolchain with 1.10. please refer to the roadmap for the latest information.

And i am just not fast enough. 🐌

Create the project inside $GOPATH

Same as before, let's create a new project with root path $GOPATH/src/gitlab.com/ykyuen/dep-example and add the following file.

main.go

package main

import humanize "github.com/dustin/go-humanize"
import "fmt"

func main() {
  fmt.Println("hello world")
  fmt.Printf("That file is %s.\n", humanize.Bytes(82854982)) // That file is 83 MB.
  fmt.Printf("You're my %s best friend.\n", humanize.Ordinal(193)) // You are my 193rd best friend.
  fmt.Printf("You owe $%s.\n", humanize.Comma(6582491)) // You owe $6,582,491.
}

The dep way

Gopkg.toml and Gopkg.lock

dep reads two files called Gopkg.toml and the Gopkg.lock. Let's initialize these 2 files.

[ykyuen@camus dep-example]$ dep init
  Using master as constraint for direct dep github.com/dustin/go-humanize
  Locking in master (bb3d318) for direct dep github.com/dustin/go-humanize

As you can see, the dep init command scans the source codes and downloads all the packages needed for the project into the vendor folder.

The Gopkg.lock serves exactly the same function as the glide.lock file which locks the version of the packages EXCEPT the version should be maintained in the Gopkg.toml. In short, the Gopkg.lock file is auto-generated and it depends on the import statements in the source with version controlled by Gopkg.toml.

Update dependency’s version

Let's edit the Gopkg.toml and use a slightly older version of the go-humanize package instead of the latest master branch.

# Gopkg.toml example
#
# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
#   name = "github.com/user/project"
#   version = "1.0.0"
#
# [[constraint]]
#   name = "github.com/user/project2"
#   branch = "dev"
#   source = "github.com/myfork/project2"
#
# [[override]]
#  name = "github.com/x/y"
#  version = "2.4.0"


[[constraint]]
  #branch = "master"
  name = "github.com/dustin/go-humanize"
  revision = "0b19b17f90333e44518aa31bbf8126017960aee3"

Then run dep ensure to update the package to the desired version. The following is the diff of the updated Gopkg.lock.

--- /home/ykyuen/go/src/gitlab.com/ykyuen/dep-example/Gopkg.lock
+++ # This file is autogenerated, do not edit; changes
@@ -2,14 +2,13 @@


 [[projects]]
-  branch = "master"
   name = "github.com/dustin/go-humanize"
   packages = ["."]
-  revision = "bb3d318650d48840a39aa21a027c6630e198e626"
+  revision = "0b19b17f90333e44518aa31bbf8126017960aee3"

 [solve-meta]
   analyzer-name = "dep"
   analyzer-version = 1
-  inputs-digest = "a5d65a2bdf47e41b99ad71813142ae93970f5806c77630b2aea665fe631dde23"
+  inputs-digest = "0023bfe634a061b89ae0fbd71e3236f3f75f0843a0c974eeb9822040c0ea2dc4"
   solver-name = "gps-cdcl"
   solver-version = 1

Add a new dependency

New package could be added using the dep ensure -add command.

[ykyuen@camus dep-example]$ dep ensure -add github.com/leekchan/accounting
Fetching sources...

"github.com/leekchan/accounting" is not imported by your project, and has been temporarily added to Gopkg.lock and vendor/.
If you run "dep ensure" again before actually importing it, it will disappear from Gopkg.lock and vendor/.

Now we have the new accounting package ready in the vendor folder with new constraints written to Gopkg.toml and locked in Gopkg.lock. Let's update the main.go as follow.

main.go

package main

import humanize "github.com/dustin/go-humanize"
import accounting "github.com/leekchan/accounting"
import "math/big"
import "fmt"

func main() {
  fmt.Println("hello world")
  fmt.Printf("That file is %s.\n", humanize.Bytes(82854982)) // That file is 83 MB.
  fmt.Printf("You're my %s best friend.\n", humanize.Ordinal(193)) // You are my 193rd best friend.
  fmt.Printf("You owe $%s.\n", humanize.Comma(6582491)) // You owe $6,582,491.

  ac := accounting.Accounting{Symbol: "$", Precision: 2}
  fmt.Println(ac.FormatMoney(123456789.213123))                       // "$123,456,789.21"
  fmt.Println(ac.FormatMoney(12345678))                               // "$12,345,678.00"
  fmt.Println(ac.FormatMoney(big.NewRat(77777777, 3)))                // "$25,925,925.67"
  fmt.Println(ac.FormatMoney(big.NewRat(-77777777, 3)))               // "-$25,925,925.67"
  fmt.Println(ac.FormatMoneyBigFloat(big.NewFloat(123456789.213123))) // "$123,456,789.21"
}

And run it.

[ykyuen@camus dep-example]$ go run main.go
hello world
That file is 83 MB.
You're my 193rd best friend.
You owe $6,582,491.
$123,456,789.21
$12,345,678.00
$25,925,925.67
-$25,925,925.67
$123,456,789.21

The issue with git submodule

One major difference of dep compared to Glide is the package's submodule is ignored. For example, after adding the go-goracle/goracle package by dep, the odpi submodule inside is empty and leads to error. The reason of dropping the submodule could be found in the following link.

Are there any plans to add git submodules support ?

Update @ 2018-02-03:

The paragraph about git submodule is incorrect.

Sam Boyer wrote:

dep should be perfectly fine at pulling in git submodules in the case you describe. I just replicated what you describe here locally, and the problem isn’t submodules — it’s that there’s no Go code in github.com/go-goracle/goracle/odpi, so it can’t be imported directly.

You likely need to turn off unused-packages pruning in Gopkg.toml for that project specifically, as otherwise dep ensure will automatically remove what appears to be an unused directly (but it seems it’s actually used by cgo).

Update @ 2018-03-04:

It is found that the go-goracle/goracle package doesn't work with dep. You could follow the issue below and check the latest update from the dep team.

Fail to get git submodule of a package after the dep ensure command

Summary

~~dep is quite likely to be the official dependency management tool in the Golang community.~~
~~If you are starting a new Golang project, dep is good to go.~~
~~If you are using Glide in a legacy project. You could consider migrating to dep but i think there is no harm to keep using Glide for a while until dep is officially released.~~
~~In addition, missing package's submodule may result in malfunction of your code.~~
dep is officially released.
dep works well on pulling git submodule.
You can checkout this example on gitlab.com.

Manage Go dependencies using Glide

Yuen Ying Kit — Tue, 16 Jan 2018 02:46:57 +0000

Originally posted on Boatswain Blog.

Update @ 2018-01-17

You might want to consider using a new dependency management tool called dep as suggested by nasa9084.

Glide is a Go package management tool, like npm in NodeJS. Let's create a simple example and see how it works.

Create the project inside $GOPATH

The project folder has to be under $GOPATH in order to resolve the Go package paths. In this example, the project root folder is $GOPATH/src/gitlab.com/ykyuen/glide-example.

A hello world Go program

Let's create a simple Go program.

main.go

package main

import "fmt"

func main() {
  fmt.Println("hello world")
}

Build and run

[ykyuen@camus glide-example]$ go build -o hello
[ykyuen@camus glide-example]$ ./hello
hello world

Add a Go dependency

dustin/go-humanize is a helpful Go package to format number, string and time. Let's update our Go program as follow.

main.go

package main

import humanize "github.com/dustin/go-humanize"
import "fmt"

func main() {
  fmt.Println("hello world")
  fmt.Printf("That file is %s.\n", humanize.Bytes(82854982)) // That file is 83 MB.
  fmt.Printf("You're my %s best friend.\n", humanize.Ordinal(193)) // You are my 193rd best friend.
  fmt.Printf("You owe $%s.\n", humanize.Comma(6582491)) // You owe $6,582,491.
}

Now if we try to build it, it will throws the following error.

[ykyuen@camus glide-example]$ go build -o hello
main.go:3:8: cannot find package "github.com/dustin/go-humanize" in any of:
  /home/ykyuen/tools/go1.8.4/src/github.com/dustin/go-humanize (from $GOROOT)
  /home/ykyuen/go/src/github.com/dustin/go-humanize (from $GOPATH)

This is because we haven't downloaded the dustin/go-humanize. Without using Glide, we could solve this build error by running the go get command.

go get github.com/dustin/go-humanize

But in this case, anyone who checkouts the project has no idea what packages are needed in order to build the program. If you have already run the go get command, delete the package before proceeding.

The Glide way

Create the glide.yaml

The glide.yaml could be generated by the glide create command. It will scan the Go src files and include all the dependencies needed. Here is the output of this example.

[ykyuen@camus glide-example]$ glide create
[INFO]  Generating a YAML configuration file and guessing the dependencies
[INFO]  Attempting to import from other package managers (use --skip-import to skip)
[INFO]  Scanning code to look for dependencies
[INFO]  --> Found reference to github.com/dustin/go-humanize
[INFO]  Writing configuration file (glide.yaml)
[INFO]  Would you like Glide to help you find ways to improve your glide.yaml configuration?
[INFO]  If you want to revisit this step you can use the config-wizard command at any time.
[INFO]  Yes (Y) or No (N)?
Y
[INFO]  Looking for dependencies to make suggestions on
[INFO]  --> Scanning for dependencies not using version ranges
[INFO]  --> Scanning for dependencies using commit ids
[INFO]  Gathering information on each dependency
[INFO]  --> This may take a moment. Especially on a codebase with many dependencies
[INFO]  --> Gathering release information for dependencies
[INFO]  --> Looking for dependency imports where versions are commit ids
[INFO]  No proposed changes found. Have a nice day.

glide.yaml

package: gitlab.com/ykyuen/glide-example
import:
- package: github.com/dustin/go-humanize

Download the Go dependencies to the vendor folder

Next, we execute the glide install command to download the dependencies.

[ykyuen@camus glide-example]$ glide install
[INFO]  Lock file (glide.lock) does not exist. Performing update.
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching updates for github.com/dustin/go-humanize
[INFO]  Resolving imports
[INFO]  Downloading dependencies. Please wait...
[INFO]  Setting references for remaining imports
[INFO]  Exporting resolved dependencies...
[INFO]  --> Exporting github.com/dustin/go-humanize
[INFO]  Replacing existing vendor dependencies
[INFO]  Project relies on 1 dependencies.

This will generated the glide.lock files and the vendor folder which contains the dustin/go-humanize package. The glide.lock file contains the version of the package. It could be a tag or a git commit hash.

glide.lock

hash: de8aada0e0453f13dc438db5fb412db797d701d6afeadcc052c477fd55e01aa8
updated: 2017-12-04T23:20:53.616704682+08:00
imports:
- name: github.com/dustin/go-humanize
  version: 8929fe90cee4b2cb9deb468b51fb34eba64d1bf0
testImports: []

Update dependency's version

The 8929fe9 commit is quite outdated. We could modify version in the glide.lock file and run glide install again to get the desired version of your package.

glide.lock

hash: de8aada0e0453f13dc438db5fb412db797d701d6afeadcc052c477fd55e01aa8
updated: 2017-12-04T23:20:53.616704682+08:00
imports:
- name: github.com/dustin/go-humanize
  version: bb3d318650d48840a39aa21a027c6630e198e626
testImports: []

[ykyuen@camus glide-example]$ glide install
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Found desired version locally github.com/dustin/go-humanize bb3d318650d48840a39aa21a027c6630e198e626!
[INFO]  Setting references.
[INFO]  --> Setting version for github.com/dustin/go-humanize to bb3d318650d48840a39aa21a027c6630e198e626.
[INFO]  Exporting resolved dependencies...
[INFO]  --> Exporting github.com/dustin/go-humanize
[INFO]  Replacing existing vendor dependencies

Build and run again

[ykyuen@camus glide-example]$ go build -o hello
[ykyuen@camus glide-example]$ ./hello
hello world
That file is 83 MB.
You're my 193rd best friend.
You owe $6,582,491.

Check in the vendor folder?

For NodeJS projects there is a common practice that we will ignore the node_modules folder in the code repository. On the other hand, the Go community has a different point of view.

Should I add "vendor" directory into .gitignore if I am using tools like glide or godep?

There isn't an absolute answer. Pick the option which you feel more comfortable.

Other concerns

When the project is getting bigger, glide might take quite a while to work as the dependency tree is getting bigger and bigger. In addition, it's hard to manage to the dependency's version.

As nasa9084 suggested, you might want to consider using a new dependency management tool called dep as suggested by the Glide team. dep has a growing popularity and in active development. Do take a look on the dep's project status and pick the one which you feel more comfortable.

Summary

Project folder has to be under $GOPATH.
Dependencies would be included automatically by scanning all the source code.
Dependency's version is managed by glide.lock.
Commit the vendor folder or not is still an open question.
Performance issue when dependency tree getting bigger.
Complete example is available on gitlab.com.