Forem: Davinder

Automate Nginx conf file validation using Docker

Davinder — Wed, 04 Dec 2019 10:00:00 +0000

Nginx is an open-source web server, HTTP cache, reverse proxy server, and load balancer and is used by over 50% of the world’s busiest websites. And Docker with Nginx is a perfect combination to automate your app deployment.

Automated testing is an integral part of CI/CD pipelines. It would be nice to have one more check for the Nginx config file.

Automating Nginx configuration file validation using docker.

docker run -v $pwd/config:/data/ --name check-nginx-conf -t -a stdout nginx:latest nginx -t -c /data/nginx.conf

What’s happing here:

-v $pwd/config:/data/ : bind your $pwd/config to /data inside the container. Make sure you have nginx.conf inside config folder and other directories that your config file is using ext SSL folder.
—name check-nginx-conf : Name of the container.
-t -a stdout : Attach to standard stream stdout to get the logs.
nginx:latest : Nginx Image version.
nginx -t -c : -t to validate the config file and -c to use an alternative configuration file instead of a default file.

Your should see this output if your config is valid

Further, you could use grep command to check if test is successful

docker run -v $pwd/config:/data/ --name check-nginx-conf -t -a stdout nginx:latest nginx -t -c /data/nginx.conf | grep "test is successful" | wc -l

grep “test is successful” will return the line which contains test is successful and wc -l to count the number of lines.

if the syntax is valid, you will get 1 otherwise 0

Access Docker Using Rest API

Davinder — Mon, 05 Aug 2019 12:46:39 +0000

Most of the time we use Docker CLI to manage Docker Objects such as containers, images, volumes, and networks. But we Also can use Rest API to Manage Docker objects.

Originally published at https://devkamboj.in.

Docker CLI also uses the Docker rest API to manage Objects. As you can see in the above image. But it is also possible to use the Rest API Directly.

The Docker daemon can listen for Docker Engine API requests via three different types of Socket: unix, tcp, and fd.

By Default, unix socket is enabled. This does not help if you want to access the Docker Rest API remotely. Because unix socket works for exchanging data between processes executing on the same host operating system.
If you need to access the Docker daemon remotely, you need to enable the TCP Socket.

sudo dockerd -H tcp://0.0.0.0:2375

You can listen on port 2375 on all network interfaces with -H tcp://0.0.0.0:2375

There is also another way by using systemd

cd /lib/systemd/system
vi docker.service
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375
#just add -H tcp://0.0.0.0:2375 and restart docker

Now you can start using the api

curl -X GET http://ip:4243/containers/json
[{
  "Id":"fff8c7927e19d5a7da280424a09572729641137a8f42e116c3073c72da2978d3",
 "Names":["/rockey"],
  "Image":"busybox",
  ...
}]

But if you do not want to access Rest API remotely then you do not need to enable TCP. Then you can just use unix socket.

List containers that are running same like docker ps

$ curl --unix-socket /var/run/docker.sock http:/v1.24/containers/json
[{
  "Id":"ae63e8b89a26f01f6b4b2c9a7817c31a1b6196acf560f66586fbc8809ffcd772",
  "Names":["/tender_wing"],
  "Image":"bfirsh/reticulate-splines",
  ...
}]

You can use Python and Go as well here is the Example:

Python

import docker
client = docker.from_env()
for container in client.containers.list():
  print container.id

Go

package main

import (
    "os"

    "github.com/docker/docker/api/types"
    "github.com/docker/docker/api/types/container"
    "github.com/docker/docker/pkg/stdcopy"
    "github.com/docker/docker/client"
    "golang.org/x/net/context"
)

func main() {
    ctx := context.Background()
    cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
    if err != nil {
        panic(err)
    }

    reader, err := cli.ImagePull(ctx, "docker.io/library/alpine", types.ImagePullOptions{})
    if err != nil {
        panic(err)
    }
    io.Copy(os.Stdout, reader)

    resp, err := cli.ContainerCreate(ctx, &container.Config{
        Image: "alpine",
        Cmd:   []string{"echo", "hello world"},
        Tty:   true,
    }, nil, nil, "")
    if err != nil {
        panic(err)
    }

    if err := cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil {
        panic(err)
    }

    statusCh, errCh := cli.ContainerWait(ctx, resp.ID, container.WaitConditionNotRunning)
    select {
    case err := <-errCh:
        if err != nil {
            panic(err)
        }
    case <-statusCh:
    }

    out, err := cli.ContainerLogs(ctx, resp.ID, types.ContainerLogsOptions{ShowStdout: true})
    if err != nil {
        panic(err)
    }

    stdcopy.StdCopy(os.Stdout, os.Stderr, out)
}

There are a number of libraries available for other languages.

Language	Library
C	libdocker
C#	Docker.DotNet
C++	lasote/docker_client
Dart	bwu_docker
Erlang	erldocker
Gradle	gradle-docker-plugin
Groovy	docker-client
Haskell	docker-hs
HTML (Web Components)	docker-elements
Java	docker-client
Java	docker-java
Java	docker-java-api
NodeJS	dockerode
NodeJS	harbor-master
Perl	Eixo::Docker
PHP	Docker-PHP
Ruby	docker-api
Rust	docker-rust
Rust	shiplift
Scala	tugboat
Scala	reactive-docker
Swift	docker-client-swift

Now you can create your own User Interface by using Docker Rest API. This API could help you in automating stuff for your Apps.

JavaScript Parser and Engine

Davinder — Thu, 01 Aug 2019 08:50:19 +0000

In this article, I will just very quickly tell you. How the JavaScript codes that we write is actually executed.

Originally posted at

So, JavaScript is always hosted in some environment. And that is most typically a browser such a Google Chrome, Firefox, Safari, etc. This is where JavaScript runs. There can also be other hosts such as the NodeJS web server, or even some applications that accept JavaScript code as input. However, I will just focus on the browser in this Article.

So when we write our JavaScript code and actually want to run it, there’s a lot of stuff happening behind the scenes. So what happens is that the host where JavaScript is hosted has some kind of JavaScript engine that takes our code and executes it. So in very simple terms, a JavaScript engine is a program that executes JavaScript code. There are many different engines out there, like

Google’s V8 engine, that is used in Google Chrome.
SpiderMonkey is Mozilla’s JavaScript engine
JavaScriptCore and many more.

Now, the first thing that happens inside the engine, is that our code is parsed by a parser, which basically reads our code line by line, and checks if the syntax of the code that we gave it, is correct. So this means that the parser knows the JavaScript rules and how it has to be written in order to be correct, to be valid. And if you make some mistakes, it basically throws an error and stops the execution.

If everything is correct though, then the parser produces a data structure known as the Abstract Syntax Tree, which is then translated into machine code. So this code is no longer JavaScript code, but a code, or let’s say a set of instructions, that can be executed directly by the computer’s processor. And it’s only when our code already converted to machine code, actually runs and does its work.

I just want you to get the basic idea of what actually happens once we choose to run our code. And there is actually way more going on behind the scenes. And different engines do things in a slightly different way.

Secure Docker : Enable Docker Content Trust

Davinder — Thu, 01 Aug 2019 08:22:47 +0000

I love Docker and Security so I thought why not write about Docker Content Trust. so here you go...

Originally posted at

Docker Content Trust allows you to have an image
signing so you can make sure that you're
not running arbitrary code you're
actually running the code that you wanted
to run in the first place.

When Docker trust is Enabled And publisher pushes the image to a remote registry, Docker Engine signs the image locally with the publisher’s private key. When someone pulls this image, Docker uses the publisher’s public key to verify the image integrity and checks image is exactly what the publisher created, has not been tampered with, and is up to date.

Enabling Content Trust

Enable via Environment Variable

  $> export DOCKER_CONTENT_TRUST=1

Or CLI Flag:

  $> docker pull --disable-content-trust=false mongo:tag

As soon as you enable docker content trust using Environment Variable

  $> export DOCKER_CONTENT_TRUST=1

Every docker single docker operation is secure like

docker push
docker run
docker pull

Every single operation is now gonna have to operate on the signature. if you try to run unsigned image docker will not allow you.

If you are creating a docker image of your application and pushing it to Repository and pull it over the network. Then you should enable content trust.

Steps to Sign Docker Images.

enable docker content trust

    $> export DOCKER_CONTENT_TRUST=1

Push image

  $> docker push  [yourImage:tag]

If you are doing it first time it will generate root key, repo key and other keys.(By default this is stored in ~/.docker/trust/). you need to enter the passphrase for root key and repo key. After giving the passphrase it will push the signed image.

You should back up the root key somewhere safe. Given that it is only required to create new repositories, it is a good idea to store it offline in hardware.
manage keys for DCT.

From now on docker push operation will prompt you to enter the passphrase of your repo. if the content trust is enabled. but this can be automated.

Automation with content trust

Most of the time we build our images using the automation system. To enable content trust on our automation system we can use the Environment variable.

$> export DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE="strongpassword"

if you use this variable then you do not need to enter the passphrase every time.

Docker Content Trust is available in Docker CS Engine 1.9.0 and Docker EE. May be Docker content trust will be enabled by default in the future.

What Is Container Orchestration?

Davinder — Tue, 30 Apr 2019 10:00:00 +0000

Container orchestration is a process that automates the deployment, management, scaling, networking, and availability of container-based applications

If your application relies on other containers such as databases or messaging services or other backend services? What if the number of users increases and you need to scale your application? How do you scale down when the load decreases? To enable these functionalities, you need an underlying platform with a set of resources and capabilities. The platform needs to orchestrate the connectivity between the containers and automatically scale up or down based on the load. This whole process of automatically deploying and managing containers is known as Container Orchestration. Kubernetes is just a container orchestration technology. There are multiple such technologies available today. Docker has its own tool called Docker Swarm , Kubernetes from Google and MESOS from Apache.

While Docker Swarm is really easy to set up and get started, it lacks some of the advanced features required for complex applications.

MESOS, on the other hand, is quite difficult to set up and get started but supports many advanced features.

Kubernetes arguably the most popular of it all, is a bit difficult to set up and get started by provides a lot of options to customize deployments and supports the deployment of complex architectures. Kubernetes is now supported on all public cloud service providers like GCP, Azure, AWS, and Kubernetes project is one of the top-ranked projects in Github.

There are various advantages of Container Orchestration: Your application is now highly available as hardware failures to not bring your application down because we have multiple instances of your application running on different nodes.

The user traffic is load balanced across the various containers. When demand increases, deploy more instances of the applications seamlessly and within a matter of seconds. And we have the ability to do that at a service level when we run out of hardware resources, scale the number of underlying nodes up or down without having to take down the application and do all of these easily with a set of declarative object configuration files. And that is Kubernetes. It is a container orchestration technology used to orchestrate the deployment and management of hundreds and thousands of containers in a clustered environment.