Forem: Shriharsh

Today I Learned: A simple trick to slim down a Docker image

Shriharsh — Thu, 08 Feb 2018 12:38:45 +0000

You may want to reduce the space used by your image. If you are building on top of an Ubuntu image and during the build, you are invoking apt-get update (you would be for sure if you are installing latest versions!) then you can delete some files downloaded by apt-get that won't be required later. The following Dockerfile snippet shows a way how it should be done. Notice that RUN instruction is doing 3 things, update the package cache, installing ant, and deleting the cache. This is done in such a ways so that Docker does all of this in a single layer of the image. If these steps are split in different RUN instructions then your Docker host and/or registry will use more space.


FROM jenkins/jenkins:lts

#Change user to root and install ant version 1.9.9-1 (available in Ubuntu packages)
USER root

#
RUN apt-get update && \
        apt-get install -y ant=1.9.9-1 && \
        rm -rf /var/lib/apt/lists/* # remove the cached files.

In my case, it saved about 138M of space. However, YMMV.

By the way, what are you deleting? You are only deleting the information about latest packages and versions available for download using apt-get install or apt-get upgrade. apt-get caches this information later use. It is highly likely that you wouldn't need this information when you are running the containers based on this image. Installing software while running the container would be a big NO!.

Similar, cache files would exist in other Linux distributions as well.

I hope this helps you slim down your ubuntu images.

"Behind the Corporate Proxy"

Shriharsh — Wed, 17 Jan 2018 21:59:55 +0000

Over the years, I have struggled to get past the corporate proxy, i.e. access to the internet was blocked because proxy details weren't set up properly. Usually, most software documentation wouldn't provide these details handy and one has search thru documentation. These are useless battles a developer has to fight in order to win the war.

I wish I could do what Liam Neeson does! But I have been able to win some of my battles and have accumulated some ammunition for a short post which might come handy to many in a similar situation. Here I list down some ways to set up proxy details in the following scenarios:

Running Vagrant behind the proxy

Set up the proxy in your guest OS

This depends on the guest OS. I use Ubuntu, so I find updating /etc/environment file with proxy details as an easy option.

    > cat /etc/environment 
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
    HTTP_PROXY=http://proxy_host:port
    HTTPS_PROXY=http://proxy_host:port
    FTP_PROXY=
    NO_PROXY=localhost,127.0.0.1
    http_proxy=http://proxy_host:port
    https_proxy=http://proxy_host:port
    ftp_proxy=
    no_proxy=localhost,127.0.0.1

Use vagrant-proxyconf plugin

With this plugin, you can setup the proxy details right into the Vagrantfile. To install the plugin run the following command on your shell:

> vagrant plugin install vagrant-proxyconf

If you are not able to download the plugin behind the corporate proxy (duh!), then you can download the gem file from Ruby Gems site and use the following command to install the plugin.

> vagrant plugin install /path/to/my-plugin.gem

and add following to the Vagrantfile:

Vagrant.configure("2") do |config|
  if Vagrant.has_plugin?("vagrant-proxyconf")
    config.proxy.http     = "http://<your-proxy-host>:<port>/"
    config.proxy.https    = "http://<your-proxy-host>:<port>/"
    config.proxy.no_proxy = "localhost,127.0.0.1"
  end
  # ... other stuff
end

Now you should be able to access the internet from your vagrant guest.¹

Running Docker behind proxy

When you are working on docker behind the proxy, it becomes important that you ensure proxy settings are passed and are available at different stages - for example, image creation, container execution, and running services. There are different command options for either of these scenarios.

Setting proxy while building an image

You might want to download some software while building your image so that every container has it pre-installed. While you can hardcode the proxy details as environment variables in the Dockerfile, it isn't the recommended way of doing so. It hampers the portability of the image and unnecessarily gives the container created by the image access to the internet. The better option would be to pass the proxy information while building the image using build-arg option as shown below:

> http_proxy=<yourproxyserverhost>:<port>
> https_proxy=<yourproxyserverhost>:<port>
> docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy -t <your-image> .

The build-arg option would allow you to pass a value to an environment variable defined in your Dockerfile using ARG instruction. However, it is a bit easier to pass proxy details. For proxy details, Dockerfile allows you to skip mentioning the proxy environment variable via ARG instruction in the Dockerfile.²

Setting up proxy when running the container

Again as mentioned above, it isn't advisable to setup proxy details in the image or to enter each new container to setup proxy details. Docker provides a better way to do so using the -e or --env options with the run command. This option can be used to pass the proxy setting as environment variables to the container.

> docker container run -e http_proxy nginx #..........................(1)
> docker container run -e https_proxy=<proxyhost:port nginx #.........(2)

Notice the difference between two commands above. If http_proxy is already defined on the docker host, we can directly pass its value to the container by just passing the name of the variable to -e options as shown on line#(1) above. If you want to provide an alternate value you can pass name=value pair as shown in line#(2) above. Finally, you can also set up all the proxy variables in a file and use the --env-file option to pass all of them at once.

> cat all-proxy-vars.txt 

http_proxy=http://proxy_host:port
https_proxy=http://proxy_host:port
ftp_proxy=
no_proxy=localhost,127.0.0.1

> docker container run --env-file ./all-proxy-vars.txt nginx

Environment options on the Docker CLI commands offers quick way tweak and test an image. ³

Setup proxy for Docker Compose - The easiest thing.

While the steps described above will come handy when you are experimenting with your images and containers. Once you have finalized your container and image details, the best approach would be to create a compose file for your service which would build the image. Docker compose allows you to pass the environment details easily using the args instruction. You just need to mention the name of the proxy environment variables as shown below:

version: '2.3'

services: 
    ngnix-service:
        image: my-nginx-image
        build: 
            context: .
            args: # Environment variables available at build-time
                - http_proxy
                - https_proxy
                - no_proxy
        environment: # Environment variables available at container run-time
            - https_proxy
            - http_proxy
            - no_proxy
        volumes: 
.........

As noted in the code snippet, environment variables defined as build > arg level are only available when image is being built. Whereas, environment level list of environment variable is available at container run-time.

So these are the corporate proxy battles that I have fought and won somehow. A long time back, I had a hard time getting thru an NTLM-based proxy. But a cool tool called CNTLM saved my day. However, I don't have the scenario or the steps handy to add to this article.

One thing is for sure, once you have resolved such issues, you can quickly identify a similar issue with a new software. I would add to this article when I fight and win more such battles. I hope you this post helps you save some time.

Detailed instruction on Vagrant-proxyconf plugin can be found here. http://tmatilai.github.io/vagrant-proxyconf/ ↩
The official Docker documentation explains how to use build-args https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables-build-arg ↩
Environment options on the Docker CLI commands. https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e-env-env-file ↩

My Shell Workflow: Executing a command in lots of directories.

Shriharsh — Wed, 03 Jan 2018 14:52:07 +0000

Problem

You want to run a shell command in many directories. For example, in my case, I wanted to execute a shell command in lots of directories(approx. 200) to clean up build files.

Solution

Writing a for-loop and running it on the shell did the trick for me.

for d in ./*/ ; do (cd "$d" && mvn clean); done

Explanation

The above for-loop would run cd "$d" && mvn clean in each item of the directory list defined by the path ./*/. $d holds the directory name. You can be creative in defining the list of directories using a complex glob as well.

I hope this helps you as well.

"My Shell Workflow: Using bash history"

Shriharsh — Wed, 22 Nov 2017 16:39:24 +0000

Working on the shell with your hands tied :|

While I try to use my vagrant box for work as much as possible, there are times when I have to connect to server machines over ssh and not have my favorite z-shell. It was getting irritatingly boring to repeat the same set of command every time. One option is to create my own aliases but I will have to do so everytime I login. Our sysadmin has (strangely) ensured that everyone sudo's to a common user to do any editing work.

Finding my way out :)

So I use the following set of commands to ensure that I am not getting bogged down by the process.

Suppose I want to search a command that I have previously used. I type history | grep <command-snippet> which gives me a numbered list when I have used the command earlier. Then I use the number to run the same command - !<number> again. See below -

> history | grep vim
2795  vim scratch-pad.txt
2820  vim .git/config
2822  vim .git/config
2845  sudo vim /etc/hosts
2847  sudo vim /etc/hosts

> !2820
> vim .git/config

This might still be tedius. So the other option is to search history without calling history command.

The Savior `Ctrl+R` :D

On the bash terminal, press Ctrl+R to open reverse search dialog.
Type the snippet of the command you would have executed earlier.
Press Ctrl+O to execute the command. or Press Ctrl+Space to print the command for editing before executing.

(reverse-i-search)`install': sudo apt-get install ant #After pressing Ctrl+O
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
.... 


(reverse-i-search)`zsh': sudo apt-get install zsh #After pressing Ctrl+Space
> sudo apt-get install zsh

A nifty side-effect of using Ctrl+R on the bash prompt is that once the chosen command is completed, the next command from history is printed on the prompt for us to use if needed. Most of the time, I am restarting my server and tailing the logs. So using this option automatically returns the next command (tail -f) after I execute the restart command.

I hope this helps you as well.

Today I Learned: To use custom ClientHttpRequestFactory with MockRestServiceServer

Shriharsh — Mon, 13 Nov 2017 15:07:11 +0000

Despite having some reservations about using Spring RestTemplate for writing JAX-RS client (see here), I chose to use Spring RestTemplate to implement a JAX RS client as I liked that I could write test cases by easily mocking the REST service I was consuming. Spring provides a very nice utility to mock REST service end-point: MockRestServiceServer. However, I ran into an pesky issue.

Problem

I developed my code along with the test cases using the above nice utility by Spring framework. Next step was to log all incoming and outgoing traffic. I was able to implement the feature by registering a custom LoggingInterceptor along with the BufferingClientHttpRequestFactory. The logging worked fine but my test cases started to fail. Although I ensured that I use BufferingClientHttpRequestFactory so that the response from the server can be read twice - once for logging and next by the application code, the test code would fail on the second read.

Diagnosis

I found that the MockRestServiceServer would override the BufferingClientHttpRequestFactory setup on the RestTemplate instance passed to it while creating the server.

@Override
public MockRestServiceServer build(RequestExpectationManager manager) {
    MockRestServiceServer server = new MockRestServiceServer(manager);
    MockClientHttpRequestFactory factory = server.new MockClientHttpRequestFactory();
    if (this.restTemplate != null) {
        this.restTemplate.setRequestFactory(factory);
    }
    if (this.asyncRestTemplate != null) {
        this.asyncRestTemplate.setAsyncRequestFactory(factory);
    }
    return server;
}

Solution

On scouring further, I found that the there is a hack to work around this issue. Once the server is created, read the requestFactory instance created by the MockRestServiceServer using ReflectionTestUtils on RestTemplate instance and wrap it up with BufferingClientHttpRequestFactory.

@Before
public void setUp() throws Exception {
    mockServer = MockRestServiceServer.createServer(restTemplate);

    final ClientHttpRequestFactory requestFactory = 
        (ClientHttpRequestFactory) ReflectionTestUtils.getField(restTemplate, "requestFactory");

    restTemplate.setRequestFactory(new BufferingClientHttpRequestFactory(requestFactory));
}

I hope this helps you as well.

Today I Learned: About Docker Volume

Shriharsh — Fri, 27 Oct 2017 11:51:03 +0000

In my experiements on running Jenkins on Docker, I wondered how the volumne worked when I provided -v to the run command.

I found a great article which succintly explains a lot about the Docker Volume - which will be handy for any docker beginer.

Original article is here

My summary goes below:

Docker Volume is the way to persist data between container restarts and share it between containers.
A volume is stored on the host file system. Usually within obscure directories inside /var/lib/docker. However, a specific directory on host system can also be mounted as volume in a container.
Volumes can be managed using docker volume and its sub commands.
Docker won't delete volume unless specifically instructed to do so. This can result in stale data on the host system. So tidying up once in a while might come handy.
Important Caveat - Don't try to modify the content of the volume in the Dockerfile (via RUN) as each build step creates a new volume and your changes will be discarded. If you want to change the volume content, do so using the ENTRYPOINT command in the Dockerfile. Have a look at this link.
If you are looking for a thorough overview of different approaches to manage application data, including the use of Volume, Bind mounts, tmpfs mounts, read this longer article

Image Credit: https://docs.docker.com/engine/admin/volumes/images/types-of-mounts-volume.png

Today I Learned: Some VIM tricks

Shriharsh — Wed, 18 Oct 2017 08:41:24 +0000

VIM Trick 1 - Create new file using NERDTree plugin

Many VIM users depend on NERDTree plugin to browse directory but rare use it to create a new file on without leaving Vim (at least I was one such Vim user).

I found nice blog entry which told me how to do so - Link here

To summarize -

Open NERDTree panel
Navigate to the directory where you want to create new file or directory.
Press m - which opens up NERDTree Filesystem Menu.
Type a to create the child node and type the name of file or directory. For directory append / at the end.
That's it.

VIM Trick 2 - Create new file using VIM built-in feature.

While I was experimenting on the steps above I made a silly mistake. I didn't open the newly created file before typing my experiment with NERDTree. So I typed my content in an un-named buffer. To wiggle out, I decided to rely on VIM help. This is what I did -

Type :h save - this opened up the help window with VIM
The saveas command in VIM helps us same the current buffer into a file.
Use ! option to save to an existing file.
I will let the reader experiment with this with their own hands-on exercise.

Comparing Spring’s RestTemplate and Jersey’s Client APIs

Shriharsh — Tue, 17 Oct 2017 14:34:37 +0000

Recently I had an opportunity to use two different REST libraries to write client side code for consuming REST web services. The libraries were Spring RestTemplate (spring-web, 4.3.10.RELEASE) and Jersey Client (jersey-client, 1.18). Although the latter is an older version, it was much easier to use it. My goals were very simple and specifically the following:

Invoke different HTTP methods on the service
Setup Basic Authentication on the client code.
Log request and response.
Handle the HTTP error responses.
Write test cases which mock the server.

Spring provides RestTemplate API to invoke various HTTP methods on a REST Web Service. Although there are methods dedicated to most HTTP method such as getForObject GET, postForObjectPOST, PUT etc., the methods don't provide full control over what is to be sent out. So you have to rely on the generic methods such as exchange or execute.When using these method one has to supply parameter such as HttpEntity even when you don't want to send any and this ambiguously named class is used to setup HTTP headers on the request (some examples on this thread).

The API provides ways to build the URI dynamically with simple templating for path param and HTTP parameters. However, the approach is clunky as there are two approaches to achieve data binding on the URI template. One is using a map which would be used to replace the key in the URI template. Second is by passing var args to methods in the same sequence as the placeholders appear on the URI templates. Both the approaches require that HTTP parameter name is mentioned on the URI template. Either of the approaches can lead to a messy code where it can be difficult to track on which parameter which value is being set.You can see the examples on this thread. On the other hand, Jersey client provides a fluent API to implement the client code. The API is very intuitive and doesn't require any specific order in which one must provide data for the URI template.The query and path parameters can be easily supplied using methods on the WebResource API. Some more examples on this page.

Setting basic authentication using either of the libraries wasn't difficult. Spring requires us to setup an interceptor on the RestTemplate instance whereas Jersey requires adding filter class. Both the libraries provide built-in implementations which can be extended if required. Spring provides BasicAuthorizationInterceptor and Jersey provides HTTPBasicAuthFilter.

Logging request and response is an important feature for investigation. Like basic authentication, this feature is to be implemented using an interceptor in Spring and a filter in Jersey. Jersey gives a very handy implementation of such as filter called LoggingFilter which can help in logging all kinds of incoming and outgoing traffic. Spring requires you to implement your own interceptor. But that's not enough, you also need to configure the RestTemplate instance to use BufferingClientRequestFactory which helps in reading the response from input stream twice - once for logging and another for application code. Sample implementation can be seen here. The sample implementation of LoggingFilter in Jersey has this aspect built-in. So Jersey gives us a headstart.

Spring RestTemplate approach towards handling HTTP status codes which are not in the 2xx range isn't user-friendly in my opinion. Spring will throw an exception - HttpClientErrorException or HttpServerErrorException if it doesn't see successful response. This forces developer to implement a try-catch block to handle the error or register an error handler with RestTemplate. Whereas Jersey's approach is to hand over the ClientResponse instance and it is up to the developer to decide how to treat the response i.e. more freedom.

Finally, let's get into writing test cases. The first thing that is crucial here is to mock server response. Spring is the clear winner here. It provides a very handy set of APIs to mock a REST web service response which allows us to focus on testing the business logic. Have a look at MockRestServiceServer.With Jersey, you have to rely on mocking APIs like Mockito to mock Jersey client APIs which is very distracting and boring.

Considering how easy I found using Jersey to write code that consumes a REST web service, I would say Jersey win in comparison.Spring framework is open enough give us libraries which lets us use Jersey along with Spring. So even if you are working on a Spring based application you can safely use Jersey for your JAX-RS needs.

Disclaimer: I am writing this blog to share my experience in a fairly simple use case. So YMMV. Spring lovers may disagree with me. There are other libraries which such as RESTEasy which I haven't explored. You may share your experience/opinion so that I can learn from your experience too.

Forem: Shriharsh

Today I Learned: A simple trick to slim down a Docker image

"Behind the Corporate Proxy"

Running Vagrant behind the proxy

Set up the proxy in your guest OS

Use vagrant-proxyconf plugin

Running Docker behind proxy

Setting proxy while building an image

Setting up proxy when running the container

Setup proxy for Docker Compose - The easiest thing.

My Shell Workflow: Executing a command in lots of directories.

Problem

Solution

Explanation

"My Shell Workflow: Using bash history"

Working on the shell with your hands tied :|

Finding my way out :)

The Savior Ctrl+R :D

Today I Learned: To use custom ClientHttpRequestFactory with MockRestServiceServer

Problem

Diagnosis

Solution

Today I Learned: About Docker Volume

Today I Learned: Some VIM tricks

VIM Trick 1 - Create new file using NERDTree plugin

VIM Trick 2 - Create new file using VIM built-in feature.

Comparing Spring’s RestTemplate and Jersey’s Client APIs

The Savior `Ctrl+R` :D