[Devops / Docker] how to manage files and volume with docker

Laurent Tardif — Thu, 27 Aug 2020 09:07:25 +0000

When starting using docker, lot of people store file either in the container instance itself either on the file system of the server. in this post, will see the different option to deals with files, and we’ll discuss in which case it’s a good approach.

With docker, like in general, there’s several way to manage files :

directly in the container
via a mount point on your server
data container
using a docker volume
using a network file system (amazon S3, Hadoop, …) .

Files stored directly in the container :

The main advantage of this solution is it’s simplicity.

But there are several drawbacks:

your container is not persisted, so, when your container crash, or is removed, you may loose your files. In some case it’s not important : temporary files, demo, files generated by a script, ….
Performance, as docker use union fs, when reading and writing files it may not be as fast as expected.
file sharing : if you want to share the file among several process, either you have several process in your container (not very good), either you may link strongly two containers (or have them in the same pod if you use K8S), that’s may not be very good too.

Files mount on the host :

One of the advantage of this solution is also simplicity, and it solves partially the problem of sharing a file among container (as long as thy run on the same server).

This solution have some drawback too:

your container need to run on this server,
the container need to have access in write (or read) to your host, it may not be secure.
that’s to be avoid in production, because you create a hard link between the container and the host, and you loose some of the advantage of the isolation concept.

Files store in a data container :

Some people do that, …, I still not understand why doing that when you have volumes….

File in a cloud provider storage :

It may looks like a perfect solution if your containers run in the cloud, but you create a direct link between the container and the cloud provider. In some case you may not want that. So, use it carefully, being cautious of what your are doing. Using a volume may offer your the abstraction needed.

Files in a docker volume :

This solution may be a bit complex for beginner, as it introduce a new concept, but in fact it’s pretty efficient. The full documentation is available here. You can see it , as creating a new logical disk dedicated for your need, where you can configure the size, the name, the filesystem used.

The main advantages of using a volume are :

it’s ease file sharing among containers
it doesn’t make any link to the server where the container and the volumes are stored.
it’s allow to scale, as you can define the filesystem you want inside it, you may even imagine to have a distributed file system like GlusterFS. (you may have a look to all driver available here) . You can even use directly cloud provider solution. So, it’s my favorite solution, as it’s offer also an abstraction level between your container and the file system.

Hope it give you some hint on how to manage files, i will try to give some performance metrics soon.

Why to we need to share a file in the docker world ? I will give some common use case see on several projects :

As a container should run only one process, when you start an application, and you need to generate a configuration file at the start up (configuration file that may change over the time because you are using consul, or etcd, or vault ..). So you have a daemon process that scan change , and that update the configuration file, and then that will notify the application to read it. Having two processes in your document will be a mess, as if one of the two process die, you may be in a strange situation. A way to manage it, is too have have two side car container sharing a volume.
If you want to push your logs in a central place (ELK, ….), you may use the same side container concept.
Database files, you may store your files in a dedicated volume, that will allow you to start another psql process, using the volume in read-only mode, this will give you access to the data, without taking any risk with the process running your DB.

[Devops / Docker] How to write a good dockerfile

Laurent Tardif — Fri, 21 Aug 2020 13:37:20 +0000

Reading or writing a dockerfile, as soon as you get a bit familiar with the syntax is quiet simple. But how many time you spent some time wondering why your image is so big, or why the image you download was so big, how many time lost trying to find the port, the volume defined, have you loose. So, writing a good docker file is not so easy. I’ll try to explain the process that we have defined with Jules

If you want some kind of introduction to docker, have a look to Aurélie's post

Understanding Docker: part 1 – Retrieve & Pull images

Aurélie Vache ・ Aug 3 '20

#docker #devops #beginners #cloud

First of all, what’s a good docker file ? is it a secure one ? , a readable one ?, one that generate a small docker image ? all that ? that’s a challenge.

In this article, we’ll not focus on security aspect, but we’ll explain how to write a readable docker file, easy to maintain and that will generate a small docker image.

Let’s start with a simple example :

FROM node  
WORKDIR /usr/src/app  
COPY babel.config.js ./  
COPY package.json ./  
COPY yarn.lock ./  
COPY public ./public  
COPY src ./src  
RUN yarn  
RUN yarn lint  
RUN yarn build  
RUN yarn serve

if we docker build this image, we can notice several point :

it may download a new version of the referenced image “node”
it create several layers. A layer is an immutable set of files that can be shared among several docker images for performances and disk usage optimization. If you use several images having the same root image, you will have the files of this images present only once on your disk.

if you rebuild this image, without any modification, you will notice that docker is using a cache mechanism. To give some hints, if nothing change on your disk (for example, you do not change the package.json) it will not redo the step (and if all the previous steps have not changed). In summary, docker will do the minimal stuff needed.

So, let’s take this 2 points in account.

FROM  node: **10.15.0-Alpine**  
WORKDIR /usr/src/app  
COPY  “babel.config.js” “package.json” “yarn.lock” ./  
COPY public ./public  
COPY src ./src  
RUN yarn  
RUN yarn lint  
RUN yarn build  
RUN yarn serve

So, to avoid a random node image being picked, i defined a strict version for it. And to avoid useless layer, i defined a layer with the 3 configurations files.

Now that we have reduced the two first “issues”, let’s take advantage of the cache mechanism.

FROM  node:10.15.0-Alpine  
WORKDIR /usr/src/app  
COPY “babel.config.js” “package.json” “yarn.lock” ./  
RUN yarn  
COPY public ./public  
COPY src ./src  
RUN yarn lint  
RUN yarn build  
RUN yarn serve

So, I move up the “RUN yarn” just after the copy of the configuration file. Like that, as long as the configuration files do not change, my “RUN yarn” will not be executed, and I’ll benefit of the docker cache mechanism.

Now, I’ll not run directly use yarn to run my application. Let’s set up quickly an nginx server in front of it. So, I’ll use the multi stage feature of docker. It’s allow me to reuse the result of a docker image in an another one, define in the same dockerfile. Let’s do it.

FROM  node:10.15.0-Alpine **AS builder**  
WORKDIR /usr/src/app  
COPY “babel.config.js” “package.json” “yarn.lock” ./  
RUN yarn  
COPY public ./public  
COPY src ./src  
RUN yarn lint  
RUN yarn build  
RUN yarn serve 

# Volume inherited from nginx image  
# VOLUME /usr/share/nginx/html  
# VOLUME /etc/nginx   

FROM nginx:1.12.1-alpine  
EXPOSE 80    
HEALTHCHECK –interval=5m –timeout=3s -CMD curl -f http://localhost/ || exit 1   
COPY –from=builder /usr/src/app/dist/ /usr/share/nginx/html/

In order to do it, I’ve defined a logical name to my first image (I named it “builder”, it will allow me to refer to it in the second image).

Then I defined a new image, from nginx, and I copy the website content from the first image. So like that, my nginx image is minimal and contains only mandatory stuff (from a security stuff, the less you have, the better it is).

You can notice I’ve define the expose and healtheck in the top of my docker image definition. Because it a kind of API of my image, i make it quickly visible to the reader. You can access my image on port 80, there’s a check that insure the nginx server respond, the run and the copy element are a kind of implementation, and only few people want to read it.

Of course, we can improve this image, but if you reach this point, it’s a very good starting point for people using your image.

Forem: Laurent Tardif

[Devops / Docker] how to manage files and volume with docker

[Devops / Docker] How to write a good dockerfile

Understanding Docker: part 1 – Retrieve & Pull images

Aurélie Vache ・ Aug 3 '20