Forem: Sergio Marin

Reducing docker images size

Sergio Marin — Fri, 21 Oct 2022 18:13:14 +0000

You might have been building containers for a long time, and in all your previous builds, the security and/or the size of your containers wasn't the priority. But if you got here now you are giving priority to those topics.

For now we can focus first on the container size, and for this lets talk about a couple of options here.

Build upon a minimal base image.
Use multi-stage builds.
Create containers with statically linked applications.

Build upon a minimal base image

Instead of using a big image like ubuntu, debian, etc. Which could be a huge image for what is needed, we can use some alternatives that will allow us to start from a much smaller base.

debian latest d91720f514f7 124MB
ubuntu latest 216c552ea5ba 77.8MB

alpine latest 9c6f07244728 5.54MB
gcr.io/distroless/static-debian11 latest 561bdfb51245 2.35MB

The alpine and google distroless container are way smaller that ubuntu or debian.

In the case of google distroless container the only problem will be the package management for that container, because it wont have any package manager, another thing “missing” will be a shell. But they have a good number of base images for several runtime.

gcr.io/distroless/static-debian11
gcr.io/distroless/base-debian11
gcr.io/distroless/cc-debian11
gcr.io/distroless/python3-debian11
gcr.io/distroless/java-base-debian11
gcr.io/distroless/java11-debian11
gcr.io/distroless/java17-debian11
gcr.io/distroless/nodejs-debian11

Alpine on the other hand is a distro container, therefore it has a package manager and a shell, but it is small enough to be use has base container.

Lets start with an easy go application, you can see the source code of the application here. The container will expose a web application by default in the port 5000 with a general information of the system.

If we start with the basic way of building this container, will be something like this.

FROM golang

WORKDIR /app

COPY . /app/

RUN go mod download; \
    CGO_ENABLED=0 go build -ldflags="-s -w" -o bce -v .

EXPOSE 5000

ENTRYPOINT ["/app/bce"]

And after this we can build the container with the command:

docker build -t highercomve/bce .

After this we will end with a whooping 1.02GB size for the container.

highercomve/bce latest 614d597dbfb5 4 seconds ago 1.02GB

That doesn’t sound that good. Lets now change the the base container from golang to golang:alpine.

FROM golang:alpine

WORKDIR /app

COPY . /app/

RUN go mod download; \
    CGO_ENABLED=0 go build -ldflags="-s -w" -o bce -v .

EXPOSE 5000

ENTRYPOINT ["/app/bce"]

And with only this change we now are in only 375MB

highercomve/bce latest 1fb5114c906e About a minute ago 375MB

Same container, same application a lot less space.

Use multi-stage builds

Now the second optimization we can do is use a multi-stage build. For this we need to mark in the FROM command the name the stage will have, an then we use another base as the final runtime of the container.

FROM golang:alpine as builder # FIRST STAGE

WORKDIR /app

COPY . /app/

RUN go mod download; \
    CGO_ENABLED=0 go build -ldflags="-s -w" -o bce -v .

FROM gcr.io/distroless/static-debian11

WORKDIR /app

COPY --from=builder /app/bce /app/bce # IMPORT FROM THAT STAGE
COPY static /app/static/
COPY templates /app/templates/

EXPOSE 5000

ENTRYPOINT ["/app/bce"]

Now after this change we can see the difference it will be huge, because the resulting image will only be 8.73MB in size.

highercomve/bce latest e68c3ebb73e8 About a minute ago 8.73MB

Create containers with statically linked applications

Another thing you could do is build the application with the libraries linked and use the scratch base container.

FROM golang:alpine as builder

WORKDIR /app

COPY . /app/

RUN go mod download; \
    CGO_ENABLED=0 go build -ldflags="-s -w -extldflags=-static" -o bce -v .

FROM scratch

WORKDIR /app
COPY --from=builder /app/bce /app/bce
COPY static /app/static/
COPY templates /app/templates/

EXPOSE 5000
ENTRYPOINT ["/app/bce"]

With the change we will reduce another 2MB in size

highercomve/bce latest f32c277bf5a6 7 seconds ago 6.39MB

Now as a bonus i will add how to use this techniques to build a multi arch build container. Maybe you have the same application but you need to run it in arm or riscv architecture. For this we can use the buildx plugin from docker https://github.com/docker/buildx.

And for the same container i will use two arguments that are going to be injected by buildx.

ARG TARGETPLATFORM
ARG BUILDPLATFORM

And will allow us to take decisions about how to configure the environment variables for golang. I will use this build.sh script to map the target platform variable injected by buildx to the golang environment variables for building.

#!/bin/sh
set -e

echo "Building for $TARGETPLATFORM" 
export CGO_ENABLED=0

case "$TARGETPLATFORM" in
 "linux/arm/v6"*)
  export GOOS=linux GOARCH=arm GOARM=6
  ;;
 "linux/arm/v7"*)
  export GOOS=linux GOARCH=arm GOARM=7
  ;;
 "linux/arm64"*)
  export GOOS=linux GOARCH=arm64 GOARM=7
  ;;
 "linux/386"*)
  export GOOS=linux GOARCH=386
  ;;
 "linux/amd64"*)
  export GOOS=linux GOARCH=amd64
  ;;
 "linux/mips"*)
  export GOOS=linux GOARCH=mips
  ;;
 "linux/mipsle"*)
  export GOOS=linux GOARCH=mipsle
  ;;
 "linux/mips64"*)
  export GOOS=linux GOARCH=mips64
  ;;
 "linux/mips64le"*)
  export GOOS=linux GOARCH=mips64le
  ;;
 "linux/riscv64"*)
  export GOOS=linux GOARCH=riscv64
  ;;
 *)
  echo "Unknown machine type: $machine"
  echo "Building using host architecture"
esac

go mod download
go build -ldflags="-s -w -extldflags=-static" -o bce -v .

We need to be sure of one thing, the compilation process needs to run in the same architecture as the host building the container, if not we may lose performance trough emulation. For this we can use the BUILDPLATFORM argument in the FROM definition on the dockerfile.

Something like this.

FROM --platform=$BUILDPLATFORM golang:alpine as builder

ARG TARGETPLATFORM
ARG BUILDPLATFORM

WORKDIR /app

COPY . /app/

RUN /app/build.sh

FROM scratch

WORKDIR /app

COPY --from=builder /app/bce /app/bce
COPY static /app/static/
COPY templates /app/templates/

EXPOSE 5000

ENTRYPOINT ["/app/bce"]

With this new Dockerfile we can build the container using the buildx plugin.

docker buildx build --platform linux/arm64 -t highercomve/bce --load .

In here we introduce 3 new arguments for the build command:

platform: in here we can defined a list separated by coma of several architectures
load: this argument configure docker buildx to load the image to our local docker environment after the build process is done.
push: this will push to docker hub all the container images after the building process is done.

You can build several platform in one command and push to docker hub.

docker buildx build --platform linux/arm64,linux/amd64,linux/arm --push -t highercomve/bce .

Docker hub will support one images with multiple architectures and when some device pull with one of those architecture will get the correct one without any problem.

If we want to test the container running with one architecture different for the one of our system we will need to load the qemu binaries.

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

And after that run the container with the platform we like

docker run -it -p 5000:5000 --platform linux/arm64 highercomve/bce

In this example the output of the web page should be something like this

I hope this could be useful for you and help you to maintain more optimized containers

May the force be with you

Running Containerized GUIs on LXC with Pantavisor Linux

Sergio Marin — Sun, 12 Sep 2021 23:26:43 +0000

Pantavisor Linux is an embedded framework for building, maintaining, and managing LXC containers. It presents an infinite number of possibilities for running applications at the smart device edge.

In this use case, we’ll describe how to run containerized GUI applications that communicate via a socket to a containerized version of Wayland Composer.

Use Case

We will run two different applications for this example:

qt : An opensource application written in QT. You can see the source code for the application here. The built container for it is here: https://hub.docker.com/r/highercomve/qt-system-monitor
flutter-example : This is another application built with Flutter. Flutter is a framework developed by Google to build iOS, Android, and Desktop apps written in Dart. We will use the default example Flutter app from Flutter

Both qt and flutter-example are built with Ubuntu containers that run in another Ubuntu container.

You can see the source code for the application containers here:

Flutter-example : https://gitlab.com/highercomve/flutter-weston-example
qt : https://gitlab.com/highercomve/qt-simple-system-monitor

For the Wayland protocol and compositor, we selected weston. Weston can be described as the reference implementation of a Wayland compositor. It is also a very useful environment in and of itself.

With this, we now have the three base containers that will do the work of running the application and then displaying it on the monitor.

Before You Begin

You need a Raspberry PI 4, a display, an HDMI cable, and a mouse and keyboard connected to the Raspberry PI. By default, weston does not start without a connected keyboard and mouse.

You can download a Pantavisor image and claim it at hub.pantacor.com. If you don’t have an account on hub.pantacor.com, create one and then claim the device by following this Getting started guide.

How this works

Because this demo is fairly straightforward, and also to make things simpler, you can clone and deploy my reference device directly to your device with:

pvr clone https://pvr.pantahub.com/highercomve/gui_rpi64
cd gui_rpi64
pvr post -m "Add weston, flutter example and qt example" YOUR_DEVICE_CLONE_URL

Those commands will push all the parts needed to your device so that you can see everything running. Inside the reference device is the following folder structure:

bsp
awconnect
pv-avahi
weston
flutter-example
qt

With all those containers we are going to build a communication system for the Wayland components that will work in this way:

You can read more about how Wayland works in their documentation about the architecture. But let’s dig into how this container works from the pantavisor perspective and how every container is built and how it works.

BSP

The bsp folder contains the Board Support Package. It is also where Pantavisor runs.

awconnect

This is the main container and platform for the system. It is very similar to the base OS for all running containers. The awconnect container is also the central point in the example app architecture with weston. Weston will use any sockets configured inside the awconnect container. This container will then pass the info down to the kernel, etc.

Sharing sockets in the awconnect container

To share sockets running in the main container, you must configure the storage parameters inside the awconnect src.json file.

First, mount a permanent volume inside the /var/run/dbus folder of awconnect. This allows you to read and write from any container that wants to send its data to the monitor.

To do that the awconnect src.json should look as follows:

{
  "#spec": "service-manifest-src@1",
  "args": {},
  "config": {},
  "docker_digest": "registry.gitlab.com/pantacor/pv-platforms/wifi-connect@sha256:5c889720c6243408049b7a2f9ec75b9f4e376f6cbc28e39e091420a1e19df2aa",
  "docker_name": "registry.gitlab.com/pantacor/pv-platforms/wifi-connect",
  "docker_source": "remote,local",
  "docker_tag": "arm32v5",
  "persistence": {
    "/var/run/dbus": "boot"
  },
  "template": "builtin-lxc-docker"
}

How the Weston container is built

The weston container is built with alpine and also includes all of the packages to make the Wayland Compositor work with weston.

You can see the source of this container here.

The Dockerfile generates the container. It will look exactly like any other normal Dockerfile, since there is nothing different that needs to be done to make a container run in Pantavisor.

However, the most important part is not the container itself, but rather the Pantavisor configuration inside the src.json file. It is here that we need to be able to read and write to the socket folder kept inside the awconnect container.

{
  "#spec": "service-manifest-src@1",
  "args": {
    "PV_LXC_EXTRA_CONF": "lxc.mount.entry = /volumes/awconnect/docker--var-run-dbus run/dbus none bind,rw,create=dir 0 0\n",
    "PV_RUNLEVEL": "platform"
  },
  "config": {},
  "docker_digest": "sha256:116cc7d42d3f7e513fad55688b618e61dcdae2dd37f8164a86eea80af00c03d9",
  "docker_name": "weston",
  "docker_source": "remote,local",
  "docker_tag": "latest",
  "persistence": {},
  "template": "builtin-lxc-docker"
}

All the docker_* parts of this file are generated using:

pvr app add --from=registry.gitlab.com/pantacor/pv-platforms/alpine-weston:arm64v8 weston

Adding Docker apps to Pantavisor

You can use the pvr app add command to add more Docker applications, and to basically convert any Docker container into a Pantavisor container (just a normal LXC container but with our tooling to make it easy to manage). See: Adding Apps from Docker for more information.

Communicating with the awconnect

To enable the weston container to talk to the awconnect system, the following two arguments were added:

1.- PV_LXC_EXTRA_CONF

This configuration allows the pvr commands to extend the default LXC generated configuration. It is necessary to add extra LXC configuration in this way because when pvr updates or installs the container it also needs to generate the LXC configuration file.

In here we configure the lXC container to mount the same volume of the /var/run/dbus on awconnect inside a /run/dbus folder with read/write permissions.

The route of the awconnect volume is defined in the run.json file for that container. We will mount it using the LXC configuration parameter lxc.mount.entry

You can read more about the LXC configuration in the documentation

The end result of that configuration will be:

lxc.mount.entry = /volumes/awconnect/docker--var-run-dbus run/dbus none bind,rw,create=dir 0 0

2.- PV_RUNLEVEL

The PV_RUNLEVEL argument configures the level of priority used to run the container. By default, all applications run in RUNLEVEL: app and all apps in that run level will wait for the platforms to start.

With this configuration, the container should run correctly and it will act as the wayland composer.

flutter-example and qt

And lastly, we have the containers that will run the GUI applications. As I mentioned before, they are built with this source code:

flutter-example : https://gitlab.com/highercomve/flutter-weston-example
qt : https://gitlab.com/highercomve/qt-simple-system-monitor

And the configuration for both of them will be similar to the weston platform where we are going to modify the args value of the src.json in order to mount the awconnect volume with the sockets and configure both containers to run on RUNLEVEL platform.

Display the results

After you’ve posted everything to your device and the device consumed the revision, the Raspberry Pi will restart. You will then see the System monitor application written on QT and the Flutter example of a counter clicker running on the Raspberry PI display.

Is important to know that you can install to a container any available application from your favorite Linux distribution (example: Ubuntu) with support for wayland. That means you will install more packages than it needs, but it is possible.

I recommend that you look insides of the flutter-example and the QT system monitor containers to see the dependencies needed for the applications to run.

Final Thoughts

If you have any questions or you want to know more about Pantavisor and hub.pantacor.com you can reach out to us on @pantahub on Twitter or join our Pantavisor Slack channel. We’d be happy to help.

How to do Port Forwarding on LXC with Awall

Sergio Marin — Mon, 23 Aug 2021 22:23:17 +0000

Intro

We are going to be running our LXC containers with pantavisor.io a Linux system to run containers on embedded devices built by Pantacor, in this way we will reduce the hassle of managing LXC containers, and we are going to run everything with a Raspberry PI 4.

Pantavisor uses LXC to run containers on embedded computers. By default, all containers share the host’s network namespace and as a result, the LXC configuration for the network will be:

lxc.net.[0].type = none

LXC — Manpages — lxc.container.conf.5

This means that if you have more than one container using the same network PORT, the container initialization will fail. A way to solve that problem is to create the containers with a virtual ethernet type of network and then do a port forwarding from the host’s network to the container IP.

In Docker, port forwarding is a straightforward configuration when you run the container. But for LXC containers it is not that straightforward. In this post, we will describe how to forward ports inside the pantavisor. With Pantavisor tools you will create the LXC configuration and then use awall to build the container IP tables that will manage the forwarding.

Configuring Raspberry Pi as a DNS server for Adguard

In this use case, you will use a Raspberry PI as a DNS server with the adguard app that will filter any ads directly from DNS.

First, let’s observe the Adguard ports on Docker with:

docker run --name adguardhome \
    --restart unless-stopped \
    -v /my/own/workdir:/opt/adguardhome/work \
    -v /my/own/confdir:/opt/adguardhome/conf \
    -p 53:53/tcp -p 53:53/udp \
    -p 67:67/udp -p 68:68/udp \
    -p 3000:3000/tcp \
    -p 853:853/tcp \
    -p 784:784/udp -p 853:853/udp -p 8853:8853/udp \
    -p 5443:5443/tcp -p 5443:5443/udp \
    -d adguard/adguardhome

The ports to map are:

53 tcp/udp : Plain DNS.
67/udp, 68 tcp/udp : Add if you intend to use AdGuard Home as a DHCP server.
3000/tcp : Add if you want to use AdGuard Home’s admin panel and also run AdGuard Home as an HTTPS/DNS-over-HTTPS server.
853/tcp : Add if you plan to run AdGuard Home as a DNS-over-TLS server.
784/udp , 853 udp, 8853/udp : Add if you are going to run AdGuard Home as a DNS-over-QUIC server. You may only leave one or two of these.
5443/tcp, 5443/udp: Add if you are going to run AdGuard Home as a DNSCrypt server.

In the Pantavisor schema, the awconnect container does all of the network management for the default image for Pantavisor on Raspberry PI. This is what allows you to connect the Raspberry Pi to the internet via wifi or to share your wired internet through a wifi hotspot.

It is this container that provides DNS and DHCP. In addition to this, you may have another container using port 3000.

Before You Begin

In order to work on this, we need to have a Pantavisor-enabled Raspberry PI 3+ or a Raspberry PI 4. You will also need to install the following on your development computer:

fakeroot
squashfs
Docker
pvr cli
Sign up for an account on hub.pantacor.com

You can install fakeroot and squashfs from your distro or with your OS package manager: apt, yum, apk, homebrew, etc.

For example Ubuntu/Debian based distros install these utilities with:

sudo apt install fakeroot squashfs-tools

To install Docker for your particular OS, refer to the Docker installation documentation

Last but not least, you will also need to install the PVR cli to manage pantavisor devices inside hub.pantacor.com. You can find more about how to install PVR in Pantacor documentation.

Because we will send new revisions and configurations to the device remotely from your development computer, the device needs to connect to the hub.pantacor.com cloud service. You will need to create an account there.

Now that you have everything installed, we can move forward to solve our network isolation and port forwarding problem.

#1 Installing Pantavisor and claiming the device

A claiming process may enable an end-user to claim a device and proof ownership thereof. Such a process is initially triggered via a claim message that could be done in several ways.

After you’ve installed the Pantavisor image or if you installed Pantavisor already, you need to claim your device. There are several ways to claim your device:

1.- Using pre-claimed images

The easiest way to claim the device is by downloading an image that automatically claims it on the first boot. Download this image from: https://hub.pantacor.com/download-image

2.- Manual claiming

Another way to claim is by following the getting started guide on pantavisor.io and then claiming the device manually with the PVR CLI from your development computer:

pvr device scan

The result of that command should look something like this:

sergiomarin@penguin:~$ pvr device scan
Scanning ...
        ID: 61147e8af0e0f5000a50d11c (unclaimed)
        Host: localhost.local.
        IPv4: [192.168.68.115]
        IPv6: [fe80::d4ac:9722:ab19:7b81]
        Port: 22
        Claim Cmd: pvr claim -c marginally-optimum-midge https://api.pantahub.com:443/devices/61147e8af0e0f5000a50d11c
Pantavisor devices detected in network: 1 (see above for details)

And after that take the Claim Cmd value and run that. For example:

pvr claim -c marginally-optimum-midge [https://api.pantahub.com:443/devices/61147e8af0e0f5000a50d11c](https://api.pantahub.com:443/devices/61147e8af0e0f5000a50d11c)

After this step, you will be able to see your claimed device at hub.pantacor.com on the devices page.

You will see a new device with the status SYNCING. ** , This means your device has been claimed and is uploading to the cloud. When that process is **DONE, you will be able to clone and modify the device.

#2 Cloning the device URL to your computer

In the device details panel, you will see the Clone URL. Copy the URL and run the following command on your development laptop:

pvr clone CLONE_URL

Example:

pvr clone [https://pvr.pantahub.com/highercomve/specially\_brief\_monkey](https://pvr.pantahub.com/highercomve/specially_brief_monkey)

That creates a new folder called **specially\_brief\_monkey** much like the way git clone works. Inside the folder you should see a structure similar to this:

specially_brief_monkey
├── awconnect
│ ├── lxc.container.conf
│ ├── root.squashfs
│ ├── root.squashfs.docker-digest
│ ├── run.json
│ └── src.json
├── bsp
│ ├── addon-plymouth.cpio.xz4
│ ├── build.json
│ ├── firmware.squashfs
│ ├── kernel.img
│ ├── modules.squashfs
│ ├── pantavisor
│ ├── run.json
│ └── src.json
├── _hostconfig
│ └── pvr
│ └── docker.json
├── network-mapping.json
├── pv-avahi
│ ├── lxc.container.conf
│ ├── root.squashfs
│ ├── root.squashfs.docker-digest
│ ├── run.json
│ └── src.json
├── pvr-sdk
│ ├── lxc.container.conf
│ ├── root.squashfs
│ ├── root.squashfs.docker-digest
│ ├── run.json
│ └── src.json
└── storage-mapping.json

This is how Pantavisor defines a device with its running containers: :

BSP : In embedded systems, a board support package (BSP) is the layer of software containing hardware-specific drivers and other routines that allow a particular operating system to function in a particular hardware environment.
awconnect : Pantavisor base platform for the device and where the network configuration is and is based in this container.
pv-avahi : a container that sends all the information needed to discover the device in the network using pvr device scan and uses the avahi protocol for that. You can see the source container here: https://gitlab.com/pantacor/pv-platforms/pv-avahi
pv-sdk : container with Pantavisor SDK that allows you to maintain the device directly from the device and is the one in charge of managing the ssh connections to the device and to the container inside it.

Every one of those containers will have inside a src.json file that describes the source from where the container was created as well as some of the Pantavisor-specific configurations to run the container.

#3 Adding adguard to our device

We are going to use the adguard docker container as the source for our Pantavisor container (basically an LXC container). In order to add a new container from Docker inside the folder created by the pvr clone process, we are going to run the following:

pvr app add --from=adguard/adguardhome:latest adguard

That creates a new folder called adguard inside the device definition with the following structure inside:

adguard/
├── lxc.container.conf
├── root.squashfs
├── root.squashfs.docker-digest
├── run.json
└── src.json

The src.json should look something like this:

{
  "#spec": "service-manifest-src@1",
  "template": "builtin-lxc-docker",
  "args": {
   "PV_RUNLEVEL": "app"
  },
  "config": {},
  "docker_name": "adguard/adguardhome",
  "docker_tag": "latest",
  "docker_digest": "adguard/adguardhome@sha256:cd5e6641e969ec8a1df1ed02dc969db49d6cf540055f14346d0d5d42951f75d6",
  "docker_source": "remote,local",
  "docker_platform": "linux/arm",
  "persistence": {}
}

#4 Configuring parameters for LXC features and Adguard persistence

For now, we will discuss only two parts of this file: args and persistence.

args: This is a configuration for the pvr cli tooling that sets up some features for the LXC container.
persistence : Configuration for the volumes and persistence for the running container. By default, pvr automatically adds all the volumes defined in the Dockerfile. You can add more volumes here, even if they aren’t defined in the Dockerfile.

As discussed, pvr adds all the containers by default to the host network namespace. As a result, the first configuration that we need to do is isolate this container inside the LXC bridge network.

To do that we need to add a couple of new arguments inside the args configuration:

PV_LXC_NETWORK_TYPE : This parameter configures lxc.net.[0].type and other parameters inside the LXC configuration depending on the value we assign in there.
PV_LXC_NETWORK_IPV4_ADDRESS : This is going to be the assigned IP address for the virtual network interface.

The resulting args will be as follows:

"args": {
    "PV_LXC_NETWORK_IPV4_ADDRESS": "10.0.3.20/24",
    "PV_LXC_NETWORK_TYPE": "veth",
    "PV_RUNLEVEL": "app"
}

If you want to know more about what arguments can be used, you can read the source code of our template to build lxc configurations.

And for persistence, we need to add the /opt/adguardhome/work and the /opt/adguardhome/conf folders. Both of these are used by adguard in order to save the configuration of the service.

The resulting persistence configuration will be:

"persistence": {
    "/opt/adguardhome/conf/": "permanent",
    "/opt/adguardhome/work/": "permanent"
}

#5 Install the Adguard App

With the persistence set up and the LXC features enabled, , you are ready to install the Adguard application using pvr.

First, check that the finalized src.json looks similar to this:

{
  "#spec": "service-manifest-src@1",
  "args": {
    "PV_LXC_NETWORK_IPV4_ADDRESS": "10.0.3.20/24",
    "PV_LXC_NETWORK_TYPE": "veth",
    "PV_RUNLEVEL": "app"
  },
  "config": {},
  "docker_digest": "adguard/adguardhome@sha256:cd5e6641e969ec8a1df1ed02dc969db49d6cf540055f14346d0d5d42951f75d6",
  "docker_name": "adguard/adguardhome",
  "docker_platform": "linux/arm",
  "docker_source": "remote,local",
  "docker_tag": "latest",
  "persistence": {
    "/opt/adguardhome/conf/": "permanent",
    "/opt/adguardhome/work/": "permanent"
  },
  "template": "builtin-lxc-docker"
}

If your src.json looks fine, let’s proceed with the installation using this command:

pvr app install adguard

In order to see how Adguard is installed and isolated, push the changes to the device:

pvr add && pvr commit && pvr post -m "Install adguard into the device"

After this you will be able to see a new revision in the device at hub.pantacor.com, and similar to this one:

When the device status is DONE or UPDATED, the process is finished. The device should be running Adguard now, but because it is isolated, you won’t be able to use the web UI or even use it as DNS.

#6 Mapping the Adguard ports with awall

The awconnect platform uses iptables to manage firewall and is the one that manages to route, and awall is a utility to generate iptables and routes based on JSON configuration.

This is a good guide about how awall works. I will not enter the deepest lands of awall, instead, I will show how to use it for this use case.

Configure Awall for Pantavisor

First, let’s start by adding this small utility script to run awall without installing it on your computer, and instead, run it from a Docker container. You can see and download the script here.

wget https://gist.githubusercontent.com/highercomve/295cf75fb660be4c6b054627c330cb4b/raw/d51989dd5c57c36136814033af6d460db3024bef/awall2pvmwall &amp;&amp; chmod +x awall2pvmwall

That script runs awall for the awall.json file in the root of the device folder as well as any _awall configuration folder inside any application or platform of the device.

We are going to create several JSON files to configure awall. Port_forward_example is the folder name for where my device repository lives and also where we will add all the new files for configuration.

mkdir -p adguard/_awall
touch awall.json
touch adguard/_awall/config.json
touch bsp/_awall.json

The configuration file bsp/_awall.json defines a couple of variables for our device.

{
   "variable": {
    "containernet_if": "lxcbr0",
    "wan_if": "eth0",
    "lan_if": "wlan0"
   }
}

These are the default values that Pantavisor sets for a Raspberry PI 4. If your Raspberry Pi is connected to the Internet via an ethernet cable, the wan_if will be eth0 and maybe your wifi will be a local area network. In some cases, you may be connected via wifi to an Internet provider and will use the eth0 as an entry point to your local network.

The awall.json configuration file defines the general firewall rules for the device. We are going to add to that file everything that is directly related to the host network namespace.

{
   "description": "How to use awall",
   "filter": [
       {
           "action": "accept",
           "in": "internet",
           "service": [
               "ping",
               "dns",
               "ssh",
               "dhcp",
               "pvssh"
           ]
       },
       {
           "action": "accept",
           "in": "intranet",
           "service": [
               "dns",
               "ping",
               "ssh",
               "dhcp",
               "pvssh"
           ]
       },
       {
           "action": "accept",
           "in": "internet",
           "out": "_fw",
           "service": [
               "ssh",
               "dhcp",
               "pvssh"
           ]
       },
       {
           "action": "accept",
           "out": "internet",
           "service": [
               "http",
               "https"
           ]
       }
   ],
   "import": [
       "bsp"
   ],
   "policy": [
       {
           "action": "accept",
           "out": "internet"
       },
       {
           "action": "drop",
           "in": "internet"
       },
       {
           "action": "accept",
           "out": "intranet"
       },
       {
           "action": "drop",
           "in": "intranet"
       },
       {
           "action": "accept",
           "in": "containernet"
       },
       {
           "action": "accept",
           "out": "containernet"
       },
       {
           "action": "reject"
       }
   ],
   "service": {
       "pvssh": [
           {
               "port": 8222,
               "proto": "tcp"
           }
       ]
   },
   "snat": [
       {
           "out": "internet"
       }
   ],
   "zone": {
       "containernet": {
           "iface": "$containernet_if"
       },
       "internet": {
           "iface": "$wan_if"
       },
       "intranet": {
           "iface": "$lan_if"
       }
   }
}

With this configuration, we open access to some ports of the device:

ssh : port 22 to access the pvr-sdk container
pvssh : port 8222 to enter directly to a container
dns : port 53
ping: icmp port
dhcp: port 67 and 68

#7 Redirect traffic to the Adguard IP with DNAT

Now let’s redirect everything to the Adguard IP address by using a DNAT configuration. We will need to populate the **aguard/\_awall/config.json** file with this configuration.

{
   "description": "Forward All need it ports to this container",
   "filter": [
       {
           "action": "accept",
           "dnat": "10.0.3.20",
           "in": "internet",
           "service": "adguard-web"
       },
       {
           "action": "accept",
           "dnat": "10.0.3.20",
           "in": "intranet",
           "service": "adguard-web"
       },
       {
           "action": "accept",
           "dnat": "10.0.3.20",
           "in": "internet",
           "service": "adguard-dotls"
       },
       {
           "action": "accept",
           "dnat": "10.0.3.20",
           "in": "intranet",
           "service": "adguard-dotls"
       },
       {
           "action": "accept",
           "dnat": "10.0.3.20",
           "in": "internet",
           "service": "adguard-dnsencrypt"
       },
       {
           "action": "accept",
           "dnat": "10.0.3.20",
           "in": "intranet",
           "service": "adguard-dnsencrypt"
       }
   ],
   "service": {
       "adguard-web": [
           {
               "port": 3000,
               "proto": "tcp"
           },
           {
               "port": 3000,
               "proto": "udp"
           }
       ],
       "adguard-dotls": [
           {
               "port": 53,
               "proto": "tcp"
           },
           {
               "port": 53,
               "proto": "udp"
           },
           {
               "port": 853,
               "proto": "tcp"
           },
           {
               "port": 853,
               "proto": "udp"
           },
           {
               "port": 784,
               "proto": "udp"
           },
           {
               "port": 8853,
               "proto": "udp"
           },
           {
               "port": 5443,
               "proto": "tcp"
           },
           {
               "port": 8853,
               "proto": "udp"
           }
       ],
       "adguard-dnsencrypt": [
           {
               "port": 5443,
               "proto": "tcp"
           },
           {
               "port": 8853,
               "proto": "udp"
           }
       ]
   }
}

There, you can see the ports grouped by service. All of those services will be translated from both the WAN and the LAN interface to the container IP.

#8 Setting up the port forwarding and viewing the Adguard dashboard

Now that we have all the JSON configuration ready, we can run the awall2pvmwall script that will create a folder with this structure:

_config/
└── awconnect
    └── etc
        └── iptables
            ├── dump
            ├── rules.v4
            └── rules.v6

With this we can now post this new revision to our device:

pvr add .
pvr commit
pvr post -m "Update awconnect with port forwading to adguard"

NOTE: In here it is important to move from port 80 to 3000

After a couple of more next and next, you will be able to enter the adguard dashboard and use your Raspberry PI as DNS for your entire network and will also be blocking ads,using an upstream DNS via DoT and a lot more..

Here is alist of known DNS providers to choose from.

You can see the device I built for this guide in here or you could clone with:

pvr clone [https://pvr.pantahub.com/highercomve/port\_fordward\_example](https://pvr.pantahub.com/highercomve/port_fordward_example)

Now go and play with Pantavisor and containers and may the force be with you.

How I explain to my friends what we do at work

Sergio Marin — Thu, 25 Jun 2020 22:17:51 +0000

Pantahub and Pantavisor could sometimes be a difficult paradigm to understand because, in a way, it changes how we are related to the firmware and software of devices.

Today, when you don’t do something with an embedded device, maybe an IoT project or home automation with your Raspberry, you will go and flash your device with a Linux distribution and start to install packages and configure all of them via ssh, and if you need to replicate that you will do it all over again. Then if you want to automate that process a little bit you may create a shell script to do all the configuration and installation.

What happens if you have updated that script and you then need to update that configuration (let’s say you have only two devices), you need to go and run everything on both devices, hopefully via SSH. You can do some kind of more sophisticated thing here, there are a lot of tools for that, but maybe you are running this on a device with very little memory and your plans can fall apart.

But what we want to do is see devices and their configuration as you see Docker images and Docker containers, but you don’t even need to install and configure the device once. You only need to flash it with Pantahub. and then post revisions that will be deployed on the device. You can even set a device as leader of a pack, and the whole pack will follow the leader revisions automatically.

Let’s start by understanding all this terminology: devices, revisions, platforms, etc. For that let’s talk more about Pantahub and Pantavisor.

What is Pantahub?

Pantahub offers cloud-controlled firmware and software management for all kinds of Linux-based connected devices. In addition to this, Pantahub provides a community ecosystem for device makers, operators, and technology enthusiasts to gather around and share their work.

To use Pantahub, you will need to install Pantavisor on your devices.

What is Pantavisor?

Pantavisor is the Linux device init system that turns all the runtime into a set of container micro functions. Through Pantavisor, devices become software-defined infrastructure, through Linux container technology. All Pantavisor enabled devices can automatically connect to a Pantahub instance.

By using these two technologies you will be able to manage your devices and the software in a way similar (similar but not the same) to how you manage applications with docker.

However, instead of having a container running on the device, the device will run the PLATFORM directly on metal, allowing you to have versions of the device. This device state version is called REVISION , and inside Pantahub you will have the ability to return to a previous revision and, more importantly, the system is fault-tolerant. That means if you deploy a revision that can’t run correctly on the device, it will start the last revision that worked correctly with that, you can be sure that you won’t lose your device if you configure something in the wrong way or if the software has any kind of problem.

How the ecosystem works

The ecosystem could look similar to how you see containers on the Docker hub, but instead of containers, you will see devices.

Here you can see how a device is shown inside Pantahub.

A device in Pantahub represents a physical device, in this example a raspberry pi 3 b+ running several PLATFORMS /APPS inside.

Those PLATFORMS are built from a docker container, but inside of the device there isn’t any Docker runtime.

In this example you see a device running:

cloudflare : Platform installed by one.apps.pantahub.com to manage the DNS configuration and to set the DNS filters for your WIFI hotspot
awconnect : Manages the network and the wifi connections.
pv-avahi : Exposes an mDns configuration in order to be able to discover any Pantahub enabled device on the network using the PVR CLI or the Mobile application.
pvr-sdk : Manages some of the functionalities of the device like the SSH connection, the ability to follow devices, read user metadata and so much more.

With this, you can notice how a device is composed of several platforms and those platforms are based on docker containers. But remember there is not any docker container running on the device.

The device revision is described as a JSON file that is converted to a file system when you clone the device.

Let me explain with a visual example using the same device.

We call this JSON specification of the device revision state.json

As you can see the STATE is a normalized JSON with the folder structure of the device, where any .json file inside that structure is part of state.json and any other file format is represented by the sha256 of that file and is saved in Pantahub objects (our storage in the cloud)

When you use pvr clone URL_OF_DEVICE (just like a git clone), you will get the same structure as in the state.json but the files represented as sha256 are going to be downloaded.

The result is going to be a folder like this one.

Every platform lives in its own folder and all the configuration files live inside of the _config/${PLATFORM} folder

If you want to dive deep in all the available endpoints you can read the API reference here in the documentation or the swagger documentation here

You may need Laziness in your Javascript

Sergio Marin — Fri, 22 Feb 2019 22:47:31 +0000

Maybe some times you have overused the Array methods map, reduce, filter, find, etc. and that could make out applications to use more…

Continue reading on ITNEXT »

Back to the Browser: React Form Validation with the DOM API

Sergio Marin — Mon, 18 Feb 2019 21:28:49 +0000

Last week I wrote an article about how we can build a form validation hook with React Hooks (more about it here)

Continue reading on ITNEXT »

Form validation with React Hooks

Sergio Marin — Thu, 14 Feb 2019 14:52:44 +0000

As you may know, I’m pretty excited about the new React API for Hooks (here is an introduction about them)

Continue reading on ITNEXT »

Why I’m excited with React Hooks?

Sergio Marin — Mon, 11 Feb 2019 19:34:47 +0000

On February 6 React 16.8 was released and with React 16.8, React Hooks are available in a stable release!

That means we can start using Hooks without the fear of writing unstable code.

Here is a live example of the code in this post:

https://frontarm.com/demoboard/?id=af4f455f-1d30-4823-90a9-b15cfc3e92f9

But, what are Hooks?

Hooks are functions that allow you to use state and “life cycle” for functional Components, that means you don’t need to write classes in order to build React components.

For me, this is exciting because I don’t like to bind “this” (you saw what I did there?). And what I mean is, that Classes in React are kind of messy (I'm not the only one who think that). Most of the time it has something to do with “this”. But that is a story for another time.

So you get access almost all the state lifecycle without using classes. But how?

Which Hooks are available and what they do?

Directly from React’s page, we have this list of Hooks (https://reactjs.org/docs/hooks-reference.html)

Basic Hooks

Additional Hooks

I will try to start with the first 3 elements of that list. (at the beginning I was thinking in 4 but the post got too long, I will split it)

useState Hook

This hook lets you have an internal state inside a functional Component. Here is a small comparison side by side of a counter component written as Class Component and as Function Component with useState.

Let's use a counter as an example:

import React from 'react'

export default class Counter extends React.Component {
  constructor (props) {
    super(props)
    this.state = {
      count: props.initialCount || 0
    }
    this.add = this.add.bind(this)
    this.minus = this.minus.bind(this)
    this.reset = this.reset.bind(this)
  }
  add () {
    this.setState((state) => ({ count: state.count + 1 }))
  }
  minus () {
    this.setState((state) => ({ count: state.count - 1 }))
  }
  reset () {
    this.setState((state) => ({ count: 0 }))
  }
  render () {
    return (
      <div className="counter">
        <span className="counter\_\_count">{ this.state.count }</span>
        <button onClick={this.add}>+</button>
        <button onClick={this.minus}>-</button>
        <button onClick={this.reset}>Clear</button>
      </div>
    )
  }
}

Now the same but with useState

import React, { useState, useEffect } from 'react'

export default function CounterUseState (props) {
  const [count, setCount] = useState(props.initialCount || 0)
  const add = () => { setCount(count + 1) }
  const minus = () => { setCount(count - 1) }
  const reset = () => { setCount(0) }

  return (
      <div className="counter">
        <span className="counter\_\_count">{ count }</span>
        <button onClick={add}>+</button>
        <button onClick={minus}>-</button>
        <button onClick={reset}>Clear</button>
      </div>
    )
}

We don’t have more “this” anymore! as result the “state” is easier to use, the handlers are cleaner and it would be easier to compose with functions outside of the component scope.

Now how we can handle with the “life cycle” of the component? we will use the useEffect Hook

useEffect Hook

This Hook received two parameters, the first argument is a callback function to run when the component is rendered, the callback will be called either is the first time the component is rendered ( componentDidMount ) or every time it’s re-rendered ( componentDidUpdate ), the second parameter is very important for this because with determines what variables or props you want to “observe” for changes. Therefore using useEffect will work as componentDidMount and componentDidUpdate

And how we clear those effects? well, the function that runs in the effect should return another function that is gonna we ran when the component is unmounted. Magic!!

import React, { useState, useEffect } from 'react'

export default function FunctionalTimer (props) {
  const [count, setCount] = useState(props.initialCount || 0)
  const [running, setRunning] = useState(false)

  const start = () => setRunning(true)
  const pause = () => setRunning(false)
  const reset = () => setCount(0)
  const tick = () => running && setTimeout(() => setCount(count + 1), 1000)

  useEffect(() => {
    tick()
  }, [running, count])

return (
      <div className="counter">
        <span className="counter\_\_count">{ count }</span>
        <button onClick={start}>Start</button>
        <button onClick={pause}>Pause</button>
        <button onClick={reset}>Clear</button>
      </div>
    )
}

setTimeout and setInterval are effects outside of the pure nature of the functional component. The same applies to things that are changing other things outside the component, for example, the document.title or adding an event listener to a not synthetic event.

import React, { useState, useEffect } from 'react'

export default function FunctionalTimer (props) {
  const [count, setCount] = useState(props.initialCount || 0)
  const [running, setRunning] = useState(false)

  const start = () => setRunning(true)
  const pause = () => setRunning(false)
  const reset = () => setCount(0)
  const tick = () => running && setTimeout(() => setCount(count + 1), 1000)

  useEffect(() => {
    tick()
  }, [running, count])

  useEffect(() => {
    document.title = `${count} seconds pass`
  }, [count])

useEffect(() => {
    const logOnSizeUpdate = () => console.log({ count, running })
    window.addEventListener('resize', logOnSizeUpdate)
    return () => window.removeEventListener('resize', logOnSizeUpdate)

  }, []) // when the array is empty only runs on mount not update

return (
      <div className="counter">
        <span className="counter\_\_count">{ count }</span>
        <button onClick={start}>Start</button>
        <button onClick={pause}>Pause</button>
        <button onClick={reset}>Clear</button>
      </div>
    )
}

One of the awesome abilities of Hooks is the ability to create Custom Hooks, and custom hooks are just simple functions!

Let's rewrite that timer code and move all the state logic to custom hooks called useTimer and useSetTitle

import React, { useState, useEffect } from 'react'

function useTimer (initialCount = 0, autoStart = false) {
  const [count, setCount] = useState(initialCount)
  const [running, setRunning] = useState(autoStart)

const start = () => setRunning(true)
  const pause = () => setRunning(false)
  const reset = () => setCount(0)
  const tick = () => running && setTimeout(() => setCount(count + 1), 1000)

  useEffect(() => {
    tick()
  }, [running, count])

  return {
    count,
    running,
    start,
    pause,
    reset
  }
}

function useSetTitle (count) {
  useEffect(() => {
    document.title = `${count} seconds pass`
  })
}

export default function FunctionalTimer (props) {
  const timer = useTimer(props.initialCount, props.autoStart)
  useSetTitle(timer.count)

  return (
      <div className="counter">
        <span className="counter\_\_count">{ timer.count }</span>
        <button onClick={timer.start}>Start</button>
        <button onClick={timer.pause}>Pause</button>
        <button onClick={timer.reset}>Clear</button>
      </div>
    )
}

This allows us to build a bunch of stateful logic that live in Hooks and can be used in any functional component, enabling us to reuse and compose in a better way our code.

useRef

This hook allows us to have a persistent value between components render. And why we could need that, if you notice in the previous example the way the time is working is putting on setTimeout the next counter change. The problem with that is if we clear the timer, the next time the function that will be run, will have the value of count before the clear, and because you need to pause, clear and start again. Functions references in the effect will have the value of the variables at the moment they were set. That way we will need useRef to storage a callback every time that the component renders with the current count value that maintains the stability of the timer.

import React, { useState, useEffect, useRef } from 'react'

function useTimerInterval (initialCount = 0, autoStart = false) {
  const [count, setCount] = useState(initialCount)
  const [running, setRunning] = useState(autoStart)
  const cb = useRef()
  const id = useRef()

  const start = () => setRunning(true)
  const pause = () => setRunning(false)
  const reset = () => setCount(0)

  function callback () {
    setCount(count + 1)
  }

  // Save the current callback to add right number to the count, every render
  useEffect(() => {
    cb.current = callback
  })

  useEffect(() => {

    // This function will call the cb.current, that was load in the effect before. and will always refer to the correct callback function with the current count value.   
    function tick() {
      cb.current()
    }
    if (running && !id.current) {
      id.current = setInterval(tick, 1000)
    }

    if (!running && id.current) {
      clearInterval(id.current)
      id.current = null
    }
    return () => id.current && clearInterval(id.current)
  }, [running])

  return {
    count,
    start,
    pause,
    reset
  }
}

function useSetTitle (count) {
  useEffect(() => {
    document.title = `${count} seconds pass`
  })
}

export default function FunctionalTimer (props) {
  const timer = useTimerInterval(props.initialCount, props.autoStart)
  useSetTitle(timer.count)

  return (
      <div className="counter">
        <span className="counter\_\_count">{ timer.count }</span>
        <button onClick={timer.start}>Start</button>
        <button onClick={timer.pause}>Pause</button>
        <button onClick={timer.reset}>Clear</button>
      </div>
    )
}

Here is a live example of the code in this post:

Demoboard

And maybe you want to read a little more about why the timer needs the persistent reference. I that case this article deeps into it https://overreacted.io/making-setinterval-declarative-with-react-hooks/

This is a basic example of what we can do know with React Hooks and I recommend to see this video from the React Conf about Hooks.

I will continue this topic about React Hooks with useContext and useReducer and how we can build something like Redux with those two Hooks.

Until the next post, May the Force be with you

Esto es lo que hago y quiero seguir haciendo porque en verdad lo disfruto

Sergio Marin — Mon, 24 Aug 2015 09:09:03 +0000

Desde hace 6 años me dedico a la educación en el área de desarrollo web/móvil, como instructor día a día compruebo que las generaciones actuales son inmensamente más geniales que la mía y logran procesar información a velocidades mucho mayores a lo que lo hacíamos nosotros.

Pero esa misma cantidad masiva de conocimientos e información procesada no siempre logran encontrar el camino adecuado a las buenas prácticas y metodologías que la industria actual del emprendimiento y startup del área del desarrollo web necesitan, por lo tanto si no logra conseguir una empresa lo suficientemente bueno para entender, este joven puede terminar rápidamente contratado en el trabajo menos adecuado para su capacidad (terminando agotado en una oficina aburrida y no desarrollando la innovación del mañana).

Por eso es que nosotros queremos iniciar un proyecto que consta con dos componentes críticos, el primero alimentar la mentes geniales de estos jóvenes llevándolos de la mano por un camino educativo que los inicie por el camino a ser líderes de la industria en el futuro, usando como herramienta el déficit de desarrolladores actuales de la industria y el crecimiento acelerado de la misma, que muchas veces no puede atacar proyectos porque no tienen quien se los desarrolle en el tiempo correcto y con una buena calidad. Y porque esto es nuestra herramienta, porque vamos a usar esta necesidad para darle a nuestros alumnos proyectos reales que desarrollar mientras mantienen su cerebro en constante aprendizaje de las mejores tecnicas y practicas de la industria.

¿Cómo lo vamos a hacer? Tres meses de entrenamiento intensivo, ¡no es un curso más! ¡No! Es un entrenamiento de guerra, ocho horas diarias, donde tu único objetivo será cambiar tu vida. En uno de los mejores espacios de emprendimiento en donde además vas a tener contacto directo con la industria, estará al lado tuyo mientras aprendes. Y por el otro lado las empresas podrán desarrollar los nuevos proyectos que desean hacer, mientras a través de un sistema web pueden hacer seguimientos y control de su producto, permitiendo que además el alumno aprenda como es el proceso y metodologías de seguimiento y construcción de un proyecto web.

¿Quién lo financia? La industria, las empresas que están desarrollando proyectos con nuestros talentosos alumnos, al pagar el desarrollo de estos proyectos pagan la educación de estos alumnos.

¿Por qué no cobrarle al alumno que es el interesado en aprender?, porque normalmente estos jóvenes geniales pueden aprender solo con usar internet, entrar en comunidades, youtube y toda la información disponible en internet de manera gratuita. Para no cambiarles ese modelo mental seguirán consumiendo contenido gratuito pero ahora enfocado y dirigido, metódico y directo por el camino de las buenas prácticas.

Esto es nuestra propuesta de valor seguir haciendo lo que amamos, mientras alimentamos a la industria de manera acelerada de mentes jóvenes y geniales para que salgan a apoderarse de ella. Este es el proximo paso para seguir hackiando en escuelaweb.

¡Gracias y que las fuerza los acompañe!