Forem: Vedran Vidovic

How to run SoapUI mockservicerunner in a Docker container

Vedran Vidovic — Tue, 15 Jun 2021 06:27:13 +0000

On one of our integration projects using the IBM API Connect, we wanted to create a quick mock of our Customer's backend SOAP service. We used quite effectively a SpringBoot a few times in the past but I wondered if it would be faster (or at least more interesting) to use the mocking capabilities of the SoapUI for this purpose 🤔

TLDR;

You can use the prepared Docker image to run mock services created by the SoapUI in the soapui-mockservicerunner image:

docker run --rm -p 8080:8080 -v $PWD/soapui:/home/soapui/project vvidovic/soapui-mockservicerunner:latest -a "/" -p "8080" /home/soapui/project/my-soapui-project.xml

Creating mock service in the SoapUI

SoapUI is a great tool with an intuitive GUI that you can use to call and mock
SOAP web services. For example:

create a new project based on the WSDL
generate a Mock SOAP service (right-click on the interface in a projects navigator view)
edit generated responses and/or create new responses
open and edit the MockOperation
- select the appropriate Dispatch method (XPATH, SCRIPT, RANDOM, SEQUENCE, QUERY_MATCH)
- for example, you can select the response based on the XPath result of the request received

For more details about the SoapUI mocking features, please check SOAP Service Mocking Overview.

Running the created mock service from the command line

SoapUI doesn't stop with the GUI for your mocking needs. It have a command-line mockservicerunner.sh script which can be used to run your SOAP mock service without the UI.

The process to use it is quite simple:

create a mock service using the GUI
run a mock service using the provided script mockservicerunner.sh

For example:

./mockservicerunner.sh -a '/' -p 8080 my-soapui-project.xml

-a: the local path to listen on
-p: the local port to listen on

For more information about how to run your mock services from the command line, please check the SoapUI Test Automation documentation.

Running the created mock service as a Docker container

I tried to find out if there is a SoapUI mockservicerunner image provided on the Docker Hub. I was able to find few images that bundle the SoapUI mockservicerunner but none of them provided the script as an Docker ENTRYPOINT. That means I could not trivially migrate my logic used to start mock services on local machine to a Docker container.

Well, how hard it can be to prepare a Docker image using the SoapUI binary?

Creating the `soapui-mockservicerunner` image

Not hard but there were few obstacles on the road, the first I encountered was:

the latest SoapUI Open Source provided is version 5.6.0 but the version for the Linux is broken and it requires some work-around
- I used the version 5.5.0 available from the SoapUI Open Source Older Versions to simplify the process

After that, I had to prepare a Dockerfile based on the OpenJDK JRE in which I:

use JRE 11 or 8
copy the SoapUI archive to the container
unpack the archive, delete archive and move the resulting folder to the /opt directory
declare the mockservicerunner.sh as a Docker ENTRYPOINT
add a non-root user to the image
set a Docker WORKDIR to this user's home directory
set a newly create user as a user to use when running this Docker image

Full Dockerfile is here:

FROM openjdk:11-jre-slim
#FROM openjdk:8-jre-alpine

COPY ./soapui/*.tar.gz /opt/

RUN cd /opt && tar -xvf /opt/*.tar.gz && rm /opt/*.tar.gz && mv * SoapUI

ENTRYPOINT ["/opt/SoapUI/bin/mockservicerunner.sh"]

RUN adduser --uid 1000 --disabled-password soapui

WORKDIR /home/soapui

USER soapui

Before building this image I had to put SoapUI 5.5.0 archive to the ./soapui directory and then just execute the build command:

docker build . -t vvidovic/soapui-mockservicerunner

To make the usage of this image more simple for everyone else, I published it to a new Docker Hub repository under my user:

docker login
docker push vvidovic/soapui-mockservicerunner:latest

Using the `soapui-mockservicerunner` image

The end result is available in the published soapui-mockservicerunner Docker image which can be run using the following command (using your own SoapUI project):

docker run --rm -p 8080:8080 -v $PWD/soapui:/home/soapui/project vvidovic/soapui-mockservicerunner:latest -a "/" -p "8080" /home/soapui/project/my-soapui-project.xml

Conclusion

If you want to create a mock SOAP service for your project, an approach described here can help you to do it faster and easier:

using SoapUI GUI application to create mock operations
using soapui-mockservicerunner Docker image to run those service on your local machine or in your cloud

Some of the images used in this post are provided by Pixabay users:

AqiSDS011 - Android AQI app

Vedran Vidovic — Sun, 21 Mar 2021 20:38:38 +0000

In the previous two posts in this series, you could read a bit more about out-of-the-box ways to use the SDS011 air quality sensor. At first, I connected the sensor to my laptop but then I realized that my mobile phone could be a more portable appliance accomplishing the same purpose.

Using the existing sds011_android app

After some time spent using a sensor with my mobile phone and the sds011_android application found on the GitHub, I realized there are few features that would make this application even more usable:

UI should reflect proper colours for air quality according to the PM2.5 or PM10 levels
save/view a history of measurements
- date, time & location included
- show location for each measurement on the map
- this way we can take a ride or walk around the town and later analyze data to see if air quality differs between different locations
pause/continue measurements
- this would save the sensor laser & my phone battery
configure periodic measurements
- this would save the sensor laser & my phone battery

I didn't touch Android development for quite a long time but I felt it shouldn't be too hard to implement at least some of the above-mentioned features. Since I wanted to have such an application for my own usage and I hoped something like that could be useful to other people I decided to spend some time on the project I named the Mobile Air Quality Monitor application for the SDS011 Sensor.

AqiSDS011 - the android app

The current result is available on GitHub as the AqiSDS011 repository. I will need some more time to prepare and publish it as an application on the Google Play Android app store.

Before I do that, anyone wishing to use the application will (unfortunately) need to build and run the application herself/himself which is not an easy task for a non-developer. This task can be accomplished using the Android Studio, as described in the official tutorial.

I am thankful to the developers of the sds011_android project and the developers of the aqi project - without these two projects, it would be much harder to implement the AqiSDS011 project. I copied parts of the logic for interaction with the SDS011 sensor from the sds011_android application and logic for calculating the AQI indices for PM2.5 and PM10 from the aqi Python logic.

The modes of operation of the SDS011 sensor

The SDS011 sensor has two data reporting modes:

Active reporting (used by AqiSDS011 and the sds011_android projects)
- when this mode is enabled, the sensor sends measuring data without additional queries
Query reporting (used by aqi project)
- when this mode is enabled, the sensor waits for query before measuring and sending the measuring data back to the connected appliance

When the active data reporting is used, the sensor can work in one of two modes:

sending data continuously (each second)
sending data each n minutes (depending on settings sent to the sensor)

AqiSDS011 supports both modes of active data reporting.

The current status of the AqiSDS011

Currently, the project finally implemented the list of basic requirements through 3 screens.

The main measurements screen shows the current AQI information and enables you to start or stop the measuring process. This is a screen where you can choose between the continuous measuring mode and the periodic measuring mode.

The history screen shows all saved measurements but also enables you to:

save the history information
clear the history
open the map application to show you the location for the saved measurement

The settings screen enables you to:

enter the length of the work period (applied when you use the application in the periodic mode).
enter the number of measurements (equals the number of seconds) for averaging before saving to history
select the location data gatherings setting

The road ahead

While AqiSDS011 became an application with all initially envisioned features, there are still a few issues I would like to do or at least investigate. For example:

prepare a description and publish AqiSDS011 to the Google Play app store
check if converting a part of the application to the service would make AqiSDS011 more reliable for the long-running measurements

Cover image by Reimund Bertrams from Pixabay.

Using SDS011 particle sensor with my Android phone

Vedran Vidovic — Sat, 06 Feb 2021 09:04:12 +0000

After some testing of SDS011 particle sensor with my laptop (s described in the first post of this series) I realized it is not the most practical way to check for the air quality. I needed to take a laptop with me in a backpack, get it out of it, resume it, connect the sensor using the USB and start the Python3 script (while keeping the lighttpd server running during suspends).

At some point, I realized that I have a quite powerful and much smaller portable computer with me at all times - Android mobile phone. Fortunately, a few months ago I bought OTG cable for my phone.

I found a Serial USB Terminal for an Android phone, installed it and connected the SDS011 sensor to my phone. An application was started automatically and after I started a connection I saw some garbage characters periodically appearing in the terminal output - so, I guessed my phone can read the output from a sensor :)

First tests showed this guess was correct. Initial calculations proved that data fetched from the phone represent correct measurements. Hand-made calculations were made based on the Arduino C code from the joy-it sensor manual:

...
#define LEN 9
unsigned char incomingByte = 0;
unsigned char buf[LEN];
int PM2_5Val = 0;
int PM10Val = 0;
...
if (incomingByte == 0xAA) {
  Serial.readBytes(buf, LEN);
  if ((buf[0] == 0xC0) && (buf[8] == 0xAB)) {
    for (i=1; i<=6; i++) {
      checksum = checksum + buf[i];
    }
    if (checksum != buf[7]) {
      PM2_5Val=((buf[2]<<8) + buf[1])/10;
      PM10Val=((buf[4]<<8) + buf[3])/10;
...

Which means that for the following sequence of bytes:

hex: aa  c0  50  00  29  01  7f  bc  b5  ab
dec:         80  00  41  01 127 188 181

...I did the following math which shows same values I was able to measure using the above mentioned Python3 script on my laptop:

checksum (hex) = 50+00+29+01+7f+bc = 1b5 (which overflows to b5)
PM25 (dec) = (00*256 + 80) / 10 =  8.0
PM10 (dec) = (01*256 + 41) / 10 = 29.7

After some thinking, I realized this is a good use-case for a new Android application.

"Well, maybe I am not the first one thinking about that.."

I searched for such application on the Google Play. Unfortunately - no such application existed (at least at the time of writing this post).

I searched for Android application on the Internet and I found a GitHub repo for SDS011 Android application - much better. Unfortunately, this application is not published to the Google Play so I needed to build and install it "by hand".

I built and installed this application to my phone using the Android Developers instructions. This is not for the casual phone user, it requires a download and configuration of the Android Studio. However, the end result is nice - thanks to the developers of the SD011 Android app.

After some playing with this app, I found there are some downsides:

gauges' ranges are not showing proper colours for air quality according to the PM2.5 or PM10 levels
app doesn't record measurements history

If I find a time, I would like to create an Android application with the following functions:

save/view a history of measurements
- date-time & GPS included
- show location for each measurement on the map
- this way we can take a ride or walk around the town and later analyze data to see if air quality differs between different locations
pause/continue measurements
- this would save the sensor laser & my phone battery
configure periodic measurements
- this would save the sensor laser & my phone battery

Cover image & laptop image are created by:

Using SDS011 particle sensor from my laptop (python3)

Vedran Vidovic — Fri, 29 Jan 2021 07:11:24 +0000

Recently we started to check the quality of the air in our town using the IQAir / Air Visual and results were sometimes quite disturbing. That is why we decided to buy an air quality measuring device. Unfortunately, I was not able to find it in any of our local stores so I ordered it from the Amazon.de but we needed to wait for around 1 month (if not more) for this monitor to arrive.

I realized there is SDS011 Air Quality sensor available from our local electronics components store so we went there and bought it the same day.

It seems that I would only need a laptop with the USB port and working Python installation to use it. Of course, it was not 100% straight-forward to use it - instructions were made for usage with the Raspberry Pi machine and older Python2 version. It didn't take a long to adjust the open-source code cloned from the GitHub for my specific needs.

The forked project with changes (in case you need to use the sensor from your machine with the Python3 instead of no longer supported Python2 environment) is available on my GitHub fork of the original project.

A nice byproduct is that I realized I can add a measurements history to the sensor web page and I created a pull request to the original project which is now merged and available for everyone using this code with the Raspberry Pi :)

The cover image is created by the Foto-Rabe

Using datefudge to fake Docker date & time for testing

Vedran Vidovic — Wed, 04 Nov 2020 11:01:10 +0000

On one project we are starting docker container with our services running on the IBM WebSphere Liberty server and run tests which make sure our services are behaving according to our expectations.

Problem is that some of the scenarios we need to test include specific dates and times. For example, you can have a system which behaves differently if it is called during the working day vs. during a weekend.
Service can behave differently if it is called during working hours or during the night.

Current date and time in the docker container are connected to time on the host machine it is running on. We wanted to find a way to force our service to behave as it was called on any specific date and time - not only the current time of test execution.

Of course, we could change our service for the purpose of testing but I thought it would be nicer to find another way to accomplish the same task, without any changes to our services and our docker images.

Modifying the Linux system time without modifying the Linux system time

Fortunately, we did find some solutions to our problems:

Both of those libraries intercept various system calls that programs use to retrieve the current date and time and change results according to library configuration.

Here are examples of typical libfaketime & datefudge usage applied to a date command:

# LD_PRELOAD=/usr/lib64/faketime/libfaketime.so.1 FAKETIME="2020-10-22 12:13:14" date -R
Thu, 22 Oct 2020 12:13:14 +0200

# datefudge "2020-10-22 12:13:14" date -R
Thu, 22 Oct 2020 12:13:14 +0200

After an initial investigation, libfaketime seemed like a great solution for our task. Its configuration is quite flexible and it supports configuring it statically through the environment variable FAKETIME or dynamically through a local file.

For example, if we set the libfaketime library to be loaded by
LD_PRELOAD environment variable it will check FAKETIME environment variable for date & time configuration to use and if the environment variable is not found it will check the local file ~/.faketimerc for its configuration. This way we can change contents of this file to change date & time in a running program.

Unfortunately, libfaketime slowed down our service considerably (almost by order of magnitude) so I decided to investigate the datefudge library.

Datefudge configuration was unfortunately much less flexible - it supported only the configuration during the initial start of our service. However, initial tests showed that our service didn't show visible slow down so I checked how hard would it be to make it a bit more configurable.

Customizing datefudge

Source code of datefudge version 1.24 is ingeniously simple, it combines a single shell script and less than 100 lines of C code.
Datefudge uses shell date command to parse date string and to get the current time which makes C code quite straight-forward - C code just uses prepared values passed to it through environment variables.

Unfortunately this approach is not possible if date is configured dynamically - if configuration is changed after datefudge is initialized. In that case our date/time configuration string must be parsed by C code. GNU time.h library fortunately implements strptime
function which makes parsing quite straight-forward. With just a few tweaks partial date-time string parsing & correct daylight saving handling is possible:

// Initialize tm struct to 1900-01-01 00:00:00 & tells mktime() to
// determine whether daylight saving time is in effect
struct tm tm = (struct tm){.tm_mday = 1, .tm_isdst = -1};
strptime(datetime_str, "%Y-%m-%d %H:%M:%S", &tm);
time_t config_time = mktime(&tm);

Customized datefudge library adds the following feature to the datefudge v1.24 version:

date & time can be configured using a local file
- when this file is changed datefudge will load new configuration, however, the datefudge configuration file is checked for changes at most each 100 ms (this time can be configured using -s flag) to lower the performance costs of a file access operation

Datefudging a docker container with our service

We have multiple test scenarios containing multiple test steps. We can execute a single test scenario or all test scenarios using a single Docker container started automatically before our tests start to run.

For example we can have:

test scenario 1
- test step 1
- test step 2
test scenario 2
- test step 1
- test step 2
- test step 3

For our tests, we wanted to be able to set our service date & time for each of our test steps and every call to service during that step (and steps following it) will use new date & time.

(We prepared automated tests using Testcontainers project but that is a different story which doesn't belong to this post)

The final solution consists of the following:

copy customized datefudge command (shell script) & datefudge shared library to the container (we are using Testcontainers GenericContainer.withCopyFileToContainer to achieve that but this could be done using different strategies otherwise)
change the CMD (default docker command from the image)
- for our WebSphere Liberty image CMD is by default set to CMD ["/opt/ibm/wlp/bin/server" "run" "defaultServer"]
- on container run we set it's CMD to CMD ["datefudge" "-f" "/home/default/date.txt" "/opt/ibm/wlp/bin/server" "run" "defaultServer"]
for each test scenario step where we need to set custom date & time we copy a new file with specific date & time to the running container as a datefudge configuration file /home/default/date.txt before running tests (and wait (configurable) 100 ms to make sure datefudge will apply new date & time configuration before tests are run)

Running tests with customized system times

For above given example if we add custom datefudge configuration for scenario 1 / step 2 (date-time A) and additional configuratno for scenario 2 / step 1 (date-time B) tests are executed like this:

scenario 1
- test step 1 (container uses "normal" date & time)
- configure date-time A
- test step 2 (container uses date-time A date & time)
test scenario 2
- configure date-time B
- test step 1 (container uses date-time B date & time)
- test step 2 (container uses date-time B date & time)
- test step 3 (container uses date-time B date & time)

Conclusion

If you need to test the behaviour of your application at specific date & time (on Linux, Docker or otherwise) it is quite straight-forward to use datefudge and/or libfaketime to achieve this without ever touching your application code.

The great news is that both of those libraries are opensource under GPL-2.0 so if you need some additional features you could add them without any problems. If you do that, please let the original authors know you did those changes so they include them in the next releases of libraries. Open-source community will be thankful to you :)

The cover image is created by Kaboompics .com from Pexels.

Set up your new machine in a blink of an eye

Vedran Vidovic — Thu, 13 Aug 2020 13:39:21 +0000

When my company internal IT informed me that my new laptop arrived I was glad to finally set up a new Ubuntu LTS release. While Ubuntu comes with a lot of great applications installed out-of-the-box there are many applications which I have to install after OS installation to make my laptop useful for everyday work. Of course, there is the third part of the laptop setup equation which makes my work possible - my data :)

My laptop is set up using these scripts in a repeatable way in minutes. I just keep them in a private GitHub repo and make sure that I don't install applications manually without adding them to those scripts.

No sensitive data like passwords or similar are uploaded there but I still don't want to make this repo public because some of our company client names and our company project names are visible in some of the rules for the backup (exclude/include).

Restore of the most valuable data is ensured with backups to both, external drive & Cloud.

TLDR;

Using a collection of Bash scripts and the GNU Stow you can set up your machine quickly in an easily repeatable way. Make all of the operations you perform idempotent so you can run all scripts as many times as you want (on a new machine, on a partially set up machine or if you just want to make sure all apps are installed).

Add a proper backup application to that and you won't need to spend much time on setting up your new laptop - you will be able to start to use it almost immediately after it arrives.

OS Installation

[Ubuntu installation[(https://ubuntu.com/tutorials/install-ubuntu-desktop) is quite straight-forward and there is no need to describe it here in more details.

Adding applications & OS customization

I have read a few articles about configuring and setting up a machine using GNU Stow and a bunch of shell scripts and I wanted to check how good can it work for me.

What I came up with was a set of scripts with where I need to call only install.sh script (to be more precise, sudo install.sh) which in turn, calls all the other scripts:

#!/bin/bash
set -eu

$(dirname $0)/install-from-repos.sh
$(dirname $0)/install-custom.sh
$(dirname $0)/install-github-releases.sh
$(dirname $0)/install-custom-opt.sh

$(dirname $0)/install-config.sh

install-from-repos.sh

This script first adds custom Ubuntu repositories (using another script) and after that executes a bunch of apt install & snap install commands, something like:

#!/bin/bash
set -eu

$(dirname $0)/add-custom-repos.sh

apt install -y git
apt install -y stow
apt install -y source-highlight
apt install -y vim
...
snap install xmlstarlet --classic
snap install go --classic
...

install-custom.sh

Custom installation is for now very simple - it just enables syntax colouring in the less pager command:

#!/bin/bash
set -eu

echo -e '#!/bin/bash\ndiff -u -r "$1" "$2" | cdiff | less -R' > /usr/local/bin/ldiff
chmod +x /usr/local/bin/ldiff
echo "installed ldiff script"
echo ""

install-github-releases.sh

Installation of GitHub releases is more complex but it boils down to fetching releases list using GitHub REST API and installing the latest version (if it is not already installed). There are more of it for many different applications but this will give you enough info to make your version of a script:

#!/bin/bash
set -eu

echo ""
echo "Starting to install custom releases from GitHub."
echo ""

# Fetches available versions from GitHub and finds the latest version satisfying
# grep constraints
function find_github_latest_release_url() {
  local -r release_repo_path="$1"
  local -r release_grep="$2"

  local -r release_list_url="https://api.github.com/repos/${release_repo_path}/releases"

  local -r asset_download_url_list="$(curl -s \
    "${release_list_url}" \
    | jq -r '.[].assets[].browser_download_url')"

  local -r asset_url="$(echo "${asset_download_url_list}" \
      | grep "${release_grep}" \
      | head -n1)"

  if [[ -z "asset_url" ]]; then
    echo "Latest relase for '${release_repo_path}' with grep '${release_grep}' not found, exiting." >&2
    exit 1
  fi

  echo "${asset_url}"
}

# Get version from GitHub release download URL eg:
# https://github.com/croz-ltd/dpcmder/releases/download/v0.6.0/dpcmder-linux-amd64
function get_github_release_version() {
  local -r asset_url="$1"

  echo "$(echo "${asset_url}" | sed 's|.*/download/||; s|/.*||')"
}

function install_asset() {
  local -r local_version="$1"
  local -r release_repo_path="$2"
  local -r release_grep="$3"
  local -r installer_function_name="$4"

  local -r asset_url="$(find_github_latest_release_url \
    "${release_repo_path}" "${release_grep}")"
  local -r github_version="$(get_github_release_version "${asset_url}")"
  echo "Local version: '${local_version}', version on the GitHub: '${github_version}' (${release_repo_path})"
  if [[ "${local_version}" != "${github_version}" ]]; then
    asset_file="${asset_url##*/}"
    wget -nv "$asset_url" -O "${asset_file}"
    ${installer_function_name} "${asset_file}"
  fi
}

################################################
# Callback functions for apps installation BEGIN
################################################

#########
# dpcmder
#########
function install_dpcmder() {
  local -r downloaded_file="$1"
  chmod +x "${downloaded_file}"
  mv "${downloaded_file}" /usr/local/bin/dpcmder

  echo ""
  echo "installed dpcmder: $(dpcmder -v || true)"
  echo ""
}

#########
# oc
#########
function install_oc() {
  local -r downloaded_file="$1"
  tar -xf "${downloaded_file}"
  local -r unpack_dir="${downloaded_file%.tar.gz}"
  mv "${unpack_dir}/oc" /usr/local/bin/oc
  mv "${unpack_dir}/kubectl" /usr/local/bin/kubectl
  rm -rf "${unpack_dir}"
  rm "${downloaded_file}"

  oc completion bash > /etc/bash_completion.d/oc_bash_completion

  echo ""
  echo "installed oc: $(oc version || true)"
  echo ""
}

################################################
# Callback functions for apps installation END
################################################

################################################
# Install apps
################################################

################################################
# dpcmder
dpcmder_version_full="$(dpcmder -v || true)"
dpcmder_version="$(echo "${dpcmder_version_full}" | sed 's/.* version //; s/ .*//')"
install_asset "${dpcmder_version}" "croz-ltd/dpcmder" "linux" install_dpcmder

################################################
# oc & kubectl
oc_version_full="$(oc version || true)"
oc_version="$(echo "${oc_version_full}" | grep oc | sed 's/.*v/v/; s/+.*//')"
install_asset "${oc_version}" "openshift/origin" "v3.*client.*linux" install_oc

echo ""
echo "Finished installing custom releases from GitHub."
echo ""

install-custom-opt.sh

This script installs custom applications which I usually unpack to the /opt directory, for example Eclipse. Here I install specific version of the product and URL to each product is hardcoded in this script. There are not many of these so I find that this approach is the best fit for my requirements. For these apps following steps are performed:

check the current version installed
- does directory opt/{app_name}.{app_version} exist?
if the target version is not found:
- download an archive
- unpack an archive to opt/{app_name}.{app_version} directory
- make a symbolic link from opt/{app_name} to opt/{app_name}.{app_version}

install-config.sh

This is where the custom configuration (not installation) of my Ubuntu is performed. These are custom configuration tasks performed by this script:

setting the vim as my default editor
stowing my dotfiles
make docker run as a non-root user
changing some Ubuntu key bindings
configuring my restic backups
enabling ufw

This is a bit shortened version of this script:

#!/bin/bash
set -eu

username="my_username"

update-alternatives --set editor /usr/bin/vim.basic

sudo -u ${username} mkdir -p /home/${username}/.config/systemd/user

# Get all directories except bin
STOW_DIRS="$(ls */ -d | grep -Ev 'download|bin')"
sudo -u ${username} stow --dir=/home/${username}/dotfiles \
  --ignore=download \
  --ignore=bin \
  ${STOW_DIRS}

# run docker as a non-root user
groupadd docker || true
usermod -aG docker ${username}


# Remove <Ctrl><Alt>[<Shift>]ArrowKey mappings
sudo -u ${username} dconf write /org/gnome/desktop/wm/keybindings/move-to-workspace-down "['<Super><Shift>Page_Down']"

# Configure restic-backup timer
sudo -u ${username} loginctl enable-linger username
sudo -u ${username} systemctl --user enable restic-backup.timer
sudo -u ${username} systemctl --user start restic-backup.timer

ufw enable

dotfiles & Stow - examples

To give you the better overview of what can be configured using Stow here are some example of files which I keep in the GIT repo:

# "classic" dotfile configurations
dotfiles/bash/.bashrc
dotfiles/bash/.profile

# scripts for the setup
dotfiles/bin/add-custom-repos.sh
dotfiles/bin/install-config.sh
dotfiles/bin/install-custom-opt.sh
dotfiles/bin/install-custom.sh
dotfiles/bin/install-from-repos.sh
dotfiles/bin/install.sh

# Restic backup scripts
dotfiles/bin/restic/.gitignore
dotfiles/bin/restic/restic-backup.sh
dotfiles/bin/restic/restic-exclude
dotfiles/bin/restic/restic-include
dotfiles/bin/restic/restic-password

# Gnome desktop launcher files
dotfiles/eclipse/.local/share/applications/eclipse.desktop

# Restic service & timer configuration
dotfiles/restic-backup/.config/systemd/user/restic-backup.service
dotfiles/restic-backup/.config/systemd/user/restic-backup.timer

Restic backup setup

For the backup, I use the restic application written in Go. I won't write a lengthy description about it here but I do find it a great app with many configuration options to securely back up your encrypted data. If you want to get a good description of how to use Restic I would suggest you check it's documentation on "Read the Docs" pages. It can also make a backup to the AWS S3 which I use. In my case, this is just a few GB backup of the most valuable data. Costs of storing a few GB in AWS S3 is just around a 1$ per year and gives me a peace of mind for the worst-case scenario.

Restic is run by systemd timer running systemd service and both of those are created using Stow + custom configuration (check sytemctl enable & systemctl start in the install-config.sh script).

My Restic configuration is one of dotfiles configuration directories. Restic script run by restic-backup.service is also present there:

dotfiles/bin/restic/.gitignore
dotfiles/bin/restic/restic-backup.sh
dotfiles/bin/restic/restic-exclude
dotfiles/bin/restic/restic-include
dotfiles/bin/restic/restic-password

dotfiles/restic-backup/.config/systemd/user/restic-backup.timer
dotfiles/restic-backup/.config/systemd/user/restic-backup.service

Just make sure that you don't commit your restic-password file to GIT repository by mistake - add it to your .gitignore file :)

restic-backup.sh

#!/bin/bash
set -eu

restic backup \
  -r ~/restic-repo \
  --exclude-caches \
  --files-from $(dirname $0)/restic-include \
  --exclude-file $(dirname $0)/restic-exclude \
  --password-file $(dirname $0)/restic-password

restic-backup.timer

###################################################
# restic-backup.timer - runs restic backup each day
# loginctl enable-linger vedran
# systemctl --user list-timers
# systemctl --user enable restic-backup.timer
# systemctl --user start restic-backup.timer

[Unit]
Description=Regular restic backup

[Timer]
Unit=restic-backup.service
OnCalendar= *-*-* 15:00:00
Persistent=true

[Install]
WantedBy=default.target

restic-backup.service

##########################################################################
# restic-backup.service - runs restic backup (used by restic-backup.timer)
# systemctl --user status restic-backup

[Unit]
Description=Restic backup

[Service]
Type=oneshot
ExecStart=/home/vedran/dotfiles/bin/restic/restic-backup.sh

Which data I do back up using this setup?

I back up with this setup only the data I find hard to "reconstruct".

I don't back up application installations available from the Internet - these can be easily fetched again from the Internet if required.

I don't back up virtual machines using this setup - there is only one VM I do back up manually, periodically but I just save whole VirtualBox machine to the external drive.

Conclusion

This approach is a DevOps inspired way to set up and configure a new desktop machine (Ubuntu Linux OS). Simplicity and ease of this approach make a process of the migration to the next machine so fast you won't think twice about it.

The cover image is created by Pixabay.

Viewing more with less - color syntax

Vedran Vidovic — Sat, 22 Feb 2020 09:50:49 +0000

If you are using Linux command-line a lot (out of love or out of necessity) you probably use less command a lot too. When you need to check the contents of some file less is here to help you to do it quickly. Maybe you didn't know but less can do more than help you view your text files, for example, you can check the contents of your archive using less:

$ less go1.13.7.linux-amd64.tar.gz

Syntax highlighting for less - source-highlight

The only issue I had with the default less configuration on my Linux machines (Ubuntu 16.04 & 18.04) was that all of my source files were shown without any syntax highlighting. For me it was not just an issue of form, it was an issue of function - syntax highlighting makes reading source files much easier.

Fortunately, for Linux environments, you can use a GNU Source-highlight project which can be used by less command to add syntax highlighting to files opened using less command.

Just install syntax-highlight for your favorite Linux distribution and configure two environment variables less uses to get syntax highlighting in your favorite pager command.



sudo apt install source-highlight
export LESSOPEN="| /usr/share/source-highlight/src-hilite-lesspipe.sh %s"
export LESS=' -R'

If you want to have these settings applied permanently to your system add something like this to your ~/.bashrc or ~/.profile file:



# less syntax highlighting + source-highlight installation
export LESSOPEN="| /usr/share/source-highlight/src-hilite-lesspipe.sh %s"
export LESS=' -R'

The result can be easily seen if you open for example JSON file and compare how less command shows it with highlighting vs. without highlighting:

Customizing source-highlight - adding new languages

There are some languages that you won't find on the list of supported languages for the source-highlight project. Often, you can find a solution to this problem with some help from the development community on the Internet. For example, YAML is not supported by default but you can find a solution created by user tkfm-yamaguchi and available as a GitHub gist. Basically you have to add make two small changes (one new configuration (language) file and small change of the existing configuration file) to enable syntax highlighting for your YAML files.

The result is visible below:

Trimming go binaries of dpcmder - trimming a compression time :)

Vedran Vidovic — Wed, 12 Feb 2020 14:07:57 +0000

Preparing for the next release of dpcmder I wanted to do some more testing of release binaries compression achievable using UPX and find out I can achieve 40 times faster compression just by changing single upx compression flag.

Note: this continues where the last post about dpcmder binaries compression finished.

I realized there are some upx flags I didn't test (for example "--ultra-brute" or "--best") and read a bit more about them. In the end, it seems (for dpcmder binaries at least) best compression results are achieved using the "--lzma" flag:

$ time upx --lzma release.lzma/*
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   9146368 ->   2812760   30.75%  linux/ElfAMD   dpcmder-linux-amd64           
   9697256 ->   2789376   28.76%   Mach/AMD64    dpcmder-mac-amd64             
   8932352 ->   2610688   29.23%    win64/pe     dpcmder-win-amd64.exe         
   --------------------   ------   -----------   -----------
  27775976 ->   8212824   29.57%                 [ 3 files ]

Packed 3 files.

real    0m23.254s
user    0m22.911s
sys 0m0.196s

Seems that the "--brute" flag makes upx tries many different algorithms and selects one which achieves the best result. The "--ultra-brute" flag tries even more compression algorithms so it takes even more time. In my case (dpcmder binaries built on my Linux machine) best result is achieved using the lzma compression algorithm. As you can see from outputs of both upx run timings, result compression is exactly the same while time spent for compression is drastically faster (more than 40 times) when using "--lzma" flag.

$ time upx --brute release.brute/*
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   9146368 ->   2812760   30.75%  linux/ElfAMD   dpcmder-linux-amd64           
upx: release.brute/dpcmder-linux-amd64.upx: UnknownExecutableFormatException
   9697256 ->   2789376   28.76%   Mach/AMD64    dpcmder-mac-amd64             
   8932352 ->   2610688   29.23%    win64/pe     dpcmder-win-amd64.exe         
   --------------------   ------   -----------   -----------
  27775976 ->   8212824   29.57%                 [ 3 files ]

Packed 3 files.

real    15m39.952s
user    15m37.567s
sys 0m1.408s

Trimming go binaries of dpcmder

Vedran Vidovic — Mon, 13 Jan 2020 11:43:16 +0000

This weekend I spent some time thinking that (Golang built) binaries of dpcmder are a bit too huge for my taste. Some would say that ~10 megs is not so much for today's world but ~10 megabytes seemed to me to be too much so I set out on a mission to find out if I can trim dpcmder size a bit.

After some googling, I found this nice "shrink your go binaries" blog which explains exactly what I needed to do to achieve my goal.

It boils down to one change in the current build process (remove debugging info & system table) and one additional step after the build (compress binaries).

Removing debugging info & system table

Instead of using default go build command, for example:
GOOS=linux GOARCH=amd64 go build -o release/dpcmder-linux-amd64 dpcmder.go

You have to add additional ldflags to instruct the linker to remove all debugging info (-w) & symbol table (-s):
GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o release/dpcmder-linux-amd64 dpcmder.go

Actually, this research of ldflags brought me to one other issue on my dpcmder todo list - adding application version for each release. This caused me to use one more ldflag (-X) to change values of version variables:
GOOS=linux GOARCH=amd64 go build -ldflags="-s -w -X 'github.com/croz-ltd/dpcmder/help.Version=$(git tag | tail -n1)' -X 'github.com/croz-ltd/dpcmder/help.Platform=linux/amd64' -X 'github.com/croz-ltd/dpcmder/help.BuildTime=$(git tag | tail -n1).$(date -u -Iseconds)'" -o release/dpcmder-linux-amd64 dpcmder.go

Compressing binaries

UPX is a free, portable, extendable, high-performance executable packer for several executable formats.

After installing UPX it is easy to use it and for the best compression results I added the flag "--brute" mentioned in the blog above:
upx --brute release/*

$ upx --brute release/*
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   7925760 ->   2538980   32.03%  linux/elf386   dpcmder-linux-386             
   9105408 ->   2686192   29.50%  linux/ElfAMD   dpcmder-linux-amd64           
   8421068 ->   2772992   32.93%    Mach/i386    dpcmder-mac-386               
   9653400 ->   2785280   28.85%   Mach/AMD64    dpcmder-mac-amd64             
   7698432 ->   2449408   31.82%    win32/pe     dpcmder-win-386.exe           
   8865792 ->   2597376   29.30%    win64/pe     dpcmder-win-amd64.exe         
   --------------------   ------   -----------   -----------
  51669860 ->  15830228   30.64%                 [ 6 files ]

Packed 6 files.

This step took quite a long time to complete but results were worth waiting. As you can see below from binary files update times, some binaries were compressed faster and some were compressed slower (compression of win/386 binaries lasted almost 30 minutes while compression of Linux binaries lasted for around 3 minutes).

$ stat -c "%.19z %n" release/*
2020-01-13 10:23:02 release/dpcmder-linux-386
2020-01-13 10:26:57 release/dpcmder-linux-amd64
2020-01-13 10:29:52 release/dpcmder-mac-386
2020-01-13 10:36:54 release/dpcmder-mac-amd64
2020-01-13 11:03:45 release/dpcmder-win-386.exe
2020-01-13 11:10:54 release/dpcmder-win-amd64.exe

Note that these 6 binaries could be compressed in parallel to achieve much better times on the modern-day multi-core machines.

How much did we trimmed?

After the whole process, all 6 binaries are now much smaller. The total size of all binaries shrank from the original 65M to a more acceptable 16M. Sizes for all 6 binaries are given below.

default go build without any -ldflags

$ ls -sh1 release/
total 65M
11M dpcmder-linux-386
12M dpcmder-linux-amd64
11M dpcmder-mac-386
12M dpcmder-mac-amd64
11M dpcmder-win-386.exe
12M dpcmder-win-amd64.exe

go build with -ldflags="-s -w"

$ ls -sh1 release/
total 50M
7,6M dpcmder-linux-386
8,7M dpcmder-linux-amd64
8,1M dpcmder-mac-386
9,3M dpcmder-mac-amd64
7,4M dpcmder-win-386.exe
8,5M dpcmder-win-amd64.exe

result of running UPX on go build (with -ldflags="-s -w")

$ ls -sh1 release/
total 16M
2,5M dpcmder-linux-386
2,6M dpcmder-linux-amd64
2,7M dpcmder-mac-386
2,7M dpcmder-mac-amd64
2,4M dpcmder-win-386.exe
2,5M dpcmder-win-amd64.exe

After comparing the latest dpcmder release assets sizes to the previous dpcmder release asset sizes I must conclude that my mission from the start of this blog is successfully accomplished :)

DataPower Commander v0.3 - more useful than ever

Vedran Vidovic — Fri, 10 Jan 2020 09:12:14 +0000

DataPower Commander (dpcmder) is a command line tool I created for easier maintenance of files on DataPower appliances and easier development of DataPower appliance solutions. You can use dpcmder too – it is an open source project available on GitHub (croz-ltd/dpcmder).

It is developed using go (golang) which makes cross-platform releases much easier and cross-compilation is built in as a go tool feature :)

Since last time I wrote about dpcmder so many new features were added that I felt compelled to write a new blog and describe how can dpcmder make your day nicer and your work with DataPower easier. Let's be honest, it will make your whole life easier :)

Most interesting new features are:

DataPower appliance configuration list
domain export
object maintenance mode
- view object configuration
- edit object configuration
- delete object configuration
- view non-saved changes in object configuration
- save current domain changes
nicer input/progress dialogs
status messages history

A bit more comprehensive list of new features will be given below but let's help you to start using dpcmder before we come to that.

Getting started with dpcmder

DataPower Commander works under Linux, macOS and Windows (normal Windows cmd prompt) but since there are no default file viewer/editor/diff externals command-line programs available under Windows it is a bit "harder" to do the initial setup for Windows environment. I would suggest you to use WSL or cmder environment to make all features of dpcmder easier to use.

The Best way to start using dpcmder is to run it with "-h" flag and see all flags available and then just call with appropriate flags to connect to your DataPower appliance, for example:
dpcmder -s https://127.0.0.1:5550 -u admin -p admin -c LocalDp

This command will connect to your local DataPower instance which exposes its SOMA interface at port 5550 using the default username and password combination and will save this configuration under given configuration name "LocalDp". Don't worry if you forgot to add "-c" flag - DataPower configuration just opened is saved as "PreviousAppliance" in the DataPower appliance list where you can easily clone it to whatever configuration name you need using F9/9 key.

You can even start without using any command line parameters when starting dpcmder for the first time and create a new DataPower configuration using the F8/8 key (when the [empty] list of appliances is shown). However, this option is not recommended if you plan to save your password to dpcmder configuration file (~/.dpcmder/config.json) - passwords are saved in the configuration file as base32 encoded text so you would have to properly encode password value before saving it.

If you need more information you can pass flag "-help" to dpcmder and it will write full dpcmder help to console so you can get more information about its features and their use.

Note on appliances configuration password safety

If passwords are saved in a dpcmder configuration file (if they are passed to dpcmder using "-p" flag) they will be base32 encoded so anyone with access to your dpcmder configuration file would be able to decode it easily. If you cannot guarantee the safety of this file, please avoid using "-p" flag and enter a password when you connect to each appliance - this password will not be saved into the dpcmder configuration file.

DataPower appliance list

When you open DataPower commander without any parameters a list of saved DataPower appliance configuration is shown (which can be empty if no appliances were added through command-line flags or by using F8/8 key).

You can connect to DataPower using a given configuration (Enter key), view current configuration (F3/3 key), edit the current configuration (F4/4 key), clone current configuration (F9/9), create a new empty configuration (F8/8 key) or delete current configuration (Del/x key).

If the configuration for DataPower appliance you want to access doesn't contain password information, the first time you want to connect to that appliance (in running dpcmder program) a dialog for password input will be shown.

If you run dpcmder using proper SOMA or REST connection parameters on command line DataPower appliance list will not be shown - list of domains on DataPower you wanted to connect to (using command line parameters) will be shown instead. You can easily go back to the DataPower appliance list by going to the parent view (selecting ".." item).

DataPower domain export

One of the new features for which I felt to fit most naturally into dpcmder is exporting of the DataPower domains. Just as it is possible to copy files and/or directories from DataPower to the local filesystem it feels natural to copy the whole domain to a local filesystem. Actually, the same key is used for that purpose (F5/5 key) and the result of that operation is the DataPower export zip archive created on a local filesystem.

The exported archive contains all objects and files from the selected DataPower domain.

I am considering to add function to import domain (and even delete domain) but since these operations are quite powerful there is a bigger chance to make some "damage" to your appliance using those. However, these options are something that would nicely round out features of working with DataPower domains so these will probably be added in some future version of the dpcmder.

DataPower object maintenance mode

Object maintenance mode is very useful for making small changes to existing DataPower object configuration or viewing configuration changes (SOMA only). For example, changing the port number used for XML Firewall Service. Or maybe you don't want to change anything but just want to quickly check the current value of some configuration option.

In the new release of the dpcmder, 0 key is used to switch between (default) "file" mode and "object" mode. To use object mode we have to connect to the DataPower appliance and select one of the domains available.

When we switch to object mode a list of all "class" of objects for which at least one object in the selected DataPower domain exists is shown. Instead of file size count of available objects is shown. In case we used the SOMA management interface to access DataPower classes we mark all modified classes with "*" (if at least one object is new, modified or deleted for that class).

The same key (Enter key) used for the operation of "entering" domain/filestore/directory is used to show all available objects of the selected object class. In case we used the SOMA management interface we show if an object is new, deleted or modified (and can even view changes for a modified object using "d" key). That operation shows us names of all objects of the selected class and enables us to do more advanced operations on each of those objects:

view object (JSON/XML)
edit object (JSON/XML)
delete object
view unsaved object changes (XML)

In case we used the REST management interface to connect to the DataPower appliance object will be accessible using JSON, in case we used SOMA management interface XML will be used. The best feature we get "for free" is that if our editor (or viewer) supports syntax highlighting for JSON/XML we will have nicely highlighted code making our viewing and/or editing of DataPower configuration easier.

List of interesing changes since last blog

For a more comprehensive list of changes you can check releases history but here is a summary of changes:

fixes
DataPower appliance configuration saving and loading
diff command (comparing files / directories)
better error handling in calls to external programs (+ hints how to fix those)
better in program help
shows program help in console ("-help" flag)
creation of new (empty) file
view status history ("m" key)
replaced library for terminal interaction (termbox-go -> tcell) for better cross platform support
added nicer terminal dialog and error messages
blinking status when sync mode is on
added object mode (edit and delete operations available)
editing DataPower files can use syntax highlighting available in external editor
workaround enabling to use diff command (normal, non-blocking) - custom blocking command is still recommended

Features of dpcmder

appliance configurations
- saves and loads DataPower appliance configuration to enable fast connections to appliances
- edit/clone/create/delete DataPower appliance configuration
file maintenance
- view, edit, copy and delete file hierarchies (DataPower and local file system)
- filter and search files in current directory
- comparing files / directories (DataPower vs. local file system)
objects maintenance
- switching between filestore mode and object mode (0 key)
- existing objects can be quickly edited (as JSON (REST mgmt) or XML (SOMA mgmt) using current editor)
- existing objects can be quickly deleted
domain export
- when applying copy operation to DataPower domain it is automatically exported as zip files (exporting all local files)
sync mode
- turn on to automatically upload new and changed files from local filesystem to DataPower
- useful for development to automatically propagate your changes from any IDE/editor you are using to DataPower

REST management vs. SOMA management

Some of the later features added were available only if we use the SOMA management interface to connect to the DataPower appliance. While the REST management interface seems a bit nicer some features are not available and/or documented. For example, Diff operation seems to be available through REST but I wasn't able to find any documentation about that action or find a way to use it. From 154 actions available through /mgmt/actionqueue/{domain} (IDG.2018.4.1.3) only 3 are documented in IBM's Knowledge Center:

LoadConfiguration
Export
Import

Some functions of DataPower object mode work a bit differently in REST vs. SOMA mode - for example in the latest version available on the Git master branch, if the object's admin state is down we can't make changes to it using a REST management interface (REST call returns the HTTP error 409 Conflict).

Issues mentioned above are reasons why I would suggest you to use the SOMA management interface for your DataPower management - REST is nice but it still doesn't offer everything SOMA does.

So Long, and Thanks for All the Fish

There are still many new features we consider adding to dpcmder so its development will continue further. For example - currently, dpcmder has a feature to edit or delete DataPower objects but doesn't have a feature to create a new object (it would be a bit hard to use). The next version (already on Git master) will be able to clone the existing one or even copy the object configuration to a local file or create/update the DataPower object from the local configuration file. Currently, dpcmder can create a new domain but can't delete the existing one, it can export the DataPower domain but can't import it - these are some of the areas we consider for dpcmder further development.

The last version seems to be much nicer and contains a lot more functions compared to the first release but this doesn't mean DataPower Commander is complete so stay tuned, there will be new features for dpcmder in 2020. :-)

Forem: Vedran Vidovic

How to run SoapUI mockservicerunner in a Docker container

TLDR;

Creating mock service in the SoapUI

Running the created mock service from the command line

Running the created mock service as a Docker container

Creating the soapui-mockservicerunner image

Using the soapui-mockservicerunner image

Conclusion

AqiSDS011 - Android AQI app

Using the existing sds011_android app

AqiSDS011 - the android app

The modes of operation of the SDS011 sensor

The current status of the AqiSDS011

The road ahead

Using SDS011 particle sensor with my Android phone

Using SDS011 particle sensor from my laptop (python3)

Using datefudge to fake Docker date & time for testing

Modifying the Linux system time without modifying the Linux system time

Customizing datefudge

Datefudging a docker container with our service

Running tests with customized system times

Conclusion

Set up your new machine in a blink of an eye

TLDR;

OS Installation

Adding applications & OS customization

install-from-repos.sh

install-custom.sh

install-github-releases.sh

install-custom-opt.sh

install-config.sh

dotfiles & Stow - examples

Restic backup setup

restic-backup.sh

restic-backup.timer

restic-backup.service

Which data I do back up using this setup?

Conclusion

Viewing more with less - color syntax

Syntax highlighting for less - source-highlight

Customizing source-highlight - adding new languages

Trimming go binaries of dpcmder - trimming a compression time :)

Trimming go binaries of dpcmder

Removing debugging info & system table

Compressing binaries

How much did we trimmed?

default go build without any -ldflags

go build with -ldflags="-s -w"

result of running UPX on go build (with -ldflags="-s -w")

DataPower Commander v0.3 - more useful than ever

Getting started with dpcmder

DataPower appliance list

DataPower domain export

DataPower object maintenance mode

List of interesing changes since last blog

Features of dpcmder

REST management vs. SOMA management

So Long, and Thanks for All the Fish

Creating the `soapui-mockservicerunner` image

Using the `soapui-mockservicerunner` image