Forem: webduvet

Trouble shooting docker - permission denied

webduvet — Mon, 22 May 2023 16:55:31 +0000

After installation or just after a an OS update or some package update docker engine starts to complain about insufficient permissions.
Particularly it complains about permission to access /var/run/docker.sock

docker ps
// permission denied message

$> ls -la /var/run/docker.sock

The output should show root:docker as {ownder}:{group}.
Run also

ls -l /lib/systemd/system/docker.socket

The output should show root:docker as {ownder}:{group}.
If it shows root:root there is a problem

sudo chgrp docker /lib/systemd/system/docker.socket

Check the following:

groups

Current user should be member of these groups: docker, kvm.
if the docker group doesn't exist add it

sudo groupadd docker

If the new group is added or the group is there but current user is not a member, add it.

sudo usermod -aG kvm $USER

The current user now should be visible in group file /etc/group
or when running

getent group

or simply

groups

If not try running:

sudo su $USER -c groups

If the list of groups contain kvm and docker you need to log out and log back in or somehow restart system-auth.
After that it should work or there is further problem unrelated to the above.

verify

docker ps

Deferred Promise Queue

webduvet — Sun, 09 Apr 2023 10:29:16 +0000

Motivation

I recently published a simple npm package to defer the execution of collection of promise based asynchronous calls.

If you ever had to do multiple async calls returning a promise object, but the service or client machine was not able to handle all calls concurrently, there are chances that something like this was used.

An obvious use case would be getting data from the remote API page by page and it is not uncommon that it could be hundreds if not thousands pages long (which was by the way the initial motivation to create the library at first place).

Deferred pattern

At the core of the implementation is something which could be described as deferred Promise. We can work with an array of Promises hundreds or thousands items long, but we can't trigger them at the same time. So the the async function calls need to be deferred until the system is able to process them. How to do that I explained in this article about Deferred Promise

Publish - subscribe

The second piece in the puzzle is to implement ability to throttle the calls.
Under the hood this is done by implementing publish-subscribe pattern and schedule-task pattern.
The entire flow is triggered by calling start method which in turn emits event that the queue is available to take a call. (Fig.1). Promise Queue is scheduling the individual async function calls periodically (Fig.2) until the buffer is filled. Buffer here is the limit on currently pending promises. Once this limit is reached the scheduler postpones the next task until at least one pending promise is converted to settled (Fig.3).

Flow Chart

Fig.1, Initial call

Fig.2, periodic loop

Fig.3, coming back from blocked queue

Notice that all the parts (apart beginning) starts and end with emitting an event. This allows to add multiple consumers e.g. for logging or debugging.

Wrap up

If anybody find this useful I wrapped all of the above in the platform agnostic npm package, which can be installed by

$ npm install prmsq

The basic usage example:

// fetchPage(pageNumber) is http call returning promise
const workload = range(1, 1000)
    .map((pageNo) => () => fetchPage(pageNo));

const delayBetweenCalls = 100;
const concurentLimit = 10
const pq = new PrmsQ(workload, delayBetweenCalls, concurentLimit )

Promise.allSettled(pq.promises)
.then(doWhenAllSettled)
.catch(handleProblem)

pq.start();

It also can be found on github.com/webduvet/prsmq
with the example in /src/example/

Deferred (promise) pattern

webduvet — Sat, 08 Apr 2023 18:25:12 +0000

Deferred pattern is one of the most commonly used design patterns in JavaScript world and in programming in general. It postpones the action to a later stage for whatever reason, but mostly because the expected value is not available yet.

Promise falls nicely to this category. It boxes the future value and provides a way to unbox it whether it realizes (then) or fails (catch). The basic usage reveals how the Promise pattern does work:

const promise = new Promise(function(resolve, reject) {
    asyncCall(function(err, result) {
        if (err) {
            reject(err);
        } else {
            resolve(result);
        }
    })
})

promise.then(successHandler).catch(rejectHandler)

New Promise object is constructed and the resolver function is called some time in the constructor. From inside resolver function the asynchronous call is triggered and once the result comes back the status of the Promise changes from pending to settled by calling reject or resolved. At that point any callback provided by calling .than() pipe is executed in the same order how they were registered.

From the above is evident that the programmer or user has limited control when the promise settlement start to happen. Calling the constructor makes the things moving right away. Of course unless we inject some handlers from calling scope or we close over the provided resolve and reject levers.

const makeDeferred = () => {
    const deferred = {};
    deferred.promise = new Promise((resolve, reject) => {
        deferred.resolve = resolve;
        deferred.reject = reject;
    });
    return deferred;
}

The new object contains promise and the handlers to reject or resolve it.

{ promise, resolve, reject }

And that's it. In the simple form and shape.

Deferred as subclass of Promise

The disadvantage if this approach is hacky implementation. The Promise object needs to be thenable and as per specification it needs to return the Promise object instantiated from the same class as the calling object. This means internally the constructor is called with standard resolver method which we need to support. Hence the check for the resolver argument. Also it is going against type definition where Promise expects resolver function in the constructor. Secondly we are robbing ourselves from any potential native performance optimization regarding Promise class or it's instance.

class DeferredPromise extends Promise {
    constructor(resolver) {
        const that = {};
        super(function(resolve, reject) {
            Object.assign(that, {resolve, reject})
        });
        Object.assign(this, that)
        if (resolver) {
            resolver(this.resolve, this.reject)
        }
    }
}

var deferred = new DeferredPromise();

deferred.then(doStuffWithValue)

deferred.resolve('Hello World')

Although this is not recommended, it might be sometimes useful if you have existing middle-ware pipeline working with promise objects and you can get away with dodgy typing.

Deferred as result of a factory function

Building on the original principle, much cleaner and recommended approach is to user factory method

export function Defer(workload) {
    let _reject, _resolve;

    const _p = new Promise(function (resolve, reject) {
        _resolve = resolve;
        _reject = reject;
    })

    const trigger = (function (workload) {
        let _result = undefined;

        return function() {
            if (_result) {
                return _result;
            }
            return _result = workload()
                .then(data => {
                    _resolve(data)
                })
                .catch(reason => {
                    _reject(reason)
                })
        }
    })(workload)

    return {
        get promise() { return _p },
        trigger,
    }
}


var deferred = Defer(() => http.get(url))

// in some class or API expecting Promise value
deferred.promise.then(doStuffWithValue)

// some later stage in app
deferred.trigger()

This implementation takes the async function returning a Promise and returns the new Promise object wrapping the future promise object returned from the async function and the handler to allow as to trigger the async function in the future.

We can see that the _reject and _resolve variables are closed over and the resolve resp reject handlers from the Promise implementation are assigned.
trigger method implements simple caching in case it gets called multiple times.

Wrap up

If anybody find this useful I wrapped all of the above in the platform agnostic npm package, which can be installed by

$ npm install deferable

Or it can be found on github/webduvet/deferable.git. It offers both the Class flavor as well as the Factory flavor.

Deferred Promise pattern has many use cases e.g. api call throttling, user input de-bounce, user action timeout etc.

In this the next article I elaborate one particular use case of Deferred Promise pattern - throttling the async calls.

Vim: ignore casing in search

webduvet — Mon, 09 Jan 2023 13:07:15 +0000

escape sequence anywhere in the pattern \c

/\ccmake

will match cmake and CMake or cMake

/c\cmake

will match cMake and cmake but not CMake

the inverse of \c is \C

/\cc\CMake

will match CMake and cMake but not cmake

further options:

This will ignore casing in all searching:

:set ignorecase

unset with no prefix as usual:

:set noignorecase

In addition to the above this will ignore casing if search string is all lowercase, but will be case sensitive if the search string contains any upper case letter. The ignorecase must me set.

:set smartcase

Scanning for open ports

webduvet — Wed, 04 Jan 2023 12:13:47 +0000

lsof

list of open files

lsof -i -P -n | grep LISTEN

netstat

Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

netstat -tulpn | grep LISTEN

ss

utility to investigate sockets

ss -tulpn | grep LISTEN

nmap

network exploration tool, port scanner

nmap -open 192.168.3.34

scanning open ports on remote IP address

nc -zv 192.168.3.34 2>&1 1-1024 | grep -v refused

this scans open ports in the range of 1 to 1024 on the above IP address.
redirects the error output to standard outoput and matches all lines except refused

scanning all open ports on the subnet using nmap

nmap -n 192.168.3.0/24

Sources

Linux for Networking Professionals by Rob VandenBrink
https://www.cyberciti.biz/faq/unix-linux-check-if-port-is-in-use-command/
man pages

Setting up private network with a gateway

webduvet — Wed, 21 Dec 2022 01:02:55 +0000

Building a Kubernetes cluster at home, part One.

This chapter will focus on the preparation work. I will wire all computers together and I configure one of my laptops to serve as an access point to the cluster and also it will serve as a gateway from the cluster to the outside world. I already gathered about 5 old computers and a simple non-managed Ethernet switch with 5 ports. I will start with the simplest possible solutions and will add complexity as goes.

Step 1

Plugin the first laptop. This one will serve as a gateway as it has two network interfaces. Wifi will interface with my router network and the Ethernet will interface private network. When you connect your laptop to your home WIFI, it usually will obtain the dynamic IP address from a router, something like 192.168.0.15. The router is set as the default gateway for all internet traffic. When you plug in the computer with an Ethernet cable, it does the same (minus the WIFI security bit). The operating system usually activates only one network interface (Ethernet takes precedes if both are present). In our case, we need to enable both so the machine can communicate with the cluster and with me.

mnode@mnode:~$ ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 49:ba:4e:4a:eb:c5 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 192.168.3.34/24 brd 192.168.3.255 scope global eth0
        valid_lft forever preferred_lft forever
    inet6 fc80::4aba:4eff:fe39:ebc5/64 scope link
        valid_lft forever preferred_lft forever
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether da:64:6a:9f:cd:1f brd ff:ff:ff:ff:ff:ff
    altname wlp2s0
    inet 192.168.1.5/24 metric 600 brd 192.168.1.255 scope global dynamic wlan0
        valid_lft 86379sec preferred_lft 86379sec
    inet6 fe80::d66a:6aff:fe9c:cd1f/64 scope link
        valid_lft forever preferred_lft forever

The above is the list of all available network interfaces. Those interesting for us are eth0 which is the Ethernet interface and wlan0 which is WIFI. from the above list we can see that Ethernet is in the state DOWN. For comparison the wireless LAN is wlan0 and the state is UP.

Running the following command will turn on the networking interface: sudo ip link set eth0 up Running ip address show again will show that the eth0 interface is now with the flag UP, so the operation worked. However, doing this on the Ubuntu desktop might result in the operating system (Ubuntu) reporting an error message activation of network connection failed Ubuntu

This is because Ubuntu Desktop uses NetworkManager to manage all the network interfaces by default. It is the application responsible for network management via UI. This is not the case in Ubuntu Server which I will be using. The documentation can be found here NetworkManager It can be completely turned off by choosing a different renderer or a specific networking interface can be excluded from under the management of NetworkManager. The configuration is in the file /etc/NetworkManager/NetworkManager.conf. On UI-less versions like Ubuntu Server, there is no NetworkManager installed. The default network renderer is systemd-networkd. In both flavors of Ubuntu, the networking configuration is done via Netplan and the configuration for Netplan is stored in yaml file in the location /etc/netplan/*.yaml. There could be more than one configuration file. More information on this are to be found in the provided link.

The default configuration on the Ubuntu desktop might look like this and it basically tells the system that NetworkManager manages all networking devices:

network:
  version: 2
  renderer: NetworkManager

If there is no renderer specified, Ubuntu by default uses systemd-networkd as the network renderer. We want to customize values for the gateway and the static IP address as we need to manage two separate network interfaces and we need to have them UP at the same time.

network:
    wifis:
        wlan0:
            dhcp4: true
            dhcp6: true
            access-points:
                "WIFI-Name":
                    password: "wifi_password"
    ethernets:
        eth0:
            dhcp4: false
            addresses: [192.168.3.34/24]
            routes:
                - to: default
                - via: 192.168.1.1

The above configuration tells the system that both interfaces are managed (by default) by systemd-networkd specifies the access point for WIFI and password. It specifies that WIFI takes the IP address from the router for both ipv4 and ipv6 protocols. It specifies that the Ethernet interface does not try to obtain IP addresses from DHCP server and takes the hard-coded one. The format 192.168.3.34/24 specifies that the IP address in the domain 192.168.3.xx and the address 192.168.3.34 within the domain. There is a lot that can be done configuring the netplan.

NOTE: for the desktop version turning off the NetworkManager results in an inability to select the network from the dropdown menu in the taskbar.

An important part is that the Wifi subnet is not clashing with the ethernet subnet. In my case wifi runs on 192.168.1.0/24 and I set up the ethernet IP to 192.168.3.0/24

After this netplan's new configuration needs to be applied bash sudo netplan apply In case this is done on the Ubuntu desktop, there might be a message systemd-networkd not running, output will be incomplete. this indicates the network manager is not managing the network anymore, but the default networkd is not running yet. it needs to be started bash systemctl start systemd-networkd I have to note also that to setup the Ubuntu server is much simpler as there is no NetworkManager and the systemd-networkd is a default network renderer.

Step 2, adding another compouter to cluster

This should be fairly simple. For me the second guinea pig is raspberry pi3, running Ubuntu for raspberry, not too dissimilar to the Ubuntu server. The only configuration which I need to do here is to assign a static IP address and point to the default gateway: yaml network: version: 2 ethernets: eth0: dhcp4: false addresses: [192.168.3.40/24] after bash sudo netplan apply I can reach the Raspberry node from the gateway laptop and vice versa. I also can reach the gateway laptop on the Wifi domain. I should be good to go to plug in the rest of the computers and do the same to them After that I could start installing k3s and build the Kubernetes cluster.

It is also worth noting - when you are building the cluster out of old laptops - it is handy to prevent them to go sleep or hibernate when the lid is closed. That way is is possible to stack them on each other to save the space.
On Ubuntu server as per documentation this needs to be added to /etc/systemd/sleep.conf or preferably new file in the location /etc/systemd/sleep.conf.d/00-sleep.conf to be created as it takes precedence over default configuration in the former file.

[Sleep]
AllowSuspend=no
AllowHibernation=no
AllowSuspendThenHibernate=no
AllowHybridSleep=no

Step 3, optional - internet access

However, there is one more thing I wanted to do. It is not essential and perhaps even not recommended, but for my convenience want to have a connection to the internet from the individual machines in the cluster. This will help me with the installation of the required software on individual nodes. This is not a production environment so I can easily get away with it. The same scenario would apply if you had a desktop machine without WIFI. You could use a laptop to connect to WIFI and use the laptop's Ethernet interface to connect with your desktop and to serve as a router.

To do this we need to add a few rules in IP tables. We need to forward all internet traffic from ethernet to wifi interface and back. We also want to set up network address translation(NAT). For Linux operating system to enable IP forwarding. This can typically be done by editing the /etc/sysctl.conf file and adding the following line:

net.ipv4.ip_forward = 1

or run sysctl -w net.ipv4.ip_forward=1 We also need to add the following rules in the ip tables:

iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT

The first rule forwards all the traffic coming from the individual machines on our subnet to out via wireless LAN.
the second adds a record to NAT table to masquerade packets coming from the individual machines from the private subnet via the gateway laptop as packets coming from the gateway laptop itself (the home router does the same thing again when you are reaching out to the internet)
The third rule forwards all the traffic originating from the individual machines back to them

Save your IP tables rules and make sure they are automatically loaded at startup. This can typically be done by using the iptables-save and iptables-restore commands. For example:

iptables-save > /etc/iptables.rules

Edit the /etc/rc.local file and add the following line before the exit 0 line:

iptables-restore < /etc/iptables.rules

Once this is done it should be possible to access the internet from a machine connected to an Ethernet switch provided the configuration in step 2 is followed.

Summary

After the above steps, I do have a cluster of computers wired together via an Ethernet switch. The computers share a private network and can communicate with each other. The private network is connected to the router via a wireless connection and the internet is also reachable from inside the private network.

Efficiently Finding and Replacing Text in Multiple Files Using ripgrep and sed

webduvet — Tue, 13 Dec 2022 23:52:35 +0000

subtitle

Convenient way to search for and replace text in multiple files using the powerful ripgrep and sed tools

command

rg "some string" | xargs -d "\n" sed -i "s/some string/another string/g"

The command searches for the specified string using ripgrep and pipes the results to xargs, which executes the sed command on each line returned by ripgrep. The sed command then replaces each occurrence of "find-string" with "new string" in the specified files.

Here's the breakdown of the command:

rg "find-string": This uses ripgrep to search for the string "find-string" in the current directory and all its subdirectories.
|: This pipes the output of the previous command (the list of files containing "find-string") to the next command.
xargs -d "\n": This takes the output from the previous command and passes it as arguments to the sed command. The -d option narrows down the delimiter to a newline character, so that each line returned by ripgrep is passed as a separate argument to sed. By default it uses new line characters as well as spaces
sed -i "s/find-string/new string/g": This is the sed command, which replaces each occurrence of "find-string" with "new string" in the specified files. The -i option tells sed to edit the files in-place, so the changes are saved directly to the original files. The s command is the substitute command, which performs search and replace operations on the specified files. The g flag at the end tells sed to replace all occurrences of the search string, rather than just the first one.

How to manage versions using update-alternatives

webduvet — Thu, 24 Nov 2022 10:05:37 +0000

Background

Recently I needed to run the end to end tests in the older chrome browser as the tool the chrome driver available to me was only working with the older version of chrome.
The solution instead of downgrading my chrome installation was to install multiple versions of chrome-browser. More on this in my recent article how and where to get older versions of google-chrome.
Long story short, I would like to be able to hotswap the chrome version. Any python, node or java developer most likely uses some type or form of version manager. In java world people might be using sdkman [www.sdkman.org] or node js world us used nvm www.getnvm.org.

What is update-alternatives

update-alternatives creates, removes, maintains and displays
information about the symbolic links comprising the Debian
alternatives system.

It is a tool provided by Debian based distributions which is designed to help to manage multiple versions or different implementations of a command or program, read more on Debian Alternatives. Update-alternatives in Debian based system like Ubuntu is used as default option by apt, however, it offers the command-line interface to interact with the settings and to add and remove entries manually.

It can also provide functionality to update something what is called slave links to cater for associated software or manual pages which also need to change when the main target changes.
The analogy in the nodejs world would be corresponding npm installation or in python world it would be corresponding pip.

How to install another version with update-alternatives

if you plan to use update-alternavives to e.g. install the multiple node versions you will download the desired version and unpack it in the directory of the choice In my case I need older version of google chrome so it would be /opt/google/chromium/1027016/chrome-linux/.

This will contain binary /opt/google/chromium/1027016/chrome-linux/chrome

if you already have chrome installed and you did that from the Debian package by

sudo apt install google-chrome-stable.deb

your already have used update-alternatives and running

update-alternatives --display google-chrome

will show output:

google-chrome - auto mode
link best version is /usr/bin/google-chrome-stable
link currently points to /usr/bin/google-chrome-stable
link google-chrome is /usr/bin/google-chrome
/usr/bin/google-chrome-stable - priority 200

This indicates that there is only one version of google-chrome and it shows the location and priority for auto mode (the higher version takes precedence).
I don't want to go into too much detail here as it could be found in man pages or online debian alternatives.

Running the following lists the existing links stored by update-alternatives.

update-alternatives --list google-chrome

?> /usr/bin/google-chrome-stable

We can see also here that there is only one. If we follow the symbolic links and investigate further we get to:

/usr/bin/google-chrome --> /etc/alternatives/google-chrome
/etc/alternatives/google-chrome --> /usr/bin/google-chrome-stable
/usr/bin/google-chrome-stable --> /opt/google/chrome/google-chrome

The last one is hard link to executable file and paint a pretty clear picture how the update-alternatives work. /usr/bin/google-chrome-stable is an executable and /usr/bin/google-chrome is a symbolic link to the executable so we can just type google-chrome and it starts the browser. Update-alternatives sticks in an extra layer of symbolic links in the middle of which it has the full control.

To add another version to it execute the following

sudo update-alternatives --install /usr/bin/google-chrome google-chrome /opt/google/chromium/1027016/chrome-linux/chrome

if you want to use this as default link in auto mode you can add the priority number at the end, higher number takes precedence. For manual intervention enter the following command:

sudo update-alternatives --config google-chrome

output (depending on given priority nubmer this might differ):

There are 2 choices for the alternative google-chrome (providing /usr/bin/google-chrome).

Selection    Path                                              Priority   Status
------------------------------------------------------------
  0            /opt/google/chromium/1027016/chrome-linux/chrome   300       auto mode
  1            /opt/google/chromium/1027016/chrome-linux/chrome   300       manual mode
* 2            /usr/bin/google-chrome-stable                      200       manual mode

Press <enter> to keep the current choice[*], or type selection number:

if you choose one option the link will no longer be in auto mode, but you can return anytime to automode by running

sudo update-alternative --auto google-chrome

slave links

As mentioned before, these are associated links which should change when master link changes. The examples are node and npm or python and pip. It also works with manual pages as some packages install also manual pages. This is also the case when installing google-chrome as deb package as above.
Version managers not based on update-alternatives usually add/edit the entry in manpath pointing to the correct man page or edit (add) entry in $PATH environment variable.
update-alternatives does this via mentioned slave links.

In case of google-chrome, even though it uses update-alternatives by default, it does feature the manual entry, but it does not use slave links and the installer simply copies the man page in the man location. Which prevents update-alternatives to apply sim link here. However, even though we can't do anything with the hard link we can add another manual entry just for the specific version. I'm install chromium so I would be interested to have chromium manual entry

manual entry for google chrome can be found here:

man -w google-chrome

We could manually move man entry for google-chrome-stable elsewhere and let update-alternatives utilize slave links, but we could also add another man entry via slave link to demonstrate flexibility. Let's do it.

sudo update-alternatives --install /usr/bin/google-chrome google-chrome /opt/google/chromium/chrome-linux/chrome \
--slave /usr/share/man/man1/chromium.1.gz chromium.1.gz /opt/google/chromium/chromium.1.gz

The result of this will be following. If we use the google-chrome-stable the manual entry for chromium via man chromium will not be available, but swapping the version for chromium will make the manual entry for chromium available along the original manual entry which is/was set as hard link (actual file).

Summary

This short articles demonstrate the basic capabilities of the tool update-alternatives. More detail can be found on manual pages or on Debian website.

Where and how to get hands on an older versions of google-chrome?

webduvet — Thu, 24 Nov 2022 09:57:07 +0000

Issue

Recently I needed to run some functional tests against the older version of chrome browser as the chrome-driver available to me (part of the 3rd party test tools) was working only with slightly older version of google-chrome than my current system.
I thought it is easy, I just google where google stores the old version of their browser, download binaries and point my testing tool to the correct version.
It turned out that both of my assumptions were hiding some troubles behind and I realized that it is actually not that straightforward to get my hands on the older binaries of the chrome browser.

Where to get the old version?

Google had not made available the older version of their chrome browser to public due to security issues which are of course patched in the current version. It is possible to downgrade as per this thread up to 3 stable versions back, but that is usually not enough and it is a hassle.
After a little bit of searching I've found this www.slimjet.com website offering plethora of old stable versions releases. It doesn't cover all, only the latest of major versions. That might be just enough and if not there is also chromium https://www.chromium.org project which is essentially identical to chrome. It is available through the chromium snapshots.
The information on how the get them can be found directly on chromium website at https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html.
It offers to browse the chromium snapshots by branch base position. However this doesn't tell you a lot if you need a specific version let's say 105.0.5195.52, but you can find corresponding number by using this lookup service at omahaproxy.appspot.com.

Enter the version number you are looking for in the lookup field and get the information like commit number along with branch base position.

Take the Branch Base Position and browse the available options if you need Linux, or lookup the corresponding platform.
If there is none, as the google-chrome released versions might not exactly correspond to chromium releases, try decreasing the branch base position until you find the hit. It might be 105.0.5195.0, which most likely will work with your chrome-driver.

How to work with the older versions

I probably should note that using the outdated browser version is not advised, actually the opposite. Therefore functional and end to end testing on an out-dated version is not recommended either if not outright pointless except some edge cases where you are required to cater for a legacy systems.
In my case I needed the older version, specifically two major versions down as my third party test tool doesn't get update that often and it happens, that the available latest chromedriver is not enough to run tests against the latest version of my chrome browser. In theory I could downgrade two versions down and still enjoy the google's support, but it might be just inconvenient. I prefer to work with binaries and leave the browser installation which I use daily most up to date. More on this in my following blog

Multiple github accounts

webduvet — Fri, 14 Oct 2022 21:03:56 +0000

What is the use case?

Recently my company moved all the repositories from a privately hosted git repository to an enterprise GitHub account. That means all the company repositories have now domain github.com, the same as all my private ones.
As it soon became evident the company GitHub credentials and associated ssh key would inevitably clash with my private GitHub account on not one, but two fronts. I could not use one instance of my browser to interact with my private and my company's repositories simultaneously. This was easily solved by dedicating one of the browsers (in my case it was opera) to all company matters while I kept Firefox for all my personal related matters.
However as most others I also use ssh key to access git repositories rather than credentials and here I run into few challenges.

GIT cli

To make long story short git-cli uses ssh-agent to authenticate to git provider. It looks up the cached ssh keys and tries them one by one from top to bottom.

If the key is not cached it will look into the .ssh directory. That's why it will work even if you delete the key pair from there. More on this is to be found on manual pages of ssh-add.

The first key which manages successful authentication is to be used even if it doesn't match the organization used in the GitHub URL.

Here is the URL structure of repository hosted on GitHub:

git@github.com:Organisation/repository.git

If the first ssh key is the private one - none of the company accounts(unless made public) would be accessible.

If the first ssh key is the company one - everything will work apart accessing the private repos under the private account.
The public repos under private account will still be accessible to clone, but most likely (depending on the repository setup) github will reject any direct changes.
You probably would be able to create a new branch and to open PR albeit not under you private name, but under the name you use on the company account.

Solution 1 - gitconfig

the solution is using the local .gitconfig settings.
for private project dedicate a separate folder and create local config file.

if you are on a linux machine the default config might be in your home directory

~/user/home/myname/.gitconfig

and all the private "stuff" will need to have a dedicated .config .

~/my-private-projects/.gitconfig

Although this is not picked automatically by git. Git does not work the same way as e.g. pip in python
where it would traverse the current and parents direcotries for config.

It has to be specified in the "main" .gitcofnig file:

[user]
    email = companyID@company.com
[core]
    editor = nvim
    sshCommand = ssh -i ~/.ssh/id_rsa_company
[includeIf "gitdir:~/private-projects/"]
    path = ~/private-projects/.gitconfig

The important bits are user.email and core.sshCommand
this way we can instruct github to use specific sshKey for this entire folder and sub-folders.

If the ssh key is not specified the ssh service will match the first available ssh key working with github domain as described above.

Solution 2 - ssh config

This solution is using .ssh/config. Documentation for ssh config.
If it doesn't exist (which it probably doesn't if you get this far), create it and add something
along these lines

Host github-work
    Hostname github.com
    # Username for remote SSH user (For GitHub, everyone uses the name `git`)
    User git
    IdentityFile /home/user/.ssh/id_rsa_company
    IdentitiesOnly yes

Host github.com
    Hostname github.com
    User git
    IdentityFile /home/user/.ssh/id_rsa_private
    IdentitiesOnly yes

Few things here is worth to mention, remember this?

git@github.com:Organisation/repository.git

and your private git repository links are most likely having very a similar form:

git@github.com:your-git-name/repository.git

The host here is github.com in both cases. The host github-work actually could be anything but you would manually need to change the github.com to something else e.g. github-work in the .git/config file of the existing repository, or use it when you are cloning the repository as following:

git clone git@github-work:Organisation/repository.git

In my case I left the other repository as github.com as there would be no further actions needed for all the private repositories already residing on my machine.
You could do it the other way on you work machine or you could use "alias" for both or all organizations you need to access with separate identities.

Static Content Server with Nodejs without frameworks

webduvet — Tue, 21 Sep 2021 11:14:14 +0000

Static server with Nodejs

This tutorial will walk you through few steps how to set up simple http server for static content using only nodejs. We will add basic features as serving requested resource from file or from memory(cache) and responding with error message when no resource is available.
In reality you will almost never run any http server in this manner however, it might be very helpful to understand what frameworks like Expressjs do under the hood. It also could serve as a very simple testing tool in your local environment.

Requirements is to have installed nodejs on the system, preferably newer version (12+). The recommended environment is Unix like machine, but it is not necessary. The target audience is javascript beginner or UI developer who is curious how does http server work in nodejs.

We will go through the following:

setup http server, what is static server
adding rules how to read the request
finding resources and caching

Lets start with the simplest possible

Http server is a network application which listens to incoming network traffic. It is doing so by acquiring some system resources. Specifically it creates the process in the memory which listens to incoming traffic over the network on the dedicated port. To talk to the http server we need the physical address of the computer and the port which the application acquired. Nodejs provides all necessary functionality to do so. Let's have a look how nodesj does it.

the simplest way how to start and run the most basic http server using nodejs would be something like this:

node -e "require('http').createServer((req, res) => {res.end('hello world')}).listen(3000)"

Running the above code on Linux machine with node installed will start the server.
it can be verified by typing http://localhost:3000 in the browser URL bar.
or by typing the following in new terminal window:

> curl http://localhost:3000
// expected response is
hello world

In this basic example we can easily see the building stones. We create an object and call the listen which effectively opens up the connection on the given port and it is waiting for the incoming request complying HTTP protocol.
We can test it with netcat sending a text complying with HTTP GET request header.

printf "GET / HTTP/1.1\r\n\r\n" | nc 127.0.0.1 3000
// The expected response is again
HTTP/1.1 200 OK
Date: Tue, 21 Sep 2021 09:59:13 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Content-Length: 11

hello world%

It is a little richer because netcat prints just about everything what is received in the response including response header. curl can do it as well. Go ahead and try using -i flag.

The other main component aside createServer() and listen() is callback passed into createServer. It contains references to request and response objects. Working with these two objects we can interact with our http server.

This article is however not about networking and protocols but tutorial how to build simple static content server using only nodejs and this does not get us too far since it responses with "hello world" to any request. Let's see if we can do better.

Serving response from a file

Let's make one step further in terms of the functionality of our new http server. We are aiming towards the server which can serve static content. The word static here means similar to "static" keyword in javascript. It is something which is already known and defined prior the user request. From the web server we usually referred as static content to files like images, icons, CSS files and so on. So let's server user with the content of the file rather then hard coded message.

module.exports = function staticServer() {
  const path = './static_content';
  const port = 3000;

    // create server object as in previous example
    var server = http.createServer(function(req, res){
    const filePath = path + 'index.html';
    fs.readFile(absPath, function(err, data) {
      res.end(data);
    });
    });

  server.listen(port, function() {
    console.log("server listening on port: " + port));
  });
  return server;
};

in addition, create directory and file ./static_content/index.html containing your content:

<html>
  <body>
    <h1>
      Hello, this is very simple
    </h1>
  </body>
</html>

In the above code we define the path where the static content is, in this case it is index.html file we read the file and send the data back to user as a response to client's request.
response.end() executes the above with some [default headers]()

Finding and serving requested resource

Next in the quest serving the content based on the user request is finding the requested resource our user is asking. The server looks it up and if it exists it serves the content of the file to client.

module.exports = function staticServer() {
  const path = './static_content';
  const port = 3000;

    // create server object as in previous example
    var server = http.createServer(function(req, res){
    // get the resource from request
    const filePath = path + req.url;
    fs.readFile(absPath, function(err, data) {
      res.end(fileContents);
    });
    });

  server.listen(port, function() {
    console.log("server listening on port: " + port));
  });
  return server;
};

const filePath = path + req.url show how mapping between the requested resource and the actual resource might work. Path is relative path to location where our nodejs app is running and req.url is last bit of the URI identifying what resource user wants.

http://www.example.com/**resource**

Caching

Let's make one small addition. The cache. When we server the file from a disk it is not a big deal, as it is pretty quick, however if the file would come from some more time expensive resource we want to keep the content of the file for later requests. Here is a very simple example how it can be implemented:

module.exports = function staticServer() {
  const path = './static_content';
  const port = 3000;

  const cache = {}

    // create server object as in previous example
    var server = http.createServer(function(req, res){
    // get the resource from request
    const filePath = path + req.url;
    if (cache[filePath]) {
      sendFile(res, filePath, cache[filePath]);
    } else {
      fs.readFile(filePath, function(err, data) {
        res.end(fileContents);
      });
    }
    });

  server.listen(port, function() {
    console.log("server listening on port: " + port));
  });
  return server;
};

Basic error handling and wrap up

In this last section we add some simple error handling. In case the user specifies the resource which is not found in the given location of static content or if the resource in not readable we need to notify the user with an error. The standard way of doing it is to return response with code 404 in the response headers. We also might add some explanation in the content.

let 
    fs = require('fs'),
    path = require('path'),
    http = require('http');

const cache = {};

/**
 * lookup content type
 * infer from the extension
 * no extension would resolve in "text/plain"
 */
function lookupContentType(fileName) {
  const ext = fileName.toLowerCase().split('.').slice(1).pop();
  switch (ext) {
    case 'txt':
      return 'text/plain';
    case 'js':
      return 'text/javascript'
    case 'css':
      return 'text/css'
    case 'pdf':
      return 'application/pdf';
    case 'jpg':
    case 'jpeg':
      return 'image/jpeg';
    case 'mp4':
      return 'video/mp4';
    default:
      return ''
  }
}


/**
 * plain 404 response
 */
function send404(res){
    res.writeHead(404, {'Content-Type':'text/plain'});
    res.write('Error 404: resource not found.');
    res.end();
}

/**
 * sending file response
 */
function sendFile(res, filePath, fileContents){
    res.writeHead(200, {"Content-Type": lookupContentType(path.basename(filePath))});
    res.end(fileContents);
}

/**
 * serve static content
 * using cache if possible
 */
function serveStatic(res, cache, absPath) {
  // use cache if there is any
    if (cache[absPath]) {
        sendFile(res, absPath, cache[absPath]);
    } else {
        fs.exists(absPath, function(fileExists) {
      // attempt to read the resource only if it exist
            if (fileExists) {
                fs.readFile(absPath, function(err, data){
          // not able to read the resource
                    if(err) {
                        send404(res);
                    } else {
                        cache[absPath] = data;
                        sendFile(res, absPath, data);
                    }
                });
            } else {
        // resource does not exist
                send404(res);
            }
        });
    }
}

module.exports = function startServer(spec){
  let { path, port } = spec;

    // create server object
    var server = http.createServer(function(req, res){
    // if no resource is specified use index.html
        if(req.url === '/') {
            const filePath = path + 'index.html';
      serveStatic(res, cache, filePath);
        } else {
      const filePath = path + req.url;
      serveStatic(res, cache, filePath);
        }
    });

  server.listen(port, function(){
    console.log("server listening on port: "+port);
  });
  return server;
};

Now we can run it like this:

const startServer = require('./startServer.js')

startServer({ path: './static_content', port: 3000 });

In the above example I added very basic error handling. In the event the resource specified by the user is not found in the static content directory, or it can't be open for reading, the server response with different header with error code 404 and different content explaining what went wrong.
In order for a browser to understand better what kind of content we are dealing with, it is also a good idea to include some indication about resource content type. In lookupContentType we can do it just based on the file extension type.
Now if we try pdf the browser will have no problem opening pdf file instead downloading it.

Conclusion

This is by no means a robust product, merely a very simplified example how things do work behind the curtain in frameworks like expressjs. We leveraged the nodejs built in library http to run simple http server.
We implemented simple routing to find static content in a given location. We also implemented simple in memory caching, content type resolution and basic error handling in case the resource is not found or accessible.

Sources

Synchronous and asynchronous function composition

webduvet — Wed, 15 Sep 2021 14:48:17 +0000

what is function composition
composing simple functions
composing asynchronous functions

In this short article I am going to discuss the function composition and how the function composition might work with asynchronous code. I'm going to explain the basic principles of it and give some examples.

Intro - Function composition

Javascript is a function friendly language. What we do often is to apply multiple functions in sequence. Some can say we are chaining the functions,hence the operator called pipe is often used or some say we are composing functions. This allows for clearly defined flow control and it makes a for a robust and testable code.

h(g(f(x)))

//logical representation
x -> f -> g -> h

We start with x and we apply functions f, g, h in that sequence. Function composition is not commutative (although some functions can commute), thus the order how we apply functions matters. For more information on this topic see
the sources at the bottom of the is article

Composing functions in JavaScript

In this section we go from the basic function composition in JavaScript to more robust approach.

Example of function composition using synchronous functions:

const double = x => x + x;
const square = x => x * x;
const half = x => x / 2;

const dsh = (x) => half(square(double(x)))

This works nicely, but it is obviously hard to read when we have even slightly more complex logic.
Something like this:

const hardToRead = (x) => cleanup(Calculate(params)(evaluate['@@space'](env)(getResult(x))))

is already much harder to read, even if we try a little tidying up.

const hardToRead = (x) => cleanup(
  Calculate(params)(
    evaluate['@@space'](env)(
      getResult(x)
    )
  )
)

we would have to break some linting rules in order to give it some fasion.

const hardToRead = (x) => cleanup(
  Calculate(params)(
  evaluate['@@space'](env)(
  getResult(x)
)))
// calling the function
hardToRead(x)

it looks good, but it has few downsides. We have to read it from bottom to top which is a little counter intuitive and it is not very flexible form. What if we decide that we want to add something after the cleanup operation? Are we going to rewrite it like this?

const hardToRead = (x) => logIt(
  cleanup(
  Calculate(params)(
  evaluate['@@space'](env)(
  getResult(x)
))))

// calling the function
hardToRead(x)

It is all doable, although we would need to be careful about number of brackets at the end.

However we can do more, we can introduce a helper function to help us with the function composition. With that the above piece of code can be written this way:

const hardToRead = pipe(
  getResult,
  evaluate(env),
  calculate(params),
  cleanup,
  logIt // now adding extra functionality does not require a lot of rewritting
)
// calling the function
hardToRead(x)

The benefit of helper composition function is evident. The syntax is a lot cleaner. We can read the steps from top to bottom and we can add and remove any step without counting the closing brackets at the end.
In addition function pipe is what is called higher order function. It returns another function which can be named and passed along or executed on the spot.
Under the hood the pipe function is actually very simple and it does basically the same thing as the calling functions in sequence. It could look like this:

function pipe(...fns) {
  return function(arg) {
    return fns.reduce((acc, fn) => {
      return fn(acc);
    }, arg)
  }
}

In practice the function composition is already built in JavaScript and conceptually it can be seen as reducing a collection of functions and over an initial parameter into a new value.
Basically, all we are doing is taking the output value from previous operation as an input value of the next operation just like in the schematic diagram in the beginning. At the end we have the final result.

Asynchronous code

Composing only synchronous operation sometimes wouldn't get us too far. JavaScript is event driven programming language and asynchronous operation are at the heart of it.
Composing asynchronous code is surprisingly straight forward as well.

We can leverage already built in common constructs - Promises.
In the asynchronous world the already mention code could be written as follow:

getResult(url)
.then(evaluate(env))
.then(calculate(params))
.then(cleanup)

That is already pretty neat and personally I would use it as is as often as I can. So would we need another way to compose asynchronous functions?
Let me explain. Sometimes we need to define the set of unique sequences of functions which might not even be known during the static evaluation. For example, in one path of the execution we would want to run:

getResult > eval_1 > eval_2 > calculate(param) > cleanup

and in the other path we want:

getResult > eval_1> eval_2 > eval_3 > calculate(param) > cleanup

or somewhere else we have:

getResult > eval_1> .... > eval_N > calculate(param) > cleanup

Moreover we could have another dynamic way of defining the number and order of the composed operations.
It is easy to see that chaining promises could become cumbersome and we need some help to create the composition. We can take the pipe function from sync section and tweak it a little. Or a little more since the current implementation does not support await in Array.reduce. However, it as long the await keyword is called inside async block any plain loop will wait for promise resolution. We can leverage:

function asyncPipe(...fns) {
  return async function(arg) {
    let res = arg;
    for (fn of fns) {
      res = await fn(res);
    }
    return res;
  }
}

The pipe function in this implementation can accept both, synchronous and asynchronous function.
To tackle the above challenge we could use it as follows:

const path = [method1, method2, ..., methodN];
const doPath = (path:Array<Function>) => pipe(
  getResult,
  ...path,
  calculate(params),
  cleanup
)

const myUniquePath = doPath(path)

Now we can easily to chain the functions returning promises also in runtime when the set of required operations are not known at compile time.

Handling Exceptions?

What about catch block? Did we forget something? What if something goes wrong and we have to provide failed path option?

No news here. The asynchronous function is only a function returning promise so we have two main ways of handling this.

traditional catch block in promises
inside asynchronous block of code we have the option using try - catch construct.

doPath(url)
.then(result => { doSomethingWithResult(result) })
.catch(error => { doSomethingWithError(error) })

async asyncBock() {
  try {
    let res = await doPath(url)
    doSomethingWithResult(res)
  } catch(e) {
    doSomethingWithError(e)
  }
}

Advantages using function composition

In the ideal world of functional programming the function is completely decoupled from the environment where it runs. This makes it very easy to test as there is virtually no difference how the function is executed in the mocked
test environment, the development environment and in the production environment. The function behaves exactly the same.
Dividing the logic into independent steps gives the opportunity to combine these steps into more a complex operation without increasing the complexity of building stones and without an extra strain to increase complexity of our test environment.

Conclusion

Function composition is one of the foundation stones of functional programming. In this article we explained the basic rules of function composition and has shown how to apply composition of synchronous and asynchronous functions. It also outlined the basic implementation details leveraging the built in JavaScript language
construct.

Forem: webduvet

Trouble shooting docker - permission denied

verify

Deferred Promise Queue

Motivation

Deferred pattern

Publish - subscribe

Flow Chart

Wrap up

Deferred (promise) pattern

Deferred as subclass of Promise

Deferred as result of a factory function

Wrap up

Vim: ignore casing in search

further options:

Scanning for open ports

lsof

netstat

ss

nmap

scanning open ports on remote IP address

scanning all open ports on the subnet using nmap

Sources

Setting up private network with a gateway

Building a Kubernetes cluster at home, part One.

Step 1

Step 2, adding another compouter to cluster

Step 3, optional - internet access

Summary

Efficiently Finding and Replacing Text in Multiple Files Using ripgrep and sed

subtitle

command

Links

How to manage versions using update-alternatives

Background

What is update-alternatives

How to install another version with update-alternatives

slave links

Summary

Links

Where and how to get hands on an older versions of google-chrome?

Issue

Where to get the old version?

How to work with the older versions

Links

Multiple github accounts

What is the use case?

GIT cli

Solution 1 - gitconfig

Solution 2 - ssh config

Static Content Server with Nodejs without frameworks

Static server with Nodejs

Lets start with the simplest possible

Serving response from a file

Finding and serving requested resource

Caching

Basic error handling and wrap up

Conclusion

Further reading

Sources

Synchronous and asynchronous function composition

Intro - Function composition

Composing functions in JavaScript

Asynchronous code

Handling Exceptions?

Advantages using function composition

Conclusion

Further reading

Sources