Forem: David Rogers

Intro to Unix Shell Commands

David Rogers — Wed, 28 Sep 2022 12:45:37 +0000

TL;DR

Essential file movement:

man, ls, pwd, touch, rm, mkdir, rmdir, cd, cp, mv

Essential Knowledge:

flags change the behaviour of the command so read through the man pages to find out what is possible and what the result will be
the most important flag is -h or --help, these will print the help pages which are more concise than the man pages but at the same time more limited

Introduction

There are a few essential commands to learn to get started with a unix operating system
These are default shell commands which should be a good foundation to begin to use the terminal more, obviously this is not an extensive list but should allow you to find further resources

flags

flags are preceded by a dash and modify the functionality of the program so for example ls -a is just ls with the -a flag

man

man are the linux man pages and takes a program name as an argument and gives detailed documentation of how exactly each available flag will modify the programs functionality
for example:
we can run the following command man ls to list the flags available to the ls command, in here we will find a lot of modifiers that could make our lives easier such as the -a flag which will print hidden directories as well as the standard ones, we could also use the --color flag which will colorize the output allowing us to easily differentiate between directories and different file extensions

ls

ls simply translates to list directory contents, there are multiple flags for ls which make it more capable, from the screenshot of the ls man page you can see that there is a lot that can be edited and a lot of flags available
This command is very basic but forms the backbone of many peoples daily workflow to know where they are in the computer and whats around you

pwd

Now that we know what is around us, we need to know where we are to do that we can usepwd if the shell prompt doesn't let you know your location already. pwd takes no arguments and simply prints the current path and exits

touch

touch creates an empty file with the name as an argument and can also take a relative path

N.B. an error will be thrown unless the directory exists that you are trying to create the file in

rm

rm removes a file by passing the file name or path as an argument, it can also remove a directory by passing the -r flag before specifying the path or directory name but it is usually used to delete single files

N.B. Be aware of the power of rm -rf / as this will recursively delete every file on your computer although it may need super user priveleges sudo

sudo

sudo can precede most commands on this list and gives you access to the system as the root user temporarily, you may have to use root priveleges when installing packages from your distro package repository

~~no image here~~

mkdir

mkdir means make directory and it does exactly that so you can create a directory in the current file or you can pass a relative path to the command.

rmdir

rmdir is the opposite of mkdir and can only remove empty directories so to remove a directory and its contents we use rm with the -r flag, which stands for recursive

cd

cd means change directory and allows us to move around changing our current working directory. The most useful commands with cd are cd .. which moves to a higher directory otherwise you can cd into a directory below

cp

This is the terminal version of copy and all you have to do is pass the source and destination file to the program, i.e. cp <source> <destination>, this will copy the source file to the destination file

mv

mv is similar to cp but it moves the source file to the destination location, it could also be considered a renaming tool because renaming is equivalent to moving to a new name

Conclusion

These are some basic shell commands but with them you will be able to move around your terminal as well as start to build on top of them, the most important thing to remember is that there are man pages available with extensive documentation as well as the help flag which will sumarize everything

Thanks for reading and good luck in your future endeavours

Custom Domain Names and Subdomains for Development

David Rogers — Tue, 10 May 2022 21:21:34 +0000

Intro

Developing a Ruby on Rails application with subdomains is much more difficult than you would imagine.
Although we could just install puma-dev following the installation guide, we wouldn't understand why that works, or how, and so we close our minds to whats down that rabbit hole.

Down the Rabbit Hole

To implement any solution it certainly helps to know what is going on behind the scenes and so what we have to understand is that there is a local DNS service on our machines that is searching in the cache for any site ip addresses which we are trying to look for in our browser. This is all well and good but it doesn't consider our local ports or in our case port 80 with a custom domain name so what we need is a configurable local DNS service that we can use to map 127.0.0.1 which is localhost by default to a custom domain able to handle subdomains.
To do this we use dnsmasq, however dnsmasq on its own will just return whatever is running on port 80 so we need a way to map our server which in my case was running on port 3000 to port 80 so for that we use NGINX as a proxy server/reverse proxy.

setting up NGINX

installing NGINX on linux is pretty simple as it should be provided by default by the package manager of your distro, if not you may have to jump through some hoops to get it installed but for this blog that is outside the scope
So... assuming nginx wasn't to difficult to install we shall continue.
The next step is to configure nginx so that our application is served at http://localhost/ instead of http://localhost:3000/, the difference being that by default a url withot the port specified, is directed to port 80 if it is using the HTTP protocol and port 443 if it is using the HTTPS protocol.
To do that we to configure NGINX to pass/proxy our traffic from port 80 to port 3000.
This is achived in the following configuration which will be somewhere in the /etc/nginx/ directory (location may vary):

http {
    .
    .
    .
    server {
        listen       80;
        server_name custom.domain *.custom.domain;

        location / {
            proxy_pass http://localhost:3000;
            proxy_redirect off;
            proxy_set_header Host $host;
        }
        .
        .
        .
    }
    .
    .
    .
}

N.B there will be other configurations in the both the server and http blocks which don't need to be removed or changed

The important code being listen 80; and proxy_pass http://localhost:3000

now if everything has gone according to plan we should see our website at http://localhost/ if we exclude the line with server_name in our configuration.

to add a custom domain name we need to add dnsmasq and the line including server_name to our NGINX config so first install and configure dnsmasq.

dnsmasq

Install dnsmasq from the default package manager or follow the recommended installation for your distro from the docs.
dnsmasq generates a massive default config file with explanatory comments for every option however it is easier to just append to the file as you may want to change something else in future.
For our purposes we are going to add the following to our config file:

port=53
domain-needed
bogus-priv
address=/custom.domain/127.0.0.1
address=/.custom.domain/127.0.0.1
listen-address=127.0.0.1
bind-interfaces

port=53 runs dnsmasq as the default local DNS
domain-needed and bogus-priv prevent packets with malformed domain names and packets with private IP addresses from leaving your network, don't ask me what this means because I am not entirely sure, more info can be found here.
the address lines add custom.domain to the server redirecting them to 127.0.0.1 which is where our service is running as mentioned earlier on port 80 which is default lookup
the .custom.domain allows for subdomains to also be resolved
listen-address is self explanatory and is then listening on localhost
bind-interfaces binds all interfaces and updates if some interfaces get dropped or changed.

Setting dnsmasq as default

by default on systemd systems the default DNS resolver is systemd-resolved so to enable dnsmasq which listens on the same port we have to first disable systemd-resolved by running:

sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved

This should destroy some symlinked files so that the service won't run on start up.
we then need to enable dnsmasq by running:

sudo systemctl enable dnsmasq
sudo systemctl start dnsmasq

All this has done at this stage is enable dnsmasq as the default DNS server and allowed it to redirect custom.domain to port 80 which should allow us to use subdomains and use the custom address but if we would like to add multiple custom domains we need to restrict the proxy server to only proxy custom.domain, to do that we add the following line to our nginx config under the listen command:

server_name custom.domain *.custom.domain;

Adding 127.0.0.1 to /etc/resolv.conf

Because we are using dnsmasq as our default DNS server we need it to resolve both local requests and internet wide and so it is using the /etc/resolv.conf file to look up the DNS server of the router or internet connection we are using, so to add ourselves to the list of available nameservers we have to append nameserver 127.0.0.1 to our /etc/resolv.conf file, by doing this we have allowed our DNS server to first consider our local entries.
I have this set up in an alias in my .zshrc file:

alias nameserver:add="sudo sed -i '1s/^/# Custom Entry\nnameserver 127.0.0.1\n/' /etc/resolv.conf"

Putting it all together

Everything is in place and near ready to go so to get it all running together we may need to restart our services and visit the page:

sudo systemctl restart nginx
sudo sed -i '1s/^/# Custom Entry\nnameserver 127.0.0.1\n/' /etc/resolv.conf
# the above command can be replaced by nameserver:add if you have sourced your shell file
sudo systemctl restart dnsmasq

Adding more domains

To add an additional domain add an address to the dnsmasq config file:

address=/new.domain/127.0.0.1

And add a server to the NGINX config file:

http {
    .
    .
    .
    server {
        listen       80;
        server_name custom.domain *.custom.domain;

        location / {
            proxy_pass http://localhost:3000;
            proxy_redirect off;
            proxy_set_header Host $host;
        }
        .
        .
        .
    }

    server {
        listen       80;
        server_name new.domain;

        location / {
            proxy_pass http://localhost:4000;
            proxy_redirect off;
            proxy_set_header Host $host;
        }
        .
        .
        .
    }
    .
    .
    .
}

Conclusion

If you have any problems the official docs are the ultimate source of truth
This solution may not work for everyone but it works very well for me

Update

I have since moved away from using /etc/resolv.conf and have updated dnsmasq.conf to use a custom resolver file:

# /etc/dnsmasq.conf

port=53
domain-needed
bogus-priv
address=/custom.domain/127.0.0.1
address=/.custom.domain/127.0.0.1
listen-address=127.0.0.1
bind-interfaces
resolv-file=/etc/dnsmasq-resolv.conf

# /etc/dnsmasq-resolv.conf

# localhost
nameserver 127.0.0.1
# Google DNS
nameserver 8.8.8.8
nameserver 8.8.4.4
# Cloudflare DNS
nameserver 1.1.1.1
nameserver 1.0.0.1

by doing this i don't need to alias auto writing to resolv.conf, and hopefully these DNS servers are fast enough, but in case they are not, the last line of dnsmasq.conf can be undone and you can use your local resolv.conf file

You also have to turn off the NetworkManager DNS service by editing /etc/NetworkManager/NetworkManager.conf

# /etc/NetworkManager/NetworkManager.conf

[main]
dns=none

I updated to this latest version because my dns service was slow and impacted my work, i hope someone else can benefit from this

The Journey to Arch

David Rogers — Fri, 15 Apr 2022 23:53:42 +0000

As all Arch users began with Ubuntu I guess it is only fair that that is where I started as well. I started watching reviews on GNOME versions and soon came to realize the limitations of a stable distro: You don't get access to the cutting edge packages and features and so I simply had to change to Arch.

Nope it wasn't that simple, in fact I first changed to the 6 month release from Ubuntu which was not an LTS version in order to try out a few of the latest GNOME improvements, I then tried to install docker which is unsupported on any version of ubuntu that is not LTS which makes sense. So I thought to go to the root of the cause.

Debian Bullseye Is the father of all these Ubuntu releases so it must have access to the latest packages right? Well not unless you change the apt repository to the unstable version where you can't not have a package, They must have everything there?
Dependency errors, that's it. You are left with a system of half stable and half unstable, trying to install some of the latest software compiling from github because there is no alternative repository hosting all this custom software except on github.

I had been flirting with the idea of moving to Arch as well, looking in the mirror and telling myself I am a hardcore Linux user. So.... I have the bright idea of dual booting Arch and Debian. I had installed Arch on a virtual machine so I thought I was ready but a slight lack of judgement and understanding on my part whilst installing Debian was that I chose to go with Logical Volume Management which I still don't know what it does so my Debian installation was in a boot loop after seeing that the size of the LVM and available size on the partitition were different.
Well S**t.

As much as I told myself I was a hardcore Linux user it seemed an opportune time to live up to it so Arch it is - although it wasn't really a choice, it was now the only bootable ISO image I had on hand. I installed Arch 3 times that day, GRUB couldn't figure out where and to be honest neither could I.

After successfully installing Arch I went about getting back all my settings which at this stage were no longer dependent on GNOME because as a hardcore Linux user it has to be a window manager, and so after pulling all my dotfiles and configuring everything again, finding some packages was such a breeze after learning the dirty secrets of the Arch User Repository and so rarely had to compile untested github code

For some of us looking for the bleeding edge we all start with a user friendly distro but after changing so many times to try and get the latest and greatest software I guess the only constant is change and as Arch is always changing I hope that that will keep me away from distro hopping.