Forem: Leo Spratt

Self Hosting Experience - 2024

Leo Spratt — Mon, 12 Feb 2024 15:57:00 +0000

Intro

Since I started self-hosting a lot has changed, both in my setup and available technology. This article will document my experience (up until 2024).

How I Started And Why

Self-hosting for me started about 5 years ago, in 2019. But why did I start? Well my family was already running a 2 bay Synology NAS. However I wanted a way to host my web app that I was making during school; so that my family could use it (it was a freezer manager) however, my love for tech and expanding my knowledge was also a driving factor.

At that time I had no idea what Docker or containers were; so everything was manually installed (nginx, mariadb and Python). All of this was running on a spare Raspberry Pi 2, so I had many limitations mostly from storage (those pesky Micro SD cards) and available memory (1GB RAM).

At some point my Computer Science teacher suggested I look into Docker (and also git). I researched Docker however did nothing with it; since my setup was running fine. It wasn't until I discovered PiHole a DNS server that allowed for network-wide ad blocking and easy management through a web ui that I started to investigate further.

Learning Docker allowed me to setup PiHole on my existing server and because I was using nginx, it enabled me to access my site through a simple domain name and share port 80, a vast improvement from having each service on a different port.

The First Upgrade

After not long the performance of the Raspberry Pi 2 became a limitation, since I was hosting many more services. So I migrated to a faster Raspberry Pi 4 8GB model, allowing for booting from a 2.5 SSD making it a lot more reliable.

I also wanted an easy way to backup my server. So I bought another Raspberry Pi 4 however, a 2GB model and attached a 2.5 1TB HDD, turning it into a NAS by installing Open Media Vault. As well as backups; I also used this device as a private Docker image registry, mostly for aiding my software development hobby.

Present Day

I now have 30+ services running locally at home. However all of these services no-longer run on a Pi, instead I have moved onto a Docker Swarm cluster running on three mini-pcs.

I went with Docker Swarm as it's a lot simpler than Kubernetes and it covers fail-over well enough for a home-lab. For storage in the cluster I have setup GlusterFS which is a distributed file system, which I have found very stable. I have it setup so that all data is replicated across all machines; as I am more interested in redundancy over storage capacity. Because of this there is no real performance downsides during reads, in-fact I have it setup so a node can read from all simultaneously.

During my time using Raspberry Pis I have found that there are some software limitations with them. The main issue is that some software only runs on x86_64 however, that is changing since a lot more software is supporting ARM64. Another is that the recent versions of MongoDB do not work, due to the CPU missing a certain cryptography feature. This may change in the future with newer Pi's or other Single Board Computers (SBCs) being released.

Despite these issues I still use Raspberry Pi's for servers. I now run Home Assistant and still use my Open Media Vault NAS, which now has an offsite backup, so I can truly say I now have a 3-2-1 backup strategy.

The 2 bay Synology NAS has also been upgraded to a 4 bay, since it was almost at capacity. This is also used as another backup endpoint but mostly houses photos, videos and other documents.

Throughout all my time self-hosting I have always picked Ubuntu Server LTS where possible. I have found it most stable and having good defaults. I would have to agree it does come with some extra software installed that is not found in distros such as Debian however, they do not take much space to make me want to change. Unlike Debian it also ships with better driver support, meaning it will have a better chance at running on random machines.

Tech Specs

For those who want a clear list of the tech I currently have running, see below:

3x HP EliteDesk 800 G3 - 35W
- CPU: i5-6500T
- RAM: 8GB
- Storage: 500GB NVMe
- OS: Ubuntu Server LTS
1x Raspberry Pi 4
- RAM: 8GB
- Storage: 2.5" 250GB SSD
- OS: Ubuntu Server LTS
1x Raspberry Pi 4
- RAM: 2GB
- Storage: 2.5" 1TB HDD
- OS: Debian with Open Media Vault
1x Synology NAS - 4 bay
- Storage: 4x 4TB HDD

Conclusion

Looking back on how I started it may have not been the best idea to buy all those Raspberry Pis, since cheap low power mini-pcs have always been available on sites like eBay and would have been more cost effective. Since with Raspberry Pi's you have to buy extras such as cases, power-supplies and storage. However the ones I have already bought will not go to waste as I can always repurpose them for other projects later.

In future I will be investigating how to deploy NixOS as a NAS to replace Open Media Vault. Apart from that I don't think I will be changing anything I am currently doing, apart from making Ansible handle all installs and changes.

I hope you enjoyed and found this article useful. Be on the look-out for a related article that is in the works which is my take on "how to start self-hosting", which will go more in-depth on how to start and some of the things I have learnt along the way.

Join The Community

Stay informed about new posts or announcements about my apps join the community:

Want to help out? A great way of doing that is supporting me at: Buy Me A Coffee.

The Journey to Neovim

Leo Spratt — Sun, 02 Jul 2023 18:21:00 +0000

Intro

I started off as a Python developer using PyCharm, this is a fine editor if you only do Python programming, however like most developers; I started to branch out into other languages. Because of this I started to look for more IDEs to use. And so my journey of installing IDEs for every language started.

I realised after a while that filling my computer with many IDEs was not a good idea. Each time I switched to use a different language I had to learn all the shortcuts and buttons again, this slowed down my development quite a lot.

There must be a solution???

The Interim

Enter Visual Studio Code (in beta at the time), the editor that completely changed my development experience. This editor finally allowed me to migrate from using many editors to just one. Unlike all the other editors I used; it finally allowed me to customise it to my liking, whether that was the theme or the keymaps. It allowed me to develop in many languages and get the advanced features expected in a traditional IDE; due to the LSP (an amazing creation).

I have seen overtime the majority of developers are still using language specific IDEs, where they are: slow, consume large amounts of memory and are not easily extendable. I can now safely say that large; language specific IDEs are not necessarily needed for most users. For example Visual Studio is only really required when developing something specific to Windows like building a desktop app using their GUI frameworks. Game programming may also be one of the use-cases where the IDE specific to the engine is required for example Unity. However for the rest of us; we are free to choose from the many available options.

However after a while of being a proud Visual Studio Code user; the limitations of it caught up, the more extensions that were installed made the program feel sluggish; even on fast machines. Also it started to freeze when I typed too fast, most likely all these issues were due to it using Electron; which is known to have certain performance issues.

It was time to change, yet again...

The Last Hope?

I have now almost switched completely to NeoVim (using VsCode just for the debugger, which I rarely use). This editor allows even more customisation compared to VsCode and is much more powerful since I can quickly make my own plugins using lua. Like VsCode it also has inbuilt support for the LSP, allowing for powerful integration with nearly any programming language.

As NeoVim is based on Vim it uses vim motions, which once learnt; can make your code editing fast. Traditional editors which use the mouse to navigate and interact with the editor, slows you down since every-time you switch between the mouse and keyboard; is time not typing, putting your mouse aside and just using your keyboard to manipulate any section of text or move/switch buffers is much more efficient.

This editor by default has a lot already setup in terms of keyboard interaction, however there are a few defaults that I had to change to make it suit me. The first change was to alter the leader key to be bound to the space key, for me this is more ergonomic than the default which is the backslash character. I also changed the key for opening the directory explorer to <leader>pv, which I find easier to press more accurately and faster since both hands can be used to activate it; allowing me to open it rapidly.

To give Neovim IDE features, I have installed several plugins, one of the most important ones is "LSP Zero" which provides suitable defaults and configuration of the LSP and integrates with Mason allowing easy install of LSP servers, linters and formatters. Another plugin I enjoy using is "Harpoon" which allows you to mark files and then navigate to them quickly through keybindings; rather than repeatedly using buffer next/previous. "Telescope" is also another plugin that can allow you to quickly navigate between files, I have multiple keybindings setup so I can use it in it's different modes (file names, git files, file contents). There are many more plugins that I use which I am not going to mention, however you can navigate to my dotfiles repository to view my configurations yourself.

Whilst setting up and getting used to vim motions did take some time, I can finally say that I am satisfied with my editor. It allows me to manipulate text quicker and more accurately than ever before. I have also recently paired it with tmux, a terminal multiplexer; allowing me to quickly switch between terminals and use vim motions.

Some of you may be wondering if I have tried out Emacs (and also Doom Emacs). In fact I actually have. Whilst I found it could be customised maybe more than Neovim, I just cannot get used to it. By default in Emacs most keymaps default to using one side of the keyboard which is not ideal for me. It also uses a subset of Lisp, which I do not like writing in. In NeoVim by default the keymaps make sense and have vim-motions; it can also be customised by using lua, which I find a much nicer language to write small functions in.

Conclusion

I will now leave now with a question, will this be the last editor I use? I guess stay tuned! For others I would urge anyone wanting to improve their text editing experience to try out NeoVim. Or even migrate to use Visual Studio Code, which is still better than most IDEs out there at the moment in my opinion.

Even if you do not want to switch to Neovim for your main code editor, learning vim can be useful if you have servers or maintain them for someone else. I say this because whilst maintaining my own servers having traditional vim installed and the knowledge to use vim-motions has been very useful, since I can now edit config files with the same speed as I would on my desktop.

This post was not endorsed or sponsored by any of the mentioned parties. My views are my own.

Looking for more self-hosted related content or programming help, join the Enchanted People community here:

Essentials for Developers

Leo Spratt — Sun, 04 Jun 2023 19:33:00 +0000

Intro

In this post I hope to showcase some of the most essential items I have discovered to aid in the development of software.

Code Editor

Starting off with the code editor, whilst this may be a obvious tool to have; it is the most important as it is what you use to develop your software and can drastically effect your experience.

Most articles seen online make quite a fuss about various IDEs, and how you must have a specific one for each language or framework. Some of the IDEs spoken about being: Visual Studio or one of the many JetBrain offerings.

After several years of software development, I have come to realise how important it is to have a good keyboard driven code editor that can support many languages. These larger IDE's spoken about mostly focus on being "point and click" interfaces; which can slow down development.

I would consider Visual Studio Code to be a great editor to start off with, it features an LSP so it can integrate with many languages and supports a reasonable amount of customisation. Through these extensions you can even get vim style motions allowing faster code modification; enhancing productivity. However the more you customise; it makes sense to switch to something like NeoVim or even Emacs. These editors whilst time consuming to setup, as they are more like a blank canvas to start with, take keyboard driven development to another level. I recently migrated to NeoVim and can recommend this as a more powerful alternative to Visual Studio Code.

I would consider the most important thing to keep in mind; is your editor should be as flexible as possible, meaning it can support many languages and support vast customisation, like NeoVim has achieved with lua plugin support and the ability to use the LSP.

Code Formatting

Throughout my time as a software developer I have used various code formatters and linters, however I still see many people not using them. I believe code formatters to be essential for a developer. This is because well formatted code makes it easier to read and conventions can be followed. For example in Python development; one should follow the PEP8 spec, with a code formatter this is easy since all you have to do is run your formatter and it will transform the code to comply.

Having your code formatted to a specific spec can also reduce the arguments when working in a team. Since a convention can be decided and then automatically applied through a configuration in the project directory, meaning there is no question how the code should be formatted.

Testing Servers

This next section may not apply to all projects, however can be very useful. When developing software that uses external services such as databases. It can aid development to have your own locally hosted and always available services. For example in my dev environment I have several databases always running ready to use.

This allows me to always have the option to connect and test my project. Whilst these services can be run locally (even easier now with Docker), it is still nice to have services always available and running on a real server.

As well as hosting your own services to use, having the ability to also test your applications on these servers can also be very useful, since they are running on real hardware. Some issues will not occur until run in the real world no matter how many unit tests you have.

Containers & Docker

Containers in general are a really useful piece of technology. Here's a quick explanation of what a container exactly is: You can think of a container as a lightweight virtual machine, however it shares the host kernel, thus no emulation of hardware is done however the data is isolated. In a container you can package all of a software's dependences together separate from any other installed software, this removes any possibility of services conflicting with each other. This also means that running the same container image on a different system should perform exactly the same, minus any performance differences.

As you can see containers now open the possibility of running databases and other services locally as part of your development environment, meaning development will be easier since all the services can be run without installing any software that may conflict with others.

Working in a team environment, containers can also allow everyone on the team to have the exact same services running; you can even lock to a specific version. This can allow for a consistent environment, without any outside interference from unexpected updates. These containers can also be used in production, bringing stable and consistent services which can be deployed independent of the underlying Linux distro.

Now on to Docker. This is a piece of software that can make handling containers easy. It can build images using Dockerfiles, run containers, manage storage mounts and configure inter-container networking. It currently is the only one that offers a complete easy to use platform.

This technology is a must to learn and try out for any developer.

Personal Git Server

First off, if you are not using Git or some other version control in your project how are you still sane? Version control is a major tool in development as it allows you to maintain a precise change history of your project as you go along; this allows you to go back in time and discover why you made specific changes (through comments in the history).

If you are using Git you're most likely pushing your code to one of the major hosting sites such as GitHub, GitLab or BitBucket. However I believe that hosting your own as well will aid in the development of software. Not only will you now not depend on these hosting companies, you can now always access your data.

Consider this scenario where you only locally clone repositories using the "--depth" flag (which does not clone the full history), if your chosen site suddenly loose the data, you will not have the full history of your project. This can be quite important for discovering what changes you made along the way and possibility who made them.

Just because you host your own, it does not mean you can't still mainly use the one hosted not under your control. To keep your own hosted solution in sync you can setup a repository mirror in most solutions; such as GitLab CE or Gitea. That way all remote repositories will have the same data.

A Linux Machine

Having a Linux based machine in my opinion is important, it allows you to have a system that is highly customisable and use features only available on Linux. For example containers are easier to setup and most images are designed to work on Linux. You can also benefit from a selection of many command-line tools, which can aid in a better development experience (like my program "run-tool") since commonly performed tasks can be automated.

Demo Hosting

Another useful tool to have at your disposal; more suited to web developers, is the ability to quickly run you app and expose it publicly right from your development machine. This is where services such as Ngrok and Cloudflare tunnels come in very handy. Running their agent on your machine whilst your app is running they can expose your locally hosted app to the public internet, under a unique URL.

These http tunnelling services are great for teams or tech demos; where you don't need a dedicated server or domain, just a quick way of sharing access to your app. They also do not require port-forwarding, since your machine first opens the connection to the server; meaning they should work pretty much anywhere the internet is available.

Conclusion

There are many other tools which could be mentioned, so many in fact that the article would never end. I Hope you enjoyed reading this article, maybe you're now rushing to setup Docker or installing NeoVim; in which case I hope you have fun learning new technology!

This post was not endorsed or sponsored by any of the mentioned parties. My views are my own.

Forem: Leo Spratt

Self Hosting Experience - 2024

Intro

How I Started And Why

The First Upgrade

Present Day

Tech Specs

Conclusion

Join The Community

The Journey to Neovim

Intro

The Interim

The Last Hope?

Conclusion

Links

Essentials for Developers

Intro

Code Editor

Code Formatting

Testing Servers

Containers & Docker

Personal Git Server

A Linux Machine

Demo Hosting

Conclusion