Forem: Tom Watt

Things I learnt migrating an application to Kubernetes

Tom Watt — Tue, 13 Jul 2021 15:02:58 +0000

Let me paint the picture, an application forgotten to gain dust and bugs. Created in a rush to be deployed solely using a custom in-house deployment system on ageing and unmaintained Virtual Machines.

Python2. Django 1. Hardcoding.

It was a mess. It was hard.

But it was a rewarding experience.

Knowledge of tooling is sometimes sparse

The application used Docker - thankfully - but used a lot of scripts to do little things.

Times change and knowledge drifts. But we can still over issues with a bash script.

First thing I did was remove all these scripts.

Script to run commands in the container, was replaced with a Makefile. Serving as both documentation and one-command setup.
Script to setup local environment variables, was replaced with default values set either in the Docker Compose or the application. Local development shouldn't require extra effort to setup and should work from a fresh pull.
Script to pull additional libraries, was replaced with git submodules. Love or hate them, submodules are useful and cleaner when done right.

If you were ever think about using a script to overcome an issue, double check what you are already using and you might find a better solution.

Focus on what the Application needs when building a container

The container originally when built was over a whopping 1GB! But as I mentioned it was a Django application, so how was it so big?

"Useful tools and packages"

Database Client, Text editors, etc., you name it and it was probably there.

The application didn't need any of it. A developer did.

The convenience of pre-install tools and packages adds unneccesary bulk. This slows building time and add extra maintenance.

Ideally, you shouldn't need to shell into a container to Debug. But if you did, you could just install the necessary when debugging and then destroy the container when done.

Containers are meant to emphemeral.

After removing all those 'convenient' tools and packages, the image size dropped to about 300MB. 💪

Moving code repositories to a new host is easy?

The code needed moved, from BitBucket to GitHub. But I had never done anything like this before.

After a bit of searching around and trying GitHub's Importer, which didn't work, I came across this:

git push --mirror {destination}

I had the power.

16 repositories pulled. 16 repositories created. 16 mirror pushes. Done.

Though as easy as it was to do with fresh empty repositories., the command is destructive so be wary.

Helm + Terraform works but better used separately

We use a combination of tools. Terraform for all the infrastructure and Helm to template the application settings.

But Terraform can do the things Helm does.

The application is aged, and reliant on older versions of services e.g., Elastic Search. I tried using the Elastic Search Helm chart but I couldn't get the exact version that the application was previously using and it added extra things that I didn't need e.g., multiple pods.

Using Terraform, I created a simple Kubernetes Deployment for the Elastic Search service and that's all I needed. Job Done.

Another challenge I encountered, as I had created the Helm template for the application and referenced it locally with Terraform, is that any update to the template wouldn't cause Terraform to update it.

A way around this was to ensure Values passed into the helm_release resource were referenced and easily updatable e.g., Image Tag.

But looking back, if I had just created it all in Terraform, I wouldn't have had to do such things and I feel it would have been nicer overall.

Plus Helm templates are painful to debug. Too many spaces here, too little there. 🙃

Embracing Failing Fast

I thought I was being smart when taking out the hardcoded settings for Django e.g., Database Name, User, etc. and using Environment Variables but providing Defaults.

DATABASE_NAME = os.environ.get('DATABASE_NAME', 'postgres')

It worked fine for the local development setup as I didn't need to update the Compose file with an extra Environment Variable.

But then I noticed an issue in the Live environment on the Cluster, that the application was using the wrong Database. But how?!

...

I forgot to include it in the Helm Values.

...

A quick fix.

DATABASE_NAME = os.environ['DATABASE_NAME']

And now I'll never forget to include important settings.

You won't always get thanks but should always take on the challenge

This took me a lot of time. Research, distractions, failures, repeat.

And from the Client and User side, nothing changed. The site is up and running.

But it was worth it.

There didn't need to be any fanfare. When you put in a lot of effort to make something a little bit better, more secure, reliable and can be proud of what you've managed, then that's all you need.

The relief from finishing it and merging all the code after a clean was pure bliss.

I've learned a lot doing this but I would happily do it again.

Going "Repository Native" with Continuous Integration (CI)

Tom Watt — Fri, 02 Apr 2021 14:13:09 +0000

The idea of Continuous Integration in Software Development has become so popular over the years that it almost seems like everyone knows about it. With a decade of worldwide growth in people searching for it, and now its declining trend in the past few years.

Data source: Google Trends (https://www.google.com/trends)

This has led to the creation of many opinions and many more tools to implement Continuous Integration. To date, the Cloud Native Computing Foundation recognises 36 tools under the term "Continuous Integration & Delivery" but there is many more in the wilds of the Internet.

All of them are not equal, either specific to a Platform or Cloud Provider or requiring additional setup. Some for unique use cases or generic enough that switching between can be easy.

So, what do I mean by "Repository Native" CI? I'm talking about the CI tools provided by a Repository Host e.g., BitBucket Pipelines, GitHub Actions and GitLab CI/CD.

These CI tools provide ease of use and require minimal setup and maintenance. Most of them only require adding an additional file, e.g., bitbucket-pipelines.yml, to the code repository and it 'just' works, providing an appropriate insight into the running tasks and feedback.

New commits and branches can easily trigger your CI process. Environment Variables for the CI process can be set and controlled in the Code Repository. And most of these Repository Native CI tools can be easily linked to external applications for Notifications.

If hosting your Repositories on a Cloud Provider and using their CI offering, you'll get a similar experience but with the additional benefit of easily integrating with other services that the Cloud Provider offers. For example, using Google Cloud Build and pushing a Container Image to a Google Container Registry without needing set up Service Accounts - in most cases.

In terms of maintenance, there is very little and pretty much seen in a 'hands-off' approach. Keeping your CI process up to date should always be the main focus.

Compared to an external CI tool e.g., Concourse, Jenkins and Tekton, "Repository Native" CI offerings are somewhat more generic. External CI tools usually have unique conventions but appeal to creating more complex CI processes but suffer from adding complexity, which requires additional knowledge.

There is an initial cost to external CI tools, usually in the form of access to the code - "Repository Native" CI tools already have access. Which, while usually, is little, it can quickly build with more repositories needing accessed and how complex the CI process is. For instance, if a task requires access to multiple repositories to run.

Another cost comes from giving appropriate access to the external CI tool to other services e.g., Container Registries or a Cloud Provider. Though, this is also experienced when using Code Repository Hosting - e.g., BitBucket - and needing create a VM on a Cloud Provider - e.g., AWS but is slowly easing with integrations being developed.

One benefit of external CI tools is usually the option to self-host. Allows for enhanced security and tweaking them to run how and where you'd like. But that comes with the greatest cost comes the maintenance: when your CI server is down or slow, you've got to fix it.

Though, in the world of more complex systems and intensive CI tasks, I’ll say that external CI tools do offer some distinct advantages. If self-hosting by being able to add more compute resources, as some "Repository Native" CI tools are limited. And by being able to create unique triggers for tasks e.g., using Slack.

I’ve usually found that the CI process will do short lived simple tasks – style checks, static code analysis and unit tests. These simple tasks benefit more from less setup. Less setup means less maintenance, and less maintenance means more productivity. This promotes the use of “Repository Native” CI tools – following the 80-20 rule.

Creating a k3s Cluster with k3sup & Multipass 💻☸️

Tom Watt — Thu, 28 Jan 2021 13:39:16 +0000

This started off due to frustration with my current Single Board Computers setup and limitations using Docker Desktop meaning I couldn't fully utilise all the features of Kubernetes.

I'd previously setup up a k3s cluster using k3sup on my Raspberry Pi's and ASUS Tinkerboard running Ubuntu 20.04. Due to using older models – Pi 2 & 3 -, with limited resources, it wasn't a great experience for testing demanding applications or load testing.

Docker Desktop Kubernetes is great for starting out but the limitations kick in when you want to test out multiple nodes and installing additional features like Ingress Controllers. Plus, running it on a MacBook Pro 2019 I found the battery life draining quickly – running just Docker Desktop is bad enough at times.

So here comes in Multipass. I'd seen it a few times and was very curious. I'd previously used Vagrant combined with VirtualBox with various levels of success and agony. So the idea of being able to run an Ubuntu VM and Kubernetes Cluster without additional programs to manage and debug got me hooked. I like things to be as native as possible and Multipass can run on HyperKit and Hyper-V.

Installing Multipass is pretty slick and having a CLI – like Vagrant – meant there was a way to wrap it all up in a script for repeatability. And install k3sup is even easier.

Bonus feature of Multipass is being able to pass in cloud-init files to do additional configuration setup, which adds many benefits beyond this example.

The plan is to create a 3 node – 1 master, 2 workers – k3s cluster, which is disposable and portable.

Tools used:

So let's begin!

k3sup uses SSH to install and by default Multipass uses a predefined User – ubuntu – and SSH Key-Pair, which can be found after bit of digging - see Issue #913. This is where cloud-init comes in.

Using cloud-init, we can easily create a user, assigned it to groups, and pass in a Public SSH Key of our choice. And this is the minimal to do so:

users:
- name: tom
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - ssh-rsa...

Next is to spin up the nodes, which can all be done with the Multipass CLI. We'll leave the VMs with the default values for CPU, RAM, Disk Space and OS - latest LTS - but include our cloud-init configuration:

multipass launch -n master --cloud-init - <<EOF
users:
- name: tom
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - ssh-rsa...
EOF

multipass launch -n node1 --cloud-init - <<EOF
users:
- name: tom
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - ssh-rsa...
EOF

multipass launch -n node2 --cloud-init - <<EOF
users:
- name: tom
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - ssh-rsa...
EOF

Using the Multipass CLI, we can then view the IP Addresses of our new VMs:

multipass list

Name                    State             IPv4             Image
master                  Running           192.168.64.8     Ubuntu 20.04 LTS
node1                   Running           192.168.64.9     Ubuntu 20.04 LTS
node2                   Running           192.168.64.10    Ubuntu 20.04 LTS

Now to get our k3s cluster built, again using the default values, first by setting up our master node:

k3sup install --ip 192.168.64.8 --context k3s-cluster --user tom --ssh-key ./demo-key

This will install k3s, setup the instance as the master node and return the kubeconfig to the current directory.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...
    server: https://192.168.64.8:6443
  name: k3s-cluster
contexts:
- context:
    cluster: k3s-cluster
    user: k3s-cluster
  name: k3s-cluster
current-context: k3s-cluster
kind: Config
preferences: {}
users:
- name: k3s-cluster
  user:
    client-certificate-data: ...
    client-key-data: ...

Next, to set up our worker nodes to join the cluster:

k3sup join --server-ip 192.168.64.8 --ip 192.168.64.9 --user tom --ssh-key demo-key

k3sup join --server-ip 192.168.64.8 --ip 192.168.64.10 --user tom --ssh-key demo-key

Finally, we can now check our k3s cluster is up and running:

KUBECONFIG=kubeconfig kubectl get nodes

NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   7m17s   v1.19.7+k3s1
node1    Ready    <none>   6m47s   v1.19.7+k3s1
node2    Ready    <none>   6m24s   v1.19.7+k3s1

And now time to tear it all down now:

multipass delete --all && multipass purge

The Next Level

I mentioned that I wanted this setup to be disposable and portable. So let's wrap this all up in a good ol' Shell script.

So with a bit of refactoring and Command Line tooling - Piping & Parsing - we can end up with a script like so:

#!/bin/sh

master="master"
nodes=("node1" "node2")
context="k3s-cluster"

createInstance () {
    multipass launch -n "$1" --cloud-init - <<EOF
users:
- name: ${USER}
  groups: sudo
  sudo: ALL=(ALL) NOPASSWD:ALL
  ssh_authorized_keys: 
  - $(cat "$PUBLIC_SSH_KEY_PATH")
EOF
}

getNodeIP() {
    echo $(multipass list | grep $1 | awk '{print $3}')
}

installK3sMasterNode() {
    MASTER_IP=$(getNodeIP $1)
    k3sup install --ip "$MASTER_IP" --context "$context" --user "$USER" --ssh-key  "${PRIVATE_SSH_KEY_PATH}"
}

installK3sWorkerNode() {
    NODE_IP=$(getNodeIP $1)
    k3sup join --server-ip "$MASTER_IP" --ip "$NODE_IP" --user "$USER" --ssh-key "${PRIVATE_SSH_KEY_PATH}"
}

createInstance $master

for node in "${nodes[@]}"
do
    createInstance "$node"
done

installK3sMasterNode $master

for node in "${nodes[@]}"
do
    installK3sWorkerNode "$node"
done

And with only needing two Environment Variables, we can now bring up a disposable cluster within minutes - I clocked just under 2 minutes!

PUBLIC_SSH_KEY_PATH=./demo-key.pub PRIVATE_SSH_KEY_PATH=./demo-key ./minimal-k3s-multipass-bootstrap.sh

Making a script as well allows for easy use on multiple systems without needing to install more than the necessary.

Multipass offers extra configuration options for the VMs and cloud-init can do more internally on the OS. Plus k3sup has even more options, but I'll keep it simple for now!

Enjoy!

tomowatt / k3s-multipass-bootstrap

Bootstrap script to get a k3s Cluster created with Multipass for local development.

Why we need Code Reviews

Tom Watt — Sun, 08 Nov 2020 10:19:44 +0000

I’ll start off by saying that this is more of a personal vent. I’ve not been in the coding game for too long but I’ve experienced different cultures when it comes to software engineering.

So here’s why I think we need Code Reviews - in no specific order.

Security

Code Reviews allow us to check for any potential security issues. As systems become more complex, we all want to make it easier to ship the code or want to test a new feature and, sometimes, we let security standards slip. We might accidentally include personal credentials in the code, expose APIs or misunderstand the consequences of our code which could have security implications.
Code Reviews allow us to work together to ensure these potential issues don't become reality.

Quality

We all should want to produce great quality code, but we are faced with reality that it's not going to happen all the time. We are faced with combinations of deadlines, obscure requirements, growing complexity and most importantly: being human.
Code Reviews allow our peers to let us know if the code isn't up to standards and improve it before it becomes a burden later on in maintenance.
We have all written code that we look back at and question why.

Reflection

Knowing that our Code is going to be reviewed allows us the opportunity to reflect on what we've done. We don't want to send bad code to get told to redo it.
When preparing our Code for review, we should reflect on what we've done, question if it meet the requirement(s), does the Commit messages make sense, does it meet internal/external standards, etc.

Personal Development

Being on both sides of a Code Review - receiving and giving - has its benefits.
As someone who's put their code up for review, we can be given knowledge and tips from others about how to improve it, told about functionality we never knew about that simplifies things, or told about the dreaded typo that we've totally overlooked and are now embarrassed about.
When giving the Code Review, we can learn from overlooking new code and styles which we can implement back into our own work.
Code Reviews, if done properly, enable us to learn and develop to become better.

Teamwork

As systems grow and become more complex, we will start having less understanding of all the moving parts. We may have to use code that we never built but someone else in our team did.
Using Code Reviews, we can enable team members to share their work and give more people a better understanding of the changes being made that we might have to use or work on in the future.

👨‍💻📝👩‍💻

New Job, New Challenges

Tom Watt — Sat, 24 Oct 2020 09:00:26 +0000

So during this chaotic year I managed motivate myself and seek new pastures. As much as I liked working at my previous employer, I wanted to grow and learn.

I managed to go from a Junior DevOps title to DevOps Engineer. I was mainly focused on working on the infrastructure and maintaining pipelines to now being brought in to do a whole lot more.

It's been 3 weeks since I started my new role and it has been tough. So much going on and so much to learn from the code, culture to working practises. Reflecting back, at my previous employment, I learnt so much from AWS, containers, CI/CD, etc., but mainly the way a team works and grows to deal with new challenges.

The biggest change I noticed was the culture. From a small business where I knew everyone and had a working relationship with them all, being able to chat and grow as a team. To a bigger business - which is currently rearranging its structure - and there is multiple historic cultures, different practises and noticeably, a lack of chat.

And now, with a bit of insight and pushing from my new manager, I'm starting to see the true nature of the DevOps role. I've got a lot to learn still but I'm starting to see what I'll need to do to fulfil my role.

Previously, I focused on automation, infrastructure and keeping the Developers 'happy' - either by simplifying things or taking away some of the burdens.

Now I'm challenged with setting new practises, standards and creating a new productive culture. I know it's going to be long and uphill struggle at times but I'm looking forward to putting good and sustainable standards in place and building a more connected team.

There will always be new technologies but I see building a strong culture of knowledge sharing, standards and shared responsibility as the biggest driver of software development.

P3 - Personal Porting Project - Update

Tom Watt — Sun, 06 Sep 2020 09:17:08 +0000

So little follow up on my Personal Project.

Progress

I've added two more Languages: Ballerina and Rust. Both of them presented unique challenges but very satisfying to finally get something working.

Ballerina, always stood out to me as an 'modern' language. Previous studies using Java allowed me to get a good start on Porting over the program but only got me so far.

Dealing with @tainted and @untainted annotations stumped me when dealing with the Error handling, so I feel my port doesn't truly represent how Ballerina should be written.

With that, I didn't use the @docker annotation - which I feel that's where Ballerina excels as a language - but that's just to be able to build and run with Docker Compose.

Rust. This was a whole new level of learning for me. I started off wrong and ended up taking a lot longer to Port compared to the other Languages - so far.

I restarted probably about 5 or 6 times just with the basics of opening a File, reading lines, etc. Not realising some examples of how to do such linear actions had been deprecated as Rust has evolved.

But it did then present me with an issue, I presented my Tasks for the Program in a abstract manner that's not easily implemented in Code. I updated these into smaller, more manageable Tasks and that allowed finish porting into Rust.

Once, I finished and complied the Rust code into the single Binary and saw how small the entire application and Docker Image could be, I was amazed.

Future Plans

Definitely aiming to add in Ruby soon. With possibility of Swift and see what if I can deal with another 2 more, to get a total of 9 different Languages.

Still need to refactor, add tests and HTML & CSS.

P3 - Personal Porting Project

Tom Watt ・ Aug 16 '20

#personal #project #docker #productivity

tomowatt / P3

P3 - Personal Porting Project

Tom Watt — Sun, 16 Aug 2020 08:38:55 +0000

What

It's a small idea based on the Password Strength - XKCD comic to create a simple application that will generate a random password from a collection of nouns and then recreate it in other programming languages.

Why

I like to take on new challenges and learn at the same time. I felt this is a great way to be able to learn new skills and techniques - and different Programming Languages - but also reinforce what I already know.

How

To keep it simple and quick to be able to test and run locally, I've used Docker Compose. This allows me to bring up multiple containers for the various Programming Languages and keep them separate.

I started with Python, as that's what I'm most comfortable with, and was able to get a skeleton/template to implemented in the other languages.

Progress

I've started to make some progress and recreated the application in Go and Node.

Go presented me with the challenge of being entirely new to me. The Syntax was somewhat familiar to Python but with the added Static Typing, needed use of Scanners to read files and needing to make Random Random(!?). It was more verbose with needed Error Handling but I have to say I liked it.
Additionally, building the binary at the end and being able to shrink the Docker Image from 300MB+ to 13MB using Multi-Stage Building was satisfying.

Node, again was entirely new to me as well. I originally intended for it to be Deno for TypeScript but came across no official Docker Image for Deno. So to support long term use, defaulted to Node. The use of CallBacks and Async totally threw me and I gave up and tried Rust - that didn't work out.
Then returned after thinking it through and reading more examples and documentation. The Syntax caught me out a few times but I managed to work my way through by going back to the skeleton/template to simplify my way of thinking.

Future Plans

Going to aim to get add in Ballerina, Ruby & Rust. Possibly, add more but I feel that's a good range of different languages and styles to start with. Next moving onto refactoring and making the applications suited to the languages.

Tests will need to be added as you always need tests.

Finally, adding use of HTML and CSS to create a UI and making the Docker Images as small as possible.

tomowatt / P3

P3 - Personal Porting Project

Tips for using Ansible

Tom Watt — Tue, 14 Jul 2020 18:38:03 +0000

I use Ansible on a daily basis and it brings me so much joy when I finally write up a new Role or update a Playbook and everything works.

But that’s usually after a while of failures - a lot of failures. But that’s okay, if Ansible is used in the right way then you’ll always make progress.

So here’s some tips that help!

1. Use ‘ansible-lint’

Such a simple but useful tool will help keep your Playbooks and Roles clean of errors and provide useful hints to improve them e.g. ensuring Tasks are named.
Best run when creating or editing Roles and Playbooks.

ansible-lint playbook.yaml

2. Be Verbose

With certain Modules there is Default values set and whilst it may be fine to not mention them when writing up Tasks, there can be consequences for your future self or others when they come to update the Task. For instance apt install packages by default but doesn't update the cache. So you can end up with a Task like this:

- name: Install Nginx
  apt:
    name: nginx

But future self or others might hit an issue with this if the cache hasn't been updated prior, so by making it more verbose the state and Task provide more clarity of what's actually being run:

- name: Install Nginx
  apt:
    name: nginx
    state: present
    update_cache: yes

3. Don’t be afraid to use ‘shell’ or ‘command’

There is lot of Modules out there but sometimes it's easier to write a simple one liner to do what you want done - although not recommended by Ansible. ACME Certificate Module is very complex to use, and it has to be run twice to work. When I can tried to implement Let's Encrypt/Certbot with a Wildcard, I found it easier to use shell instead:

- name: Get TLS Certificates
  shell: "certbot certonly --dns-route53 -d {{ server_name }} -n --agree-tos --email {{ email }}"

It's clearer what's being run and it can easily be changed in the future by others.

4. Enable Timer

Add this to your ansible.cfg:

callback_whitelist = timer, profile_tasks

And this will allow you to see how long each Task and the Playbook took to run. This gives both estimates for when others run them but also allows insight into what tasks are slow and could be improved.

PLAY RECAP
**************************************************************
rpi2                       : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
rpi3                       : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
tinker                     : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Tuesday 14 July 2020  19:20:11 +0100 (0:00:00.603)       0:01:19.673 ********** 
==============================================================
Gathering Facts - 13.72s
/Users/tomwatt/DEV/Ansible-Local/playbooks/test.yaml:2 
--------------------------------------------------------------
test-connection : Ensure the right sudo password - 6.13s
/Users/tomwatt/DEV/Ansible-Local/roles/test-connection/tasks/main.yaml:2 
--------------------------------------------------------------
debug-host-info : Print Host Information - 0.60s
/Users/tomwatt/DEV/Ansible-Local/roles/debug-host-info/tasks/main.yaml:2 
--------------------------------------------------------------
Playbook run took 0 days, 0 hours, 1 minutes, 19 seconds

5. Keep Ansible up to date

Newer versions have more functionality and fixes. One caveat, though, is being aware of changes between versions e.g. modules renamed, deprecations, etc. So check the Porting Guides when prior to updating.
Running your Playbooks with Verbose - -vvvvv - on will give hints of what’s being changed/removed in later versions.

6. Many and Small Roles

I've done it and it's easily done, creating a Playbook that does everything that needs doing. It will work but will it work in the future? What happens when it needs a minor change to work on a different server? This is where having more, smaller Roles helps.

Keeping a Role to a simple grouping of Tasks, with any needed variables, handlers or templates improves the reusability of it. It's also organises the code, for easier testing and maintenance.

7. Use ‘ansible-galaxy role init’ to create Roles

Following on, be lazy and use it to create the basic structure for your new Roles, then remove what you don’t need.

Hope some of these are helpful for those who use Ansible or looking to use it. But let me know and share any tips you all have as well!

Running an Unity WebGL game within Docker

Tom Watt — Sun, 05 Jul 2020 10:10:38 +0000

So whilst learning on how to make Unity games, I got curious of what would be a good way to test, share and get feedback on.

There are various websites that will host the games for you but I like the idea of making a game that’s Open for people to contribute to, learn from and ultimately play.

So this is where I went down the path of looking into using Docker to host a WebGL game. After searching the web I came across a few others that looked to do something similar.

So to start, I've built and exported the Unity game and kept a simple file structure.



.
├── Dockerfile
├── docker-compose.yaml
├── webgl
└── webgl.conf

This allowed me to easily copy necessary files with a single COPY within the Dockerfile.

To host game within Docker and keeping things simple, I've used Nginx as the base image as the HTML files only needed served.

But the default configuration needed to be updated to point to the copied files. This resulted in the following for the Nginx configuration, just using the index.html created by Unity and update the location root to where the files were copied to.

Next part is the Dockerfile itself, putting all the pieces together to host the WebGL game.

Finally, using Docker Compose, I can finally launch the Docker image and play the game within a browser with a single docker-compose -d up

All the code can be found here:

tomowatt / unity-docker-example

Example of running Unity WebGL within Docker

unity-docker-example

An example of how to run a Unity WebGL game using Docker. Although not the most exciting game, it could prove useful to be able to build and share games using Docker.

Run the game:

docker-compose up -d

Then visit localhost:8080 in a Browser.

Stop the game:

docker-compose down

View on GitHub

Hope this helps anyone who is curious about doing a similar thing and I'll hope to improve this as I learn more about Unity, WebGL and Docker.

2022 Update: Add linked code examples via gists and embedded repository, updated Dockerfile code

AWS - Save Costs with Lambda

Tom Watt — Fri, 05 Jun 2020 13:35:43 +0000

When I first started at my current role, AWS was completely new to me. And I mean everything. I'd worked previously dealing with physical servers and now I was just dealing with the CLI and Web Console to interact and manage servers.

One thing that caught me by chance was when I saw the billing every month and wondered why were we paying for development instances, databases, etc. when no one was there to work on them?

Whilst learning more as time went by, I found inspiration and use from previous colleagues works to deal with these unused resources. And that's when I found use and power of AWS Lambda.

I won't go in-depth but keeping things simple, using Crons from CloudWatch Events to trigger Lambdas that shutdown or terminate instances saves money.

Why is that? Because in most cases, unless you use Lambda a lot, it's free under the AWS Free Tier and such simple tasks such as deleting EC2 instances run take less than 1 second. Finally, it is automated.

Here's an example of what is needed to filter and terminate EC2 instances with specific tags:

import boto3

EC2_CLIENT = boto3.client('ec2')
EC2_RESOURCE = boto3.resource('ec2')

PROJECTS = ['super', 'special', 'ultra',]
ENV = ['dev']

def get_ec2_instances():
    ec2_instances = EC2_RESOURCE.instances.filter(
        Filters=[
            {
                'Name': 'tag:project',
                'Values': PROJECTS
            },
            {
                'Name': 'tag:env',
                'Values': ENV
            },
        ]
    )

    return [instance.id for instance in ec2_instances]

def delete_ec2_instances(instance_ids):
    if instance_ids:
        EC2_CLIENT.terminate_instances(InstanceIds=instance_ids)

def delete_instances(event=None, conext=None):
    delete_ec2_instances(get_ec2_instances())

You can easily use environment variables, add logging and error handling if needed but sometimes keeping it simple is the best.

And if you need to change when instances get terminated, just update your CloudWatch Event.