Forem: James G. Best

Essential terminal packages in 2020

James G. Best — Fri, 26 Jun 2020 08:44:51 +0000

This is the start of a new series about maximising your terminal experience. I will write about what I think are essential packages for the CLI, a simple but super effective Vim setup and setting some sane Tmux defaults.

In this first article, I am going to list some of what I think are essential CLI packages. These are ones I use every day and could not live without. I am leaving Vim and Tmux off this list as they will get there own articles.

I spend all my time in a terminal. This is where I do all my development, hacking and note-taking. Spending so much time there I want it to work for me, providing a buttery smooth experience.

Here is a list of my top CLI packages.

1. ZSH and oh-my-zsh:

This is my chosen shell. It has way too many features to list here but here are a choice few:

automatic cd: Just type the name of the directory
recursive path expansion: For example “/u/lo/b” expands to “/usr/local/bin”. This is not one I use much because I lead on the tools mentioned below.
spelling correction and approximate completion: If you make a small mistake typing a directory name, ZSH will fix it for you
plugin and theme support: ZSH includes many different plugin frameworks. This is where oh-my-zsh comings in. Its a community drive framework for managing your ZSH configuration.

2. FZF

FZF is a command-line fuzzy finder. It is super fast, light and works so well with all my other tools.

3. RipGrep (rg)

RipGrep is a stupid fast replacement for grep. It does away with lots of the hard to remember flags that grep requires. By default, it respects your .gitignore, and skips hidden files.

4. Autojump

Autojump is a super-fast way to navigate your filesystem. It learns the directories you visit most and remembers them for next time.

5. HTTPie

HTTPie is a user-friendly command line HTTP client. It has JSON support, syntax highlighting and lots more.

6. Bat

Bat is a drop-in replacement for cat but with superpowers. It's highly configurable and supports syntax highlighting.

7. exa

exa is a modern replacement for ls. It is highly configurable, provides better defaults.

8. twf

twf is a tree view explorer inspired by fzf.

You can find my dotfiles here. This is how I use them, how will you?

Thanks for reading 🙏

If there is anything I have missed, or if there is a better way to do something then please let me know.

Check out our software focused podcast - Salted Bytes

Running a Kali VM on a retina display with zero lag

James G. Best — Fri, 12 Jun 2020 08:13:58 +0000

This weeks post is more of a tip than an article but it is something I recently came across and thought it might help others.

In my free time, I like to learn about hacking and often spend my time completing CTFs. Most of the time I will use a distro like Kali as it is quick to set up and has all the tools that I am likely to need. Kali Linux is a Debian-based Linux distribution aimed at advanced Penetration Testing and Security Auditing.

I have always run Kali in a virtual machine (VM) in Virtualbox and I had never had any issues.

But when I installed it on my new (to me) 2019 MacBook Pro, there was so much lag that it was pretty much unusable. But if I ran it on my second display, a non-retina AOC 27in screen it ran fine. It boiled down to it being run on a retina display.

After a bit of digging, I found this thread which seemed to be describing a similar situation to mine.

So, now I knew it was an issue with the pixel density of my display, I needed to find a program that could change it. There were several options but EasyRes seemed to fit the bill and was free.

With EasyRes installed I was able to play around with various screen resolutions and densities to find the sweet spot and found that 1440 x 900 at standard resolution (1x) seems to work great. I am sure you could push it a little further but I did not need to, this worked fine.

Off the back of this issue, I also explored some other all-purpose infosec distros and found Parrot OS. I think I am in love. It's so lightweight. I run like a dream without the need for EasyRes, has all the same tools as Kali and looks stunning. So will I change? I am not sure. Parrot seems to do some strange things with network adapter names that make doing wireless hacking stuff a real pain. Other than that I am sold. Watch this space for a future update.

Thanks for reading 🙏

If there is anything I have missed, or if there is a better way to do something then please let me know.

Check out our software focused podcast - Salted Bytes

This was originally posted on my website

Javascript robotics: The Johnny Five REPL

James G. Best — Fri, 05 Jun 2020 06:54:27 +0000

In the previous article, we discussed how to get up and running with Johnny Five and the ESP8266 micro-controller. If you are not sure how to get set up I recommend you read that article first.

Previously we created a simple script that allowed us to control the onboard LED with our keyboard. This required us to install the keypress package and listen in process.stdin. This is not a bad approach but there is a better one.

Johnny Five comes packaged with a REPL. A REPL (read-eval-print-loop) is an interactive language shell giving a simple interactive interface to the Johnny Five API. We can control our robots with the REPL but there is some set up first.

Setting up the REPL

We need to make the REPL aware of our hardware by injecting the instance of the hardware into our script. We are using the script from before but with the keyboard code stripped out:

const { EtherPortClient } = require("etherport-client")
const { Board, Led, Pin } = require("johnny-five")

const board = new Board({
  port: new EtherPortClient({
    host: "192.168.1.109",
    port: 3030,
  }),
  repl: false,
})

const LED_PIN = 2

board.on("ready", () => {
  console.log("Board ready")
  var led = new Led(LED_PIN)
})

Now, let's add the REPL code. Update the board ready callback to look like this:

[...]
board.on("ready", function() {
  /*
    Initialize pin 2, which
    controls the built-in LED
  */
  var led = new Led(LED_PIN);

  /*
    Injecting object into the REPL
    allow access while the program
    is running.

    Try these in the REPL:

    led.on();
    led.off();
    led.blink();

  */
  board.repl.inject({
    led: led
  });
});

With this simple addition, we now have access to all the functions available on the LED object.

Limiting access

But what if we don't want to give our users unfettered access? What if we only want to give access to specific functions or write functions that do more than just control the hardware. Maybe we want to add logging or provide more appropriate function names. Well, we can write our own functions that are available in the REPL and inject those instead.

[...]
board.on("ready", function() {
  /*
    Initialize pin 2, which
    controls the built-in LED
  */
  var led = new Led(LED_PIN);

  board.repl.inject({
    // Allow limited on/off control access to the
    // Led instance from the REPL.
    on: function() {
      led.on();
    },
    off: function() {
      led.off();
    },
    flash: function () {
      led.blink();
    },

  });
});

This script will make on(), off() and flash() functions available in the REPL.

And that is it. A nice, short intro to the Johnny Five REPL. Until you start hooking your scripts up to WebSockets or a REST API, I think this is one of the better ways to control your robots.

Thanks for reading 🙏

If there is anything I have missed, or if there is a better way to do something then please let me know.

Check out our software-focused podcast - Salted Bytes

This was originally posted on my website

Wireless javascript robotics with Johnny five and the ESP8266

James G. Best — Fri, 29 May 2020 13:49:51 +0000

Orignally post on my website

In this article, I will talk about the first steps needed to start building robots with Javascript. I will be using the infamous ESP8266 microcontroller, this is because it is super cheap and allows it not to be tethered to your machine in the way that an Arduino would be.

To allow us to write our robotics scripts in Javascript with will be using the Johnny-Five library written by Rick Waldron. The library supports a huge selection of boards and hardware. Although not all boards support all of the hardware.

Several other boards could be used as alternatives (Proton, Tessel) but these are considerable more expensive and not as easily available. It is also possible to use an Arduino attached to a Raspberry Pi (RPi) and you then interface wirelessly with the RPi but that seems a little unnecessary now.

There are better languages to build robots with, but being an engineer working primarily in Javascript, I was keen to keep things close to home. In this setup Javascript is not run on the microcontroller but is run through some custom firmware called the Firmata protocol. This does mean it runs slower than something like C would but generally for things like robots this is not much of an issue. The first step in this process is to upload the StandardFirmataWifi sketch onto our board but to be able to do that we need the Arduino IDE and for it to recognise our ESP8266 based boards.

Getting set up

The following instructions are based on using a mac. They will be very similar to other platforms.

Copy the following URL http://arduino.esp8266.com/stable/package_esp8266com_index.json. Open the IDE and go to the files menu and click on preferences. Add the URL to the ‘Additional Boards Manager URL’.

Close the preferences panel and click tools. Under tools select boards and then board manager. Navigate to esp8266 by esp8266 community and install the software for Arduino.

Once this has been done you should be able to program your ESP8266 based board, I am using the NodeMCU board.

Click the examples panel and select the Wireless Firamta sketch. We will be updating so make a copy now.

We need to update the header file wifiConfig.h with our network credentials. Update the values of char ssid[] ="" and char wpa_passphrase[] ="".

With this done we can now upload the sketch to our device. Once uploaded you can close the Arduino IDE.

Don’t forget that the ESP* boards have a different pin layout to the Arduino boards. See the picture below for an example.

Now that we are all set up on the microcontroller we need to create a new node project and install the needed packages.

Our first robot script

Create a new folder for the project and initialise a new node project

mkdir helloWorld && cd $_ && npm init -y

Now we need to install Johnny-Five and the ethernet client that allows us to connect wirelessly.

npm install johnny-five ethernet-client keypress

With this done we are set to write our first script.

const { EtherPortClient } = require("etherport-client")
const { Board, Led, Pin } = require("johnny-five")
const keypress = require("keypress")

const board = new Board({
  port: new EtherPortClient({
    host: "192.168.1.109",
    port: 3030,
  }),
  repl: false,
})

keypress(process.stdin)
const LED_PIN = 2

board.on("ready", () => {
  console.log("Board ready")
  var led = new Led(LED_PIN)
  console.log("Use Up and Down arrows for On and Off. Space to stop.")

  process.stdin.resume()
  process.stdin.setEncoding("utf8")
  process.stdin.setRawMode(true)

  process.stdin.on("keypress", (ch, key) => {
    if (!key) {
      return
    }

    if (key.name === "q") {
      console.log("Quitting")
      process.exit()
    } else if (key.name === "up") {
      console.log("Blink")
      led.blink()
    } else if (key.name === "down") {
      console.log("Stop blinking")
      led.stop()
    }
  })
})

We need to replace 10.0.0.49 with the IP assigned to our board. I use an app called Fing but this information can be found out from the Serial monitor in the Arduino IDE.

This simple script will make the allow you to turn the onboard led on and off. Nothing fancy but paves the way for much more exciting things. To execute the file node index.js. You should see something similar to this:

$ node hello.js
1590554783332 SerialPort Connecting to host:port: 192.168.1.109:3030
1590554783334 Connected Connecting to host:port: 192.168.1.109:3030
1590554793338 Use Up and Down arrows for On and Off. Space to stop.

Now that the board is set up we are ready to create some more interesting projects. Johnny Five provides such a gentle introduction to robotics in Javascript but allows you to do so much as you have the wealth of packages in NPM and hundreds of public APIs that you can lean on to create great projects.

The next article will be a nice short one, introducing the Johnny Five repl and why it is great for prototyping your next project.

Thanks for reading 🙏

If there is anything I have missed, or if there is a better way to do something then please let me know.

Check out our software focused podcast - Salted Bytes

Fixing coc-tsserver errors due to Deno

James G. Best — Thu, 28 May 2020 13:56:18 +0000

This is a quick post detailing how to fix issues with coc-tsserver.

I am a long term Neovim user and feel that the only Intellisense engine to use is CocVim. It is simply amazing. It breathes so much power into you Vim setup.

I write Typescript daily and so use the coc-tsserver plugin. It gives all the power of VSCodes typescript engine in NeoVim. But recently it broke. It was reporting errors when there was nothing wrong. I made sure all my dependencies we up to date and still nothing. All I was seeing was this:

The error message is: The error message is: [tsserver 2307] [E] Cannot find module <import-path>

Something was wrong.

Thankfully someone was having the same issue and had created an issue in the CocVim git repo. It turns out that there is a bug in the coc-deno plugin that has some problems with non-deno projects.

Thankfully the fix is simple. Disable the deno plugin for non deno projects.

With in Vim: :CocList extensions and find the Deno plugin. Press tab and select d to disable it. Restart Vim and all your issues should have disappeared.

Thanks for reading 🙏

If there is anything I have missed, or if there is a better way to do something then please let me kno

Time controlled colour schemes for Vim and Iterm2

James G. Best — Tue, 26 May 2020 20:46:19 +0000

This was originally posted on my own website

If you have read any of my other articles you will know that I live in a terminal. All my work is done in Iterm2, using NeoVim, Tmux and a multitude of other tools. You will probably also know that I have taken the time to get my terminal and Vim looking just how I like it. I have written a theme (Ariake) for both Iterm2 and Vim with light and dark variants.

The reason for the light and dark variants are to help with working through the day into the evening. During the day I find it easier on the eyes to have a light theme. In the evening, a dark is better. But I would rarely remember to make the switch. I figured there must be a way to automate the process. It turned out to be pretty simple.

Setting up Vim

The update to Vim was super simple. Simply add the following lines to your init.vim

if strftime("%H") < 17
  set background=light
else
  set background=dark
endif

This follows the 24hr clock so update the 17 (5 pm) accordingly. I have mine change at around 5 pm. Now every time you openVim the time would be checked and the background colour updated accordingly.

Setting up Iterm2

This was a little more involved but nothing to hard. Iterm2 gives the user access to a Python API. With this, we have access to a huge array of functionality and so we can programmatically update it. In our case, we want to update the colour schema of our profile.

From the menu select Scripts > New Python Script and follow the wizard. It does not matter what you select as we are gonna replace the whole lot anyway. Once you have completed the wizard, open the script and replace the contents with the script below.

#!/usr/bin/env python3.7

import asyncio
import datetime
import iterm2

# Clock time to change colours.
LIGHT_TIME=(7, 0)
DARK_TIME=(17, 0)

# Color presets to use
LIGHT_PRESET_NAME="ariake-light"
DARK_PRESET_NAME="ariake-dark"

# Profiles to update
PROFILES=["Ariake"]

def datetime_after(t, time):
    today = datetime.datetime(t.year, t.month, t.day, time[0], time[1])
    if today > t:
        return today
    # Same time tomorrow
    return today + datetime.timedelta(1)


def next_deadline_after(t):
    light_deadline = datetime_after(t, LIGHT_TIME)
    dark_deadline = datetime_after(t, DARK_TIME)
    print("light {} dark {}".format(light_deadline, dark_deadline))
    if light_deadline < dark_deadline:
        return (LIGHT_PRESET_NAME, light_deadline)
    return (DARK_PRESET_NAME, dark_deadline)

def get_duration():
    now = datetime.datetime.now()
    preset_name, deadline = next_deadline_after(now)
    duration = (deadline - now).seconds
    print("Sleep for {} seconds until {}".format(duration, deadline))
    return duration, preset_name

async def set_colors(connection, preset_name):
    print("Change to preset {}".format(preset_name))
    preset = await iterm2.ColorPreset.async_get(connection, preset_name)
    for partial in (await iterm2.PartialProfile.async_query(connection)):
        if partial.name in PROFILES:
            await partial.async_set_color_preset(preset)

async def main(connection):
    while True:
        duration, preset_name = get_duration()
        await asyncio.sleep(duration)
        await set_colors(connection, preset_name)
        await asyncio.sleep(1)

iterm2.run_forever(main)

The only things that need changing are marked with comments. I found the script console handy for debugging the scripts. Once the script is complete you can activate it Scripts > [you script name]. If you want it to run automatically, move the script to $HOME/Library/ApplicationSupport/iTerm2/Scripts/AutoLaunch. You may need to create this folder.

Now that the scripts are in place you should have a lovely self-changing development environment allowing you to code all day and night 😀.

Thanks for reading 🙏

If there is anything I have missed, or if there is a better way to do something then please let me know

Avoiding that awkward BBC interview moment with Google Calendar, IFTTT, Tasker and a Raspberry Pi.

James G. Best — Mon, 23 Mar 2020 17:13:35 +0000

Having experienced this in my first week of working at home I knew I needed a solution for letting my wife and kid know when I was going to be in a meeting/pair programming. I could just shout downstairs but where is the fun in that? By automating it, it is also one less thing to have to remember in what is shaping up to be a stressful time for everyone.

I wanted this to be a quick Sunday evening project that only took a couple of hours so needed it to be simple and use the kit I already had. I also wanted it to work without the Raspberry Pi (RPi) being accessible from outside my home network.

I could have completed this project in a much simpler way if I had allowed access to the RPi from the outside world but in a world where IoT devices are the target of cyber attacks I wanted to limit the exposure.

On the hardware side of things, it seemed obvious to use an RPi and Mote LEDs. I have them already and they are simple to work with.

Mote is a collection of components allowing you to easily work with LEDs. The visual side of this project could be anything you have knocking around. I just happened to have these.

I am using an RPi 3 for this but you could any Pi that has is network connected.

All my meetings are in Google Calendar so I just needed a way to trigger an event when my meetings start and finish. If This Than That (IFTTT) is perfect for this. Google Calendar is the THIS of IFTTT and a webhook is the THAT.

I wanted this to be a quick project with as little programming as possible so I needed a way to accept the webhook but also be able to perform actions on a device in my local network. This part took a little longer to figure out, but a combination of Tasker and AutoRemote was able to perfectly meet my needs.

Tasker is an automation app that allows users to perform actions based on an event.

AutoRemote allows devices to communicate with each other without any user interaction.

N.B This article assumes some previous knowledge of Tasker and AutoRemote and so the instructions may be brief

Tasker and AutoRemote are built to work with each other and so we can use AutoRemote as a way of triggering an action in Tasker. AutoRemote is available as a plugin in Tasker.

AutoRemote acts as our "server" for the IFTTT webhook to callout to. When it receives a request, an event is triggered in Tasker which in turn sends a REST request to a local flask app that is running on the RPi. This Flask app controls the Mote LEDs.

Setting up Tasker and AutoRemote

In AutoRemote we need to register IFTTT as a device. Click the phone icon in the tab bar and then click the IFTTT logo and follow the instructions. This will give you a URL that can be added to the IFTTT webhook. It will look something like this:

https://autoremotejoaomgcd.appspot.com/sendmessage?key=[***yourKey***]&sender=[***yourId***]&message=IFTTT_MEETING_BUSY=:=true

Notice message=IFTTT_MEETING_BUSY=:=true this is how we pass properties to AutoRemote. The left side of =:= is how we can filter incoming requests and the right side are any params you want to pass in.

With this set up we now need to trigger events in Tasker. Under the profiles tab create a new one profile selecting event -> plugin -> AutoRemote -> AutoRemote. Edit the configuration and add the name of the event (the left side of the request from IFTTT e.g IFTTT*MEETING_BUSY). Next, click back and add the action. In my case, I wanted to perform an HTTP request* to my RPi but there is a multitude of things that Tasker can do for you.

Once set up your profiles will look like this

The final step is setting up the local server to trigger the Mote lights. I wrote a really simple Flask app that listens to requests and calls the correct Mote methods. It looks something like this:

import motephat
from flask import Flask, jsonify, make_response

app = Flask(__name__)

motephat.set_brightness(0.5)
motephat.configure_channel(1, 16, False)
#motephat.configure_channel(2, 16, False)
#motephat.configure_channel(3, 16, False)
#motephat.configure_channel(4, 16, False)

colour = 'FFFFFF'
status = 0

def mote_on(c):
    for channel in range(1, 5):
            for pixel in range(16):
                motephat.set_pixel(channel, pixel, 255, 0, 0)
    motephat.show()

    return True

def mote_off():
    motephat.clear()
    motephat.show()
    return True

def get_status():
    global status
    for channel in range(4):
        for pixel in range(16):
            if motephat.get_pixel(channel + 1, pixel) != (0, 0, 0):
                status = 1
    return status

@app.route('/')
def hello():
    return 'hello'

@app.route('/mote/api/v1.0/<string:st>', methods=['GET'])
def set_status(st):
    global status, colour
    if st == 'on':
        status = 1
        mote_on(colour)
    elif st == 'off':
        status = 0
        mote_off()
    elif st == 'status':
        status = get_status()
    return jsonify({'status': status, 'colour': colour})

@app.errorhandler(404)
def not_found(error):
    return make_response(jsonify({'error': 'Not found'}), 404)

if __name__ == '__main__':
    mote_off()
    app.run(debug='true', host='0.0.0.0')

Now when you are about to go into a meeting your lights should look like this

Conclusion

I hope this was of some interest. I really enjoy these little projects that have limited time and scope. Obviously, there are so many ways that this could be done and some many improvements that could be made but this was only meant to be a little bit of fun on a Sunday night.

My 2020 plan

James G. Best — Wed, 15 Jan 2020 13:06:18 +0000

The 2020 plan

At the beginning of 2019 I set about creating a list of all the things I wanted to learn. It was huge! I was kidding my self if I thoughtI was ever going to be able to do all the things I had written down.

Obviously life happened, events occurred that I could not have planned for and my long list remained largely unfinished. I think I probably "completed" about 3 of the 10 tasks I had set myself. That is not to say that I failed in 2019. Far from it. But where I directed my time had to shift. I interviewed for and got a new job, something I was not planning on and something that consumed an awful lot of my time. But it was worth the input and sacrificing some of the tasks I had set at the beginning of the year.

Whats in store for 2020

2020 is already shaping up to be a good year. I am 5 weeks into my new role, which I love. We have already recorded the first episode of Salted Bytes and I have a new plan in place.

This year, I am gonna be more realistic. I am gonna concentrate on few things in the hope that I will be able to finish. But if I don't, then that is also OK.

Salted Bytes

We are planning on stepping things up a gear in 2020. The plan is to record 2 episodes a month and to try and get more guests (if you fancy coming on the show please get in touch). I am also planning on trying to write 2 articles a month. These will be as varied as ever but will still center around the software industry.

Health

In the last year I have been using the mornings before work to swat up on my current interests. But this year I am going to use this time to exercise a little self care. It is all to easy to neglect your physical and mental health and this takes its tole.

So this year I will be exercising, doing yoga and practising a little mindfulness in the hope that an increase in mental and physical health will help me become a better engineer.

Programming

Software is a tough one. I just want to learn everything! It's all so interesting but I will be taking some learnings from last year, I think it is far too difficult to plan a years worth of learning and exploration. Stuff changes so fast and my interests and where I need to concentrate time does too. So I am going to approach it in a different way. I will only be looking 3 months ahead in the hope that this is more maintainable. I still a rough idea of what I want to learn but I a realistic in the knowledge that this will change.

Here are a few things that I will be concentrating on:

Getting better at the stuff I already know.
Application security.
Making stuff - Hardware and software.

Thats it! A start contrast from last years plan but one that I feel is more achievable and one that is less likely to make me feel like I have failed. What do you have have in store for 2020.

Thanks for reading 🙏

If there is anything I have missed, or if there is a better way to do something then please let me know.
Check out our software focused podcast - Salted Bytes

Build a Content-based recommendation engine in JS

James G. Best — Mon, 23 Dec 2019 13:09:29 +0000

Machine learning has been on my radar for a long time but I have never really knuckled down and actually started learning it. That is until recently. I am a serial learner and with nothing lined up I decided to tackle some machine learning. I set myself the task of creating a recommendation engine. We interact with these ever day, through social media and online shopping as well as so many other places. I used a simple dataset from the web that consisted on 20 images with the results from a Google Vision API request. My goal was to recommend other images from the collection when a single image is selected.

I realise that Python would probably have been a better choice of language for this task but I know Javascript very well and did not want the extra burden of having to piece together the engine in a language I am not 100% comfortable with.

According to Wikipedia, a content-based recommendation engine is:

“Recommender systems or recommendation systems (sometimes replacing “system” with a synonym such as a platform or engine) are a subclass of information filtering system that seeks to predict the ‘rating’ or ‘preference’ that user would give to an item.”

Recommendation engines are active filtering systems that personalise the information coming to a user based on information known about a user. In our case, this information is the image initially selected and the data that was returned from Google Vision.

Best the end of this article we will be able to recommend a user more images based on their initial image selection.

The pros and cons

Before we run through how. Let's talk about why. There is a reason this type of engine is so popular but there will be reasons not to use it as well.

Pros

Unlike other methodologies, content-based filtering does not need the data of other users, since the recommendations are specific to the user. This avoids the issue of cold starts where there is limited data
The model captures the specific interests of the users and so can recommend niche items that may not be popular with other users

Cons

The model can only make recommendations based on existing interests. This limits the recommendations to known interests, stopping the broadening of the user's interests
You are reliant on the accuracy of the labels
Does not take into account a user's quirks. They like something but only in a very specific circumstance.

How do content-based recommendation engines work

A content-based recommendation engine works with data that a user provides (in our case, selecting an image). Based on this data we can make suggestions to the user.

In our case, our script will progress through the following steps:

Training
- Format data into a usable state
- Calculate TF-IDF and create vectors from the formatted docs
- Calculate similar documents
Use trained data to make a recommendation based on the user's image selection.

Before we start writing our recommendation engine we need to talk about a few key concepts. Namely, how are we going to decide what data to recommend?

The concepts of Term Frequency (TF) and Inverse Document Frequency (IDF) are used to determine the relative importance of a term. With that, we can use the concept of cosine similarity to determine what to recommend. We will discuss these throughout the article.

TF is simply the frequency a word appears in a document. The IDF is the frequency of a term in a whole corpus of documents. It signifies the rarity of a word and helps to boost the score of rarer terms. TD-IDF is used because it takes into account not only the isolated term but also the term within the whole corpus of documents. This model combines how important the word is in the document (local importance), with how important the word is int the whole corpus (global importance)

Cosine similarity is a metric used to determine the similarity of documents irrespective of size. Mathematically it is measuring the cosine angle between 2 vectors. In our context, the vectors will be objects containing the term as the key and the TF-IDF as the value. The value is also referred to as the vector's magnitude.

1. Training

The first step in "training" our engine is to format the data into a structure that is usable and easy to manage. The labels data that comes back from Google Cloud Vision looks something like this:



{
  "1.jpg": [
    {
      "locations": [],
      "properties": [],
      "mid": "/m/0c9ph5",
      "locale": "",
      "description": "Flower",
      "score": 0.9955990314483643,
      "confidence": 0,
      "topicality": 0.9955990314483643,
      "boundingPoly": null
    },
    {
      "locations": [],
      "properties": [],
      "mid": "/m/04sjm",
      "locale": "",
      "description": "Flowering plant",
      "score": 0.9854584336280823,
      "confidence": 0,
      "topicality": 0.9854584336280823,
      "boundingPoly": null
    },
    [...]
  ]
}

1.a Formatting

For the purpose of this exercise, we are only concerned with the top-level key of the object (1.jpg) and the description of each of the objects in the array. But we want all of the descriptions in a single string. This will allow us to process them more easily later on.

We want the data to be in an array of objects like this:



const formattedData = [
  {
    id: '1.jpg',
    content: 'flower flowering plant plant petal geraniaceae melastome family geranium wildflower geraniales perennial plant' 
  }
]

To format our data we will run it through the following function. This will return an array of all the data we need to continue training our engine. We use Object.entries to allow us to iterate more easily. MDN states that:

The Object.entries() method returns an array of a given object's own enumerable string-keyed property [key, value] pairs...

We then loop over the array created bt Object.entries plucking the necessary properties and add them to a desc array. Finally, we join the contents of the desc array and write it to the content property. This formatted array is our corpus.



const formatData = data => {
  let formatted = [];

  for (const [key, labels] of Object.entries(data)) {
    let tmpObj = {};
    const desc = labels.map(l => {
      return l.description.toLowerCase();
    });

    tmpObj = {
      id: key,
      content: desc.join(" ")
    };

    formatted.push(tmpObj);
  }

  return formatted;
};

1.b TF-IDF and Vectors

As mentioned above the TF is just the number of times a term features in a document.

For example:



// In the data set below the TF of plant is 3
{ 
  id: '1.jpg',
  content: 'flower flowering plant plant petal geraniaceae melastome family geranium wildflower geraniales perennial plant' 
}

The IDF is slightly more complex to work out. The formula is:

In javascript this is worked out with:



var idf = Math.log((this.documents.length) / docsWithTerm );

We only need to above values (TF and IDF) so we can calculate the TF-IDF. It's simply TF multiplied by the IDF.



const tdidf = tf * idf;

The next step in our process is to calculate the TF-IDF of our documents and create a vector containing the term as the key the value (vector) as the TF-IDF. We are leaning on natural and vector-object npm packages to allow us to do this easily. tfidf.addDocument will tokenise our content property. The tfidf.listTerms method lists our new processed documents returning an array of objects containing the TD, IDF and TD-IDF. We are only concerned with the TF-IDF though.



/**
* Generates the TF-IDF of each term in the document
* Create a Vector with the term as the key and the TF-IDF as the value
* @example - example vector
* {
*   flowers: 1.2345
* }
*/
const createVectorsFromDocs = processedDocs => {
  const tfidf = new TfIdf();

  processedDocs.forEach(processedDocument => {
    tfidf.addDocument(processedDocument.content);
  });

  const documentVectors = [];

  for (let i = 0; i < processedDocs.length; i += 1) {
    const processedDocument = processedDocs[i];
    const obj = {};

    const items = tfidf.listTerms(i);

    for (let j = 0; j < items.length; j += 1) {
      const item = items[j];
      obj[item.term] = item.tfidf;
    }

    const documentVector = {
      id: processedDocument.id,
      vector: new Vector(obj)
    };

    documentVectors.push(documentVector);
  }

Now we have an array of objects containing the id of the image (1.jpg) as the id and our vector. Our next step is to calculate the similarities between the documents.

1.c Calculating similarities with Cosine similarity and the dot product

The final step in the "training" stage is to calculate the similarities between the documents. We are using the vector-object package again to calculate the cosine similarities. Once calculated we push them into an array that contains the image id and all the recommended images from the training. Finally, we sort them so that the item with the highest cosine similarity is first in the array.



/**
* Calculates the similarities between 2 vectors
* getCosineSimilarity creates the dotproduct and the 
* length of the 2 vectors to calculate the cosine 
* similarity
*/
const calcSimilarities = docVectors => {
  // number of results that you want to return.
  const MAX_SIMILAR = 20; 
  // min cosine similarity score that should be returned.
  const MIN_SCORE = 0.2;
  const data = {};

  for (let i = 0; i < docVectors.length; i += 1) {
    const documentVector = docVectors[i];
    const { id } = documentVector;

    data[id] = [];
  }

  for (let i = 0; i < docVectors.length; i += 1) {
    for (let j = 0; j < i; j += 1) {
      const idi = docVectors[i].id;
      const vi = docVectors[i].vector;
      const idj = docVectors[j].id;
      const vj = docVectors[j].vector;
      const similarity = vi.getCosineSimilarity(vj);

      if (similarity > MIN_SCORE) {
        data[idi].push({ id: idj, score: similarity });
        data[idj].push({ id: idi, score: similarity });
      }
    }
  }

  // finally sort the similar documents by descending order
  Object.keys(data).forEach(id => {
    data[id].sort((a, b) => b.score - a.score);

    if (data[id].length > MAX_SIMILAR) {
      data[id] = data[id].slice(0, MAX_SIMILAR);
    }
  });

  return data;

Under the hood, the getCosineSimilarity method is doing a number of things.

It generates the dot product, this operation takes 2 vectors and returns a single (scalar) number. It is a simple multiplication of each component in both vectors added together.



a = [1.7836, 3]
b = [4, 0.5945]

a.b = 1.7836 * 4 + 3 *0.5945 = 8.9176

With the dot product calculated we just need to reduce the vector values of each document down to scalar values. This is done by taking the square root of each value multiplied by itself added together. The getLength method below is doing this calculation.



const getLength = () => {
  let l = 0;

  this.getComponents().forEach(k => {
    l += this.vector[k] * this.vector[k];
  });

  return Math.sqrt(l);
}

The actual cosine similarity formula looks like this:

and in javascript looks like this:



const getCosineSimilarity = (vector) => {
  return this.getDotProduct(vector) / (this.getLength() * vector.getLength());
}

The training is complete!!

2. Getting our recommendations

Now we have completed the training phase we can simply request the recommended images from the training data.



const getSimilarDocuments = (id, trainedData) => {
  let similarDocuments = trainedData[id];

  if (similarDocuments === undefined) {
    return [];
  }

  return similarDocuments;
};

This will return an array of objects containing the recommended images and their cosine similarity score.



// e.g

[ { id: '14.jpg', score: 0.341705472305971 },

  { id: '9.jpg', score: 0.3092133517794513 },

  { id: '1.jpg', score: 0.3075994367748345 } ]

Wrap up

I hope you were able to follow along. I learned so much from this exercise and it has really piqued my interest in machine learning.

A simple Elm, SCSS toolchain

James G. Best — Mon, 28 Oct 2019 08:51:30 +0000

Recently I have been knee deep in Typescript, React and Redux code. It is what I write every day at work and I love it! Typescript feels like one of those love hate things. You hate it until you try it and then you feel like writing Javascript without types is modern day savagery.

One morning during my daily morning scroll through Twitter I found this post. It talks about how if you use the stack I mentioned above (Typescript, React and Redux) that you would love Elm. I had looked at Elm before and although it interested me it never captured me enough to stick it out but this time I thought it might be different. This post is not about my journey into Elm as it is still in its infancy but about how I set up a simple project with Elm, SCSS and hot-reloading.

This post is not about my journey into Elm as it is still in its infancy. I am gonna run through how I set up a simple project with Elm, SCSS and hot-reloading. I realise there are lots of boilerplates out there but many of them just do too much. I want to learn in a simple understandable environment but with the perks of using something like CRA (create-react-app).

I required my starter package to do a few things.

Compile Elm
Compile SCSS
Create a dev server and hot reload I also wanted to stay away from the usual bundlers that we use in the JS world, like Webpack and Parcel. These tools are great but add a load of packages.

The project folder structure is as follows:

This is obviously personal taste, and this is how I like to set things up. SCSS folder structure is also personal taste but I use this as a starting point.

Most of the action happens in the package.json. We run and build our project with a number of npm scripts.

Elm

Watching Elm

The first task we are gonna cover is the compilation of our Elm code. We also need the compiler to be able to watch for changes and recompile. This is standard in JS projects these days and so why should it be any different in Elm. We are going to be using a file watcher called chokidar-cli. This is the same file watcher used in many of the most popular bundlers and task runners. First, let’s install it:

 # npm
 npm install — save-dev chokidar-cli

 # yarn
 yarn add -D chokidar-cli

and then add the following to the scripts section of the package.json

"scripts": {
   […]
   "watch:elm": "chokidar './src/**/*.elm' -c 'elm make ./src/Main.elm — 
     output ./public/js/elm.compiled.js' — initial"
   […]
 }

Here we are using chokidar to watch for changes to any file matching this regex ./src/*/.elm and then running the Elm make command elm make ./src/Main.elm — output ./public/js/elm.compiled.js

Building Elm

While we are dealing with Elm, let’s write the script to build the Elm project. This is very similar to the script above except we won’t use chokidar as we are not watching for changes. We are also adding the — optimize flag so that the js output is minified and optimised.

"scripts": {
   […]
   "build-elm": "elm make ./src/Main.elm — output 
     ./public/js/elm.compiled.js — optimize",
  […]
 }

SCSS

Watching and building SCSS

Next, we are gonna tackle watching and building our scss files. When watching we want the scss to recompile when changes are made. We will be using node-sass-chokidar for this. Let’s install the package:

 # npm
 npm install — save-dev node-sass-chokidar

 # yarn
 yarn add -D node-sass-chokidar

Dev server and hot reloading

This would not be a serious modern frontend toolchain without a dev server and hot reloading. For this we are gonna use browser-sync. Let’s install it.

 # npm
 npm install -g browser-sync

 # yarn
 yarn global add browser-sync

Let’s add another script. This once will start a server and watch all files in our public folder. Because the other watch scripts all compile into the public folder anytime we make any changes browser-sync will detect them and restart the server and refresh the browser.

"scripts": {
   […]
   "dev-server": "browser-sync start — server 'public' — files 
     'public/**/*.*'"
   […]
 }

Tying it all together.

Let’s tie it all together with a build and start script. For these, we want to be able to run multiple npm scripts at once. For this we will use one last package — npm-run-all. This allows you to run scripts sequentially or in parallel.

 # npm
 npm install — save-dev npm-run-all

 # yarn
 yarn add -D npm-run-all

With this installed we can write 2 final scripts. Let’s tackle the build script. We want to be able to compile both the scss and elm in a single command. All we do is pass the names on the scripts and we want them to run in parallel so we use the -p flag.

"scripts": {
   […]
   "build": "npm-run-all -p build-css build-elm "
   […]
 }

The last script is the one we will use the most. This starts our dev server and compiles (with file watching) both the Elm and SCSS.

"scripts": {
   […]
   "start": "npm-run-all -p watch-css watch:elm dev-server"
   […]
 }

And that is it. The complete scripts section should look like this:

{
  "scripts": {
   "start": "npm-run-all -p watch-css watch:elm dev-server",
   "watch:elm": "chokidar './src/**/*.elm' -c 'elm make ./src/Main.elm -  
     output ./public/js/elm.compiled.js' - initial",
   "build-css": "node-sass-chokidar sass/ -o public/css",
   "watch-css": "npm run build-css && node-sass-chokidar sass/ -o 
     public/css - watch - recursive",
   "build-elm": "elm make ./src/Main.elm - output 
     ./public/js/elm.compiled.js - optimize",
   "build": "npm-run-all -p build-css build-elm ",
   "dev-server": "browser-sync start - server 'public' - files 
     'public/**/*.*'"
   }
}

I am still really new to Elm development and I am sure there are better ways to build your projects but I wanted something simple that I could easily understand. I hope this helps in some way.

Running a python script on boot on a RPi

James G. Best — Mon, 14 Oct 2019 09:44:21 +0000

Once you are able to successfully run your python script from the terminal, what are your options to run the same script on booting the Raspberry Pi? This post details the issues I had getting my scripts to run on boot and how I fixed them.

TL;DR Make sure you are running your scripts as the correct user. Python scripts that use Pip packages will need to be run as the user used to install said packages or else they might not be available.

Writing python scripts is a relatively new thing for me. I love playing with the Raspberry Pi and generally write Nodejs. For most of the stuff I have been doing up until this point, Nodejs has been fine. But the Raspberry Pi is blessed with a wealth of hats performing a multitude of weird and wonderful things. The problem is that most of the libraries for these hats are written in Python. So I thought it was about time I start learning some python.

My first python script was controlling the Mote LEDs through MQTT. Writing the script did not take too long as the examples in the Pimoroni repos gave me enough to get started. I was able to run the script from the terminal and had it working fine. Obviously, we don't want to have a terminal session open all the time and we want to have the script start on boot (or reboot).

I had read that there are a number of ways to run a script on boot but that the preferred way was to add an entry into the /etc/rc.local file. This file is loaded when the Pi is booted and so should achieve our goal.

Have you ever seen the IP of the Pi print in the terminal output when the Pi is booting? This is where that command is run from.

So I added the following line to the file:

/usr/bin/python3 /home/pi/Code/desklights/desklights.py

and rebooted the Pi. Nothing! No logs, no error. Nothing! Very unhelpful. I tried every possible way of calling the script. And each one proved fruitless. Nothing.

What I needed was to see so sort of log output. I hoped that this would give me an idea of what was wrong. I updated the command to the following:

/usr/bin/python3 /home/pi/Code/desklights/desklights.py > /tmp/desklights.out > /tmp/desklights.err

This would write anything logging from the program to /tmp/desklights.out and and errors to /tmp/desklights/err. This was a key to solving the issue. The .out file was empty which indicated the script was failing. The `.err` file, on the other hand, contained this:

It was now clear what I needed to do. The python packages I had installed were install using the Pi user. This file runs as root and so does not have access to the installed packages. I need to make sure I am running my script as the Pi user and all should be fine and dandy.

sudo -H -u pi /usr/bin/python3 /home/pi/Code/desklights/desklights.py/home/pi/Code/desklights/desklights.py

Updating the script to include the change of user fixed my issue and allowed the script to run successfully on boot.

Thanks for reading 🙏

Giving my first technical talk, the good, the bad and the ugly

James G. Best — Thu, 10 Oct 2019 09:52:43 +0000

Giving my first technical talk, the good, the bad and the ugly

I recently had the opportunity to give a talk at a local meetup, South West Mobile. I have always feared public speaking, especially to my peers and so set myself a goal this year to try it out. I am discuss the wild mental journey I went through. I appreciate that speaking at a meetup to 45 people is not a big deal some but to me it was huge.

It all started with CFP (Call for papers) that I had forgotten I had submitted. Once the confusion was cleared up and I had remembered that I had submitted it, I was confirmed. I began planning the talk. The title was "GraphQL in React Native with AWS AppSync".

A quick rundown of tech

React Native is a technology we use a great deal at Gravitywell. The speed of development it allows is fundamental to producing the ever-important MVP or prototype. It allows us to iterate quickly and easily. But then when we are ready to push the full version it is able to scale and provides us with all the functionality we need to produce enterprise-grade mobile applications. In one of our recent, I give a quick (~2 mins) overview of what React Native is:

We really appreciate the benefits of producing an MVP and getting your product on the market as quickly as possible. If you are interested in learning more about MVPs then check out some of our other articles:

GraphQL is another favourite of ours. GraphQL is a query language that allows you to describe the data we want. I discuss further in the following video:

The final piece from the title is AppSync. AppSync is a managed service that uses GraphQL to make it easy for applications to get exactly the data they need, give the ability to pull data from multiple data sources in a secure manner.

Thats a brief overview of what I discussed in the talk but this article is about the process and my mindset.

Preparation

To prepare for the talk, I started with an app, that I would use to help show how great the React Native, GraphQL combo is. I was not concerned that the example was slightly contrived as it was perfect for showing how they work together without confusing things with a complicated app.

The demo app code can be found here

I then created the slides. Initially I started this with MDX Deck but this proved to time consuming and I was not able to get the hot reloading working correctly so I resorted back to Google slides. This was definitely a good choice.

I started with setting out the basic structure and then added content where needed. I had heard that you should have roughly 1 slide a minute. The talk was meant to be about 20 mins and I was doing a live demo so I settled on 15 slides.

The big day - ABSOLUTE TERROR

The day arrived and although I knew I was prepared, I was terrified. My stomach was in knots, I could not eat and I struggled to think straight! But before I knew it, it was 7pm, I was set up and ready to go. I was second on the bill. The first talk was on accessibility, it was very polished and the speaker had ~50 slides. This immediately made me nervous. I only had 15. Was my talk gonna be 3 mins long? Was I gonna get laughed out of the room?

Then my turn came.

Thankfully, I was not laughed at. The talk got a few laughs and I did not pass out. My live coding demo did break, right at the last moment (I later found it was due to miss spelling a GraphQL type) but that was fine. On the whole it felt like it went ok but I had already decided in my head that I was not gonna do another. I had found the whole experience really stressful and time consuming. With my mind made up, I went home happy that I had done it.

But the next day, the recording of the talks was released. I tentatively watch my session and I was pleasantly surprised, it was much better than I was expecting. On the outside I was as calm as a Hindu cow but I knew that on the inside I was screaming! But this gave me the strange feeling that I should do another. My wife said the same thing! She was trying to get me to sign up to do another one straight away.

Now I am confused, should I do another? Will it be easier? Will it be less terrifying?

All of these can be answered with "probably" and I wont find out until I try again. But that is where I have left it. I am still deciding if I want to put myself through it again even though it will probably be easier!

Watch this space.

A few takeaways

If like me, you are scared of public speaking. Start at a meetup with a topic you know really well. It should make it a little easier. I was really lucky that SouthWest mobile were a really lovely bunch of humans.
If you are not first, try not to worry about what the previous speaker is doing. Generally your talks will be very different so it should not really matter.
If you do decide to do one and it bombs, try not to worry about it, don't listen to people if they are being critical. Stuff like this happens all the time. At least you stepped up and gave it a go. There is a fantastic quote from Teddy Roosevelt that puts this better than I could

It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly...

Here is the recording and slides from my talk:

Slides