Forem: Dan Garland

Come out of your Shell and learn Bash

Dan Garland — Tue, 30 Oct 2018 00:00:00 +0000

One thing I noticed whenever I'd kick off a new cohort at We Got Coders, was that despite all my efforts, often students would arrive without knowing much about how to use a shell. The shinier the laptop, the greater the reluctance to let go of the GUI and get their hands on the keyboard.

I'd put together a cheat sheet. Yet I can't think of many other technologies more likely to sap the enthusiasm of a new coder than Bash (maybe SQL?). It would make a boring first day: let's learn how to make a directory and then remove it again! Especially when ultimately the goal would be to learn web development, or get a job as a programmer; why go over these basic command line commands?

There are some concepts that are assumed when you learn programming. Take the filesystem. It's not obvious to new starters what a path is, how a relative path works or how to navigate around the concept of a tree. It's not hard - but it's enough to deter newcomers who will wrongly believe they are up against 'clever' people, and it's enough to distract from an actual high-level concept we might cover in class.

Trouble is, the longer we'd leave it, the more of a stumbling block the shell would become. That first couple of weeks, where I'm bombarding them with new concepts about Ruby and Object-orientated programming, they'd be wrestling with their operating systems, using a text editor to rename files and create directories, or the finder to delete folders and lose their work. Everything was slower, and all of that extra effort was taking away from learning to code.

While there is a lot to learn about bash programming, unless you're planning on being a Debian package maintainer, I found over the last fifteen years that I'd stick to a relatively small subset of Bash commands, and the odd bit of scripting on top. I never listened to those proponents of other shells, and how zsh would change my life and so on. I felt like I had learnt just enough, and that enabled me to be much better at my job than if I never bothered to learn at all.

In the end, many of us learn shell programming as a subtask of something else, a necessary obstacle to be overcome. Once we've learnt Ruby, JavaScript, Go or whatever we're focussing on, we're not going to spend much time on our bash scripting. But for automating repetitive tasks, converting files, creating initial project directory structures, navigating the filesystem and finding where we left our code, knowing the basics around shell is a vital skill for any serious programmer.

I decided to put together a short lesson for beginners, that covers some basic Bash concepts. I tried to focus on that subset of knowledge that I'd come to rely on over the years, rather than regurgitating a bash bible. I wanted to make a essential guide - something that would underpin any online course, bootcamp or training program. The content is free to use and download - and you can request my help if you get stuck on any of it.

To give you a flavour, here is one part of the exercises that we'll work on during the course.

Generating a HTML project skeleton in Bash

One of the nice things about Bash is that it doesn't require any other dependencies to run, it usually comes with a UNIX machine. Therefore it's a good choice for code that creates a new project, as we can run it straight away without any further setup.

Suppose we wanted to achieve a directory structure ready for web development. We might want to keep separate folders for our different assets, and a starter HTML file. To begin with, I'll aim for something like this:

site/
├── css
├── images
├── index.html
└── javascripts

Using the commands we had before, we could easily achieve this using mkdir and touch. You can try any of these snippets by just dropping them in your shell, or you might want to create a script (For this example I'll assume you've created a new_html_project script and made it executable).

mkdir -p site/images
mkdir -p site/javascripts
mkdir -p site/css
touch site/index.html

But it is a bit repetitive. Once our file list gets longer, we might want to avoid writing the mkdir command many times. This is where the for loop comes in. Any code inside the loop gets repeated, so that the code can be run many times with different inputs. Let's roll these up into a loop and see if it saves any repetition:

for folder in "site/css" "site/images" "site/javascripts"
do
  mkdir -p $folder
done

Here we're using a bash variable (the bit with the dollar sign before it) to store the values of "site/css", "site/images", "site/javascripts" consecutively, and using value that as the argument to mkdir.

As for the index page, we need to use a different approach. You might use touch to create a blank file, or even echo to add some content using redirection:

echo "<h1>My amazing webpage</h1>" > index.html

Since we generally need a bit of content, we could consider using a here document to build a larger string, and using that as the input to cat, before redirecting the output into the index file. A here document uses an arbitrary delimiter (here HTML) to denote the start (<<HTML) and end (HTML on it's own line) of the string.

cat > site/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML

So far we've got a script that looks like this:

for folder in "site/css" "site/images" "site/javascripts"
do
  mkdir -p $folder
done
cat > site/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML

So far we've hit on a nice coincidence, that the commands we've chosen work even on subsequent uses - that is, even if we ran the script twice. However, we probably don't want to keep running the command to set the index.html, otherwise there is a chance that we'd lose our work if we re-ran the script. Fortunately, this is a good use case for the if statement, to check whether the file already exists before creating it. Let's update our script to use if statements, using the -f flag to determine whether a file exists.

for folder in "site/css" "site/images" "site/javascripts"
do
  mkdir -p $folder
done
if [ ! -f site/index.html ]
then cat > site/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML
fi

Finally, we're assuming that we always want to create a site folder. The script would be considerably more useful if we could parameterise it, to allow us to name to subfolder in question. Bash scripts do accept arguments, which are space delimited on the command line. Each argument is delimited by the dollar sign, and a number index of which argument in the list you want to use. If you try typing this into your shell, you'll see that $1 becomes hello, and $2 becomes there:

echo $1 $2 hello there

The arguments are zero-indexed, which means there is also an argument $0, which represents the program you called, in this case echo.

We might therefore use this to make our site folder more dynamic:

#!/bin/bash
for folder in "$1/css" "$1/images" "$1/javascripts"
do
  mkdir -p $folder
done
if [ ! -f $1/index.html ]
then cat > $1/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML
fi

We can now generate multiple projects using ./new_html_project my_amazing_webpage, replacing the argument for each folder. But there's a bug! If we try to call this program, but forgot to pass in an argument, the value of $1 will be blank, leaving a shell script that notionally looks like this:

#!/bin/bash
for folder in "/css" "/images" "/javascripts"
do
  mkdir -p $folder
done
if [ ! -f /index.html ]
then cat > /index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>
  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML
fi

You may have spotted that without our dynamic site folder name, we're now referencing paths that start with /, which means the root of the entire filesystem. This folder is protected from ordinary users (partly for this reason!), so unless you had root privileges, you will probably see permission errors.

To fix this, we can use the $# operator to provide the number of arguments given, and display a message if we got it wrong. Given that the program name counts as an argument, we are looking for exactly two as the result.

#!/bin/bash
if [ ! $# == 2 ]; then
  echo "Usage: $0 <folder_name>"
  exit
fi

for folder in "$1/css" "$1/images" "$1/javascripts"
do
  mkdir -p $folder
done

if [ ! -f $1/index.html ]
then cat > $1/index.html <<HTML
<!DOCTYPE HTML>
<html>
  <head>
    <title>My Amazing HTML document</title>
  </head>

  <body>
    <h1>My Amazing HTML document</h1>
  </body>
</html>
HTML
fi

I reckon this example contains enough concepts for further study in other languages. In the course we look at some more examples of basic bash commands, and I leave you an exercise to build a directory structure to support a bootcamp syllabus.

I'd welcome any feedback you have, beginner or pro, to develop this course further. If there is enthusiasm for it, I could go further and explore more concepts, like how to list and kill processes, some administration, background tasks, networking and so on; but for now I think it's good enough for anyone looking to take a training course or bootcamp to help them arrive prepared and raring to go. The sign up link is here.

Learn To Code The Jedi Way

Dan Garland — Tue, 23 Oct 2018 17:58:39 +0000

TLDR: If you’re going to be a Jedi, you need to know how to construct your own lightsaber. If you’re going to learn to code, you’ll need to know how to setup your programming environment.

When a recent change to an API we relied on for providing in-browser exercises dropped support for creating user accounts, I was left without a seamless way for providing you with programming exercises. Next to the plush, effortless and glossy e-learning websites out there, what chance would I have to promote my own on-line courses?

I was worried that without interactive repls, I’d lose some of the new members who’d sign up for the site. But apart from being very easy to get started, coding within an iframe running in a web browser is actually not much fun. Even the best repls don’t stand up next to a good text editor, and no doubt you've experienced the frustration of a crashing exercise, losing our work.

The deceptive simplicity of many interactive coding exercsies tends to be for very beginner, very basic tasks - nothing like the real world. In providing something immediate and visually appealing, all too often interactive repls over simplify, teaching you how to type; rather than how to code.

“If you choose the quick and easy path, as Vader did - you will become an agent of evil”

Jedi Master Yoda

Truth is, unless you work with the same dev tools that developers use, you are not learning how to become a developer. No amount of interactive exercises will convince an employer to give you a job, if you don’t know your command line shell commands, some basic UNIX administration and how to install software on your machine.

It’s sometimes painful to watch novice developers fumble around with a text editor. I’m used to it – I’ve been teaching for five years. But there’s nothing that says ‘newbie’ more than holding down the → key for 10 seconds to reach the end of a file, where an easy keyboard shortcut exists. That’s not to say that you have to be some Vi ninja before you apply for your first job; just that being spoonfed exercises on-line robs us of the chance to get our hands dirty with our programming environment.

Now that we’ve chosen coding as our career, there is also something deeply personal about our programming environment. The operating system, text editor, web browser, even our shell prompt says much about us as programmers, and it needs to be a comfortable and safe space for us.

A New Hope

After some mad refactoring and a lot of work, I migrated onto our existing git provider, and brought the exercises in line with how I was delivering the assignments. In the end I found that many ways my lessons are now simpler: everything is delivered via git, encouraging you to learn some basic git concepts and commands, and becoming accustomed to the dev tools you will be using in your next programming job.

Learning to run the test suite before you commit your work, learning to make small, regular commits, rather that committing everything at the end; these apparently innocuous things make a huge difference to the ease with which you can be absorbed into a development team, and they are difficult to teach, becuase they're really a habit that you have to practice more than a concept.

The way my assignments are structured on my courses is the same approach that I’d take to any client or open source work. I create a git repository, with a README that explains how to get going. In the case of an exercise, the README is a walkthrough guide that you can follow, making the changes as you go. There are also assignments that are more open-ended and challenging, allowing you to tackle a more protracted task, knowing that is achievable, whilst giving you some room for creativity.

Each exercise has some sample code to get you started, and a test suite to check that you got the result that I was expecting. In this way, you’re also implicitly practicing the git workflow used by all the best software companies in the world. Your journey would therefore go something like this:

Fork the repository - This obtains your own version of the exercise, so you can make edits without affecting anyone else.
Clone the repository - This brings down a copy of the exercise to your local machine.
Open a terminal in the cloned directory – This is where a bit of shell programming comes in handy, using cd to navigate your filesystem to the right place.
Reading the instructions – perhaps with less to paginate through the README. How many developers skip this step ;)
Make an edit to the code – in your favourite editor (I recommend Sublime text for beginners)
Run the test suite – making sure that your recent changes made something pass. Otherwise, what was the point of adding it?
Save your progress - git add and git commit to selectively add the files that changed, and logging your progress as you went along. Why did you need to make the changes you did? Writing good commits and committing often is a great habit to get into!
Repeat steps 5-7 - until you’ve completed the exercise
Upload your work for review – with git push
Reward yourself - with a tasty snack or a piece of fruit

While I accept that for someone entirely uninitiated, this process requires quite a bit of going sideways before someone can focus on the actual coding, I would argue that this will make you a better coder in the long run.

Learning to deal with this frustration is also part of the daily routine for developers. Not being phased when task A depends on task B which depends on task C, but learning to set realistic expectations of yourself is a big part of developing a good coding temperament. Just like Luke Skywalker, we must learn to deal with our anger if we're to avoid turning to the Dark Side.

A padawan must learn how to construct a lightsaber, as part of their training to become a Jedi. In the same way, learning how to setup your environment is an essential part of becoming a coding Jedi – and if you’re up for the challenge, that’s exactly how I approach the learning on this course.

Ruby Unconference 2018 Protobufs, gRPC and Ruby

Dan Garland — Tue, 09 Oct 2018 18:32:36 +0000

One of my favourite Ruby meetups is the London Ruby Unconference, which took place last Saturday, just round the corner from Buckingham Palace.

Unlike conferences where the audience is mostly passive, and speakers are representing big-name tech companies, stealthily (or openly) promoting something, an 'unconference' tries to do the opposite: encouraging participation from its attendants.

A real-world demonstration of anarchy in action, the conference schedule is planned on the morning of the event. Anyone can propose a session, and the audience votes with their feet. In this way, even a rough idea for a session can make it onto the agenda.

What I like about this is that I might take an idea of something that I'd like to explore in Ruby, float it as a session and spend time with like minded people working it out.

Enter Protobuf

I'd recently been tasked in a FinTech project with learning about Google's protobuf definitions language for describing binary formats in web services. In high-volume, real-time trading, time is of the essence. Algorithmic trading relies on micro changes in data in order to make decisions, and every nano second counts towards your competitive advantage.

Coming from a Ruby and Rails background, I thought we'd slayed the RPC / SOAP / XML dragon long ago. HTTPS and JSON formats are prevalent all over the web, with RESTful conventions making designing APIs straightforward and predictable. However, when speed is key, protobufs are well worth a look. I wanted to know whether the tooling and workflow would stand up to the standards set by Rails, Padrino and Sinatra, or whether it would be a hark back to my Enterprise JavaBean days.

Although I'd worked with Go on the FinTech project, one of the nice things about protobufs is that by defining your API endpoints and the messages they deliver in one standard syntax, it is possible to have clients and services to be written in various languages, and still be interoperable with each other.

Therefore, I decided to pitch a session on getting started with gRPC and protobufs in Ruby, and see how it compared with the RESTful approach I was used to with Rails.

Getting Started

Since I was attending an event, I thought I'd use the example of attending events and demo a calendar application, using grpc to handle the backend code. With a time limit of around 45 minutes, I thought I could just about demo adding an event to a calendar, and maybe show how we could retrieve them too.

All of the code in this example is on my GitHub

Initially, I created a git repo and some folders to separate my proto definitions from the Ruby code.

  
git init calendar_demo
cd calendar_demo
mkdir proto
mkdir lib

To begin fleshing out what our calendar web service can do, I started with a protobuf definition file, a separate API language definition that defines what endpoints the API will support, and what messages they will communicate to each other. This differs from the RESTful approach, where we might start with a resource representing an Event, and expecting API endpoints to match up by convention.

The first draft of the protobuf looked like this:

  
syntax = 'proto3';

service Calendar {
  rpc AddEvent(Event) returns (EventAdded) {}
}

message Event {
  string name = 1;
  string date = 2;
}

message EventAdded {
  bool added = 1;
}

Here, we're defining a web service called Calendar, which represents the entire API. Within that, we can have a number of endpoints, which perform separate tasks or services. It's somewhat similar to a class and it's methods; except the method calls are triggered by the underlying protobuf binary messages, over a transport layer (more on that later).

Our AddEvent endpoint needs to know information about the event we are adding. We define this as a message, which I've named Event. Each message consists of a number of fields, each with basic data types, such as string, floats, integers; types that can be serialised (converted into binary), and deserialised at the other end. To keep things simple, I'm just considering an event as storing a name (what we're doing) and a date (when we're doing it).

Note that each field on the message is numbered in ascending order. This is to maintain backwards compatibility with other clients / services that use our service, but might not have the latest codebase, rather than changing the format of the message. In this way, we can add new fields with higher numbers, or change their types, without confusing other clients and services.

As RESTful developers, we might expect such an endpoint to simply acknowledge that the event was added successfully, perhaps with a HTTP status code of 200 or 201. With protobufs, we need to explicitly define the response too, and so here I define another message EventAdded, which encapsulates a boolean value which indicates whether the event was added or not. In practice, we might wish to include more information here, for instance whether the event was added immediately or queued for processing later.

So with this rather concise definition, we've defined a request / response for an endpoint that is designed to add an event to our calendar.

Generating Clients and Servers

Whilst the protobuf definition is short and readable, it won't do very much for us until we put it into action. That means we need to create the code that implements this definition, taking care of serialisation / deserialisation, sending / receiving messages, and routing to the right endpoint. Fortunately, most of this hard work is done for us, using the grpc toolset, leaving us to focus just on our business logic and think in terms of events and calendars.

Using the grpc-tools gem, we can side-step the setup and install of the protoc compiler and plugins, and start from a pre-made binary. I added it to my Gemfile, along with the grpc dependency.

  
source "https://www.rubygems.org"

gem 'grpc'
gem 'grpc-tools'

I could then generate the code in a couple of commands:

  
bundle
grpc_tools_ruby_protoc -Iproto calendar.proto --ruby_out=lib --grpc_out=lib

This second command asks the bundler-managed, protoc compiler to generate the client and server stubs from our API definition, looking for the protobuf file in the proto folder, and to output the resulting code in the lib folder.

If you sift through the generated code, you will see that it generated:

calendar_services.pb, containing a module named after our service, registering our AddEvent endpoint
calendar_pb.rb, registering the events that we had defined.

Defining the Server

Ultimately, we can't have the grpc toolset define everything for us; otherwise we'd be out of a job! We need to define what happens when the service receives the AddEvent message, using the generated service as a template. I stored the following code in lib/calendar_server.rb:

  
require 'grpc'
require 'calendar_services_pb'
require 'calendar_pb'

module Calendar
  class Server < Service
    def add_event(event, _call)
      # TODO - Business logic!
    end
  end
end

I used good old fashioned inheritance to subclass the Service, inheriting all the methods and functionality generated by the toolset. We are then expected to provide methods for each endpoint that we've defined, so I wrote an add_event method. The method is passed an Event object, which encapsulates our name and date fields, and a _call, which encapsulates the request.

Whilst this is a great start, we have not yet got anything that implements our service, or anything to define the transport layer - that is, how messages are sent and received by the service. The below snippet is recommended from the grpc tutorial as a way of getting started; in practice this might be called elsewhere but for now I'll add it to the server class.

  
require 'grpc'
require 'calendar_services_pb'
require 'calendar_pb'

module Calendar
  class Server < Service
    def add_event(event, _call)
      # TODO - Business logic!
    end
  end
end

addr = "0.0.0.0:8080"
s = GRPC::RpcServer.new
s.add_http2_port(addr, :this_port_is_insecure)
puts("... running insecurely on #{addr}")
s.handle(Calendar::Server.new)
s.run_till_terminated

This is just enough code to run the server, but it would not do very much; any connecting clients would hang forever, because our add_event method does not return anything. Let's add a little business logic and naively store our Event in an array:

  
require 'grpc'
require 'calendar_services_pb'
require 'calendar_pb'

module Calendar
  class Server < Service
    def initialize
      @events = []
    end

    def add_event(event, _call)
      @events << event

      return EventAdded.new(added: true)
    end
  end
end

addr = "0.0.0.0:8080"
s = GRPC::RpcServer.new
s.add_http2_port(addr, :this_port_is_insecure)
puts("... running insecurely on #{addr}")
s.handle(CalendarServer.new)
s.run_till_terminated

Now that we've got our first draft of the server, you might consider splitting your terminal in half and running the code with:

ruby -Ilib lib/calendar_server.rb

You should see the caveat that the service is running, albeit insecurely. It is sitting in an infinite loop waiting for your requests to arrive!

Sending Client Requests

One drawback of protobufs from a developer's perspective, is that as a low-level, binary protocol, we can't easily test out our API using something like curl. Instead we need to use the client stub provided to make a Ruby client. In practice, this code would also likely be embedded within a larger application, but for now it will suffice just to make a calendar_client.rb in the lib folder with the following code:

  
require 'grpc'
require 'calendar_services_pb'
require 'calendar_pb'

stub = Calendar::Stub.new('localhost:8080', :this_channel_is_insecure)
event = Event.new(name: 'Ruby Unconference', date: '2018-10-06')
response = stub.add_event(event)
puts response.added

Here, we're instantiating a suitable client that knows how to 'speak' to our endpoint, and passing it a message. The client handles the response, and gives us an object which provides a getter for the added field, so we can see whether the event was added or not.

ruby -Ilib lib/calendar_client.rb

Streaming Events

One of the nice features of protobufs is that it has support for streaming APIs, which is great for handling lots of data gradually, rather than building huge responses and leaving the client waiting around.

To illustrate this, I demonstrated how you might implement recurring monthly events in a new API endpoint called ShowEvents. This endpoint would stream events one at a time, generating recurring events if necessary from an initial event.

I began by extending our protobuf definition with a new endpoint. All we need to do to make this endpoint a streaming API is to add the stream directive before the return value. I added an EventRange message that specified a range of dates to select from, and I also updated the Event message to include a recurring field.

  
syntax = 'proto3';

service Calendar {
  rpc AddEvent(Event) returns (EventAdded) {}
  rpc ShowEvents(EventRange) returns (stream Event) {}
}

message Event {
  string name = 1;
  string date = 2;
  bool recurring = 3;
}

message EventAdded {
  bool added = 1;
}

message EventRange {
  string from = 1;
  string to = 2;
}

Since the API definition has changed, we'd need to update the generated code as before. This gives rise to an important consideration, documented in the generated code, that we must not edit the generated files by hand; otherwise those changes would be overwritten the next time we (or our teammates) update the protobuf definition file.

After running the generator, we only need to add a new method to our server (lib/calendar_server.rb) to support the new endpoint (I'll leave out the rest of the file for brevity). The Ruby grpc implementation makes good use of the Enumerator class here, allowing us to define our stream as an Enumerator - i.e. we define the sequence, and the underlying code will call each repeatedly until it runs out. Thus we can trivially return each of the events in our array in a stream:

  
def show_events(search, _call)
  @events.each
end

Using the build-in select method, we can easily add the date range filtering, using the dates supplied in the EventSearch message:

  
def show_events(range, _call)
  from = Date.parse(range.from)
  to = Date.parse(range.to)

  @events.select {|event| Date.parse(event.date) <= to && Date.parse(event.date) >= from }.each
end

Since Ruby is weakly typed, there is quite a bit of type conversion going on, between String and Date, which is making the code a bit repetitive. Although Ruby is ambivalent about a lot of these considerations, when it comes to API design, we'll be working with strongly typed code more often.

I found that a good approach for avoiding this casting was to open the Event class provided, and add a getter that provides the Ruby Date equivalent of the underlying data. This allows us to keep using strings for the underlying serialisation, but providing dates for use in our logic

  
class Event
  def actual_date
    @actual_date ||= Date.parse date
  end
end

def show_events(range, _call)
  from = Date.parse(range.from)
  to = Date.parse(range.to)

  @events.select {|event| event.actual_date <= to && event.actual_date >= from }.each
end

Using a somewhat obscure, but very powerful feature of the Enumerator class, we can also pass a block to the initialize method of the Enumerator class, and define what each iteration's yielded value is. This is useful for instances when we want to define our own custom sequences.

Using this approach, I tested whether an event was recurring or not, and if so, would generate new dates one month ahead of the original. Sticking this code in a while loop meant I could generate an arbitrary number of events in my stream:

  
def show_events(range, _call)
  from = Date.parse(range.from)
  to = Date.parse(range.to)

  Enumerator.new do |y|
    @events.select {|event| event.actual_date <= to && event.actual_date >= from }.each do |event|
      y << event
      while event.recurring && event.actual_date < to
        # assume monthly its only a demo :)
        event = Event.new(name: event.name, date: (event.actual_date >> 1).to_s, recurring: true)
        y << event
      end
    end
  end
end

What's more, the client-side code is provided the stream in the form of a Ruby Enumerable, making it trivial to iterate over the streamed responses:

  
range = EventRange.new(from: '2018-01-1', to: '2018-12-31')
events = stub.show_events(range)
events.each do |event|
  puts "#{event.name} is happening on #{event.date}"
end

How fast?

So we can see that using the grpc approach is pretty straightforward to work with, but how fast is it? Part of the problem with our standard HTTP/JSON APIs is the same problem that HTTP/JSON itself was proposing to solve when compared to XML/SOAP web services: serialisation time. The amount of extra bandwidth, both in terms of networking, and computing power to serialise and deserialise data, whether HTTP headers or JSON structures, is an overhead that must be taken into account when designing an API.

Whilst JSON is leaps and bounds simpler and faster than XML, I did a quick benchmark to see how our calendar app compared against an API written in Sinatra. Note that both are using HTTP transport, so really that is just comparing the JSON overhead and the internals of both systems. Below are the results for sending and receiving 1,000,000 requests to add events to our Ruby array:

Time to process 1 million requests (seconds)

| user | system | total | real | reqs / sec |
| 113.786016 | 27.136001 | 140.922017 | (342.784571) | 7092 |
| 300.176772 | 119.946387 | 420.123159 | (1050.995664) | 238 |

That's 3 times faster with protobufs! Powering away with over 7,000 requests per second, grpc shortcuts the JSON serialisation and takes the prize.

What I learned

GRPC isn't a throwback to the WSDL, SOAPy hell we experienced in the 90's.
It's pretty easy to work with and define new services
It's interoperable and a good way to break down Rails apps into microservices written in other backends like Go
It's fast - 3x faster than a JSON API
It's fun to meet new people at hackdays! :)

Thanks

Most of the information in this post was inspired by the excellent tutorial on grpc.io, which also has good links and documentation references on protobufs.

Had it not been for London Ruby Unconference, I might not have spent an hour or so hacking away on this demo, so thanks to Jairo and CodeScrum for putting on the event.

If any of this doesn't make sense to you, you need to take a look at my Ruby course!

Learn Coding to Break Your Programming

Dan Garland — Wed, 03 Oct 2018 13:12:03 +0000

Kryten is my favourite Red Dwarf character. He's logical, collected and responsible, the sardonic guardian without whom the hapless crew would certainly fall apart.

He wasn't always the sage, scientific know-it-all, who would be called upon to devise escape plans, analyse trajectories through supernova and decipher alien technology. His first job was sanitary droid aboard the Nova 5, responsible for cleaning the ship (too much for its crew - they were killed as a consequence of his meticulous cleaning of the ship's computer with soapy water). And importantly, he was programmed to never question authority.

There is a terrific backstory behind his character, where Lister encourages Kryten to break his programming and become more generalised. Rather than just cleaning, he was inspired to try other things too; and it turns out that Kryten is as capable of handling space shuttles as he is a mop.

I relate to Kryten's character, because he tells me that I too should never accept being labelled as performing one function, nor seek to specialise in one role. As humans we are evolved to improvise in circumstances and arrive at creative solutions to problems that machines would not be able to deal with. Yet all too often I hear people talk and act like robots, who identify with one singular function, whether their job, their role as a parent, their talent or their passion.

What do you do for a living?

It would be easy to overlook Kryten's talents, if he described himself as the sanitary droid. Although when people meet a new acquaintance, they often ask: "What do you do for a living?". This notion of being identifiable from one's job is a very limited view on the plurality of human endeavour. Our job comes with a description and expectations that stem from that; and often these expectations place limits on us and what we are expected to do.

Of course, it would be unworkable for a doctor to spontaneously provide the catering at her hospital, or the caterer to spontaneously provide the surgery. When organisational politics and hierarchical management play out, our roles are constrained further. Hence our energy is channelled into performing that one role, at the expense of our creativity.

Fortunately as a coder I find that the nature of my role is considerably more pluralistic. A good web developer needs to know about marketing, accessibility, business operations, user interface design, algorithms, data design, network protocols, security, deployment, administration, automation, testing and quality assurance. If you're self-employed, add to that list accountancy, law, tax, network building, public speaking and human resources. When I consider that I work in various industries: tourism, music, health, e-commerce, law, education; and for companies big and small, learning to code was the single most important thing behind breaking my programming; turning me into an active, creative generalist. I am widening my scope and learning new things all of the time. I am making my own mistakes, and learning from those too.

Being a generalist has meant I've run a hotel business whilst running a bootcamp. I've debugged responsive CSS bugs to display the same page on a mobile or a desktop site. I've normalised data to make it more efficient to store and retrieve. I've created pipelines to deploy my website automatically when I commit to source control. I've written design patterns in Ruby to reduce code coupling between my classes. I've deployed apps in React whilst learning how it worked earlier the same day. It's hard to say therefore where my one strength lies - I'm a jack of all trades.

The Singular Role of Consumers

However I wasn't always trying my hand at new things. I was schooled and brought up to accept the role of a consumer. The extent of my economic participation was simply to choose products and services, and sell my time to earn wages in order to pay for them. A consumer is necessarily passive; because rather than envisaging and creating our own products and services, we can only choose from what is there already, or make very narrow personalisation choices. Think of choosing a colour of an iPhone, its background image, its ringtone - ultimately you're still consuming the same product.

If we limit ourselves to the role of a consumer, accept everything as we find it, and neglect to develop our creative sides, we are the sanitary droid aboard the Nova 5. We need to learn to code to break our programming!

To learn coding is to suddenly find the ability to create products and services that didn't exist, with ostensibly little or no barrier to entry, other than some cheap hardware, or any limitations (in the case of Open Source software). The LEAN methodology promotes starting a skeleton business, allowing you to test ideas before putting more time and energy into them. You can write software that generates personalised, data-driven webpages for your customers, allowing you to grow without hiring staff, machinery or premises.

In this way, coding is an opportunity to move beyond our passive role as consumers and to try our hand at producing products and services ourselves.

The Price of Creativity

If you Google for long enough on this topic, you'll doubtless fall into one of the clickbait A-B tests of the redoubtable Tim Ferris, who for many is a paragon of the LEAN business approach and globalisation. In his book The 4 Hour Work Week, he reveals how to outsource all non-essential aspects of life, focussing on the most important 20% of tasks. He achieves this mainly by delegating tasks to offshore workers in developing countries, who work for less pay then he does. In this way, he only focusses on those important tasks that promote his business, and delegates everything else. Given the vast inequality in incomes globally, it's cost-effective to outsource all manner of day to day tasks, and he even cites the example of tasking an outsourcing firm to send personal messages to his girlfriend.

Quite apart from the dubious ethics of this, the idea that pricing our time and delegating every task that is 'beneath us' is also short-termist and anti-creative; because we rob ourselves of a chance to learn and grow. Given the hourly rate that I charge for web development, if I followed such advice and delegated those daily tasks that I could pay someone less to do, then I'd never do any of the following:

Look after my son
Play musical instruments (let alone practice or write music)
Learn new things about coding or business
Cook
Organise a day out with friends or family
Pay attention to my business accounts
Plan my exercise regime
Walk or cycle anywhere
Speak French
Write this blog!

The point Ferris makes is that if I outsource all the unnecessary, boring tasks, I'd have more time for the tasks that are important to me. In practice, easily delegatable tasks, that are simple to convey and to verify the progress on, make up a very small part of my working week. But even if they didn't, I'm not convinced that arranging my working week so that I'm focussed on just one part of my job is necessarily desirable, even if it were feasible. I want to be a generalist, not a sanitary droid!

And which 20% of my job would I choose? Should I label myself as a front-end developer, and outsource the back end. Or, outsource the design, and just take care of the coding. Or, outsource the copy for my webpages and focus on the UX. With startup projects, learning how to write good copy and make good design choices is vital, and not something I'd want to outsource.

I'd rather try my best to deliver something, and iterate on the feedback. Whatever choice I make, by specialising I'm losing the chance to learn and become self-sufficient in others areas, and am becoming a consumer of other services instead.

So should we follow Kryten's example and break our coding? Definitely! Not just our code that we write (that's the fun part), but our ideas of who we are and the roles we play.

Ready to get started? Check out my courses!

How I Teach Coding - The Monolithic Approach

Dan Garland — Tue, 25 Sep 2018 13:39:40 +0000

Every time I’d welcome a new cohort of my coding bootcamp We Got Coders, I’d give a speech about learning to code, and why we approached our learning the way we did. As part of this I’d usually baffle the audience with a somewhat obscure reference to Stanley Kubrick’s 2001: A Space Odyssey, and show the scene where an ape picks up a bone and proceeds to destroy the rest of the skeleton with it. After Strauss’s musical accompaniment builds to a glorious crescendo, the YouTube clip stops and I’m generally met with an awkward silence. Even if you’ve seen the film, taken out of context it is a striking image (forgive the pun!); but requires a bit of follow up to explain how on earth it has anything at all to do with learning to code.

The part that requires the most explanation is the shot of the imposing black monolith, who seems to have some kind of effect upon the ape. Nothing is said nor written down; but somehow an idea is imparted to that ape, a wonder: what might happen if I pick up that bone and hit something with it? His first attempts are timid, abortive. But as he gains confidence, he begins to see a more distinctive effect, which reinvigorates and encourages him to try more. By the end of the scene, he’s obliterating carcasses, felling beasts, fighting off other tribes, and several thousand years later, he’s orbiting the moon in a spaceship.

For the more cynical of my new recruits, there might have been something of a demotivating take from this; namely being likened to an ape that’s going to take thousands of years to reach modern standards of technological ability. However, I wanted instead to emphasise the more reassuring message, that everything we need in order to grow, to create and become who we want to be, we already have and already are. The monolith doesn’t teach the ape how to do anything. It doesn’t provide the benefits of self-paced, online coding courses with amazing interactive material. It just suggests a way forward. While the monolith is a towering and mysterious alien intelligence that we do not possess, we are the ones who do the work.

Hence our ape follows the age-old, tried and tested approach of trial and error. Start with a premise; test it out, learn from the experience.

Select bone => Pickup bone => Hit other bone => See what happens

What’s important about trial and error, is that it completely bypasses the notion of right and wrong, of success and failure. Whatever the hypothesis and however it was executed, there is something to learn from the experience. Even if that conclusion is never to do that ever again, that is a valuable lesson. We learn even from our failures and our mistakes. In fact it’s especially from our failures and mistakes; if we somehow fluke success in every experiment we do, there is nothing to compare the results against to know if the outcome is dependable.

“A person who never made a mistake never tried anything new.”

Albert Einstein

So the approach that I’ve always adopted for teaching coding is to be the Monolith. I provide suggestions, I am a guide. I know from experience that the direction is good, the exercise worth persuing. I know when you’ll run into obstacles and how to overcome them. One of the biggest challenges in teaching coding is knowing how to isolate a subject, so that when we sit down to learn, we can focus on one topic. Programming relies on a huge subset of presumed knowledge: how to use a keyboard or a mouse, how to use a command line shell, basic UNIX user administration, how to get on the Internet. The Monolith would have a huge task on its hands if it had to try and explain coding to an ape!

So instead I prefer to lay out a bit of theory, some ideas (see that bone over there – it can be used for hitting things with), create some programming exercises that illustrate the concept and isolate or reduce the number of irrelevant dependencies and sub tasks required in order to attempt them (try hitting things for yourself) and then leaving you an assignment to do in your own time (practice hitting bones and let me know how you get on). It’s this approach that all self-taught engineers adopt, and the one that works for many of my students.

Test Driven Development as a Teaching Tool

I didn’t learn the word pedagogy (and took longer to learn to pronounce it properly) until well after I’d established my training courses. The science of how we learn is an interesting branch of psychology and social sciences, but one that is not as well understood as many might think. There is a huge gulf of difference between how you will learn from a University, a bootcamp or on-line courses and different approaches suit difference people at different times. I don’t know anyone in the bootcamp industry who is a qualified teacher in the classic sense and many more have no teaching experience at all. Yet student outcomes for bootcamps and on-line courses is very promising.

One reason that might be is that software developers have the enormous fortune to possess an array of learning aids that they use on a daily basis, without necessarily realising it. Debugging and profiling software provides useful insight into programs, and lets their authors understand how things work by seeing the moving parts in isolation and testing their assertions. This has been perfected in the development approach known as Test Driven Development (TDD), which is industry best practice for many disciplines of software development, especially Agile development.

Knowing TDD gave me an unexpected boost at the start of my training career. Using tests to demonstrate functionality provides a great way of examining a piece of code in isolation from the rest of a program, setting up a context and all the requirements to get the program running in a simple way. By focussing on a method or function at a time, we encourage our students to break down hard problems into easier ones, which can be solved one at a time. Rather than expecting them to create their own tests, I provide repl exercises on my courses with a detailed walkthrough, which allow my students to experiment on some code and see whether their changes worked or not with instant feedback, almost like observing the effect of hitting the skeleton with the bone. Once they see results, the exercise will add more functionality, requiring further adjustment.

In this way, we frame coding as a problem solving exercise that is solved iteratively. It’s this process of feedback on gradual changes with the benefit of hindsight, rather than some divine inspiration, that supercharges the learning process and makes programming fun; all whilst learning some industry best practices along the way. After all, there were quite a few iterations between smacking a skeleton to rocketry and space flight; but this feedback loop is daily life for many developers.

So if any of this resonates with you, I dare you to try some of the ideas put forward on my courses – will you go beyond infinity?