Forem: William Gough

Simple CLI To-Do application with Twirp RPC

William Gough — Mon, 23 Jul 2018 00:00:00 +0000

Introduction

You've seen it on Twitter, you've read about it on Medium, it's in all your newsletters... so what is RPC? In this post I'll introduce you to the technology using TwitchTv's fantastic RPC library (Twirp) and quench your curisosity. We'll be building a classic tutorial application... the To-Do list, but this time, making use of a CLI for interaction. Do you even know how to use a framework if you don't build a To-Do list with it?! Jokes aside, a To-Do list is the perfect size and requires simple CRUD utility to work, making it a great way to dip your feet into something new.

Why RPC?

Remote Procedure Call (RPC) is mostly used for system-to-system communication, e.g microservice to microservice. If you just want a normal public API then you still want to go with REST. However, if you have an API which needs to poll for data from elsewhere, you can use RPC to communicate with that external service if that implements it too. Lots of large organisations use it for all their system communications, for example Google does this with an infrastructure called Stubby which includes an open source version you're probably already familiar with - gRPC.

Benefits of RPC:

Bi-directional streaming of data over HTTP/2 transport asynchronously.
Ability to generate both client & server stubs for most common languages.
Easier to serialise structured data.
Can be 3 - 10 times smaller than XML/JSON.
Can be anywhere from 20 - 100 times faster.

Set up & Prerequisites

Before we can do anything, we need to install a few things:

// Install Twirp protoc generator
go get github.com/twitchtv/twirp/protoc-gen-twirp

// Install Protoc - The protobuf compiler
https://github.com/google/protobuf/releases/tag/v3.6.0

// Mac users can install the above with HomeBrew
brew update && brew install protobuf

// Install Golang Protobuf support
go get -u github.com/golang/protobuf/protoc-gen-go

So it begins

Now that we have everything installed, we can crack on with the fun part, writing the code! First, we need to create our project directory and flesh out the structure:

cd $GOPATH/src/github.com/<your_username>
mkdir rpc-todo && cd $_
mkdir rpc
cd rpc && mkdir todo && cd $_

We're doing a couple of things here to separate our application structure and I'll explain each part as we progress through the tutorial. For now, let me explain the rpc directory. This is an important convention that sometimes appears under different namespaces; for example, you may see it named proto. The convention serves as a location for protobuf service definitions and their generated code. In our rpc directory we've created our first service directory todo . Now we can write out our service definition in a new file, service.proto:

syntax = "proto3";

package <your_github_username>.rpctodo.todo;
option go_package = "todo";

service Todo {
    rpc MakeTodo(Todo) returns (Empty);
    rpc GetTodo(TodoQuery) returns (Todo);
    rpc AllTodos(Empty) returns (TodoResponse);
    rpc RemoveTodo(TodoQuery) returns (Empty);
    rpc MarkComplete(TodoQuery) returns (Empty);
}

message TodoQuery {
    int32 id = 1;
}

message TodoResponse {
    repeated Todo todos = 1;
}

message Todo {
    int32 ID = 1;
    string title = 2;
    bool complete = 3;
}

message Empty {
}

There's a few things going on here so let's start at the top. First, we're declaring the syntax version of our protobuf file as version 3. Secondly, we declare the proto package. This works in the following format:

package <user/organisation>.<repository_name>.<service_name>;

// For example, mine is:
package williamhgough.commando.todo;

Then, we provide the name we wish to appear when importing the service as a Go package. Once all that is done, we can begin defining our service. Think of this as defining the interface for our API. Each method in our service takes parameters and returns a value, these values must be of a message type. In the same way we're equating the service keyword to an interface, we can think of a message as a struct type in Go. We are simply defining a new type and outlining its fields. There are some clear differences though. You may have noticed the = 1, = 2, = 3 after each message field, these represent the order of the fields. This allows us to come back later and add more fields or change the order.

It's also important to take note of both the Empty message type and the keyword repeated inside our TodoResponse. Since protobuf does not use the assumed return types void or null, we must define Empty instead. Inside TodoResponse, the repeated key word is used to return more than one item of type Todo. Since protobufs don't have array types for comparison in a Golang struct, you would have used a slice of Todo like so []Todo. For more information on protobuf syntax, check out the documentation here.

Now that we understand our way around a proto definition, let's generate our Golang service code using the protoc command. In the root of the project, execute the following:

protoc --proto_path=$GOPATH/src:. --twirp_out=. --go_out=. ./rpc/todo/service.proto

You should now see, inside your rpc/todo service directory, two new files service.pb.go and service.twirp.go. These are generated files. Do not edit them! Twirp and the Protoc compiler have generated client and server stubs for interacting with your RPC service.

We're about to make use of the generated files so take a minute to read through them and inspected the generated Go code. The next step for us is to implement our Todo service interface. In the root of your project directory, execute the following:

mkdir internal && cd $_
mkdir todoserver && cd $_
touch server.go

Here we create our internal directory, this is to hold our service implementations, one directory for each, as you can see we've named ours todoserver. Inside the server.go file you just created, let's add the following code:

package todoserver

import (
    "context"
    "github.com/twitchtv/twirp"
    // Import our protobuf package, namespace it to pb. This is a convention!
    pb "github.com/<your_username>/rpc-todo/rpc/todo"
)

// Create our Server type
type Server struct{}

// Seed our application with one To-Do to start with.
// The map holds the To-Do ID as the key to a To-Do value.
var todos = map[int]*pb.Todo{
    1: &pb.Todo{ID: 1, Title: "hello world", Complete: false},
}

// MakeTodo creates a To-Do
func (s *Server) MakeTodo(ctx context.Context, todo *pb.Todo) (*pb.Empty, error) { return nil, nil }

// GetTodo returns a single To-Do for given ID.
func (s *Server) GetTodo(ctx context.Context, query *pb.TodoQuery) (*pb.Todo, error) { return nil, nil }

// AllTodos returns all To-Dos.
func (s *Server) AllTodos(ctx context.Context, e *pb.Empty) (*pb.TodoResponse, error) { return nil, nil }

// RemoveTodo deletes the To-Do with the given ID.
func (s *Server) RemoveTodo(ctx context.Context, q *pb.TodoQuery) (*pb.Empty, error) { return nil, nil }

// MarkComplete sets the To-Do with given ID to completed.
func (s *Server) MarkComplete(ctx context.Conext, q *pb.TodoQuery) (*pb.Empty, error) { return nil, nil }

The code above creates a new Server type and makes it implement our Todo service interface. For now, our code just satisfies the interface and won't actually function. Now we have the base implementation, let's finish them off with the actual code we need. Make sure you read through the comments in the code as they should explain what is happening at each point:

func (s *Server) MakeTodo(ctx context.Context, todo *pb.Todo) (*pb.Empty, error) {
    // Validate the given ID, return error if not > 0
    if todo.ID < 1 {
        return nil, twirp.InvalidArgumentError("ID:", "No ID provided")
    }
    // Validate Title, should not add empty To-Do.
    if todo.Title == "" {
        return nil, twirp.InvalidArgumentError("Title:", "No title given")
    }
    // Set To-Do in the Todos map, passing ID as key.
    todos[todo.ID] = todo
    // Return Empty response and nil error.
    return &pb.Empty{}, nil
}

func (s *Server) GetTodo(ctx context.Context, q *pb.TodoQuery) (*pb.Todo, error) {
    // Validate the given ID, return error if not > 0
    if q.Id < 1 {
        return nil, twirp.InvalidArgumentError("ID:", "Must be greater than zero")
    }
    // Return To-Do with given ID from map and nil error
    return todos[q.Id], nil
}


func (s *Server) AllTodos(ctx context.Context, e *pb.Empty) (*pb.TodoResponse, error) {
    // Create new slice of To-Dos
    t := []*pb.Todo{}
    // for each one in our map, add to slice
    for _, v := range todos {
        t = append(t, v)
    }
    // If there are no todos, return error
    if len(t) < 1 {
        return nil, twirp.InternalError("No todos found!")
    }
    // Otherwise return Todos and nil error
    return &pb.TodoResponse{
        Todos: t,
    }, nil
}

func (s *Server) RemoveTodo(ctx context.Context, q *pb.TodoQuery) (*pb.Empty, error) {
    // Validate ID is greater than 0
    if q.Id < 1 {
        return nil, twirp.InvalidArgumentError("ID:", "Must be greater than zero")
    }
    // Remove To-Do with given ID from map
    delete(todos, q.Id)
    // Both return nil
    return &pb.Empty{}, nil
}

func (s *Server) MarkComplete(ctx context.Context, q *pb.TodoQuery) (*pb.Empty, error) {
    // Validate ID greater than 1
    if q.Id < 1 {
        return nil, twirp.InvalidArgumentError("ID:", "Must be greater than zero")
    }
    // Find To-Do with given ID in Todos map and set complete to true
    todos[q.Id].Complete = true
    // Return nil for both
    return &pb.Empty{}, nil
}

Alright, we're almost there. The final stage of this application involves building our CLI programs. We need a program to run the server and a CLI to interact with it. Both applications are very straightforward. Let's go ahead and create the server. In the project root, execute the following:

mkdir cmd && cd $_
mkdir todo-server && cd $_
touch main.go

Here we've created our client application directory and the main program. If you aren't familiar with Go application structure and using the cmd directory, it's a simple pattern to structure applications where a command-line interface is used. We name the directories inside cmd with the desired program name so that when we run go build, we get a binary named todo-server instead of main. Inside main.go, add the following code:

package main

import (
    "fmt"
    "net/http"

    "github.com/<your_username>/rpc-todo/internal/todoserver"
    pb "github.com/<your_username>/rpc-todo/rpc/todo"
)

func main() {
    // Create new Server from our server package.
    server := &todoserver.Server{}
    // Create Twirp Handler for our server from generated server stub.
    // This takes the form pb.New<SERVICE_NAME>Server()
    twirpHandler := pb.NewTodoServer(server, nil)

    // Start HTTP server on 8080, passing our twirpHandler as the servemux.
    http.ListenAndServe(":8080", twirpHandler)
    fmt.Println("To-Do server listening on :8080")
}

Next, we create the client program:

cd cmd
mkdir todo && cd $_
touch main.go

The client application has more functionality than our server as we need to parse any command line flags, validate them, and then execute the desired process. All of which can be achieved with the flags package and a simple switch statement. Nice and easy:

package main

import (
    "context"
    "flag"
    "fmt"
    "net/http"

    "github.com/<your_username>/rpc-todo/rpc/todo"
)

// Create global To-Do counter
var todoCount int32 = 1

func main() {
    // Define our interaction flags.
    action := flag.String("command", "list", "Command to interact with Todos. create, get, list, delete")
    title := flag.String("title", "", "The todo title")
    id := flag.Int("id", 1, "The ID of the todo you wish to retrieve")
    flag.Parse()

    // Create the generated Twirp Protobuf Client for our Todo service
    client := todo.NewTodoProtobufClient("http://localhost:8080", &http.Client{})

    switch *action {
    case "create":
        _, err := client.MakeTodo(context.Background(), &todo.Todo{
            ID:       todoCount + 1,
            Title:    *title,
            Complete: false,
        })
        if err != nil {
            fmt.Println("could not create todo:", err)
        }
    case "list":
        res, err := client.AllTodos(context.Background(), &todo.Empty{})
        if err != nil {
            fmt.Println("could not fetch todos:", err)
        }
        for i, t := range res.Todos {
            fmt.Printf("%d. %s [%v]\n", i+1, t.Title, t.Complete)
        }
    case "get":
        todo, err := client.GetTodo(context.Background(), &todo.TodoQuery{
            Id: int32(*id),
        })
        if err != nil {
            fmt.Println("could not fetch todo:", err)
        }

        fmt.Printf("%d. %s [%v]\n", todo.ID, todo.Title, todo.Complete)
    case "delete":
        _, err := client.RemoveTodo(context.Background(), &todo.TodoQuery{
            Id: int32(*id),
        })
        if err != nil {
            fmt.Println("could not remove todo:", err)
        }
    case "complete":
        _, err := client.MarkComplete(context.Background(), &todo.TodoQuery{
            Id: int32(*id),
        })
        if err != nil {
            fmt.Println("could not complete todo:", err)
        }
    default:
        fmt.Println("invalid command, please try again.")
    }
}

Voila! That's it! See how we automagically generated Twirp client uses the same API as our server code? This makes it super intuitive to use and interact with. All that remains for us to do is to verify it's all working in our command line. Open up two terminal windows in your project root:

Window 1 - Start the To-Do service server.

cd cmd/todo-server
go run main.go

// Output
fmt.Println("To-Do server listening on :8080")

Window 2 - Interact with To-Do client.

cd cmd/todo

// list To-Dos
go run main.go
1. hello world [false]

// create To-Do
go run main.go -command="create" -title="Hello DevTheWeb"

// get specific To-Do
go run main.go -command="get" -id=2
2. Hello DevTheWeb [false]

// Complete To-Do
go run main.go -command="complete" -id=1

// Remove To-Do
go run main.go -command="delete" -id=1

And that's a wrap! You've done it! You can now add basic RPC to your repertoire of skills.

Okay, so that was a lot to take in and hopefully, I've been able to explain in a way that leaves you feeling clear and confident! I'd also like to recommend trying out Google's gRPC library. I opted for Twirp here because I find it more intuitive and easy to use. The best advice I could give you is to go ahead and try writing another RPC service with this freshly acquired knowledge. Maybe a library for finding books or a movies search tool using OMDB? There is an undeniable truth in the old proverb:

"practice makes perfect"

Thanks again for reading and please consider sharing or reaching out to me on twitter @whg_codes.

The original application source code for this tutorial can be found on my GitHub here: github.com/williamhgough/comman-do

How to test TCP/UDP connections in Go - Part 2

William Gough — Sun, 15 Jul 2018 00:00:00 +0000

This was originally posted on my blog: devtheweb.io
Before continuing please ensure you've read the first installment of this series - How to test TCP/UDP connections in Go - Part 1

Introduction

In the last installment of this mini-series we took a look at testing TCP connections in Golang and how we can verify expected outputs.
In Part 2 we're going to look at taking the application one-step further by adding UDP capability. There are several things to be aware of when working with UDP or other stream-based protocols, all of which I will elaborate on below:

UDP connections use the net.PacketConn interface, instead of net.Listener
A UDP client connection in golang is just a net.UDPAddr instead of net.Conn, meaning we can't conn.Close()
Since net.PacketConn does not implement io.writer, we can't simply write to a connected client.

As someone with little network programming experience in any language, I came across quite a few “gotchas” trying to test the UPD connections. I started by creating an interface I assumed each protocol would satisfy - how wrong I was! Now that I’ve had more experience, utilised some fantastic documentation on the std lib, and learned from my mistakes, I will demonstrate how I solved the problem of meeting the same API and how to best communicate with a UDP service in hopes of saving you from having the same “gotchas” moments I had.

Update the tests

In true test-driven manner, there is one thing we need to do before we do anything else - update the tests from Part 1. Let’s update the tests to reflect the desired addition of the UDP.

var tcp, udp Server

func init() {
    // Start the new server
    tcp, err := NewServer("tcp", ":1123")
    if err != nil {
        log.Println("error starting TCP server")
        return
    }

    udp, err := NewServer("udp", ":6250")
    if err != nil {
        log.Println("error starting UDP server")
        return
    }

    // Run the servers in goroutines to stop blocking
    go func() {
        tcp.Run()
    }()
    go func() {
        udp.Run()
    }()
}

func TestNETServer_Running(t *testing.T) {
    // Simply check that the server is up and can
    // accept connections.
    servers := []struct {
        protocol string
        addr     string
    }{
        {"tcp", ":1123"},
        {"udp", ":6250"},
    }
    for _, serv := range servers {
        conn, err := net.Dial(serv.protocol, serv.addr)
        if err != nil {
            t.Error("could not connect to server: ", err)
        }
        defer conn.Close()
    }
}

func TestNETServer_Request(t *testing.T) {
    servers := []struct {
        protocol string
        addr     string
    }{
        {"tcp", ":1123"},
        {"udp", ":6250"},
    }

    tt := []struct {
        test    string
        payload []byte
        want    []byte
    }{
        {"Sending a simple request returns result", []byte("hello world\n"), []byte("Request received: hello world")},
        {"Sending another simple request works", []byte("goodbye world\n"), []byte("Request received: goodbye world")},
    }

    for _, serv := range servers {
        for _, tc := range tt {
            t.Run(tc.test, func(t *testing.T) {
                conn, err := net.Dial(serv.protocol, serv.addr)
                if err != nil {
                    t.Error("could not connect to server: ", err)
                }
                defer conn.Close()

                if _, err := conn.Write(tc.payload); err != nil {
                    t.Error("could not write payload to server:", err)
                }

                out := make([]byte, 1024)
                if _, err := conn.Read(out); err == nil {
                    if bytes.Compare(out, tc.want) == 0 {
                        t.Error("response did match expected output")
                    }
                } else {
                    t.Error("could not read from connection")
                }
            })
        }
    }
}

All we've done here is bootstrap the new UDPServer and then add a slice of servers to each test. This allows us to run the tests for each type of server connection with unique protocol and network address. If you run go test -v now, you should see the tests failing or erroring, but don't worry about that for now, we're going to fix it.

Next, at the bottom of net.go, we're going to write the minimum amount code we need to create our new type UDPServer and to implement the Server interface we defined in Part 1:

// UDPServer holds the necessary structure for our
// UDP server.
type UDPServer struct {
    addr   string
    server *net.UDPConn
}

// Run starts the UDP server.
func (u *UDPServer) Run() (err error) { return nil }

// Close ensures that the UDPServer is shut down gracefully.
func (u *UDPServer) Close() error { return nil }

The tests will still fail, however, we now have our foundations to build the UDPServer on. The next step is to implement the above two methods, like so:

// Run starts the UDP server.
func (u *UDPServer) Run() (err error) {
    laddr, err := net.ResolveUDPAddr("udp", u.addr)
    if err != nil {
        return errors.New("could not resolve UDP addr")
    }

    u.server, err = net.ListenUDP("udp", laddr)
    if err != nil {
        return errors.New("could not listen on UDP")
    }

    for {
        buf := make([]byte, 2048)
        n, conn, err := u.server.ReadFromUDP(buf)
        if err != nil {
            return errors.New("could not read from UDP")
        }
        if conn == nil {
            continue
        }

        u.server.WriteToUDP([]byte(fmt.Sprintf("Request recieved: %s", buf[:n])), conn)
    }
}

// Close ensures that the UDPServer is shut down gracefully.
func (u *UDPServer) Close() error {
    return u.server.Close()
}

That's it! Our tests should now all pass, and we have a working UDP connection Notice the differences here? First, we have to resolve the UDP address before we can start listening for connections and starting the server. Then we start accepting requests from the server. This is a little different with a UDP server. With a net.Listener, we can just .Accept() individual connections, however with UDP connections, we read from the server connection for requests and write each one to a buffer, we can then use the buffer to parse commands etc. ReadFromUDP returns three variables:

An integer representing the number of bytes written
UDP address of remote connection
Any errors encountered

We can use the first to parse only the number of written bytes, as shown in the example above with buf[:n]. Having the buffer sized to 2048 bytes allows us to listen for larger requests. It's also important to note that instead of writing to the connection, we have to use the server to write to the UDP addr of the connection. Since net.UDPConn doesn't implement io.Writer or io.Reader, we can't use the same approach we did with TCP by using the bufio package. Although I found this approach to be successful, I would love to hear your suggestions on how to overcome this problem, as there are bound to be cleaner ways of solving it.

Okay, so we've achieved the functionality we want, but can we abstract away some functionality? Yes, we can. Using the same approach as we did for TCP with a handleConnections method, we can reduce the responsibility of the Run method and ensure both servers have the same internal API. This benefits any other developers of the package as it provides a consistent way of working with different network protocols. Let's add the following:

func (u *UDPServer) Run() (err error) {
    laddr, err := net.ResolveUDPAddr("udp", u.addr)
    if err != nil {
        return errors.New("could not resolve UDP addr")
    }

    u.server, err = net.ListenUDP("udp", laddr)
    if err != nil {
        return errors.New("could not listen on UDP")
    }

    return u.handleConnections()
}

func (u *UDPServer) handleConnections() error {
    var err error
    for {
        buf := make([]byte, 2048)
        n, conn, err := u.server.ReadFromUDP(buf)
        if err != nil {
            log.Println(err)
            break
        }
        if conn == nil {
            continue
        }

        go u.handleConnection(conn, buf[:n])
    }
    return err
}

func (u *UDPServer) handleConnection(addr *net.UDPAddr, cmd []byte) {
    u.server.WriteToUDP([]byte(fmt.Sprintf("Request recieved: %s", cmd)), addr)
}

Voila! We've broken down the original Run method into two more functions that will achieve the same results and make the tests pass, but in a much cleaner way.

Conclusion

That's all for this series folks! I hope over the course of both posts I've clarified some of the differences and problems you may face when attempting to offer a tested, reliable, and simple service via TCP & UDP. Thanks again for reading, please consider sharing or reaching out to me on twitter @whg_codes.

The source code for this example can be found on my github here: github.com/williamhgough/devtheweb-source

How to test TCP/UDP connections in Go - Part 1

William Gough — Fri, 06 Jul 2018 00:00:00 +0000

This article was originally posted on my blog - How to test TCP/UDP connections in Go - Part 1

Introduction

For a recent work task, I had to expose a key-value store (which was normally accessed through REST API) via packet and stream based protocols such as TCP and UDP in a REST-esque manner. This presented an exciting challenge for me as I am relatively new to writing Go professionally and I wanted to ensure a number of things:

The overall integrity and reliability of the software I was writing
Cleaner code
Rapid development cycle
Regression detection

I had no specific requirements given on how to interact with the exposed interface, but I knew writing tests would be the best way to solve a new challenge in the most effective and efficient way.

This tutorial will demonstrate my approach to testing TCP/UDP connections in Golang in a simple yet effective way. The step-by-step instructions will provide guidance to verify your network connections produce the desired output, especially if you’re new to Go or testing (like me).

I’m not going to re-implement the original project for the sake of this post, however, I am going to create something similar to demonstrate the problem and what I found to be a sensible and quick approach to solving it.

Setting up

Firstly, let's create a new working directory and create 2 new Go files. I normally work inside /go/src/github.com/me, but it's totally up to you:

cd $GOPATH/src/github.com/<you>
mkdir net-testing && cd $_
touch net.go net_test.go

Fire up your favorite editor (I'm using VS Code; it has great support for Go with auto formatting and auto-test on save) and let's start building out our application. First up, in net_test.go, we need to bootstrap the application to start when the tests run.

N.B: Running a network server, whether its TCP/UDP/HTTP, will block the main Goroutine until the application is shut down or the server errors.

package net

import (
    "log"
)

var srv Server

func init() {
    // Start the new server.
    srv, err := NewServer("tcp", ":1123")
    if err != nil {
        log.Println("error starting TCP server")
        return
    }

    // Run the server in Goroutine to stop tests from blocking
    // test execution.
    go func() {
        srv.Run()
    }()
}

The above shows how we declare a global variable of type Server, which we haven't created yet, then use the init function to bootstrap our server in a new Goroutine. Next, we're going to write a test to verify that the server has started.

// Be sure to update your imports
import (
    "log"
    "net"
    "testing"
)


// Below init function
func TestNETServer_Run(t *testing.T) {
    // Simply check that the server is up and can
    // accept connections.
    conn, err := net.Dial("tcp", ":1123")
    if err != nil {
        t.Error("could not connect to server: ", err)
    }
    defer conn.Close()
}

Running the tests will inevitably result in a failed build right now, but don't worry about that. Now that we have our test ready to check our server runs, let's start the server implementation. Inside net.go, add the following code:

package net

import (
    "errors"
    "net"
    "strings"
)

// Server defines the minimum contract our
// TCP and UDP server implementations must satisfy.
type Server interface {
    Run() error
    Close() error
}

// NewServer creates a new Server using given protocol
// and addr.
func NewServer(protocol, addr string) (Server, error) {
    switch strings.ToLower(protocol) {
    case "tcp":
        return &TCPServer{
            addr: addr,
        }, nil
    case "udp":
    }
    return nil, errors.New("Invalid protocol given")
}

// TCPServer holds the structure of our TCP
// implementation.
type TCPServer struct {
    addr   string
    server net.Listener
}

// Run starts the TCP Server.
func (t *TCPServer) Run() (err error) {
    t.server, err = net.Listen("tcp", t.addr)
    if err != nil {
        return
    }
    for {
        conn, err := t.server.Accept()
        if err != nil {
            err = errors.New("could not accept connection")
            break
        }
        if conn == nil {
            err = errors.New("could not create connection")
            break
        }
        conn.Close()
    }
    return
}

// Close shuts down the TCP Server
func (t *TCPServer) Close() (err error) {
    return t.server.Close()
}

Whether you're new to Go or a comfortable Go programmer, bear with me whilst I break this down. First, we declare a new interface type named Server, this is so we can ensure all Clients use the same API, since in the next part of this series we'll build a UDPServer type. Adding interfaces to your application too early normally results in over-engineering, always consider YAGNI... "you ain't gonna need it". Next, we write a builder function that will return a Server implementation based on the chosen protocol. Finally, we create our TCPServer type that implicitly satisfies the Server interface because we have defined Run and Close methods. For now, all we've achieved is passing the test we defined. As the saying goes... Red, Green, Refactor.

With that in mind, let's write a test that covers handling output from the server. This time we're going to make use of table testing, we will run a series of test criteria inside a for-loop, meaning we can efficiently test different outcomes, like so:

func TestNETServer_Request(t *testing.T) {
    tt := []struct {
        test    string
        payload []byte
        want    []byte
    }{
        {
            "Sending a simple request returns result",
            []byte("hello world\n"),
            []byte("Request received: hello world")
        },
        {
            "Sending another simple request works",
            []byte("goodbye world\n"),
            []byte("Request received: goodbye world")
        },
    }

    for _, tc := range tt {
        t.Run(tc.test, func(t *testing.T) {
            conn, err := net.Dial("tcp", ":1123")
            if err != nil {
                t.Error("could not connect to TCP server: ", err)
            }
            defer conn.Close()

            if _, err := conn.Write(tc.payload); err != nil {
                t.Error("could not write payload to TCP server:", err)
            }

            out := make([]byte, 1024)
            if _, err := conn.Read(out); err == nil {
                if bytes.Compare(out, tc.want) == 0 {
                    t.Error("response did match expected output")
                }
            } else {
                t.Error("could not read from connection")
            }
        })
    }
}

The tests above send a payload (byte slice) to our server and then attempt to read from the connection to see what the response was. By utilizing the bytes package we can test for an Index of a substring with bytes.Index or simply to test for a substring with bytes.Contains. If we want to test for a direct match between the expected response and actual response, we can use the approach outlined above using bytes.Compare(a, b), where 0 return value means a == b.

As the application currently stands, the tests will fail, so now we need to add the code that will actually handle input in an idiomatic way:

func (t *TCPServer) handleConnections() (err error) {
    for {
        conn, err := t.server.Accept()
        if err != nil || conn == nil {
            err = errors.New("could not accept connection")
            break
        }

        go t.handleConnection(conn)
    }
    return
}

func (t *TCPServer) handleConnection(conn net.Conn) {
    defer conn.Close()

    rw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))
    for {
        req, err := rw.ReadString('\n')
        if err != nil {
            rw.WriteString("failed to read input")
            rw.Flush()
            return
        }

        rw.WriteString(fmt.Sprintf("Request received: %s", req))
        rw.Flush()
    }
}

So, what are we doing here? Well, we start by defining a new method receiver on TCPServer that is going to continually listen and accept any connection requests to the server. If there are any problems doing so, it breaks the loop and returns the error. If all goes to plan, the connection is passed to the second method, handleConnection, where we continually read messages and respond to the client. Using the go keyword allows us to handle each connection inside its own Goroutine. Most importantly, let's not forget to update our Run method to use the new handleConnections. Inside Run, replace the line: conn.Close() with return t.handleConnections(), since handleConnections also returns an error, we can use the method call as a return value.

If you run the tests now with go test -v -cover. You should see the following output:

=== RUN   TestNETServer_Running
-------- PASS: TestNETServer_Running (0.00s)
=== RUN   TestNETServer_Request
=== RUN   TestNETServer_Request/Sending_a_simple_request_returns_result
=== RUN   TestNETServer_Request/Sending_another_simple_request_works
-------- PASS: TestNETServer_Request (0.00s)
    --- PASS: TestNETServer_Request/Sending_a_simple_request_returns_result (0.00s)
    --- PASS: TestNETServer_Request/Sending_another_simple_request_works (0.00s)
PASS
coverage: 68.6% of statements

Awesome, we have a working TCP server! We can be confident will return us the output, regardless of how simple the implementation is. Hopefully, if you're new to network programming in Go or testing in general, you can draw some inspiration from this tutorial. As I mentioned earlier, in the next installment of the series I'm going to look at adding a UDPServer and updating the tests to efficiently cover both network protocols. Thanks for reading, if you liked the content, please consider sharing!

The source code for this example can be found on my GitHub here: github.com/williamhgough/devtheweb-source