Forem: elesq

Super simple connecting to postgres with Go.

elesq — Fri, 06 Aug 2021 22:58:01 +0000

Hi again,

I recently wrote a small article on making a really easy connection to mongoDB with Go. This article intends to cover the same ground but for postgres because not everyone wants to go with a document model engine.

Let's map it out

We want to achieve a couple of things here:

separate the DB code into a separate package.
make clear that this is demo quality and I'll be taking the easy route with usernames and passwords and so on. ie. Do not repeat these shortcuts in production code.
clearly mark the bits that are whiffy for prod code.

Let's get a Dockerised postgres instance

We want to grab a docker image, my particular one was 10.8 because it matched a requirement I had. You can take a later version without any hassles.

docker pull postgres

# I used
docker pull postgres:10.8

To run this postgres image in a container you can use the following.

docker run --name postgres-local -p 5432:5432 -v ~/.postgres-data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=STRONGPASSWORD -d postgres:10.8

This connects you to the default user. Therefore to setup your database you can psql into the postgres shell and create a database CREATE DATABASE databasename and a role/user.

If you don't like the CLI for this sprt of thing you can use postico, or pgadmin apps to have a graphical tool for the ops here.

Lets go

Ideally you want your project to resemble the following structure:

.
├── go.mod
├── go.sum
├── models
│   └── models.go
└── main.go

We'l do most of the database work in the models.go file and it will look like this. First you'll have to go get the driver package called pq. This can be achieved with go get github.com/lib/pq. Easy!

package models

import (
    "database/sql"
    "fmt"

    _ "github.com/lib/pq"
)

const (
    host     = "127.0.0.1"
    port     = 5432
    user     = "your_username_here"
    password = "make_it_a_good_on"
    dbname   = "your_database_name_here"
)

func InitDB() (*sql.DB, error) {
    var connectionString = fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=disable",
        host, port, user, password, dbname)
    var err error

    db, err := sql.Open("postgres", connectionString)
    if err != nil {
        return nil, err
    }

        // example of an operation we can take.
    stmt, err := db.Prepare("CREATE TABLE IF NOT EXISTS your_table_name(ID SERIAL PRIMARY KEY, ATTR TEXT NOT NULL);")
    if err != nil {
        return nil, err
    }

    _, err = stmt.Exec()
    if err != nil {
        return nil, err
    }

    return db, nil
}

and to be able to run the demo we have a trivial main.go, it's as simple as:

package main

import (
    "log"

    models "github.com/USERNAME/postgres-go-driver/models"
)

func main() {
    _, err := models.InitDB()
    if err != nil {
        log.Println(err)
    } else {
        log.Println("Database schema successfully initialized")
    }
}

Short, hopefully sweet, and a little bit of reference boilerplate for you, (or for me a few months from now) and remember the takeaway about it being demo code, this means you might consider:

making username and password environment variables
Not using a magic string to hardcode SQL in your code. You could separate this to a package, or packages for DML and DDL respectively.
remember to not use admin & password.

Best wishes.

Go - Protocol Buffer, Protoc tool & generated structs

elesq — Thu, 05 Aug 2021 16:23:52 +0000

Protocol Buffers with Go

So, today I'm having a look at protocol buffers with generated output in Go. Let's start at the top with:

Q. What is a protocol buffer?
Well, google says that it's a flexible, efficient, automated mechanism for serializing structured data, like XML but smaller, faster and simpler.

Q. That's great, thanks. Now what does it mean?
Well, seemingly we can define once how we'd like our data to be structured. Then we just simply use the specially generated source code to read and write the data a variety of streams in a variety of languages. From this we can summarize that they allow us to define the data contract between systems or services. Whereas data in formats like XML or JSON are still text-based protocol buffers are binary.

Q. What are the benefits of this?

Stronger interfaces are a good thing.
Smaller size than text based alternatives are another advantage.
typically they are speed optimised.
type and order are in place so ambiguity is minimised.
data access classes generated are considered to be easier to work with.

A tight interface with our data is absolutely essential for designing microservices well.

Protocol buffers in Go can be used on different transport layers such as HTTP/2 or AMQP (Advanced Message Queuing Protocol) where the binary data being transferred can only be understood between the client and the server.

Protocol Buffer language

I know this is a bit wordy with no code yet and it can feel a bit TL;DR but stick with me. A protocol buffer is a file with a minimalist language syntax we use to define our data structures. We compile this file and it generates a new file for a given programming language - in our case this means generating .go files. The language gives us types that we will use to create interfaces. This is called a protobuf which is one of the terms of the technology. Let's see an example of this for version 3.



syntax 'proto3';

# note that we have type, name, index position in the file for each entry
message SomeInterface {
    int id = 1;
    int code = 2;
    string name =3;
    string nickname = 4;
}

This file when compiled will give us Go style structures.

Key types:

Scalar values.
Enumerations and repeated values.
nested fields.

Enumerations

A numerical ordering of a given set of elements, defaults from 0 to n. This is fairly standard and should remind you of an iota in Go. Where it differs is by using option allow_alias = true we can have the same numeric value assigned to more than one option. essentially allowing duplicates.



syntax = 'proto3';

message Move{
    enum Moves{
        ROCK = 0
        PAPER = 1
        SCISSORS = 2
    }
}

Repeated

Repeated values are much like an array in JSON. We use the [values, values, values] syntax.

Nested fields

We can see below how to define nested types. Below we have a person who can have multiple contact numbers.



syntax = 'proto3';

message MyPerson {
    string name = 1;
    string nickname = 2;
    repeated ContactType contacts = 3;
}

message ContactType {
    string desc =1;
    string number =2;s
}

Protoc and Protocol Buffer compilation

Install the protoc tool for your system.
save your protobuf files with a .proto extension.
compile to target a particular language, for us that is the mighty Go!
Import structs from generated file and add the required data

Here we go, the generating part....

So now you want to generate the Go files, this can be real headache. I must have dropped a few pounds in the swear jar here but if you're trying to do this (or you're me several months from now) remember to put the option go_package = "github.com/USERNAME/MODULE_NAME"; in the .proto file.

By now this will look like:



syntax = 'proto3';

option go_package = "github.com/USERNAME/MODULE_NAME";

package protofiles;

message MyPerson {
    string name = 1;
    string nickname = 2;
    repeated ContactType contacts = 3;
}

message ContactType {
    string desc =1;
    string number =2;s
}

to do the generation of our files we can run the following in the CLI.



protoc --go_out=. --go_opt=Mpaths=source_relative  FILENAME.proto

Afterwards, you'll have generated a new folder set in the protofiles directory and in the project name subfolder you'll find a person.pb.go. This autogenerated code should look similar to below.



// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
//  protoc-gen-go v1.27.1
//  protoc        v3.17.3
// source: mock.proto

package protocol_buffers

import (
    protoreflect "google.golang.org/protobuf/reflect/protoreflect"
    protoimpl "google.golang.org/protobuf/runtime/protoimpl"
    reflect "reflect"
    sync "sync"
)

const (
    // Verify that this generated code is sufficiently up-to-date.
    _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
    // Verify that runtime/protoimpl is sufficiently up-to-date.
    _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)

type MyPerson struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    Name     string         `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
    Nickname string         `protobuf:"bytes,2,opt,name=nickname,proto3" json:"nickname,omitempty"`
    Contacts []*ContactType `protobuf:"bytes,3,rep,name=contacts,proto3" json:"contacts,omitempty"`
}

func (x *MyPerson) Reset() {
    *x = MyPerson{}
    if protoimpl.UnsafeEnabled {
        mi := &file_mock_proto_msgTypes[0]
        ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
        ms.StoreMessageInfo(mi)
    }
}

func (x *MyPerson) String() string {
    return protoimpl.X.MessageStringOf(x)
}

func (*MyPerson) ProtoMessage() {}

func (x *MyPerson) ProtoReflect() protoreflect.Message {
    mi := &file_mock_proto_msgTypes[0]
    if protoimpl.UnsafeEnabled && x != nil {
        ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
        if ms.LoadMessageInfo() == nil {
            ms.StoreMessageInfo(mi)
        }
        return ms
    }
    return mi.MessageOf(x)
}

// Deprecated: Use MyPerson.ProtoReflect.Descriptor instead.
func (*MyPerson) Descriptor() ([]byte, []int) {
    return file_mock_proto_rawDescGZIP(), []int{0}
}

func (x *MyPerson) GetName() string {
    if x != nil {
        return x.Name
    }
    return ""
}

func (x *MyPerson) GetNickname() string {
    if x != nil {
        return x.Nickname
    }
    return ""
}

func (x *MyPerson) GetContacts() []*ContactType {
    if x != nil {
        return x.Contacts
    }
    return nil
}

type ContactType struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields

    Desc   string `protobuf:"bytes,1,opt,name=desc,proto3" json:"desc,omitempty"`
    Number string `protobuf:"bytes,2,opt,name=number,proto3" json:"number,omitempty"`
}

func (x *ContactType) Reset() {
    *x = ContactType{}
    if protoimpl.UnsafeEnabled {
        mi := &file_mock_proto_msgTypes[1]
        ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
        ms.StoreMessageInfo(mi)
    }
}

func (x *ContactType) String() string {
    return protoimpl.X.MessageStringOf(x)
}

func (*ContactType) ProtoMessage() {}

func (x *ContactType) ProtoReflect() protoreflect.Message {
    mi := &file_mock_proto_msgTypes[1]
    if protoimpl.UnsafeEnabled && x != nil {
        ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
        if ms.LoadMessageInfo() == nil {
            ms.StoreMessageInfo(mi)
        }
        return ms
    }
    return mi.MessageOf(x)
}

// Deprecated: Use ContactType.ProtoReflect.Descriptor instead.
func (*ContactType) Descriptor() ([]byte, []int) {
    return file_mock_proto_rawDescGZIP(), []int{1}
}

func (x *ContactType) GetDesc() string {
    if x != nil {
        return x.Desc
    }
    return ""
}

func (x *ContactType) GetNumber() string {
    if x != nil {
        return x.Number
    }
    return ""
}

var File_mock_proto protoreflect.FileDescriptor

var file_mock_proto_rawDesc = []byte{
    0x0a, 0x0a, 0x6d, 0x6f, 0x63, 0x6b, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x70, 0x72,
    0x6f, 0x74, 0x6f, 0x66, 0x69, 0x6c, 0x65, 0x73, 0x22, 0x6f, 0x0a, 0x08, 0x4d, 0x79, 0x50, 0x65,
    0x72, 0x73, 0x6f, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01,
    0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x1a, 0x0a, 0x08, 0x6e, 0x69, 0x63, 0x6b,
    0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x6e, 0x69, 0x63, 0x6b,
    0x6e, 0x61, 0x6d, 0x65, 0x12, 0x33, 0x0a, 0x08, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x73,
    0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x66, 0x69,
    0x6c, 0x65, 0x73, 0x2e, 0x43, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x54, 0x79, 0x70, 0x65, 0x52,
    0x08, 0x63, 0x6f, 0x6e, 0x74, 0x61, 0x63, 0x74, 0x73, 0x22, 0x39, 0x0a, 0x0b, 0x43, 0x6f, 0x6e,
    0x74, 0x61, 0x63, 0x74, 0x54, 0x79, 0x70, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x64, 0x65, 0x73, 0x63,
    0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x64, 0x65, 0x73, 0x63, 0x12, 0x16, 0x0a, 0x06,
    0x6e, 0x75, 0x6d, 0x62, 0x65, 0x72, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x6e, 0x75,
    0x6d, 0x62, 0x65, 0x72, 0x42, 0x23, 0x5a, 0x21, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63,
    0x6f, 0x6d, 0x2f, 0x65, 0x6c, 0x65, 0x73, 0x71, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x63, 0x6f,
    0x6c, 0x2d, 0x62, 0x75, 0x66, 0x66, 0x65, 0x72, 0x73, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f,
    0x33,
}

var (
    file_mock_proto_rawDescOnce sync.Once
    file_mock_proto_rawDescData = file_mock_proto_rawDesc
)

func file_mock_proto_rawDescGZIP() []byte {
    file_mock_proto_rawDescOnce.Do(func() {
        file_mock_proto_rawDescData = protoimpl.X.CompressGZIP(file_mock_proto_rawDescData)
    })
    return file_mock_proto_rawDescData
}

var file_mock_proto_msgTypes = make([]protoimpl.MessageInfo, 2)
var file_mock_proto_goTypes = []interface{}{
    (*MyPerson)(nil),    // 0: protofiles.MyPerson
    (*ContactType)(nil), // 1: protofiles.ContactType
}
var file_mock_proto_depIdxs = []int32{
    1, // 0: protofiles.MyPerson.contacts:type_name -> protofiles.ContactType
    1, // [1:1] is the sub-list for method output_type
    1, // [1:1] is the sub-list for method input_type
    1, // [1:1] is the sub-list for extension type_name
    1, // [1:1] is the sub-list for extension extendee
    0, // [0:1] is the sub-list for field type_name
}

func init() { file_mock_proto_init() }
func file_mock_proto_init() {
    if File_mock_proto != nil {
        return
    }
    if !protoimpl.UnsafeEnabled {
        file_mock_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} {
            switch v := v.(*MyPerson); i {
            case 0:
                return &v.state
            case 1:
                return &v.sizeCache
            case 2:
                return &v.unknownFields
            default:
                return nil
            }
        }
        file_mock_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} {
            switch v := v.(*ContactType); i {
            case 0:
                return &v.state
            case 1:
                return &v.sizeCache
            case 2:
                return &v.unknownFields
            default:
                return nil
            }
        }
    }
    type x struct{}
    out := protoimpl.TypeBuilder{
        File: protoimpl.DescBuilder{
            GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
            RawDescriptor: file_mock_proto_rawDesc,
            NumEnums:      0,
            NumMessages:   2,
            NumExtensions: 0,
            NumServices:   0,
        },
        GoTypes:           file_mock_proto_goTypes,
        DependencyIndexes: file_mock_proto_depIdxs,
        MessageInfos:      file_mock_proto_msgTypes,
    }.Build()
    File_mock_proto = out.File
    file_mock_proto_rawDesc = nil
    file_mock_proto_goTypes = nil
    file_mock_proto_depIdxs = nil
}

I'll be using these in a further examples where we get to write some Go and use this, but we've seen the phaff and pain required to setup and generate your files. I hope if you're dabbling in this area at all that is somewhat useful to you.

Best wishes.

Go, Gorilla/mux and mongo demo

elesq — Mon, 02 Aug 2021 22:13:33 +0000

What are we going to do?

Setup some tools for starting a containerised MongoDB and connecting to the shell.
Create a connection to the database with Go
perform some trivial CRUD.
get an MVP knocked up, not paint the Sistine Chapel.

Simple. Let's go.

Docker setup

I'm assuming you have docker installed, if not head over to the docker site, follow the installation instructions for you system. Please forgive anything in the following section that is Mac specific and an oversight to make it OS agnostic. OK so all good, Docker installed. Let's get an image of MongoDB.

docker pull mongo

Now we have an image successfully pulled you'll want to be able to start a container running it. There's couple of things here, you should read up on detached/interactive modes for docker. It's fairly self explanatory and you can control it with cli flags when starting a container -d detached mode, -it interactive mode respectively. Another thing we have to be mindful of is binding a container port to a local port. So to get a container running that is in detached mode and bound locally to the usual mongoDB port of 27017 and with the ubiquitous admin/password combo you can use a command like this

docker run -d --name mongodb -v ~/data:/data/db -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=password -p 27017:27017 mongo:4.4.3

This is quite a handful to memorise, so what I like to do with stuff like this is take the pain one time, drop that command in a runnable script that I can dump in my scripts folder and assign an alias to it so I can weave magic with a one or two keystroke combo.

To do so, I'm creating a script called startmongo.sh

#!/bin/sh

echo " ┌─┐┌┬┐┌─┐┬─┐┌┬┐        "
echo " └─┐ │ ├─┤├┬┘ │          "
echo " └─┘ ┴ ┴ ┴┴└─ ┴          "
echo " ┌┬┐┌─┐┌┐┌┌─┐┌─┐┌┬┐┌┐  "
echo " ││││ │││││ ┬│ │ ││├┴┐  "
echo " ┴ ┴└─┘┘└┘└─┘└─┘─┴┘└─┘ "
echo " "
echo " Starting a docker container running MongoDB...."


docker run --name postgres -e POSTGRES_DB=vapor_database -e POSTGRES_USER=ed -e POSTGRES_PASSWORD=foolsgold -p 5432:5432 postgres

This gives a jazzy little header and starts my container. Great stuff. Next I want to be able to connect to the shell incase I have any brut force operations that I want to do on the CLI. Again, I'm doing this once, sticking it in an easy script and hiding that behind an alias. The command is docker exec -it mongodb bash to connect interactively to the container called mongoDB with a bash shell.

#!/bin/sh

echo " ┌┬┐┌─┐┌┐┌┌─┐┌─┐    "
echo " ││││ │││││ ┬│ │     "
echo " ┴ ┴└─┘┘└┘└─┘└─┘    "
echo " ┌─┐┌┐┌               "
echo " │ ││││               "
echo " └─┘┘└┘               "
echo " ┌┬┐┌─┐┌─┐┬┌─┌─┐┬─┐ "
echo "  │││ ││  ├┴┐├┤ ├┬┘  "
echo " ─┴┘└─┘└─┘┴ ┴└─┘┴└─ "
echo " run mongo --username admin --password password for authenticated mongo shell"

# open bash in contaner to connect to mongo shell
docker exec -it mongodb bash

Now we can add these to your profile (.bashrc, .zshrc, .profile or .aliases however you're doing it) and source your profile script to ensure these new aliases are part of the env. You can now start them with easier to remember commands you've created yourself. Hurrah!

Now we've got docker installed, mongo pulled and the ability to bring it up and connect to the shell with our own wizardry we can say that's section 1 of the post complete.

Go, Go Go....

Now we move to the Go part. We'll be using the go-mongo driver, you can get this by creating your project folder and running your go mod init github.com/username/projectname command and then running the go get for your dependencies.

go get go.mongodb.org/mongo-driver/mongo
go get -u github.com/gorilla/mux
go get gopkg.in/mgo.v2/bson

We're good to go. (see what I did there?)

Get connected

OK let's get the main.go working.

package main

import (
    "context"
    "encoding/json"
    "io/ioutil"
    "log"
    "net/http"
    "time"

    "github.com/gorilla/mux"
    "go.mongodb.org/mongo-driver/bson"
    "go.mongodb.org/mongo-driver/bson/primitive"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

type DB struct {
    collection *mongo.Collection
}

func main() {

    // dont put this in real code. Nobody will speak to you
    credentials := options.Credential{
        Username: "admin",
        Password: "password",
    }

    clientOptions := options.Client().ApplyURI("mongodb://localhost:27017").SetAuth(credentials)
    client, err := mongo.Connect(context.TODO(), clientOptions)
    if err != nil {
        panic(err)
    }
    defer client.Disconnect(context.TODO())

    collection := client.Database("appDB").Collection("movies")
    db := &DB{collection: collection}

    r := mux.NewRouter()
    r.HandleFunc("/v1/movies/{id:[a-zA-Z0-9]*}", db.GetMovie).Methods(http.MethodGet)
    r.HandleFunc("/v1/movies/{id:[a-zA-Z0-9]*}", db.UpdateMovie).Methods(http.MethodPut)
    r.HandleFunc("/v1/movies/{id:[a-zA-Z0-9]*}", db.DeleteMovie).Methods(http.MethodDelete)
    r.HandleFunc("/v1/movies", db.PostMovie).Methods(http.MethodPost)

    srv := &http.Server{
        Handler:      r,
        Addr:         "localhost:8080",
        WriteTimeout: 15 * time.Second,
        ReadTimeout:  15 * time.Second,
    }

    log.Fatal(srv.ListenAndServe())
}

Things to note:

This is a demo, for the love of god don't put your credentials into any "real" code like I have done in this above, you'll get blasted for it (rightly so), this is just for demo purposes. You'll want to put that sort of stuff into env variables and read them in.
read #1 again!
Note the DB structure. That's for convenience.
We're using gorilla/mux to define and configure our multiplexer.
hark at me with my fancy regex to grab any alphanumeric part of the url for matching IDs.

Next up we're going to cut some CRUD functions to be able to do some stuff with or database.

func (db *DB) GetMovie(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)

    var movie Movie
    objectID, _ := primitive.ObjectIDFromHex(vars["id"])
    filter := bson.M{"_id": objectID}

    err := db.collection.FindOne(context.TODO(), filter).Decode(&movie)
    if err != nil {
        w.WriteHeader(http.StatusInternalServerError)
        w.Write([]byte(err.Error()))
    } else {
        w.Header().Set("Content-Type", "application/json")
        response, _ := json.Marshal(movie)
        w.WriteHeader(http.StatusOK)
        w.Write(response)
    }
}

func (db *DB) PostMovie(w http.ResponseWriter, r *http.Request) {
    var movie Movie

    postBody, _ := ioutil.ReadAll(r.Body)
    json.Unmarshal(postBody, &movie)

    result, err := db.collection.InsertOne(context.TODO(), movie)
    if err != nil {
        w.WriteHeader(http.StatusInternalServerError)
        w.Write([]byte(err.Error()))
    } else {
        w.Header().Set("Content-Type", "application/json")
        response, _ := json.Marshal(result)
        w.Write(response)
    }
}

func (db *DB) UpdateMovie(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    var movie Movie

    putBody, _ := ioutil.ReadAll(r.Body)
    json.Unmarshal(putBody, &movie)

    objectID, _ := primitive.ObjectIDFromHex(vars["id"])
    filter := bson.M{"_id": objectID}
    update := bson.M{"$set": &movie}
    result, err := db.collection.UpdateOne(context.TODO(), filter, update)
    if err != nil {
        w.WriteHeader(http.StatusInternalServerError)
        w.Write([]byte(err.Error()))
    } else {
        w.Header().Set("Content-Type", "application/json")
        response, _ := json.Marshal(result)
        w.Write(response)
    }
}

func (db *DB) DeleteMovie(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    objectID, _ := primitive.ObjectIDFromHex(vars["id"])
    filter := bson.M{"_id": objectID}

    result, err := db.collection.DeleteOne(context.TODO(), filter)
    if err != nil {
        w.WriteHeader(http.StatusInternalServerError)
        w.Write([]byte(err.Error()))
    } else {
        w.Header().Set("Content-Type", "application/json")
        response, _ := json.Marshal(result)
        w.Write(response)
    }
}

things to note:

There's a lot in these functions that could be outsourced to clean the functions and reduce code. Outside the scope of this basic connect and get a quick and dirty demo together.
We can enhance the use of context to avoid using the .TODO context which is a default.

Lastly we need to add the structs to make up a movie entity.

type Movie struct {
    ID        interface{} `json:"id" bson:"_id,omitempty"`
    Name      string      `json:"name" bson:"name"`
    Year      string      `json:"year" bson:"year"`
    Directors []string    `json:"directors" bson:"directors"`
    Writers   []string    `json:"writers" bson:"writers"`
    BoxOffice BoxOffice   `json:"boxOffice" bson:"boxOffice"`
}

type BoxOffice struct {
    Budget uint64 `json:"budget" bson:"budget"`
    Gross  uint64 `json:"gross" bson:"gross"`
}

Things to note:

we define the json and bson names/refs
we have a relationship between movie and BoxOffice.
we have attributes that take a slice of values.

you can run the server and tap into this with some postman requests or simply with some curl requests. That choice is yours. That's hopefully enough for you to see how this works without focusing too much on code beauty by adding more abstraction at this stage.

Quick & dirty but I hope if you need a quick smash & grab demo of using the mongo driver for Go with gorilla mux then it should be enough to get you started on the right path with one eye on creating cleaner, better code as you get to grips with it..... ps. don't embed credentials in code outside of a demo.

Best wishes.

More Go RPC, using gorilla/rpc/json

elesq — Wed, 28 Jul 2021 21:49:37 +0000

Following on from an earlier post about Go and RPC, this time we're looking at gorilla/rpc and gorilla/rpc/json. One weakness of the prior example in the post linked above was it required both parts, client and server to be written in Go, a tight coupling not immediately apparent to someone learning either Go or RPC. Enter JSON...

Using JSON allows us to send a request as JSON and receive a response as JSON thus removing a tight coupling that we had previously.

Setup the project

OK let's get started. You're going to need a project structure much like below, obviously the modules files are optional if you're a GOPATH user and you don't strictly need a readme but you should by force of habit include one in your projects.

gorilla-jsonrpc-demo
├── go.mod
├── go.sum
├── projects.json
├── readme.md
└── server
    └── main.go

go get github.com/gorilla/rpc
go get -u github.com/gorilla/mux

The json data file

Here I've created a dummy data file with some sample projects structures in it. Extremely trivial to avoid complicating the demo.

[
    {
        "name": "go/golang",
        "author": "google",
        "stars": "88200",
        "forks": "12900"
    },
    {
        "name": "chi",
        "author": "go-chi",
        "stars": "9800",
        "forks": "665"
    },
    {
        "name": "go-me",
        "author": "me",
        "stars": "1",
        "forks": "0"
    },
    {
        "name": "go-you",
        "author": "you",
        "stars": "100",
        "forks": "10"
    },
    {
        "name": "go-them",
        "author": "them",
        "stars": "2000",
        "forks": "112"
    }
]

You can adapt your file, structures to suit your case or just to play around with more complex structures.

Implementing the RPC server

We're going to make a couple of changes to the previous server we wrote and although the changes are slight they make a world difference.

package main

import (
    jsonparse "encoding/json"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "os"
    "path/filepath"

    "github.com/gorilla/mux"
    "github.com/gorilla/rpc"
    "github.com/gorilla/rpc/json"
)

type Args struct {
    Name string
}

type Repo struct {
    Name   string `json:"name,omitempty"`
    Author string `json:"author,omitempty"`
    Forks  string `json:"forks,omitempty"`
    Stars  string `json:"stars,omitempty"`
}

type JSONServer struct{}

func (t *JSONServer) GiveRepoDetails(r *http.Request, args *Args, reply *Repo) error {
    var repos []Repo

    // get the datafile
    absPath, _ := filepath.Abs("projects.json")

    raw, readerr := ioutil.ReadFile(absPath)
    if readerr != nil {
        log.Println("error:", readerr)
        os.Exit(1)
    }

    // unmarshal the raw data from the file into our array for the repos
    marshalerr := jsonparse.Unmarshal(raw, &repos)
    if marshalerr != nil {
        log.Println("error:", marshalerr)
        os.Exit(1)
    }

    for _, repo := range repos {
        if repo.Name == args.Name {
            // we found a hit
            fmt.Println(repo)
            *reply = repo
            break
        }
    }
    return nil
}

func main() {

    port := ":8080"

    s := rpc.NewServer()                                 // Create a new RPC server
    s.RegisterCodec(json.NewCodec(), "application/json") // Register the type of data requested as JSON
    s.RegisterService(new(JSONServer), "")               // Register the service by creating a new JSON server

    r := mux.NewRouter()
    r.Handle("/rpc", s)
    log.Fatal(http.ListenAndServe(port, r))

}

Things to note:

we've had to rename the standard encoding/json to jsonparse to avoid clashing with the gorilla/rpc json library in our code.
we're using a gorilla mux
we are registering a codec for application/json
we register the service
in the service we're unmarshaling a file into an array
our results are sent back in JSON format.

Testing our server

So, how do we run this? Where is the client application this time? Of course we could create one but we can just as easily demo it with a fairly simple curl request.

curl -X POST \
   http://localhost:8080/rpc \
   -H 'cache-control: no-cache' \
   -H 'content-type: application/json' \
   -d '{
   "method": "JSONServer.GiveRepoDetails",
   "params": [{
   "name": "go-me"
   }],"id": "1"}'

Things to note about the curl request:

To get a response you need an ID, I too learned this the hard way after way too much time thinking where is my response? Why is there no output? Thanks to this post on SO I resolved my issue by adding the "id": "1" section.

I hope this has been simple enough for you get an easy takeaway from, about why and how you might use the gorilla libs to help you knock up an RPC json solution.

Best wishes.

RPC with Go, what is it?

elesq — Wed, 28 Jul 2021 14:18:33 +0000

OK, so we're going to have a look at RPC with Go. So what is RPC? RPC (remote procedure call) is a distributed computing concept where a call and response mechanism applies to the execution of a function to achieve an outcome and receive a result.

Say what?

But what does that mean? Well, we could look at it as follows:

We have a client who preps a function name and the arguments required to be sent with it.
An RPC server hosting this function receives this.
The server executes the remote process request.
The server responds to the client with the outcome and result.
the client receives the response (outcome) and data (result) and goes about its business merrily on to the next task.

OK. Show me in Go

Prerequisites

The server needs to expose the service. This is needed for the client to call and connect to the service.
go provides two libraries of interest net/rpc and net/jsonrpc.

We're going to have a project that contains two sub-projects, so let's create a folder for the project and within it create two subfolders, client and server.

Ideally we now have a structure that looks like the following:



rpc-banner-demo
├── client
│   └── main.go
└── server
    └── main.go

The RPC server

We're going to create an RPC server with a function that returns a message that could be used as a banner message or some other purpose for which you need to return a string.



package main

import (
    "log"
    "net"
    "net/http"
    "net/rpc"
)

// an RPC server in Go

type Args struct{}

type BannerMessageServer string

func (t *BannerMessageServer) GetBannerMessage(args *Args, reply *string) error {
    *reply = "This is a message from the RPC server"
    return nil
}

func main() {

    // create and register the rpc
    banner := new(BannerMessageServer)
    rpc.Register(banner)
    rpc.HandleHTTP()

    // set a port for the server
    port := ":1122"

    // listen for requests on 1122
    listener, err := net.Listen("tcp", port)
    if err != nil {
        log.Fatal("listen error: ", err)
    }

    http.Serve(listener, nil)
}

Things to note:

we're creating a function called GetBannerMessage of the BannerMessageServer type.
we need to register it.
Our server requires a port, 1122 in our case.

The RPC client

Now we create the client that will call our server and handle/use the response and result.



package main

import (
    "log"
    "net/rpc"
)

// rpc client

type Args struct{}

func main() {

    hostname := "localhost"
    port := ":1122"

    var reply string

    args := Args{}

    client, err := rpc.DialHTTP("tcp", hostname+port)
    if err != nil {
        log.Fatal("dialing: ", err)
    }

    // Call normally takes service name.function name, args and
    // the address of the variable that hold the reply. Here we
    // have no args in the demo therefore we can pass the empty
    // args struct.
    err = client.Call("BannerMessageServer.GetBannerMessage", args, &reply)
    if err != nil {
        log.Fatal("error", err)
    }

    // log the result
    log.Printf("%s\n", reply)
}

Things to note:

note that we have to specify the hostname and port in our client. localhost and :1122 in our demo case.
note that we have an empty args struct because we don't need it for our demo. If we needed it we can define the argument struct at the top of the file and make more complex use of our setup.
we call the service.function and the reply will be assigned to the address of of our local variable reply.
we simply log the outcome and end. In a more complex example we might act on this data, it may even feed into another RPC function call.

Run, Forest, run...

OK so now we're in a position to run our code and see this magic unfold before our eyes. There is a couple of options here, I'm opting to run both programs from the root folder of our project. We'll need two terminals, so here you can split your terminal in VSCode.

in one run go run server/main.go, in the other run go run client/main.go and marvel at your results

I hope you found this fairly trivial RPC example useful as a RPC client/server setup overview and as a potential springboard to your own functions and setups that actually have a real purpose.

Best wishes.

Chaining middleware in Go using justinas/alice package

elesq — Tue, 27 Jul 2021 21:05:59 +0000

Following on from https://dev.to/iamelesq/custom-middleware-basics-with-go-no-frameworks-29eg

We can make a couple of improvements to the simple API we created that included a couple of middleware wrapping functions.

We're going to split the project into separate files.
We're going to use a 3rd party package to chain the middleware functions using a popular community package called alice.

Let's go

create a new project and run go mid init <name-of-your-project>
Run go get github.com/justinas/alice on the cli.
Create main.go, handlers.go & middlewares.go files.

add our handler

In the handlers.go file we want to place the following code.

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
)

type Lesson struct {
    Title   string
    Summary string
}

func handle(w http.ResponseWriter, r *http.Request) {
    if r.Method == http.MethodPost {

        // create a variable of our defined struct type Lesson
        var tempLesson Lesson

        decoder := json.NewDecoder(r.Body)
        err := decoder.Decode(&tempLesson)
        if err != nil {
            panic(err)
        }

        defer r.Body.Close()

        fmt.Printf("Title: %s. Summary: %s\n", tempLesson.Title, tempLesson.Summary)
        w.WriteHeader(http.StatusCreated)
        w.Write([]byte("201 - Created"))

    } else {
        w.WriteHeader(http.StatusMethodNotAllowed)
        w.Write([]byte("405 - Method Not Allowed"))
    }
}

Add the middleware functions

In the middlewares.go file we can add the following code.

package main

import (
    "log"
    "net/http"
    "strconv"
    "time"
)

func filterContentType(handler http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Println("Inside filterContentType middleware, before the request is handled")

        // filter requests by mime type
        if r.Header.Get("Content-Type") != "application/json" {
            w.WriteHeader(http.StatusUnsupportedMediaType)
            w.Write([]byte("415 - Unsupported Media Type. JSON type expected"))
            return
        }

        // handle the request
        handler.ServeHTTP(w, r)

        log.Println("Inside filterContentType middleware, after the request was handled")
    })
}

func setServerTimeCookie(handler http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Println("Inside setServerTimeCookie middleware, before the request is handled")

        // handles the request
        handler.ServeHTTP(w, r)

        cookie := http.Cookie{Name: "Server Time(UTC)", Value: strconv.FormatInt(time.Now().Unix(), 10)}
        http.SetCookie(w, &cookie)

        log.Println("Inside setServerTimeCookie middleware, after the request is handled")
    })
}

add the main function

in main.go we need to add the following

package main

// chaing middleware with alice.
// here we will use the alice package to demo chainging middleware
// around our original or main handler.

import (
    "log"
    "net/http"

    "github.com/justinas/alice"
)

func main() {

    port := ":8080"

    originalHandler := http.HandlerFunc(handle)
    chain := alice.New(filterContentType, setServerTimeCookie).Then(originalHandler)

    http.Handle("/lesson", chain)
    log.Fatal(http.ListenAndServe(port, nil))
}

As you can see our refactor has added a 3rd party dependency but increased readbility. The fluent API of the alice package makes for clear and readable code when used.

Splitting the project into multiple files here is trivial as we have a tiny codebase and it is purely for demo purposes, however it is reasonably good, common practice to split your files in Go projects. This lowers the cognitive load from each file on the reader.

Custom middleware basics with Go. No frameworks.

elesq — Tue, 27 Jul 2021 20:26:33 +0000

So you want to add some custom middleware to a Go api? There's many elegant and advanced ways to do this, which is great! - but who wants to simply use a pre-cooked black-box when you can cut your own and have some sort of idea how others are doing it?

Here we're going to demo some fairly trivial hand-rolled middleware functions that will wrap the original function handling our request.

Let's create a fairly standard looking function that takes a ResponseWriter and a Request.

func postHandler(w http.ResponseWriter, r *http.Request) {
    if r.Method == http.MethodPost {

        // create a variable of our defined struct type Lesson
        var tempLesson Lesson

        decoder := json.NewDecoder(r.Body)
        err := decoder.Decode(&tempLesson)
        if err != nil {
            panic(err)
        }

        defer r.Body.Close()

        fmt.Printf("Title: %s. Summary: %s\n", tempLesson.Title, tempLesson.Summary)
        w.WriteHeader(http.StatusCreated)
        w.Write([]byte("201 - Created"))

    } else {
        w.WriteHeader(http.StatusMethodNotAllowed)
        w.Write([]byte("405 - Method Not Allowed"))
    }
}

Thing to note:

we can use the http types instead of r.Method == "POST" for better clarity.
In the snippet above you can see we mention the type we create called Lesson, you will see this depicted in the snippet that shows the package, imports and the main function below.

Adding Middleware

So, now the middleware we'd like to apply is firstly to check that we're accepting JSON format, or prevent the request from going anywhere. We can do that like this.

func filterContentType(handler http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Println("Inside filterContentType middleware, before the request is handled")

        // filter requests by mime type
        if r.Header.Get("Content-Type") != "application/json" {
            w.WriteHeader(http.StatusUnsupportedMediaType)
            w.Write([]byte("415 - Unsupported Media Type. JSON type is expected"))
            return
        }

        // handle the request
        handler.ServeHTTP(w, r)

        log.Println("Inside filterContentType middleware, after the request was handled")
    })
}

Things to note:

we take an http.Handler and return an http.Handler
we can do stuff before and after handling the request. Here we're doing a but of useless printing but you get the idea.

The next thing we want to do is apply another middleware and this time we're going to add a cookie attribute with the server timestamp. This magic can be woven like so...

unc setServerTimeCookie(handler http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Println("Inside setServerTimeCookie middleware, before the request is handled")

        // handles the request
        handler.ServeHTTP(w, r)

        cookie := http.Cookie{Name: "Server Time(UTC)", Value: strconv.FormatInt(time.Now().Unix(), 10)}
        http.SetCookie(w, &cookie)

        log.Println("Inside setServerTimeCookie middleware, after the request is handled")
    })
}

Things to note:

again you can see we can have a before and after routine demonstrating we have standard wrapping capabilities yet again.
we create and set the cookie value using an http in-built method.

Testing our work

OK, so no we need a way to run this so we can flesh out our main method like this.

 package main

import (
    "encoding/json"
    "fmt"
    "log"
    "net/http"
    "strconv"
    "time"
)

type Lesson struct {
    Title   string
    Summary string
}

func main() {
    port := ":8080"

    originalHandler := http.HandlerFunc(postHandler)
    http.Handle("/lesson", filterContentType(setServerTimeCookie(originalHandler)))

    //http.HandleFunc("/city", poastHandler)
    log.Fatal(http.ListenAndServe(port, nil))
}

things to note here:

the originalHandler is an http.HandlerFunc type.
you can see that each middleware effectively wraps what it receives, this allows us to chain the middlware functions around the original handler meaning we can have a a line like filterContentType(setServerTimeCookie(originalHandler))
we could even do it on a one liner, but clarity starts to dwindle. If you want to try it swap to http.Handle("/lesson", filterContentType(setServerTimeCookie(http.HandlerFunc(postHandler))))

so, now we're going to need some positive and negative tests. To do so easily we can manually test on the CLI using curl.

So, let's see if we can process non-json input as the request.

curl http://localhost:8080/city -d '{"title": "middleware with Go", "summary": "learn some middleware concepts with a quick and dirty demo"}'

like me, you should be met with a 415 - Unsupported Media Type. JSON type expected% response to your test.

Now let's try what should hopefully be a successful attempt.

curl -i -H "Content-Type: application/json" -X POST http://localhost:8080/city -d '{"title": "Middleware demo with Go", "summary": "Hopefully a successful demo now that we have specified mime type and the method"}'

Hopefully your output is similar to below. 

`HTTP/1.1 200 OK
Date: Tue, 27 Jul 2021 20:09:52 GMT
Content-Length: 13
Content-Type: text/plain; charset=utf-8

201 - Created%`

That wraps up the post. I hope you found this quick overview to be of some use if you too are messing about with Go and looking to grasp what middleware is and how it works.

Best wishes