Forem: Santosh Kumar

str vs repr in Python and When to Use Them

Santosh Kumar — Fri, 24 Feb 2023 18:30:00 +0000

Introduction

In a previous post, we have done a comparison between __new__ and __init__. In this post, we are going to do something similar, but with a different pair.

For those who don't know, __str__() and __repr__() are two dunder methods which you can override in a class to change the way class is represented.

These two little functions have been a source of confusion in my early career. Let us first go through them one by one and let us understand what they do first. Then we'll do comparison among them with respect to their similarities and differences.

str() and str()

If you have worked with other language before coming to Python, you might be familiar with str() or string() function, which would cast any other data type to String data type or class.

Python behaves the same. Here is an example in Python REPL.

>>> my_int = 2
>>> type(my_int)
<class 'int'>
>>> my_str = str(my_int)
>>> type(my_str)
<class 'str'>

Simlar is the repr() function. repr also returns a string.

>>> my_int = 2
>>> type(my_int)
<class 'int'>
>>> my_repr = repr(my_int)
>>> type(my_repr)
<class 'str'>

repr() and repr()

By now you understand that both the dunder method has their corresponding function symbols in which you can pass another data type.
We also saw in the previous section that when you pass anything to repr(), it simply turns into a str data type.

str() and repr() in a Class

If you still don't know about dunder methods, this section might help you undestand it.

There is a massive list of Python dunder methods. Dunder methods allow the way your handwritten class to behave in a certain manner.

The difference between the two methods is that __str__ is intended to provide a human-readable representation of the object, while __repr__ should provide a representation that can be used to recreate the object.

Let's see the difference between the two with the help of a simple code example:

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __str__(self):
        return f"({self.x}, {self.y})"

    def __repr__(self):
        return f"Point({self.x}, {self.y})"

p = Point(1, 2)

print(p) # Output: (1, 2)
print(repr(p)) # Output: Point(1, 2)

In the example above, __str__ returns a string representation of the Point object that is intended to be human-readable. The __repr__ method returns a string representation that could be used to recreate the object.

In general, it is recommended to always define a __repr__ method for your objects, so that you have an unambiguous representation of the object that can be used for debugging or in a REPL environment. On the other hand, __str__ is typically only defined when a more human-readable representation is needed. If __str__ is not defined for an object, repr(obj) is used as a fallback.

It's worth noting that the __str__ method should return a string, while the __repr__ method should return a string that, when passed to the eval() function, returns an object with the same value. The recommended format for __repr__ is a string that, when passed to eval(), creates an object with the same value as the original. In the example above, the Point object can be recreated by calling Point(1, 2).

Conclusion

In conclusion, __str__ and __repr__ are two important methods in Python that allow you to define string representations of objects. Understanding the difference between the two and when to use them is important in creating robust and well-designed code.

Keep reading:

https://www.geeksforgeeks.org/str-vs-repr-in-python/

How to Control Philips Wiz Bulb Using Go

Santosh Kumar — Fri, 17 Feb 2023 18:30:00 +0000

Introduction

I have always been conscious about privacy and it was the sole reason I wasn't early adapter of smart home devices. This changed when I bought Raspberry Pi. I learned that I can make my Pi hub to all the smart devices at my home.

And this is how I bought 9W Philips Wiz light; the cheapest one I could get in the Indian market.

To use this product, you at least need to have a home network (a wifi router will work). The router does not need to have internet connection.

Control Philips Wiz using App

The most common way to control the Wiz light is using the mobile app. It is available for both Android and iOS.

I don't want to cover how to operate the bulb using the app. I would leave it upto you. I want you to explore the app completely before proceeding towards the next section.

As we are going to control the bulb using code, I want you to practice doing these actions:

Turn on/off
Set colors
Set brightness
Set temperature
Set modes

Control Philips Wiz using Go

While there are libraries available to interact with the bulb, I wanted to show you guys what happens internally with the bulb. In a home automation scenario, you'd mostly be using such libraries.

Find the bulb IP

Before we write any piece of code, it is crucial to find out the IP of your bulb. In most latest routers, they show what clients are connected to the router, the device name should appear as wiz_xxxxxx where xxxxxx is the starting mac address of the bulb.

If somehow your router does not supports showing the IP of connected devices, you'll have to do some hit and trial.

Get the IP of your PC/Laptop. You can run ip a if you are on Linux. My PC's IP is 192.168.0.100.
Turn off the light from the app and send the Hello Philips Wiz code described in next section to each of the IPs near to your PCs IP.

So I'd send following message to 192.168.0.95-105. How many machines you have to try depends on how many devices you have on your network.

We need the IP because we are going to communicate with bulb by sending and receiving UDP messages.

Hello Philips Wiz using UDP

package main

import "net"

var wizIP = "192.168.0.101"  // this is address on my bulb in my network
const wizPort = "38899"

func main() {
    c, err := net.Dial("udp", fmt.Sprintf("%s:%s", wizIp, wizPort)
    if err != nil {
        panic("could not connect to wiz light")
    }

    c.Write([]byte(`{"method": "setPilot", "params":{"state": true}}`))
}

Starting at line 5, first we have to define the IP of the bulb. It is 192.168.0.101 in my case.

Next we have to define the port on which Wiz light listens on. This port ID is constant and won't change unlike the wizIP.

Line 14 is the most important. The message is actually a JSON. Let's beautify it before discussing about it.

{
  "method": "setPilot",
  "params": {
    "state": true
  }
}

"method": The first key in the JSON object is method. This is either getPilot or setPilot depending on whether you want to get the state of the bulb, or set a state to the bulb. In our case, we are going to set some state. I have never really used the getPilot method, yet.
"params": Next in the JSON object is parameters we are sending; it is denoted by params in the JSON body. This is going to complement the method.

Now depending on what we are trying to do with the bulb, the params are going to change. In above code, we are just saying to set the state to true, which literally means to start the bulb.

If your bulb was off, it would have turned on after running the above code.

Similarly, if you want to turn it off, change the state to false and send the message again.

Set custom colors

Wiz accepts RGB colors values. If you don't know what that means, go to google and type in "rgb color picker", then choose a color and then note the RGB value. For example, value for pure red would be 255, 0, 0. Here 255 is the value of red, and 0 is for both green and blue.

To set Wiz to red color, this message can be sent:

{
  "method": "setPilot",
  "params": {
    "r": 255,
    "g": 0,
    "b": 0
  }
}

You have already seen how to send that message.

Set brightness

In the app UI, you must have seen the brightness slider? If not, consider the following screenshot.

First of all, I have warm white light selected. The top slider is for temperature, which we are going to talk about in next section.

The lowest value for brightness is 10, and max is 100. You guessed it right, lowest is not 0 because at 0, the bulb will be off.

You can club brightness setting with scene settings as well as any color setting. In following message, I'm setting the color to green and brightness to 50%.

{
  "method": "setPilot",
  "params": {
    "r": 255,
    "g": 0,
    "b": 0,
    "dimming": 50,
  }
}

Set temperature

Setting temperature is similarly easy. There is a params called temp. The range is between 2200 to 6200.

To the best of my knowledge, temperature can only be clubbed with brightness.

Here I'm presenting you an example payload for the message demonstrating 2200 kelvin temperate with a intensity of 20%.

{
  "method": "setPilot",
  "params": {
    "temp": 2200
    "dimming": 20,
  }
}

Conclusion

When I bought my Philips Wiz, I had no intention of controlling it programatically. It would such an overkill.

But shortly after this buy, I also bought a Raspberry Pi. I became aware that this programatic access can be integrated by home automation.

Libraries already exist in Python. One of them is pywizlight, which seems to be most mature. But there are many more.

Maybe we could write some similar library in Go.

First Month With My Raspberry Pi

Santosh Kumar — Fri, 17 Feb 2023 10:59:00 +0000

Introduction

It's no longer than a month since I bought a Raspberry Pi. Pi 4B to be precise. In this post, I'm going to log how I spent my 1st month with it.

Setup the Pi

When I decided to buy Raspi, I decided not to go with the kit. The reason was the model I was looking for, which is 8GB Model B wasn't available in India. And finding a kit for it would have been more challenging.

So I bought the Pi itself from OL Electronics. It cost me ₹10,700 INR with taxes and delivery charge.

As you might know, Pi does not ship with a power adapter. So I bought one from robu.in. It cost me ₹315 with delivery charges.

Pi also does not comes with a memory card. I tried searching for 64 micro sd cards offline and online. The best deal was from Amazon. It cost me ₹539. You will also need a card reader, which I bought from the local market at a price of ₹30.

While flashing, I choose to use the Lite variant of OS which comes without a desktop environment and has a footprint of 300 MB on disk.

Setting up the OS

The Pi board, a 5V 3A Type-C power adapter, a micro sd, and a card reader is all you need to boot up a Pi. You don't need a monitor or HDMI adapter if you already have a laptop because you can also SSH to your Pi to do your work remotely and that is how I usually work on my Pi.

As you know, I use Ubuntu on my machine. If you are also using Ubuntu, you can install it by:

sudo apt install rpi-imager

Instruction for other operating systems can be found on their GitHub page.

Put a micro sd card in the card reader and Please consider watching How to use Raspberry Pi Imager.

Blink Light

I prefer to use the gpiozero package for dealing with GPIO. But there is also an RPi module.

import time
from gpiozero import LED

led = LED(18)

while True:
    led.on()
    time.sleep(1)
    led.off()
    time.sleep(1)

Make sure you attach the long head of the LED to pin 18.

There is also an equivalent code using the RPi package:

import time
import RPi.GPIO as GPIO

led_pin = 18
GPIO.setmode(GPIO.BCM)
GPIO.setup(led_pin, GPIO.OUT)

while True:
    GPIO.output(led_pin, True)
    time.sleep(1)
    GPIO.output(led_pin, False)
    time.sleep(1)

I have also short a video I have posted on Instagram. Here it is:

Actuate on Proximity

Here I have used both packages to run something when the proximity sensor detects any object near the sensor.

The sensor I am using is an IR proximity sensor.

import time

from gpiozero import LED
import RPi.GPIO as GPIO

GPIO.setmode(GPIO.BOARD)
GPIO.setup(8, GPIO.IN)

led = LED(18)

while True:
    if GPIO.input(8) == 0:
        print("turning light on")
        led.on()
    else:
        print("turning light off")
        led.off()

    time.sleep(0.1)

Controlling LED brightness, 0-100 and back

Below is Python code to change the brightness of a LED light using Raspberry Pi.

import time
import RPi.GPIO as GPIO

led_pin = 18
GPIO.setmode(GPIO.BCM)
GPIO.setup(led_pin, GPIO.OUT)

pwm_led = GPIO.PWM(led_pin, 500)
pwm_led.start(100)

while True:
    start = 0
    end = 100
    while start < end:
        pwm_led.ChangeDutyCycle(start)
        time.sleep(.025)
        start += 1
    break

I don't shoot a video for this one because the intensity was so low that it was very hard to distinguish between different strengths of light.

Controlling an RGB light (failed)

I have also bought a common cathode LED. But somehow, I'm not able to turn on the LED. Not sure if the problem is in the LED, my code, or the pins.

Here is the code I was using.

from gpiozero import RGBLED

led = RGBLED(22, 27, 17)

led.color = (1, 0, 0)

And here is the setup I was using.

In the above image, the blue, green, and red are placed at 6, 7, and 8 on the left column, and the common cathode is placed at 3rd from the right.

Please let me know if you have a solution for this one.

Conclusion

This is not the entirety of the things I have done with my Raspberry Pi in the first month. I have a DHT11 sensor which I mistakenly bought without the IC. And now I need a 4.7K resistor to work with it. I am yet to order the resistor.

Alongside Raspi accessories, I have also bought some NFC tags to play with. I will post another blog if I make something interesting with it.

Installing Home Assistant is another thing that I have done on my Pi. This board is going to be the hub of other smart devices on my network. I'm planning to buy some ESF32 to help me with my home automation journey.

If you are also keen on home automation and IoT, I'd recommend you IoT for Beginners.

Let me know if you want more articles like this one.

How to Write Classes in Golang

Santosh Kumar — Thu, 16 Feb 2023 15:01:00 +0000

Introduction

I have seen many people coming from Python background struggling with Go. What I have obeserved it they write code that is limited to writing functions and calling them. They never leverage design constructs such as abstraction, encapsulation, inheritance, composition etc. I can understand their pain because I've been in the same boat.

In today's post I'll compare how we write classes in Python vs how we do the same thing in a Go way.

What is a class composed of?

Attributes in form of variables
Behavior in form of methods

Example in Python

Consider the following code in Python that defines a simple Person class with a name attribute and a greet method:

class Person:
    def __init__(self, name):
        self.name = name

    def greet(self):
        print(f"Hello, my name is {self.name}")

p = Person("John")
p.greet()  # prints "Hello, my name is John"

If you're a Python programmer looking to learn Go, you'll be happy to know that Go has a similar concept to classes called "structs". Structs are similar to classes in that they allow you to define a custom data type with its own fields and methods.

Struct to define the structure and attributes

To define a struct in Go, you use the type keyword followed by the name of the struct and a set of curly braces containing the fields. Here's an example of a simple struct called Person:

type Person struct {
    name string
    age  int
}

This defines a struct called Person with two fields: name and age, both of which are of type string and int respectively.

Method or behaviour using receiver function

You can also define methods on your structs by using the func keyword and attaching the method to the struct using the (p *Person) syntax, where p is the receiver of the method and Person is the type of the receiver. Here's an example of a method called SayHello that prints a greeting to the console:

func (p *Person) SayHello() {
    fmt.Printf("Hello, my name is %s and I am %d years old\n", p.name, p.age)
}

To create an instance of a struct, you can use the new function or the shorthand syntax :=. Here's an example of both:

// Using the new function
p1 := new(Person)
p1.name = "John"
p1.age = 30

// Using the shorthand syntax
p2 := Person{name: "Jane", age: 25}

You can also define a "constructor" function to create instances of your struct. This can be useful if you want to perform some initialization logic when creating a new instance. Here's an example of a constructor function for the Person struct:

func NewPerson(name string, age int) *Person {
    p := new(Person)
    p.name = name
    p.age = age
    return p
}

You can then use this constructor function to create a new instance of the Person struct like this:

p := NewPerson("Bob", 35)

I hope this tutorial has helped you understand how to define and use structs in Go as a Python programmer. Structs are a powerful and flexible way to define custom data types in Go, and I encourage you to explore them further.

Thank you for reading!

Testing HTTP Requests with Postman

Santosh Kumar — Wed, 15 Feb 2023 06:20:00 +0000

Introduction

In Postman Premier - The Minimal Postman you need to Learn, I posted the minimum of Postman you need to learn. In this article, I'm going to expand on that knowledge and dig more into the response. In this post, I'll be asserting the response on certain values and response codes and more.

What are assertions and how to use it

Assertions are nothing but test cases we give to a particular request. In the previous post, we already have seen that each of the requests has a Tests tab. There we write code in JavaScript, judging specific parameters of the response.

Let's first test for a specific status code. Just for context, while testing, I'm going to use my gingo server.

Looking for a specific status code

Testing for a specific status code is one of the easiest tasks to do within Postman. I'm going to use my GET {{baseUrl}}/books endpoint for this purpose.

When you open the Tests tab, you'll see snippets on the right side column. Click on Status code: Code is 200.

The code area will be populated with the following:

pm.test("Status code is 200", function () {
    pm.response.to.have.status(200);
});

When you send the request, you'll see Test Results (1/1) under the response section. That area shows how many of your tests have passed or failed.

I've recorded a gif for you.

This was for the positive assertions, you can send wrong data and do a negative assertion as well.

Looking for a specific string in the body

For this test, I'll be using the GET {{baseUrl}}/books/:isbn endpoint, with isbn's value to be 9781612680194.

Looking for a specific string in the body is not much different from checking for a status code. You'll find a snippet for it as well. Snippets are a great way to explore assertion in Postman.

Look for an assertion called Response body: Contains string. It will spit this source code into the test panel.

pm.test("Body matches string", function () {
    pm.expect(pm.response.text()).to.include("string_you_want_to_search");
});

I'm going to modify the body:

pm.test("Body includes Rich Dad Poor Dad", function () {
    pm.expect(pm.response.text()).to.include("Rich Dad Poor Dad");
});

Now I'm not going to include a gif for this as this would be very identical to the previous one.

Looking for a specific header

Another thing we can test in a REST API is headers. Some use cases to look for a header are:

When you are expecting a specific content type
When you are expecting an authentication token as a response to a request.

For a header, the testing code would look like this:

pm.test("Content-Type is application/json", function () {
    pm.response.to.have.header("Content-Type", "application/json");
});

Adding tests at the collection level

Response time is another thing to test if your API is time critical. This time we don't add this test on a request basis. We are programmers and we DRY. Checking for responses is going to be for each and every request, it's better to add it at the collection level.

To add a test at the collection level, click on the collection name on the left panel. As already discussed, you'd see Authorization, Pre-request Script, Tests and Variables.

Navigate to Tests, and put this in the code area:

pm.test("Response time is less than 500ms", function () {
    pm.expect(pm.response.responseTime).to.be.below(500);
});

500ms is good enough for local testing. If your server is over the internet, you might need to modify it to suit your need.

This test will run for every request which is there in the collection and for each new request, you add to the collection.

Run all tests from a collection

When you click on a collection, you'll see a Run button at the top right corner. When you click that, you'll be asked which tests you want to run.

This is the Collection Runner UI. When you are done with all the config. Click on the Run Postman Premier to run all the tests and see the results.

Conclusion

We have learned enough Postman to get our wheels rolling. There's much more to Postman than we have discussed in 2 articles on Fullstack with Santosh. Lots of features keep getting added in each release. You can keep yourself updated by following Postman Docs.

Postman Premier - The Minimal Postman you need to Learn

Santosh Kumar — Mon, 13 Feb 2023 15:32:27 +0000

Introduction

Welcome to my Fullstack with Santosh. Today we are going to learn about how to use Postman as a backend engineer to make our daily life easier. Postman is an API testing tool where you can test your REST APIs. It has now developed into handling many different kinds of interfaces, but today we are going to focus on the REST APIs.

Make sure you have Postman already installed. I work on Ubuntu and the way I install is sudo snap install postman. You can find instructions about your platform on the downloads page: https://www.postman.com/downloads/

When you open Postman for the first time, you might see a screen similar to this:

Your home screen might appear a little different because 1. I have already signed in. 2. I have already created some workspaces. 3. I have changed the theme to dark.

Create a collection

Before you even start to test endpoints in your server, the first thing you'd do is create a collection. Press that little + button on the left column.

Give it a name. I'm going to name it Postman Premier.

There are a lot more things that can be done with a Collection. But for now, let's not get deep into it.

Note: If you don't have an API to test, you can refer to my previous blog posts: Building a Book Store API in Golang With Gin and How to Integrate Swagger UI in Go Backend - Gin Edition where I show how you can develop your simple CRUD server in Go. It's worth a look.

Working with Requests

I am going to test my gingo server, which is a CRUD server I have developed before. I already have linked them above.

Creating a GET Request to get introduced to UI

I have already run my gingo server and I am going to make a GET request at http://localhost:8090/api/v1/books.

Creating a request is straightforward in Postman. Let's start with creating a GET request.

When you expand your created collection on the left column, you'll see a link called Create a request. Click on that, and a tab will open in the main column.

In that window, a text will already be selected. This is the identifier of the request. I like to name my request the same as the path of the endpoint. In my case, I'm going to name it /books.

The next thing you do is select a verb. There is a dropdown just below the name of the request. This will by default be GET. For this particular request, we don't modify that.

Next, we'll input the address of the endpoint we need to hit. I've input http://localhost:8090/api/v1/books.

We don't need any other setting for now. Don't be intimidated by the UI if you don't understand it yet.

I have recorded a gif for your reference. Here it goes.

What we covered in this section: Creating a request, request name, request verb, and request path.

Path Parameters

We can get all books. Let us get a specific book; by ISBN.

The quickest way to create our next request is to duplicate the previous one and modify the duplicated one.

Let us name this new request /books/:isbn. For the path, we're going to use localhost:8090/api/v1/books/:isbn. As soon as you write :isbn in the path input, Params section will be highlighted.

When you navigate to that tab, you'd see isbn is already added as KEY in Path Variables section. You need to provide value. You can get ISBN from the response of the previous endpoint we tested. E.g. 9781612680194.

When you do this much, you should see a single book in the response, unlike the previous response, which had all the books.

Below is a gif for this section.

Creating a POST Request and working with Body and Headers

Duplicate the first endpoint we created again, but this time, change the verb to POST.

In the previous section Params section was highlighted. This time you need to highlight the Body section.

When you highlight Body, you'll see a group of radio buttons. Click on the one labeled with raw. A dropdown will appear at the right with the text Text. Click on the dropdown and select JSON.

When you select JSON from the dropdown, a new header is added to the request. You can see the header when you highlight the Headers tab. In this tab, you can see many other headers already added by Postman.

When the header is set, use the following body to make a POST request:

{
    "isbn": "9780137081073",
    "title": "The Clean Coder",
    "author": "Robert C. Martin"
}

Here is a gif of me doing the same thing.

For those who don't know. Headers are not limited to POST requests. It might be required for a GET endpoint to have an auth token in its header.

Making requests for other verbs such as DELETE and PUT is no different. I left it over for you to try with those.

Working with variables

Using Collection variables

We have different variable scopes available when working with Postman:

Global > Collection > Environment > Local

They all have their use case:

Global - Global variables are global to all the collections you have in Postman. This should be mostly avoided.
Collection - This one is my favorite. You can have collection-level variables for say credentials, baseurl, etc.
Environment - If you are testing a collection with multiple data sets or conditions. Environment variables come to use.
Local - Local variables are local to request and will override all other scopes variables.

I'll demonstrate Collection variables, but all behave the same way and have similar UI.

Click on the Postman Premier collection and you'll see 4 tabs. Authorization, Pre-request Script, Tests and Variables. Click on the variables.

Collection Variables use case: We are doing a lot of repetitions in each of the requests. We are inputting the base URL of the server each time. If we had to somehow modify the port address, we would have to modify it for all.

Let us change that to use a variable. If we ever need to change the value of the baseurl, we'll only be changing that in a single place.

When you see the variables tab, you'll see a table with 3 columns. Namely VARIABLE, INITIAL VALUE, and CURRENT VALUE. Set VARIABLE = baseUrl, INITIAL VALUE = http://localhost:8090/api/v1, CURRENT VALUE = http://localhost:8090/api/v1. INITIAL_VALUE and CURRENT_VALUE are the same here, but CURRENT_VALUE can be programmatically changed during request-response life cycle.

Now change all the occurrences of http://localhost:8090/api/v1 in the collection's URL section to {{baseUrl}}. This will be a manual process, but this is worth it.

Saving response data in variables

Why would you want to save response data in a variable?

That is a good question to ask. One such use case is user login. When you log in to a service, you'd get some kind of token with a certain validity time. That token is needed to make further requests. We can add a header using Pre-request script using the variable we save.

Our gingo server does not have a user authentication module right now. Let us try saving isbn from a GET request.

This may sound a little bit misleading, but anything you want to do after a request goes inside the Tests tab. This also includes setting a variable for a session.

Go to the Tests section of /books/:isbn and add this javascript code:

const body = pm.response.json()
pm.collectionVariables.set("isbn", body.isbn);

The code is straightforward. We are storing the JSON response in a const called body. Then we access the collection variable using pm.collectionVariables. We then set the variable name isbn with the value of body.isbn.

Here is me doing the same thing.

Similar to collection variables, we can set any scope of variable we discussed above.

Conclusion

In this post, we saw some basic use cases of Postman. In upcoming posts, we'll see more advanced use cases. Until then, subscribe and stay updated.

Best Logging Library for Golang

Santosh Kumar — Sun, 12 Feb 2023 11:24:47 +0000

Introduction

I recently conducted a poll in the Go community regarding which logging library they use in their personal or professional project. The results were interesting. In this post I have jotted down the most favorite logging libraries and here is my opinion about them.

Poll Results

Here are the poll results which I conducted on r/golang, The Go Programming Language community on Twitter and my LinkedIn following.

Analysis of Results

Selection of the candidates

For choosing the candidates for the poll, I didn't do any thorough research. I was looking for a library to use in my project at work, and I ended up at sirupsen/logrus which was already being used by one of the dependencies in that project.

After reading the README of the logrus, I noted it was in maintenance mode and no new feature was actively being developed. I still decided to use this library because logrus is a mature project and it has all the features which a logging library should have.

logrus README recommended using other libraries such as Zerolog, Zap, and Apex.

I started a poll on r/golang with these four candidates, but also came to know about glog which was a go port of a C++ project by Google. I used that option in the poll conducted on LinkedIn.

I haven't personally used any one of them yet.

logrus

Let's talk about logrus first, it was the first library I landed upon when looking for one. The reason I looked at it first is that logrus was already one of the indirect dependencies of one of the dependencies in my project.

logrus stands 2nd in the place. zerolog is faster than logrus but there's a reason it stands seconds among 4. It has been there for a while.

zerolog

The main reason you'd want to use zerolog if you are switching from logrus is the performance. Go is blazingly fast; but if your logger is making it slow, your application gets no benefit from being implemented in Go.

This is what zerolog's README says:

The zerolog package provides a fast and simple logger dedicated to JSON output.

Zerolog's API is designed to provide both a great developer experience and stunning performance. Its unique chaining API allows zerolog to write JSON (or CBOR) log events by avoiding allocations and reflection.

zap

zap is probably the most loved logging library for Go. Among 817 people who have voted on r/golang, 315 voted for zap.

According to the stats on their GitHub, zap is the fastest logger available there.

Also, zap only supports the two most recent minor versions of Go. So you might be locked with an older version of zap if your project for some reason is not able to update to the latest version of Go.

Conclusion

To have an opinion, I feel that I use them all before coming to a conclusion. But after this poll, I think I'm going to start using zap right away. What is your opinion? Why factor was responsible when you were choosing your logging library.

Release Dangling Elastic IPs using Lambda and Go SDK

Santosh Kumar — Thu, 15 Sep 2022 05:42:55 +0000

Introduction

Today we are going to do a hands-on on how to kill a dangling EIP or in other words, deallocate an unassigned Elastic IP. But before that, let us know...

What is an Elastic IP and why do we need it?

Elastic IPs are simply IP addresses provided by Amazon AWS. These are usually attached to internet gateways such as load balancers, or even EC2 instances. I have been using it with EC2 instances directly for my testing use case.

Normally when you configure your EC2 instance to have a public IP but at the same time, you don't use EIP. The problem is that when you stop and start your instance, your IP address is changed.

This is where Elastic IP shines. You attach an EIP to the EC2 instance and when you stop and start your instance, your IP won't be changed.

This is extremely important if the EC2 you are running is hosting an internet-facing web server. Web servers shouldn't generally frequently change their IPs.

But why do we need an EIP killer?

There is 1 good thing about EIP and 1 bad thing. The good thing is EIP is free to use. You don't need to pay anything for the static IP you are using. How cool!

The bad thing is, that when EIP is not attached to an instance, it cost you money. How warm!

It won't cost you much if you leave it detached for a few hours. But as a human, we tend to forget if EIP is in use and only get to know about it when we get our monthly bill.

In the Asia Pacific (Mumbai) region, if I forget to kill an EIP for a month, I get around a $4 bill. This was my main motivation to write this article.

With an EIP killer, I will be able to use Elastic IP with confidence by releasing it when not in use.

Get Started

What we'll work with:

AWS Lambda
Go
AWS Go SDK
AWS CLI

What we're going to do:

Run a Lambda on a predefined interval.
This Lambda will use AWS SDK to check if any dangling EIP is lying around.
If dangling EIP is found, deallocate, or in our language, kill it.

Initially, I was going to use Python for the Lambda runtime, but Go is faster than Python. We are going to benefit from this decision as we can run our function for a smaller time span. This is important because we plan to run our Lambda at frequent intervals.

With this post, I mostly plan to expand my knowledge in the field of serverless computing. I intend to touch Lambda, EventBridge, and related constructs.

Phase 1: Provision a Lambda function

The main task in this phase is to manually create a function. In the second phase, we'll see what is the logic for killing the EIPs. Let's get started.

Go through the web UI and choose to create a function.
Keep "Author from scratch" selected.
Fill in the name. I liked to call it "eip-killer".
Select the runtime to be Go 1.x.
Keep everything else on default. Hit Create function.

Note: Creating Lambda with default settings also creates an IAM role. This IAM role dictates how this Lambda interacts with other AWS services. By default, this role allows Lambda to talk with CloudWatch to put logs. You can add more policies to this role when working with other AWS services.

Let's go ahead and do a quick invoke to test if the function is working.

Our EIP-killer is alive. But doesn't know how to kill. Let's teach it in the next phase.

Phase 2: Write the EIP killing logic

While testing, you might have got a view of the Code tab of the function which says, "The code editor does not support the Go 1.x runtime". What this means for us developers is that we have to use our text editor to write the logic and upload the pre-built binary to Lambda.

The prime documentation for Go AWS SDK is https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html, but don't worry. I'll filter out the required material required for this post.

Phase 2.1: The initial deployment process

Before we write the actual logic, let's write a simple "Hello EIP Killer" code to get the awareness of the process required for uploading the code.

Create a workspace for our lambda code.

mkdir eip-killer
cd eip-killer

Initialize a go module in the folder.

go mod init EIP-killer

You could also put the GitHub/GitLab URL of the repo where you intend to put your lambda code at.

Create a main.go and put this code into it:

package main

import (
    "github.com/aws/aws-lambda-go/lambda"
)

func hello() (string, error) {
    return "Hello EIP Killer!", nil
}

func main() {
    // Make the handler available for Remote Procedure Call by AWS Lambda
    lambda.Start(hello)
}

As we are using external dependency, get it with this command:

go get github.com/aws/aws-lambda-go/lambda

Build a binary out of main.go:

GOOS=linux GOARCH=amd64 go build -o main main.go

Remember while creating the function we had selected architecture to be x86_64. So we need to reflect that with GOARCH=amd64.

Zip the binary. Upload the zip to function.

zip main.zip main

Upload using this command:

aws lambda update-function-code --function-name eip-killer --zip-file fileb://main.zip

While this could have been enough, the default handler (where lambda begins executing) in function with factory settings is hello. But our code uses main. So we need to update the handler config to match that.

We need this update only once.

aws lambda update-function-configuration --function-name eip-killer --handler main

After this, when you run the func:

You can roam around this path: https://github.com/aws/aws-lambda-go if you want to dig deeper into Go+Lambda. But for now, I'm going to use AWS SDK to do our magic.

Phase 2.2: Logic to deallocate unused EIPs

Open up the main.go again and write this:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/ec2"
    "github.com/aws/aws-sdk-go-v2/service/ec2/types"
)

type MyEvent struct {
    Name string `json:"name"`
}

func Handler(ctx context.Context, name MyEvent) (string, error) {
    // Load the Shared AWS Configuration (~/.aws/config)
    cfg, err := config.LoadDefaultConfig(ctx)
    if err != nil {
        log.Fatal(err)
    }

    // Create an Amazon EC2 service client
    ec2ServiceClient := ec2.NewFromConfig(cfg)

    // DescribeAddressesInput with no filter means list all addresses
    IpListFilter := &ec2.DescribeAddressesInput{}

    // Ask the EC2 client to list all Elastic IPs.
    result, err := ec2ServiceClient.DescribeAddresses(context.TODO(), IpListFilter)
    if err != nil {
        fmt.Println("Got an error retrieving information about your Amazon Elastic IPs:")
        return "", err
    }

    // Out of all IPs, check if they have an allocation ID. If not, they are a candidate for deletion
    for _, a := range result.Addresses {
        fmt.Println("Allocation ID: " + *a.AllocationId)
        fmt.Println("Public IP: " + *a.PublicIp)

        // if this EIP is not associated with any AWS resources, proceed with killing.
        if a.AssociationId == nil {
            // extra step for flexibility
            if isThisEIPKillable(a.Tags) {
                fmt.Println("About to be killed")
                // Release/kill the Elastic IP
                ReleaseAddressFilter := &ec2.ReleaseAddressInput{AllocationId: a.AllocationId}
                ec2ServiceClient.ReleaseAddress(context.TODO(), ReleaseAddressFilter)
                fmt.Println(*a.PublicIp + " is not more")
            } else {
                fmt.Println(*a.PublicIp + " has KILLERHOLD enabled and is going to stay")
            }

        } else {
            fmt.Println("Association ID: " + *a.AssociationId)
            fmt.Println("Instance ID: " + *a.InstanceId)
            fmt.Println("PrivateIpAddress: " + *a.PrivateIpAddress)
        }

        fmt.Println("")
    }
    return "Done execution", nil
}

// isThisEIPKillable scans to tags of Elastic IP and only returns false when
// EIPKILLER is tag is set to true.
func isThisEIPKillable(tags []types.Tag) bool {
    for _, tag := range tags {
        if *tag.Key == "KILLERHOLD" && *tag.Value == "true" {
            return false
        }
    }
    return true
}

func main() {
    lambda.Start(Handler)
}

Quite a large source code. Let us understand it line by line.

Line 14-16 declares an event struct. This struct is going to be a parameter to the Handler function.

On line 18, we see function with signature Handler(ctx context.Context, name MyEvent) (string, error). Handler here is the name of the function.

The first parameter to the handler is the context object. If you are working with web services in golang, you must be aware of that.

Next is our MyEvent which is defined above. This is as per Lambda's spec.

Our handler function also returns a string and an error. Which is then bubbled to Lambda logs.

Line 20 deals with configuration. Please note that we are using AWS SDK. And the SDK can run on platforms other than Lambda. On the local system, the default config is located at ~/.aws/config. But on Lambda, it automatically loads the region it's running in.

On line 26 we create a new EC2 client to interact with our AWS account. We initialize this client with the config instance we created on line 15.

On line 29 we have initialized a filter for IPs. This seems to be an added step. But this variable is going to be a mandatory parameter for our next command.

On line 32 we get a list of all Elastic IPs associated with our account. There are no filters to the result as we have not specified any filter on line 29.

It's time to look at the isThisEIPKillable function which is from 69 to 76. This is an extra feature for flexibility. This will forgive any EIP which has the tag KILLERHOLD set to true. So when you have an EIP you don't want to kill, you set this tag to it. This is good for the experiment as our Lambda will be running on a 1 hours (configurable by user) interval and it might not be desired outcome to delete them every 1 hour.

Let us get back to our main flow. We have a result variable with all the addresses in it. On line 40 we check if the IP is associated with any EC2 or not. Then we run the helper function we just discussed. If all is okay, meaning that KIP can be released, we go ahead and release the address back to the AWS pool.

On lines 57-59, if EIP is associated with any EC2, we print some metadata about the EIP association.

That's it. It was not that hard to understand, isn't it? Before we deploy it, let's create some EIP and not allocate it to any EC2. We'll also have some EIP having KILLERHOLD set to true.

Phase 2.3: Test our EIP killer Give Lambda permission to release and read EIPs

I hope you have updated your lambda function and pushed the zip file to the function. Review Phase 2.1 if you have not. Continue reading if you did.

Before I test it with actual EIPs, I wanted to show something else. Right now if you try to test the function, you'd see something like this:

If you clearly read the error message, it says that API call returned 403. Meaning that Lambda was not authorized to make the call to EIP APIs.

To overcome this problem, we need to give access to Lambda to read some metadata from EC2.

When no Lambda Web UI for the function. If you go to Configuration > Permission > Execution role as described in the image below. You'll see a role that was created while we created the function.

Click on that and you'll be taken to the IAM page for that role.

When on that page, look for a dropdown named Add permissions and click on Create inline policy.

Switch from Visual editor to JSON. And paste this policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:ReleaseAddress",
                "ec2:DescribeAddresses"
            ],
            "Resource": "*"
        }
    ]
}

As you can see, in the Action section, we have added permission for ec2:ReleaseAddress, and ec2:DescribeAddresses.

Review policy and give it a name, and then Create policy.

When done, you'd see this policy in the list of policies attached to the role.

Phase 2.3: Test our EIP killer

We are not set for the actual testing.

We are going to allocate 3 Elastic IPs for our testing purpose.

One we'll attach to the EC2 instance.

The second will be dangling.

And the third one will be also dangling but will have our KILLERHOLD set to true.

To actually invoke the function (apart from testing it), one way is to create a public URL. We can also invoke through the AWS CLI, but I'm not choosing that for now. You can do it.

Then I went ahead and opened that URL to execute the function.

You can configure AWS API Gateway in front of the function. But that's out of the scope of this post.

Next, we see the result of the invocation.

That's the end of Phase 2. We can kill EIPs now. How wonderful.

Phase 3: Trigger Lambda at a 1-hour interval

The main part of the EIP killer is already developed. Let's now execute it at a 1-hour interval as planned before.

Phase 3.1: Configure EventBridge aka CloudWatch Events

We are going to emit an event at our 1-hour interval and we are going to use EventBridge for this.

I have recorded a screencast gif to preserve some space on this post.

Once done, this event should appear in the Trigger section of the Lambda function; as seen in the below image

Phase 3.2: Create an EIP and go to sleep

As I'm writing this post at 4 am. I'm going to create an EIP and go to sleep. Will check in the morning if our setup is working.

A few hours later...

I woke up this morning with only 1 EIP.

And this is how we gonna achieve this killing at 1 hour. You can configure the interval, but I think this is optimal. You also don't need to have a public IP for this. So if you have, you can delete it now from the web interface.

How to Integrate Swagger UI in Go Backend - Gin Edition

Santosh Kumar — Mon, 30 May 2022 07:08:42 +0000

Introduction

Unlike FastAPI, Gin does not have OpenAPI integration built in. With FastAPI when you add a route, documentation is already generated. With Gin, this is not the case. But recently I integrated Swagger UI in one of my Go backends and I wanted to document that process.

Prerequisites

An already existing Gin server

You might consider Building a Book Store API in Golang With Gin if you are starting from scratch. I'll be using the same book store and extending over it to integrate Swagger UI.

How Swagger works with Gin

The way Swagger works with other Go backend is not different they all have the same mechanism. But how does it really work and what do we have to really do?

Swagger uses the Go comment system which is very well integrated with documentation already as we know. We write comments in a pre-defined way, details of which we will see further ahead in the post. But mostly it is divided into 2 parts. The server itself and the routes.

Swagger has a CLI binary which when runs converts these comment documents into OpenAPI compliant documentation. The resultant file also includes OpenAPI server specs in JSON and YAML format.

Lastly, we route the generated content via a handler in our backend.

How do we do that? Let's see.

Setup Swagger CLI

The prime repository you should keep under your pillow is https://github.com/swaggo/swag. Both in terms of CLI and documentation.

We'll install a CLI application called swag. Here's how:



go get -u github.com/swaggo/swag/cmd/swag

# 1.16 or newer
go install github.com/swaggo/swag/cmd/swag@latest

The first command will download the dependencies to integrate in the server application.

The second one is where we install the CLI.

Now you should be able to run the swag command:



$ swag -h
NAME:
   swag - Automatically generate RESTful API documentation with Swagger 2.0 for Go.

USAGE:
   swag [global options] command [command options] [arguments...]

VERSION:
   v1.8.1

COMMANDS:
   init, i  Create docs.go
   fmt, f   format swag comments
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help (default: false)
   --version, -v  print the version (default: false)

Documenting the Gin Server

When you look at Swagger UI documentation for any OpenAPI enabled website, you'd see something like this:

You might be familiar with the low half part of the UI. This is what Swagger docs are for i.e. to document the routes.

But before we deal with the lower 50% part I want you to notice in the above picture is the top 50% part. Right from the Swagger Petstore header to Find out more about the Swagger anchor link.

This part shows the metadata about the API server itself. In this section, we are going to build that part. You can find the code to start with here. We'll be continuing on that code.

First of all, we need to go get the packages we need to work with:

go get -u github.com/swaggo/files
go get -u github.com/swaggo/gin-swagger

Now after we have done that, let's look at our main.go file. This is the main.go at the moment:



package main

import "github.com/santosh/gingo/routes"

func main() {
    router := routes.SetupRouter()

    router.Run(":8080")
}

Add a route for Swagger Docs

We add a new router in the main function:

router.GET("/docs/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))

The modified file would look like this:



-import "github.com/santosh/gingo/routes"
+import (
+       "github.com/santosh/gingo/routes"
+       swaggerFiles "github.com/swaggo/files"
+       ginSwagger "github.com/swaggo/gin-swagger"
+)

 func main() {
        router := routes.SetupRouter()

+       router.GET("/docs/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))
+
        router.Run(":8080")
 }

swaggerFiles.Handler here is the handler which has all the assets embedded in it. Assets like the HTML, CSS, and javascript files are shown on the documentation end.

ginSwagger.WrapHandler is a wrapper around http.Handler, but for Gin.

This should be enough for testing. Let's see how the docs render at <server address>/docs/index.html.

The good news is API documentation site is up. The bad news is it does not look like a normal API documentation site. What did we miss?

Generate swagger docs every time you modify doc string

Swagger documentation is stored in Go's own docstring. We have a special syntax that we follow. More on that later. But first, let us learn how to generate docs.



$ swag init
2022/05/25 23:59:16 Generate swagger docs....
2022/05/25 23:59:16 Generate general API Info, search dir:./
2022/05/25 23:59:16 create docs.go at  docs/docs.go
2022/05/25 23:59:16 create swagger.json at  docs/swagger.json
2022/05/25 23:59:16 create swagger.yaml at  docs/swagger.yaml

We run swag init every time we update docs for our API. This generates 3 files inside a sub directory called docs/.



$ tree docs
docs
├── docs.go
├── swagger.json
└── swagger.yaml

0 directories, 3 files

swagger.json and swagger.yaml are the actual specification which you can upload to services like AWS API Gateway and similar services. docs.go is a glue code that we need to import into our server.

Let us now import the documentation to our main.go.



 package main

 import (
+       _ "github.com/santosh/gingo/docs"
        "github.com/santosh/gingo/routes"
        swaggerFiles "github.com/swaggo/files"
        ginSwagger "github.com/swaggo/gin-swagger"
 )

+// @title     Gingo Bookstore API
 func main() {
        router := routes.SetupRouter()

As you can see we have imported docs the module by giving the full path of the module. We also have to prepend this import using _ because it is not explicitly used in main.go file.

You might also have noticed // @title Gingo Bookstore API line. Please note its position as this is important. It is just above the main() func. Also @title is not random here. It is one of the keywords which is documented under General API Info. We'll see more of these as we go, but for now, let's regenerate our docs and review the API docs.

Looks like we are getting somewhere. :D

Add General API Info to Gin API Server

We saw @title annotation in the last section. It is used to set the title for the API server. But it is not the only annotation available. There is a wide array of annotation in Swagger. You can find them in use in real life here.

I'm going to use some of them to construct metadata on my doc site.



// @title           Gin Book Service
// @version         1.0
// @description     A book management service API in Go using Gin framework.
// @termsOfService  https://tos.santoshk.dev

// @contact.name   Santosh Kumar
// @contact.url    https://twitter.com/sntshk
// @contact.email  sntshkmr60@gmail.com

// @license.name  Apache 2.0
// @license.url   http://www.apache.org/licenses/LICENSE-2.0.html

// @host      localhost:8080
// @BasePath  /api/v1

Run swag init.
Re-run the server.
Check for update:

That's some metadata in there.

You might also have noticed that the API spec inside docs/ directory has changed. For example, here is the swagger.yaml file from the docs/ dir.



 {
     "swagger": "2.0",
     "info": {
-        "title": "Gingo Bookstore API",
-        "contact": {}
+        "description": "A book management service API in Go using Gin framework.",
+        "title": "Gin Book Service",
+        "termsOfService": "https://tos.santoshk.dev",
+        "contact": {
+            "name": "Santosh Kumar",
+            "url": "https://twitter.com/sntshk",
+            "email": "sntshkmr60@gmail.com"
+        },
+        "license": {
+            "name": "Apache 2.0",
+            "url": "http://www.apache.org/licenses/LICENSE-2.0.html"
+        },
+        "version": "1.0"
     },
+    "host": "localhost:8080",
+    "basePath": "/api/v1",
     "paths": {}
 }

Now it's time to add docs for API endpoints.

Add API Operation Info to API Endpoints

Just like General API Info for the main server, there also is API Operation for individual routes/endpoints handlers.

They are more diverse than what I'm going to use here. The ones I'm going to use are just a subset of them. If you want to check out some real examples, you can find them here on each of the route handlers.

Here is modified handlers/books.go:



        "github.com/santosh/gingo/models"
 )

-// GetBooks responds with the list of all books as JSON.
+// GetBooks             godoc
+// @Summary      Get books array
+// @Description  Responds with the list of all books as JSON.
+// @Tags         books
+// @Produce      json
+// @Success      200  {array}  models.Book
+// @Router       /books [get]
 func GetBooks(c *gin.Context) {
        c.JSON(http.StatusOK, db.Books)
 }

-// PostBook takes a book JSON and store in DB.
+// PostBook             godoc
+// @Summary      Store a new book
+// @Description  Takes a book JSON and store in DB. Return saved JSON.
+// @Tags         books
+// @Produce      json
+// @Param        book  body      models.Book  true  "Book JSON"
+// @Success      200   {object}  models.Book
+// @Router       /books [post]
 func PostBook(c *gin.Context) {
        var newBook models.Book

@@ -28,7 +41,14 @@ func PostBook(c *gin.Context) {
        c.JSON(http.StatusCreated, newBook)
 }

-// GetBookByISBN locates the book whose ISBN value matches the isbn
+// GetBookByISBN                godoc
+// @Summary      Get single book by isbn
+// @Description  Returns the book whose ISBN value matches the isbn.
+// @Tags         books
+// @Produce      json
+// @Param        isbn  path      string  true  "search book by isbn"
+// @Success      200  {object}  models.Book
+// @Router       /books/{isbn} [get]
 func GetBookByISBN(c *gin.Context) {
        isbn := c.Param("isbn")

Run swag init.
Re-run the server.
Check for updates:

And here is the updated swagger.yaml:



 basePath: /api/v1
+definitions:
+  models.Book:
+    properties:
+      author:
+        type: string
+      isbn:
+        type: string
+      title:
+        type: string
+    type: object
 host: localhost:8080
 info:
   contact:
@@ -12,5 +22,58 @@ info:
   termsOfService: https://tos.santoshk.dev
   title: Gin Book Service
   version: "1.0"
-paths: {}
+paths:
+  /books:
+    get:
+      description: Responds with the list of all books as JSON.
+      produces:
+      - application/json
+      responses:
+        "200":
+          description: OK
+          schema:
+            items:
+              $ref: '#/definitions/models.Book'
+            type: array
+      summary: Get books array
+      tags:
+      - books
+    post:
+      description: Takes a book JSON and store in DB. Return saved JSON.
+      parameters:
+      - description: Book JSON
+        in: body
+        name: book
+        required: true
+        schema:
+          $ref: '#/definitions/models.Book'
+      produces:
+      - application/json
+      responses:
+        "200":
+          description: OK
+          schema:
+            $ref: '#/definitions/models.Book'
+      summary: Store a new book
+      tags:
+      - books
+  /books/{isbn}:
+    get:
+      description: Returns the book whose ISBN value matches the isbn.
+      parameters:
+      - description: search book by isbn
+        in: path
+        name: isbn
+        required: true
+        type: string
+      produces:
+      - application/json
+      responses:
+        "200":
+          description: OK
+          schema:
+            $ref: '#/definitions/models.Book'
+      summary: Get single book by isbn
+      tags:
+      - books
 swagger: "2.0"

Notice the new definitions and paths section added. It is for models and the endpoints respectively.

Bonus: Router Group for /api/v1

It is always a good idea to prepend your routes with api/v1. v1 bit has a logic behind it. It is for the time when you have to introduce a breaking change which is not backward compatible. Maybe the input or the output has changes that might break millions of dependent client.

In these situations, you increment the version to something like api/v2 and let the older API server serve old from the old handler.

Right now all of the routes in gingo server start with /book. We are going to change that.

Here is the modified routes/routes.go.




 func SetupRouter() *gin.Engine {
        router := gin.Default()
-       router.GET("/books", handlers.GetBooks)
-       router.GET("/books/:isbn", handlers.GetBookByISBN)
-       // router.DELETE("/books/:isbn", handlers.DeleteBookByISBN)
-       // router.PUT("/books/:isbn", handlers.UpdateBookByISBN)
-       router.POST("/books", handlers.PostBook)
+
+       v1 := router.Group("/api/v1")
+       {
+               v1.GET("/books", handlers.GetBooks)
+               v1.GET("/books/:isbn", handlers.GetBookByISBN)
+               // router.DELETE("/books/:isbn", handlers.DeleteBookByISBN)
+               // router.PUT("/books/:isbn", handlers.UpdateBookByISBN)
+               v1.POST("/books", handlers.PostBook)
+       }

        return router
 }

Here we use router.Group to create a group with a path of /api/v1. We then move all the route definitions into the group and surround it with braces. That is how we create a path route in Gin.

Conclusion

API documentation is an essential part of API documentation. Instead of documenting the endpoints anywhere else, we can document the routes right in the code. That way we only have 1 single source of truth. No need to maintain code and documentation separately. In turn, we get always up-to-date documentation.

Every backend server has some sort of support for Swagger UI. I have covered the basics for Gin, but if you use any other framework, I encourage you to look for your own framework.

gRPC for Absolute Beginners, in Go

Santosh Kumar — Sat, 28 May 2022 15:43:57 +0000

Introduction

The Internet has evolved in the last 2 decades a lot. HTTP/1.1 was not enough so we have HTTP2 now. The specification we used to transfer data between the client and the server has also evolved. From XML to JSON, now we have Protocol Buffer, which is a binary spec. Let's dive deeper.

The picture below from Wikipedia shows how exponential data is growing.

Prerequisites

Go
HTTP2

Enough HTTP2 to know gRPC

During the early days of the web, we only had static content, hardly text files, and HTML files. After then, the web grew a bit and the web server and clients started to deal with rich media like rich texts, images, and videos. Servers also started to serve dynamic content based on what clients request.

This was the start of the RPC era. A client would hit a certain endpoint with certain data, and the server's work was to respond to that request. In the early time of RPC, XML was extensively used. We still use XML in some systems. But most of the world has moved to JSON as a communication format between the server and the client.

Now, over time, everything has evolved. One of the things which evolved is HTTP spec. Now we have HTTP2. HTTP2 is the base of gRPC. Certain properties make HTTP2 a nurturing ground for gRPC.

HTTP/1.1	HTTP/2
Header is plaintext and not compressed	Header is compressed and binary
Spawns a new TCP connection on each request	uses already existing TCP connection
TLS is not required	Requires TLS by default, enhanced security

Let's learn about Protocol Buffer first

We have seen how HTTP2 is essential for gRPC, now let's see where Protocol Buffer stands.

Although you can use gRPC with JSON, Protocol Buffer brings new things to the table. We saw that for a long time, JSON was used to send data back and forth between servers and clients. Now, what has happened is that we have taken yet another step to move from JSON to its successor.

Protocol buffers are building block of gRPC and is a replacement for JSON. Protocol Buffer inherits a lot from HTTP/2.

JSON	Protocol Buffer
Plaintext	Binary
Larger payload over wire	Smaller payload over wire
Write your server	.proto files can create server stub

As we already know, HTTP/2 is binary. So are protocol buffers. Now you don't need to pass a date as a string in JSON. You can pass BSON over the wire.
It is easy on a network as we only use the space we need to use. Say int32 only uses 4 bytes of data. The same data in JSON (string) could have used multiple bytes for a longer integer value.
We write .proto files. the proto compiler generates stub files which can be used to write the server as well as the client. The key takeaway here is we can use the same proto files to generate clients and servers in many different languages.
Protocol Buffers are agnostic to the language you are working with to develop your server or the client. .proto files have their syntax and data types which convert to specific data types in a destination programming language. This is my favorite reason to use gRPC in my projects.

I recommend you to read through the proto syntax, keywords, and data types here: https://developers.google.com/protocol-buffers/docs/proto3

What is gRPC?

Ever heard of RPC? It stands for Remote Procedure Call. It's an old way of running a remote procedure on a remote machine. Let me make it a little familiar for you. When you hit an endpoint from a frontend to a backend, you are making a remote call or a remote procedure call.

SOAP and REST both are an example of RPC. You can send data in the body and hit an API on the other end. That's how it has been happening since the start.

gRPC is a continuation of that SOAP and REST. gRPC brings all the advancements that their ancestors can't. With gRPC, a client application can directly call a method on a server application on a different machine as if it were a local object.

Types of Service Methods in gRPC

We'll practically see what service methods are when we do hands-on with code. But right now I want to specify that in gRPC we have 4 kinds of method definitions.

Unary - Unary is similar to a normal REST call. A client initiates a TCP connection, sends a message, waits for the server to respond, and finally, the server responds.
Server Streaming - Server streaming RPCs where the client sends a request to the server and gets a stream to read a sequence of messages back. For example, you searched for a keyword. Instead of returning a static page, Twitter returns a stream of tweets, including whatever is being tweeted in real time.
Client Streaming - Client streaming RPCs where the client writes a sequence of messages and sends them to the server. Once the client has finished writing the messages, it waits for the server to read them and return its response. Again gRPC guarantees message ordering within an individual RPC call. An example of it would be an IoT device (e.g. a car) streaming its device location to the central server (e.g. Uber).
Bidirectional Streaming - Bidirectional streaming RPCs where both sides send a sequence of messages using a read-write stream. The two streams operate independently, so clients and servers can read and write in whatever order they like. An example of this would be a chat application. But I know there are more complex use cases available in the wild. Please let me know in the comments if you do so.

For a detailed explanation of what happens when a gRPC client calls a gRPC server method, please consider reading RPC life cycle.

Developing with gRPC

We have had enough of theories. Let's develop some code to see this thing in action.

We are going to have an example in Go. Although as we already know, the proto files have a language of their own which is used to generate code in multiple languages.

We are going to write a simple calculator app.

Write the proto file

.proto files are contracts between the server and the client. This is similar to the REST API you have already experienced with.

Before I proceed further, I'd like to inform you that having the whole code into a module will make your life easy. That's the reason, I've created a go module at the root of the directory named github.com/santosh/example.

calculator/calculator.proto

syntax = "proto3";

option go_package = "github.com/santosh/example/calculator";

package calculator;

// The calculator service definition.
service Calculator {
    // Adds two number
    rpc Add(Input) returns (Output);
}

// Input message containing two operands
message Input {
    int32 Operand1 = 1;
    int32 Operand2 = 2;
}

// Output message containing result of operation
message Output {
    int32 Result = 1;
}

Explanation

Protocol buffer data is structured as messages, where each message is a small logical record of information containing a series of name-value pairs called fields. Line 7-10 is an example of a message. So is 12-14. Messages are comprised of data types, identifiers, and index positions.

You'd also note messages are composed inside Services. Service is simply what this web service does. We currently have a Calculator service from lines 3-5. Calculator service comprises of a method called Add. Add takes 2 parameters as defined in Input message and emits an Output message.

Generate code from a proto file

You'd need a protoc compiler to generate code from proto files. If you are on Debian based system, you can use sudo apt install protobuf-compiler. If you are on any other OS, please read Protocol Buffer Compiler Installation

I am going to use Go for this tutorial, so I'm going to install the Go plugin for the protoc compiler.

go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2

Now with everything in place, I'd issue this command from the root of the module:

protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative calculator/calculator.proto

If this command is successful, this would generate 2 more go files.

$ tree calculator
calculator
├── calculator_grpc.pb.go
├── calculator.pb.go
└── calculator.proto

0 directories, 3 files

If this is your first time with gRPC, and you look at the generated code now, you'd probably be lost.

We are going to use both files to create our server and the client. The generated code will start to make sense.

Implement calculator server and the client

We all know that the output generated by the protoc compiler is a stub. We need to use that stub as a guide to implementing/writing our client and the server code. The stub work as a guideline for us.

Right now, the generated code lives inside calculator package in calculator.pb.go which mostly deals with data part (i.e. Input, Output, Operand1, Operand2, their getters etc) and calculator_grpc.pb.go which mostly deals with method implementation (i.e Add in both server and client). The first thing in both server and client code is to import this package.

I will keep referencing the proto file as we define our client/server code.

Write gRPC calculator server

If you look into calculator_grpc.pb.go, you'd find that there is a struct called UnimplementedCalculatorServer. This struct represents our server. Now there is a reason it is named Unimplemented. If you look at methods attached to this struct, you'd see a method named Add. This is the same method we defined in our proto file. Here is a refresher:

// Adds two number
rpc Add(Input) returns (Output);

What we are going to do is we are going to take this UnimplementedCalculatorServer and implement the Add method.

// server is used to implement calculator.CalculatorServer.
type server struct {
    pb.UnimplementedCalculatorServer
}

// Add implements calculator.CalculatorServer
func (s *server) Add(ctx context.Context, in *pb.Input) (*pb.Output, error) {
    log.Printf("Received: %v %v", in.GetOperand1(), in.GetOperand2())
    result := in.GetOperand1() + in.GetOperand2()
    return &pb.Output{Result: result}, nil
}

Pay attention to the signature of the method. We are taking Input in form of *pb.Input and returning Output in form of *pb.Output. This is very the same as we declared in the proto file.

The Add implementation is incomplete without Add logic. On line 26 we are using GetOperand1 and GetOperand2 which is available from the calculator.pb.go file. At last, we use the Output struct to return the result.

Implementing the method is not enough, we also need to start the gRPC server and start listening.

func main() {
    flag.Parse()
    lis, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    s := grpc.NewServer()
    pb.RegisterCalculatorServer(s, &server{})
    log.Printf("server listening at %v", lis.Addr())
    if err := s.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

On line 32 I'm starting to listen to TCP connection on a given port. grpc.NewServer() calls the internal library function to create a new gRPC server, this call returns a grpc.ServiceRegistrar object. This object is then passed to RegisterCalculatorServer along with our implemented methods.

The whole code looks like this:

package main

import (
    "context"
    "flag"
    "fmt"
    "log"
    "net"

    pb "github.com/santosh/example/calculator"
    "google.golang.org/grpc"
)

var (
    port = flag.Int("port", 50051, "The server port")
)

// server is used to implement calculator.CalculatorServer.
type server struct {
    pb.UnimplementedCalculatorServer
}

// Add implements calculator.CalculatorServer
func (s *server) Add(ctx context.Context, in *pb.Input) (*pb.Output, error) {
    log.Printf("Received: %v %v", in.GetOperand1(), in.GetOperand2())
    result := in.GetOperand1() + in.GetOperand2()
    return &pb.Output{Result: result}, nil
}

func main() {
    flag.Parse()
    lis, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    s := grpc.NewServer()
    pb.RegisterCalculatorServer(s, &server{})
    log.Printf("server listening at %v", lis.Addr())
    if err := s.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

In above implementation, we have used flag library to pass port from command line.

Write gRPC calculator client

Like we have done with server code, we'll start by defining command line flags.

var (
    addr     = flag.String("addr", "localhost:50051", "the address to connect to")
    operand1 = flag.Int("op1", 2, "1st operand")
    operand2 = flag.Int("op2", 2, "2nd operand")

    operand1int32 = int32(*operand1)
    operand2int32 = int32(*operand2)
)

Line 19-20 here are kind of a hack as flag module does not have a way to accept int32 which is requirement for our Output.Result.

Next is connecting with the server part:

    // Set up a connection to the server.
    conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()

grpc.Dial takes address of the server as well as other variadic parameters. As gRPC works over HTTP2 which is by default requires TLS, we are using insecure credentials as we have not configured our server to use certificates.

On the next line, we are going to create a new gRPC client:

    c := pb.NewCalculatorClient(conn)

On preceding lines, we are going to invoke the Add endpoint:

    // Contact the server and print out its response.
    ctx, cancel := context.WithTimeout(context.Background(), time.Second)
    defer cancel()
    r, err := c.Add(ctx, &pb.Input{Operand1: operand1int32, Operand2: operand2int32})
    if err != nil {
        log.Fatalf("could not add: %v", err)
    }
    log.Printf("Add result: %v", r.GetResult())

The entirity of the code looks like this:

package main

import (
    "context"
    "flag"
    "log"
    "time"

    pb "github.com/santosh/example/calculator"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

var (
    addr     = flag.String("addr", "localhost:50051", "the address to connect to")
    operand1 = flag.Int("op1", 2, "1st operand")
    operand2 = flag.Int("op2", 2, "2nd operand")

    operand1int32 = int32(*operand1)
    operand2int32 = int32(*operand2)
)

func main() {
    flag.Parse()
    // Set up a connection to the server.
    conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()
    c := pb.NewCalculatorClient(conn)

    // Contact the server and print out its response.
    ctx, cancel := context.WithTimeout(context.Background(), time.Second)
    defer cancel()
    r, err := c.Add(ctx, &pb.Input{Operand1: operand1int32, Operand2: operand2int32})
    if err != nil {
        log.Fatalf("could not add: %v", err)
    }
    log.Printf("Add result: %v", r.GetResult())
}

Let's go ahead and test our service.

Demo our server and the client

I have recorded a gif for the demo.

Here I have demonstrated with the default flag value, but you can override -op1 and -op2 flag to see different results.

Conclusion

This was merely an introduction to gRPC and protocol buffer. What I've found is different than a conventional REST API is the endpoints. In REST, we have a predefined endpoint to hit. Such as /calculation/add. We would then pass JSON with the first and the second operand.

This is not the case with gRPC. We get the stub files as the protoc artifact. The only communication is from there.

In this post, I have only covered the Unary service method types. I'll leave streaming for some other time. If you'd like to read and practice more gRPC, I'd found this series very helpful.

External Links

Building a Book Store API in Golang With Gin

Santosh Kumar — Fri, 18 Mar 2022 04:16:18 +0000

Introduction

Go community believes that we need no framework to develop web services. And I agree with that. When you work without using any framework, you learn the ins and outs of development. But once you learn how things work, you must reside in a community and don't reinvent the wheel.

I have created POCs in Go before and I had to deal with HTTP headers, serializing/deserializing, error handling, database connections, and whatnot. But now I've decided to join the Gin community as it is one of the widely accepted in the software development community.

Although I'm writing this post as a standalone article and would keep things simple here. But I want to continue building on these examples to have authentication, authorization, databases (including Postgres, ORM), swagger, and GraphQL covered. Will be interlinking the posts when I create them.

Why Gin

There are numerous reasons you may want to use Gin. If you ask me, I'm a big fan of Gin's sensible defaults.

Another thing I like about Gin is that it's an entire framework. You don't need a separate multiplexer and a separate middleware library and so on. On top of that, there are many common things already available that you don't have to reinvent. It does enhance our productivity. Although I'm not using it in production, I already have started to feel it.

Hello World in Gin

package main

import (
    "net/http"

    "github.com/gin-gonic/gin"
)

func main() {
    router := gin.New()

    router.GET("/ping", func(ctx *gin.Context) {
        ctx.JSON(http.StatusOK, gin.H{
            "message": "pong",
        })
    })

    router.Run()
}

Let's get familiar with Gin a little bit.

router := gin.New()

This creates an instance of Engine. gin.Engine is the core of gin. It also acts as a router and that is why we have put that Engine instance into a variable called router.

router.GET("/ping", func(ctx *gin.Context) {

This binds a route /ping to a handler. In the example above, I've created an anonymous function, but it could be a separate function as well. The thing to note here is the parameter to this function. *gin.Context. Context is yet another important construct besides Engine. Context has almost 100 methods attached to it. A newcomer should be spending most of their time understanding this Context struct and their methods.

Let's now look at the next few lines:

        ctx.JSON(http.StatusFound, gin.H{
            "message": "pong",
        })

One of the *gin.Context method is JSON. This method is used to send a JSON response to the client. Meaning that it automatically sets response's Content-Type to application/json. JSON method takes an HTTP status code and a map of response. gin.H is an alias to map[string]interface{}. So basically we can create an object which can have string key and whatever value we want.

Next is:

router.Run()

Engine.Run simply takes our router along with the route handler and binds it to http.Server. The default port is 8080, but if you want, you can have another address passed here.

The Book Store API

I've already done a POC on bookstore before, at that time, I wanted to prototype a connection between MongoDB and Go. But this time my goal is to have Postgres and GraphQL incorporated.

So first of all, I'd like you to set up a directory structure like this:

$ tree
.
├── db
│   └── db.go
├── go.mod
├── go.sum
├── handlers
│   └── books.go
├── main.go
└── models
    └── book.go

And let's start filling up those files.

db/db.go

package db

import "github.com/santosh/gingo/models"

// Books slice to seed book data.
var Books = []models.Book{
    {ISBN: "9781612680194", Title: "Rich Dad Poor Dad", Author: "Robert Kiyosaki"},
    {ISBN: "9781781257654", Title: "The Daily Stotic", Author: "Ryan Holiday"},
    {ISBN: "9780593419052", Title: "A Mind for Numbers", Author: "Barbara Oklay"},
}

Instead of going into the complexity of setting up a database right now, I've decided to use an in-memory database for this post. In this file, I've seeded db.Books slice with some books.

If models.Book makes, you curious, the next file is that only.

models/book.go

package models

// Book represents data about a book.
type Book struct {
    ISBN   string  `json:"isbn"`
    Title  string  `json:"title"`
    Author string  `json:"author"`
}

Nothing fancy here, we only have 3 fields as of now. All of them are strings and with struct tags.

Let us see our main.go before we go onto handlers.go.

main.go

package main

import (
    "github.com/gin-gonic/gin"
    "github.com/santosh/gingo/handlers"
)

func setupRouter() *gin.Engine {
    router := gin.Default()
    router.GET("/books", handlers.GetBooks)
    router.GET("/books/:isbn", handlers.GetBookByISBN)
    // router.DELETE("/books/:isbn", handlers.DeleteBookByISBN)
    // router.PUT("/books/:isbn", handlers.UpdateBookByISBN)
    router.POST("/books", handlers.PostBook)

    return router
}

func main() {
    router := setupRouter()
    router.Run(":8080")
}

Almost similar to the hello world example we saw above. But this time we have gin.Default() instead of gin.New(). The Default comes with defaults which most of us would like to have. Like logging middleware.

Frankly speaking, I haven't used much of Gin's middleware yet. But it's damn simple to create your middlewares. I'll put some links at the bottom of the post for your exploration. But for now, let's look at our handlers.

handlers/books.go

package handlers

import (
    "net/http"

    "github.com/gin-gonic/gin"
    "github.com/santosh/gingo/db"
    "github.com/santosh/gingo/models"
)

// GetBooks responds with the list of all books as JSON.
func GetBooks(c *gin.Context) {
    c.JSON(http.StatusOK, db.Books)
}

// PostBook takes a book JSON and store in DB.
func PostBook(c *gin.Context) {
    var newBook models.Book

    // Call BindJSON to bind the received JSON to
    // newBook.
    if err := c.BindJSON(&newBook); err != nil {
        return
    }

    // Add the new book to the slice.
    db.Books = append(db.Books, newBook)
    c.JSON(http.StatusCreated, newBook)
}

// GetBookByISBN locates the book whose ISBN value matches the isbn
func GetBookByISBN(c *gin.Context) {
    isbn := c.Param("isbn")

    // Loop over the list of books, look for
    // an book whose ISBN value matches the parameter.
    for _, a := range db.Books {
        if a.ISBN == isbn {
            c.JSON(http.StatusOK, a)
            return
        }
    }
    c.JSON(http.StatusNotFound, gin.H{"message": "book not found"})
}

// func DeleteBookByISBN(c *gin.Context) {}

// func UpdateBookByISBN(c *gin.Context) {}

The real juice is in this handlers file. This might need some explanation.

handlers.GetBooks, which is bound to GET /books dumps the entire book slice.

handlers.GetBookByISBN, which is bound to GET /books/:isbn does the same thing, but it also accepts isbn as a URL parameter. This handler scans the entire slice and returns the matched book. Scanning a large slice would not be the most optimal solution, but don't forget that we'll be implementing a real database while we continue to develop this bookstore.

The most interesting one here is handlers.PostBook, which is bound to POST /books. c.BindJSON is the magic method, which takes in the JSON from the request and stores it into previously created newBook struct. Later on

Tests

We need a little change here at the moment. We need to remove these contents from main.go:

@@ -1,17 +1,9 @@
 package main

-import (
-       "github.com/gin-gonic/gin"
-       "github.com/santosh/gingo/handlers"
-)
+import "github.com/santosh/gingo/routes"

 func main() {
-       router := gin.Default()
-       router.GET("/books", handlers.GetBooks)
-       router.GET("/books/:isbn", handlers.GetBookByISBN)
-       // router.DELETE("/books/:isbn", handlers.DeleteBookByISBN)
-       // router.PUT("/books/:isbn", handlers.UpdateBookByISBN)
-       router.POST("/books", handlers.PostBook)
+       router := routes.SetupRouter()

        router.Run(":8080")
 }

And put it into a new file.

routes/roures.go

package routes

import (
    "github.com/gin-gonic/gin"
    "github.com/santosh/gingo/handlers"
)

func SetupRouter() *gin.Engine {
    router := gin.Default()
    router.GET("/books", handlers.GetBooks)
    router.GET("/books/:isbn", handlers.GetBookByISBN)
    // router.DELETE("/books/:isbn", handlers.DeleteBookByISBN)
    // router.PUT("/books/:isbn", handlers.UpdateBookByISBN)
    router.POST("/books", handlers.PostBook)

    return router
}

I have changes that make sense to you. We did this because we need to start the server from our tests.

Next, we create a books_test.go in handlers.

handlers/books_test.go

package handlers_test

import (
    "bytes"
    "encoding/json"
    "net/http"
    "net/http/httptest"
    "testing"

    "github.com/santosh/gingo/models"
    "github.com/santosh/gingo/routes"
    "github.com/stretchr/testify/assert"
)


func TestBooksRoute(t *testing.T) {
    router := routes.SetupRouter()

    w := httptest.NewRecorder()
    req, _ := http.NewRequest("GET", "/books", nil)
    router.ServeHTTP(w, req)

    assert.Equal(t, 200, w.Code)
    assert.Contains(t, w.Body.String(), "9781612680194")
    assert.Contains(t, w.Body.String(), "9781781257654")
    assert.Contains(t, w.Body.String(), "9780593419052")
}

func TestBooksbyISBNRoute(t *testing.T) {
    router := routes.SetupRouter()

    w := httptest.NewRecorder()
    req, _ := http.NewRequest("GET", "/books/9781612680194", nil)
    router.ServeHTTP(w, req)

    assert.Equal(t, 200, w.Code)
    assert.Contains(t, w.Body.String(), "Rich Dad Poor Dad")
}


func TestPostBookRoute(t *testing.T) {
    router := routes.SetupRouter()

    book := models.Book{
        ISBN: "1234567891012",
        Author: "Santosh Kumar",
        Title: "Hello World",
    }

    body, _ := json.Marshal(book)

    w := httptest.NewRecorder()
    req, _ := http.NewRequest("POST", "/books", bytes.NewReader(body))
    router.ServeHTTP(w, req)

    assert.Equal(t, 201, w.Code)
    assert.Contains(t, w.Body.String(), "Hello World")
}

Also, again, pretty much self-explanatory. I don't think the above code needs any explanation. We are testing for response codes and response bodies for a specific string.

Let's also run the tests and check how it goes:

$ go test ./... -cover
?       github.com/santosh/gingo        [no test files]
?       github.com/santosh/gingo/db     [no test files]
ok      github.com/santosh/gingo/handlers       (cached)        coverage: 83.3% of statements
?       github.com/santosh/gingo/models [no test files]
?       github.com/santosh/gingo/routes [no test files]

Exercise

Yeah, let's this blog post more interesting by adding some interactivity. I have some tasks for you, which you need to solve on your own. Please have a try on them. They are:

Implement DeleteBookByISBN and UpdateBookByISBN handlers and enable them.
Write tests for handlers mentioned above.
Our tests are very basic. So are our handlers. We are not doing any error handling. Add error handling to handlers and write tests to validate them.

Conclusion

We have seen how simple is it to create a hello world application in Gin. But this journey does not end here. I'll come back with more tutorials next time. Until then, goodbye.

Related Link

init vs new and When to Use Them

Santosh Kumar — Sun, 06 Mar 2022 13:18:08 +0000

Introduction

I recently got asked to me what did __init__ dunder method did in Python. And I was like, "it is used to initialize variables inside a class". And just after that, the follow-up question was, "then what is __new__ used for. And I was completely blank and was not able to answer that.

I was not able to answer that question because there are not many tutorials out there that talk about __new__. I didn't want to happen this with you. And that is why I came up with this blog post for you.

Similarities

Let's start with similarities.

Both of them are called/invoked during the creation of the instance.

Differences

Let's start with differences.

	new	init
1	Called before init	Called after new
2	Accepts a type as the first argument	Accepts an instance as the first argument
3	Is supposed to return an instance of the type received	Is not supposed to return anything
4	Used to control instance creation	Used to initialize instance variables

Talking about the first point. __new__ is called when the instance is first created. This happens before the initialization of the class.

By the way, did you note that the first argument to __init__ is always self? This self is the instance of the class. self is what __new__ returns.

Coming to the third point, __new__ is supposed to return an instance of the class. Note that if __new__ does not returns anything, __init__ is not called.

Which one of them is a constructor?

If you are coming from another language, you might be surprised that there are two similar things doing the same kind of work. Most languages you might have worked with would have something called a constructor.

In Python, that concept is broken down into constructor and initializer. And you might have guessed, __new__ is the constructor and __init__ is the initializer.

Please note that __new__ is implicit. Meaning that if you don't actually need to modify the creation of an instance of the class, you don't need to have a new method.

One more thing I'd like to add is... instance variables are local to an instance. So anything you are doing in init is local to that instance only. But anything you are doing in new will be affecting anything created for that type.

Execution flow with an example

I am going to add some code to make this more engaging. Consider this example:

class Demo:
    def __new__(cls, *args):
        print("__new__ called")
        return object.__new__(cls)

    def __init__(self):
        print("__init__ called")

d = Demo()

This is the simplest example of both __new__ and __init__ in action. If you save the above code in a file and run it, you'd see something like this:

$ python3 in_it.py 
__new__ called
__init__ called

As you can see, the new method is called first and then execution is passed to the init method.

Use Cases

Use case for `new`

One of the best use cases I can take an example of is when creating a Singleton. As we know, Singleton ensures a class only has one instance and provides a global point of access to it.

Some of the places I have seen singleton in action are in game programming where there is only one instance of the player. Another place, if you have used frontend libraries like Vuex (or Redux), there is only one global instance of the store. Doesn't matter how many instances you create, you'll end up having only one.

Let's see how to achieve similar behavior in Python:

class Singleton:
    __instance = None

    def __new__(cls):
        if cls.__instance is None:
            print("creating...")
            cls.__instance = object.__new__(cls)
        return cls.__instance

s1 = Singleton()
s2 = Singleton()

print(s1)
print(s2)

Output:

$ python3 singleton.py 
creating...
<__main__.Singleton object at 0x7f943301d350>
<__main__.Singleton object at 0x7f943301d350>

As you can see, creating... is printed only once. they both point to the same memory location. In my case, it's 0x7f943301d350.

You might be guessing that can't we do the same thing with __init__? No! That's because __init__ does not return anything. We'll see in the next section what __init__ is well suited for.

But first, I wanted to show you another use case of new.

class Animal:
    def __new__(cls, legs):
        if legs == 2:
            return Biped()
        else:
            return Quadruped()

class Biped:
    def __init__(self):
        print("Initializing 2-legged animal")

class Quadruped:
    def __init__(self):
        print("Initializing 4-legged animal")

anim1 = Animal(legs=4)
anim1 = Animal(legs=2)

Output:

$ python3 newexample.py 
Initializing 4-legged animal
Initializing 2-legged animal

I am by no means a zoologist. But you get my point here. You can use __new__ to conditionally create an instance from a class.

Use case for `init`

As we have already seen previously. init is there to initialize an instance variable. These instance variables can later be used in different methods of the instance.

I have extensively used __init__ when I used to work with Qt framework. Qt is a framework for desktop-based UI development. When initializing UI objects, you can set how wide or long the window could be. You can also read preferences from a file and apply that during the initialization phase of an application. Setting the window title could be another example.

Here I'll demonstrate one such example:

class Window(QWidget):
   def __init__(self, parent = None):
      super(Window, self).__init__(parent)
      self.resize(200, 100)
      self.setWindowTitle("My App")

The above example is by no means a complete example, but when set up correctly, it will show a window similar to this.

Key Takeaways

In most cases, you don't need
__new__ is called before __init__.
__new__ returns a instance of class.
__init__ receives the instances of the class returned by __new__.
Use __init__ to initilize value.

The same concept can be used to answer the question of abstraction vs encapsulation.

Conclusion

That is all I know about __init__ vs __new__. If you have something in your mind that I missed. Please let me know.

I'll also list some references I look to write this blog post so that you can really the real source of information.

Forem: Santosh Kumar

__str__ vs __repr__ in Python and When to Use Them

Introduction

__str__() and str()

__repr__() and repr()

__str__() and __repr__() in a Class

Conclusion

How to Control Philips Wiz Bulb Using Go

Introduction

Control Philips Wiz using App

Control Philips Wiz using Go

Find the bulb IP

Hello Philips Wiz using UDP

Set custom colors

Set brightness

Set temperature

Conclusion

First Month With My Raspberry Pi

Introduction

Setup the Pi

Setting up the OS

Blink Light

Actuate on Proximity

Controlling LED brightness, 0-100 and back

Controlling an RGB light (failed)

Conclusion

How to Write Classes in Golang

Introduction

What is a class composed of?

Example in Python

Struct to define the structure and attributes

Method or behaviour using receiver function

Testing HTTP Requests with Postman

Introduction

What are assertions and how to use it

Looking for a specific status code

Looking for a specific string in the body

Looking for a specific header

Adding tests at the collection level

Run all tests from a collection

Conclusion

Postman Premier - The Minimal Postman you need to Learn

Introduction

Create a collection

Working with Requests

Creating a GET Request to get introduced to UI

Path Parameters

Creating a POST Request and working with Body and Headers

Working with variables

Using Collection variables

Saving response data in variables

Conclusion

Best Logging Library for Golang

Introduction

Poll Results

Analysis of Results

Selection of the candidates

logrus

zerolog

zap

Conclusion

Release Dangling Elastic IPs using Lambda and Go SDK

Introduction

What is an Elastic IP and why do we need it?

But why do we need an EIP killer?

Get Started

Phase 1: Provision a Lambda function

Phase 2: Write the EIP killing logic

Phase 2.1: The initial deployment process

Phase 2.2: Logic to deallocate unused EIPs

Phase 2.3: Test our EIP killer Give Lambda permission to release and read EIPs

Phase 2.3: Test our EIP killer

Phase 3: Trigger Lambda at a 1-hour interval

Phase 3.1: Configure EventBridge aka CloudWatch Events

Phase 3.2: Create an EIP and go to sleep

How to Integrate Swagger UI in Go Backend - Gin Edition

Introduction

Prerequisites

How Swagger works with Gin

Setup Swagger CLI

str vs repr in Python and When to Use Them

str() and str()

repr() and repr()

str() and repr() in a Class

init vs new and When to Use Them

Use case for `new`

Use case for `init`