Forem: Bruce Du

7 common pitfalls in JSON operations in Golang

Bruce Du — Fri, 17 May 2024 08:06:39 +0000

0 Preface

JSON is a data format often used by many developers in their work. It is generally used in scenarios such as configuration files or network data transfer. And because of its simplicity, easy to understand, and good readability, JSON has become one of the most common formats in the entire IT world. For such things, Golang, like many other languages, also provides standard library level support, which is encoding/json.

Just like JSON itself is easy to understand, the encoding/json library for manipulating JSON is also very easy to use. But I believe that many developers may have encountered various strange problems or bugs like I did when I first used this library. This article summarizes the problems and mistakes I personally encountered when using Golang to operate JSON. I hope it can help more developers who read this article to master the skills of Golang to operate JSON more smoothly and avoid unnecessary mistakes’ pit.

BTW, the content of this article is based on Go 1.22. There may be slight differences between different versions. Readers should to pay attention when reading and using it. At the same time, all cases listed in this article use encoding/json and do not involve any third-party JSON library.

1 Basic use

Let’s simply talk about the basic use of encoding/json library first.

As a kind of data formats, the core operations of JSON are just two: Serialization, and Deserialization. Serialization is to convert a Go object into a string (or byte sequence) in JSON format. Deserialization is the opposite, converting JSON format data into a Go object.

The object mentioned here is a broad concept. It not only refers to structure objects, but also includes slice and map type data. They support JSON serialization as well.

Example:

import (
    "encoding/json"
    "fmt"
)

type Person struct {
    ID   uint
    Name string
    Age  int
}

func MarshalPerson() {
    p := Person{
        ID:   1,
        Name: "Bruce",
        Age:  18,
    }
    output, err := json.Marshal(p)
    if err != nil {
        panic(err)
    }
    println(string(output))
}

func UnmarshalPerson() {
    str := `{"ID":1,"Name":"Bruce","Age":18}`
    var p Person
    err := json.Unmarshal([]byte(str), &p)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", p)
}

The core is the two functions json.Marshal and json.Unmarshal, which are used for serialization and deserialization respectively. Both functions will return error, here I simply panic.

Readers who have used encoding/json might know that there is another pair of tools that are often used: NewEncoder and NewDecoder. A brief look at the source code shows that the underlying core logic calls of the two are the same as Marshal, so I won’t give examples here.

2 Pitfalls

2.1 Public struct fields

This is probably the most common mistake made by developers who are new to Go or encoding/json. That is, if we're manipulating JSON with structs, then the member fields of the struct must be public, i.e., capitalized, and private members cannot be parsed.

Example:

type Person struct {
    ID   uint
    Name string
    age  int
}

func MarshalPerson() {
    p := Person{
        ID:   1,
        Name: "Bruce",
        age:  18,
    }
    output, err := json.Marshal(p)
    if err != nil {
        panic(err)
    }
    println(string(output))
}

func UnmarshalPerson() {
    str := `{"ID":1,"Name":"Bruce","age":18}`
    var p Person
    err := json.Unmarshal([]byte(str), &p)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%+v\n", p)
}

// Output Marshal:
{"ID":1,"Name":"Bruce"}

// Output Unmarshal:
{ID:1 Name:Bruce age:0}

Here, age is set as a private variable, so there is no age field in the serialized JSON string. Similarly, when you deserialize a JSON string into Person, you can't read the value of age correctly.

The reason for this is simple. If we dig deeper into Marshal's source code, we can see that it actually uses reflect underneath to dynamically parse struct objects:

// .../src/encoding/json/encode.go

func (e *encodeState) marshal(v any, opts encOpts) (err error) {
    // ...skip
    e.reflectValue(reflect.ValueOf(v), opts)
    return nil
}

Golang prohibits reflective access to the private members of a structure at the language design level, so this kind of reflective parsing naturally fails, and the same applies to deserialization.

2.2 Use `map` sparingly

As mentioned earlier, JSON can manipulate not only structures, but also slice, map and other types of data. slice is special, but map and structure are actually the same in JSON format:

m := map[string]any{
    "ID": 1, 
    "Name": "Bruce",
}

// ...skip the structure code

// JSON output:
{
    "ID": 1,
    "Name": "Bruce"
}

In this case, unless there is a special circumstance or need, use maps sparingly. Because maps incur additional overhead, additional code, and additional maintenance costs.

Why?

First of all, as in the Person example above, since ID and Name are different types, if we want to deserialize this JSON data with a map, we can only assert a map of type map[string]any. any, which is interface{}, means that if we want to use Name or ID on their own, we need to convert the type of the JSON data with the type assertion if we want to use Name or ID alone:

var m map[string]any
// ...Deserialize JSON data into `m`，ignore the code...
// Fetch Name
name, ok := m["Name"].(string)

The type assertion itself is an extra step, and in order to prevent panic, we need to determine the second parameter ok, which definitely increases the development effort as well as the code burden.

In addition, a map is inherently unconstrained with respect to data. We can predefine fields and types in structures, but not in maps. This means that we can only understand what's in the map through documentation or comments or the code itself. Moreover, while a structure can restrict the key and value types of JSON data from being changed, a map can't restrict changes to JSON and can only be detected by business logic code. Just think about the amount of work and maintenance costs involved.

The reason I mention this pitfall is that before I developed in Go, my main language was Python, and Python, you know, doesn't have structures, it just has dict (map) to load JSON data. When I first got into Go, I used to use map to interact with JSON. But because Go is statically typed, you have to explicitly convert the type (type assertion), not like Python way. Those troublesome gave me a headache for a while.

In short, use map sparingly, or as little as possible, to manipulate JSON.

2.3 Beware of struct combinations

Go is object-oriented, but it doesn't have classes, only structures, and structures don't have inheritance. So Go uses a kind of combination to reuse different structures. In many cases, this combination is very convenient, as we can manipulate other members of the combination as if they were members of the structure itself, like this:

type Person struct {
    ID   uint
    Name string
    address
}

type address struct {
    Code   int
    Street string
}

func (a address) PrintAddr() {
    fmt.Println(a.Code, a.Street)
}

func Group() {
    p := Person{
        ID:   1,
        Name: "Bruce",
        address: address{
            Code:   100,
            Street: "Main St",
        },
    }
    // Access all address's fields and methods directly
    fmt.Println(p.Code, p.Street)
    p.PrintAddr()
}

// Output
100 Main St
100 Main St

Convenient, right? I think so too. But there's a small pitfall to be aware of when we incorporate combinations into our use of JSON. Take a look at the code below:

// The structure used here is the same as the previous one, 
// so I won't repeat it. error is also not captured to save space.

func MarshalPerson() {
    p := Person{
        ID:   1,
        Name: "Bruce",
        address: address{
            Code:   100,
            Street: "Main St",
        },
    }
    // It would be more pretty by MarshalIndent
    output, _ := json.MarshalIndent(p, "", "  ")
    println(string(output))
}

func UnmarshalPerson() {
    str := `{"ID":1,"Name":"Bruce","address":{"Code":100,"Street":"Main St"}}`
    var p Person
    _ = json.Unmarshal([]byte(str), &p)
    fmt.Printf("%+v\n", p)
}

// Output MarshalPerson:
{
  "ID": 1,
  "Name": "Bruce",
  "Code": 100,
  "Street": "Main St"
}

// Ouptput UnmarshalPerson:
{ID:1 Name:Bruce address:{Code:0 Street:}}

This code is slightly more informative, so let's run through it bit by bit.

Let's start with the MarshalPerson function. Here a Person object is declared and then the serialization result is beautified with MarshalIndent and printed. As we can see from the printout, the entire Person object is flattened. In the case of the Person struct, it looks like it still has an address member field, despite the combination. So sometimes we take it for granted that the serialized JSON of Person looks like this:

// The imagine of JSON serialization result
{
  "ID": 1,
  "Name": "Bruce",
  "address": {
    "Code": 100,
    "Street": "Main St"
  }
}

But it doesn't, it's flattened. This is more in line with the feeling we had earlier when we accessed the address member directly through Person, i.e., the address member seems to become a member of Person directly. This is something to be aware of, as the combination will flatten the serialized JSON result.

Another slightly counter-intuitive point is that the address struct is a private struct, and it seems that private members shouldn't be serialized? That's right, and that's one of the things that's not so great about this form of composition: it exposes the public members of the private composition object. So, it's important to note here that this exposure is sometimes unintentional, but it can cause unwanted data leaks.

Then there's the UnmarshalPerson function. With the previous function out of the way, this is easy to understand, but it's really just a matter of flattening the JSON result after combining. So, if we need to deserialize back to Person, we also need a flattened JSON data.

In fact, in my personal use of Go over the years, when I encounter these kinds of structures that need to be converted to JSON, I usually don't use combinations very much, unless there are some special circumstances. After all, it's too easy to cause the problems mentioned above. Also, since JSON is flattened and structs are not defined to be flattened (looks not), once the struct is defined to be more and more complex, the harder it is to visually compare it to the original flattened JSON data, which will make the code much less readable.

If you don't have special needs (e.g., the raw JSON data is flattened and there are multiple structs with duplicate fields that need to be reused), from my personal point of view, I would recommend that you try to write it this way:

type Person struct {
    ID      int
    Name    string
    Address address
}

Of course, if this structure doesn't involve JSON serialization, then I'd still prefer to use a combination, it's really convenient.

2.4 Care needed when deserializing part of member fields (patch update data)

Look directly at the code:

type Person struct {
    ID   uint
    Name string
}

// PartUpdateIssue simulates parsing two different 
// JSON strings with the same structure
func PartUpdateIssue() {
    var p Person
    // The first data has the ID field and is not 0
    str := `{"ID":1,"Name":"Bruce"}`
    _ = json.Unmarshal([]byte(str), &p)
    fmt.Printf("%+v\n", p)
    // The second data does not have an ID field, 
    // deserializing it again with p preserves the last value
    str = `{"Name":"Jim"}`
    _ = json.Unmarshal([]byte(str), &p)
    // Notice the output ID is still 1
    fmt.Printf("%+v\n", p)
}

// Output
{ID:1 Name:Bruce}
{ID:1 Name:Jim}

The comment explains it clearly: when we use the same structure to deserialize different JSON data repeatedly, once the value of a JSON data contains only some of the member fields, then the uncovered members will be left with the value of the last deserialization. This is actually a dirty data pollution problem.

This is a problem that can be easily encountered and, when triggered, is quite insidious. I previously wrote a post (Bugs in Golang caused by Zero Value and features of the gob library) about a similar situation encountered when using the gob library. Of course, the gob problem is related to zero values, which is not quite the same as the JSON problem we're talking about today, but they both end up behaving in a similar way, with some of the member fields being contaminated by dirty data.

The solution is also simple: each time you deserialize JSON, use a brand new struct object to load the data.

Anyway, be careful with this situation.

2.5 Handling pointer fields

Many developers get a big headache when they hear the word pointer, but you don't have to, it's not that complicated. But pointers in Go do give developers one of the most common panics in Go programs: the null pointer exception. And what happens when pointers are combined with JSON?

Look at this code:

type Person struct {
    ID      uint
    Name    string
    Address *Address
}

func UnmarshalPtr() {
    str := `{"ID":1,"Name":"Bruce"}`
    var p Person
    _ = json.Unmarshal([]byte(str), &p)
    fmt.Printf("%+v\n", p)
    // It would panic this line
    // fmt.Printf("%+v\n", p.Address.Street)
}

// Output
{ID:1 Name:Bruce Address:<nil>}

We define the Address member as a pointer, and when we deserialize a piece of JSON data that doesn't contain an Address, the pointer field is set to nil because it doesn't have a corresponding piece of data. encoding/json doesn't create empty Address object and point to it for this field. If we call p.Address.xxx directly, the program will panic because p.Address is nil.

So, if there is a pointer to a member of our structure, remember to determine if the pointer is nil before using it. This is a bit tedious, but it can't be helped. After all, writing a few lines of code may not be a big deal compared to the damage caused by a panic in a production environment.

Also, when creating a structure with a pointer field, the assignment of the pointer field can be relatively cumbersome:

type Person struct {
    ID   int    
    Name string 
    Age  *int   
}

func Foo() {
    p := Person{
        ID:   1,
        Name: "Bruce",
        Age:  new(int),
    }
    *p.Age = 20
    // ...
}

Some people say, "Well, isn't it recommended that we try not to use pointers in any of the member variables of a JSON data structure?” I don't think so this time, because pointers do have a couple of scenarios where they are better suited than non-pointer members. One is that pointers reduce some of the overhead, and the other is what we'll talk about in the next section, zero-value related issues.

2.6 Confusion caused by zero values

Zero value is a feature of Golang variables that we can simply think of as default values. That is, if we don't explicitly assign a value to a variable, Golang assigns it a default value. For example, as we have seen in the previous example, int has a default value of 0, string has an empty string, pointer has a zero value of nil, and so on.

What are the pitfalls of handling JSON with zero values?

Look at the following example:

type Person struct {
    Name        string
    ChildrenCnt int
}

func ZeroValueConfusion() {
    str := `{"Name":"Bruce"}`
    var p Person
    _ = json.Unmarshal([]byte(str), &p)
    fmt.Printf("%+v\n", p)

    str2 := `{"Name":"Jim","ChildrenCnt":0}`
    var p2 Person
    _ = json.Unmarshal([]byte(str2), &p2)
    fmt.Printf("%+v\n", p2)
}

// Output
{Name:Bruce ChildrenCnt:0}
{Name:Jim ChildrenCnt:0}

We added a ChildrenCnt field to the Person structure to count the number of children of the person. Because of the zero value, this field is assigned 0 when the JSON data loaded by p does not have ChildrenCnt data, which is misleading: we can't distinguish objects with this missing data from objects that do have 0 children. In the case of Bruce and Jim, one of them has 0 children due to missing data, and the other has 0. Bruce's number of children should be "unknown", and if we really treat it as 0, it may cause problems in business.

This kind of confusion is very fatal in some scenarios with strict data requirements. So is there any way to avoid this kind of zero-value interference? There really is, is the last legacy of the last section of the use of pointers scenarios.

Let's change Person's ChildrenCnt type to *int and see what happens:

type Person struct {
    Name        string
    ChildrenCnt *int
}

// Output
{Name:Bruce ChildrenCnt:<nil>}
{Name:Jim ChildrenCnt:0xc0000124c8}

The difference is that Bruce has no data, so ChildrenCnt is a nil, and Jim is a non-null pointer. At this point it is clear that the number of Bruce's children is unknown.

Essentially this approach still utilizes zero values, the zero values of pointers. It's kind of fighting fire with fire (laughs).

2.7 Pitfalls of tags

Finally, we're talking about tags. Tags are also a very important feature in Golang and are often accompanied by JSON. In fact, for those of you who have used Go tags, you should know that tags are a very flexible and easy to use feature. So what are the pitfalls of using such a great feature?

One is the name problem: Tag can specify the name of a field in JSON data, which is very flexible and useful, but it is also error-prone, and to some extent adds some professionalism to the programmer himself.

For example, if a programmer intentionally or unintentionally defines a structure like this:

type PersonWrong struct {
    FirstName string `json:"last_name"`
    LastName  string `json:"first_name"`
}

The Tag of FirstName and LastName are swapped. would you want to beat the programmer up if you encountered such code? I've actually encountered something like this in my production code. Of course, it was unintentional, a mistake during a code update. However, when it comes to this kind of situation, such bugs are usually not easy to locate. Mostly because, well, who the **** would have thought?

Anyway, don't do it, and be careful when you write it.

Another problem is related to the combination of omitempty + zero, see the code:

type Person struct {
    Name        string `json:"person_name"`
    ChildrenCnt int    `json:"cnt,omitempty"`
}

func TagMarshal() {
    p := Person{
        Name:        "Bruce",
        ChildrenCnt: 0,
    }
    output, _ := json.MarshalIndent(p, "", "  ")
    println(string(output))
}

// Output
{
  "person_name": "Bruce"
}

See the problem? We assigned a value of 0 to ChildrenCnt when we created the new struct object p. Because of the omitempty tag, it causes the JSON to omit empty values when it is serialized or deserialized. This is manifested in serialization by the fact that the output JSON data does not contain ChildrenCnt and looks as if it does not exist. What is an empty value? That's right, it's a zero value.

So, the familiar confusion arises: Bruce has 0 children, not unknown data. The output JSON says that Bruce's children count is unknown.

Deserialization suffers from the same problem, so I won't give you an example.

What about this omitempty problem? Since it's still essentially a zero-valued problem, use pointers.

3 Summary

In this article, I've listed 7 mistakes that I've made when using the encoding/json library, and I've encountered most of them in my own work. If you haven't encountered them yet, congratulations! It's also a reminder to be careful with JSON in the future; if you've encountered any of these problems and are confused by them, I hope this article has helped you.

If there are any errors or lack of clarity in this post, please do not hesitate to point them out, thank you!

Bugs in Golang caused by Zero Value and features of the gob library

Bruce Du — Wed, 08 May 2024 14:07:04 +0000

0 Background

Just in the last year, the department platform project I was responsible for released a new version. This version also launched a new function, which is similar to a scheduled task. Everything was ok on the first day, but on the second day there were very few tasks that were not executed as expected (paused tasks continued to be executed, but stared tasks were not executed).

The first reaction of myself and another colleagues was that there were some issues with the logic of scheduled task execution. But after we spent a lot of time debugging and testing, we found that the root of the problem was not the functional logic, but a piece of underlying public code that had been online for a year and had not been touched. The core of this code is the protagonist of this article gob, and the root cause of the problem is a feature of the go language: zero value.

I will use a more simplified example to describe this BUG later.

1 `gob` and the Zero Value

Let’s briefly introduce gob and zero value first.

1.1 Zero Value

The zero value is a feature of the Go language. Simply put, the Go language will provide a default value for some variables which have not been assigned a value yet.

For example, the following code:

package main

import (
    "fmt"
)

type task struct {
    name  string
    state int
}

func main() {
    t := task{}
    var list []byte
    var f float32
    var s string
    var m map[string]int

    fmt.Println(list, f, s, m)
    fmt.Printf("%+v", t)
}

/* Output:
[] 0  map[]
{name: state:0}
*/

As you see, all variables have been assigned a default value, those default values are the zero value.

The zero value does bring convenience to developers in many cases, but there are also many people who don't like it who think that the existence of the zero value makes the code grammatically lax and brings some uncertainty. For example, the problem I will describe in detail later.

1.2 `gob`

gob is a standard library in encoding/gob. Gob is actually the abbreviation of go binary, so from its name we can also guess that gob should be related to binary.

In fact, gob is a unique format of Golang for serializing and deserializing program data in binary form, similar to pickle in Python. Its most common usage is to serialize an object (structure) and store it in a disk file, then read the file and deserialize it when needed to achieve object persistence.

I won’t give examples, and this article is not a special topic on the use of gob. This is its Official Document. Readers who are not familiar with gob usage can take a look at the Example part of the document, or just look at the codes I use to describe the problem later.

2 Problem

2.1 Requirement

At the beginning of this article, I briefly described the origin of the problem. Here I use a simpler model to expand the description.

First we define a structure named task:

type task struct {
    // Just as json library, the fields must be public could be serialized
    Name  string
    State int // Task state, it's only 2 states for simplistic, 1: running, 0: stopped
}

Around this structure, we will enter several tasks data, each task is a task object. But for some reasons, we must use gob to persist these task information to the local disk instead of using a database like MySQL.

Next, we have such a requirement:

Iterate and deserialize the locally stored gob files, then count how many tasks are running, and how many of them are stopped.

2.2 Code

Based on the above requirements and background, the code is as follows:

defines.go

package main

// Directory where .gob files saved
const DIR = "./tasks"

type task struct {
    // Just as json library, the fields must be public could be serialized
    ID    int
    Name  string
    State int // Task state, it's only 2 states for simplistic, 1: running, 0: stopped
}

// Test tasks which will be serialized and saved into .gob files.
// We define 2 running tasks and 3 stopped tasks here.
var tasks = []task{
    {0, "StoppedTask1", 0},
    {1, "RunningTask1", 1},
    {2, "RunningTask2", 1},
    {3, "StoppedTask2", 0},
    {4, "StoppedTask3", 0},
}

serializer.go

package main

import (
    "bytes"
    "encoding/gob"
    "fmt"
    "os"
    "path/filepath"
)

// serialize serializes person objects and save to files,
// the files are ./persons/${t.Name}.gob
func serialize(t task) {
    filename := filepath.Join(DIR, fmt.Sprintf("%s.gob", t.Name))
    buffer := new(bytes.Buffer)
    encoder := gob.NewEncoder(buffer)
    _ = encoder.Encode(t)
    _ = os.WriteFile(filename, buffer.Bytes(), 0o644)
}

// deserialize deserializes .gob files to task objects
func deserialize(path string, t *task) {
    raw, _ := os.ReadFile(path)
    buffer := bytes.NewBuffer(raw)
    decoder := gob.NewDecoder(buffer)
    _ = decoder.Decode(t)
}

main.go

package main

import (
    "fmt"
    "path/filepath"
)

func main() {
    storeTasks()
    countStates()
}

func storeTasks() {
    for _, p := range tasks {
        serialize(p)
    }
}

func countStates() {
    counter := make(map[int]int)
    // Use a temporary pointer as a carrier of objects in the file
    // to save the cost of creating new objects.
    tmpT := &task{}
    for _, t := range tasks {
        filename := filepath.Join(DIR, fmt.Sprintf("%s.gob", t.Name))
        // Deserialize task into tmpT
        deserialize(filename, tmpT)
        // Count the state
        counter[tmpT.State]++
    }
    fmt.Printf("Stopped: %+v, Running: %+v\n", counter[0], counter[1])
}

After running code, I got the result below:

Stopped: 1, Running: 4

What? 1 stopped task and 4 running tasks? Bug occurred. The result is different with our test data: 3 stopped tasks and 2 running tasks. So, what went wrong?

2.3 Debug

Let’s add a print statement line in for loop of main.go/countStates() function to see what we could get:

func countStates() {
    // ...
    for _, t := range tasks {
        // ...
        deserialize(filename, tmpT)
        // Add this line
        fmt.Printf("%+v\n", *tmpT)
        // ...
    }
    // ...
}

Output:
{ID:0 Name:StoppedTask1 State:0}
{ID:1 Name:RunningTask1 State:1}
{ID:2 Name:RunningTask2 State:1}
{ID:3 Name:StoppedTask2 State:1}
{ID:4 Name:StoppedTask3 State:1}

Oh man, StoppedTask2 and StoppedTask3 are both running! But the amazing thing is that except for State, all other data are valid! Seeing this result, if you often deal with configuration files such as JSON and Yml just like me, you may take it for granted that the above gob files are read correctly, and there should be a storage problem.

But gob files are binary files, and it is difficult for us to verify them with eyes like JSON files. Even if you use tools like xxd or something, you can only get such ambiguous output:

> xxd .\tasks\RunningTask1.gob
00000000: 2b7f 0301 0104 7461 736b 01ff 8000 0103  +.....task......
00000010: 0102 4944 0104 0001 044e 616d 6501 0c00  ..ID.....Name...
00000020: 0105 5374 6174 6501 0400 0000 15ff 8001  ..State.........
00000030: 0201 0c52 756e 6e69 6e67 5461 736b 3101  ...RunningTask1.
00000040: 0200                                     ..

> xxd .\tasks\StoppedTask2.gob
00000000: 2b7f 0301 0104 7461 736b 01ff 8000 0103  +.....task......
00000010: 0102 4944 0104 0001 044e 616d 6501 0c00  ..ID.....Name...
00000020: 0105 5374 6174 6501 0400 0000 13ff 8001  ..State.........
00000030: 0601 0c53 746f 7070 6564 5461 736b 3200  ...StoppedTask2.

Maybe we can try to forcely parse these binary files to compare the differences between them; or deserialize two identical objects except State into gob files, and then compare them. If you are interested, you can try it. At that time, due to time constraints and other reasons, we did not try this approach. We continue to modify the data to test.

2.4 Find patterns

Observing the results of the previous error, we found that the error data were all StoppedTask. Programmer's intuition tells me that this may not be a coincidence. So I tried to change the order of the data, completely separate RunningTasks and StoppedTasks, and then tested:

// First group, stopped first.
var tasks = []task{
    {0, "StoppedTask1", 0},
    {3, "StoppedTask2", 0},
    {4, "StoppedTask3", 0},
    {1, "RunningTask1", 1},
    {2, "RunningTask2", 1},
}

// Output
{ID:0 Name:StoppedTask1 State:0}
{ID:3 Name:StoppedTask2 State:0}
{ID:4 Name:StoppedTask3 State:0}
{ID:1 Name:RunningTask1 State:1}
{ID:2 Name:RunningTask2 State:1}
Stopped: 3, Running: 2

// Second group, running first.
var tasks = []task{
    {1, "RunningTask1", 1},
    {2, "RunningTask2", 1},
    {0, "StoppedTask1", 0},
    {3, "StoppedTask2", 0},
    {4, "StoppedTask3", 0},
}

// Output
{ID:1 Name:RunningTask1 State:1}
{ID:2 Name:RunningTask2 State:1}
{ID:2 Name:StoppedTask1 State:1}
{ID:3 Name:StoppedTask2 State:1}
{ID:4 Name:StoppedTask3 State:1}
Stopped: 0, Running: 5

The weird phenomenon appears, when StoppedTask first, the result is all right; when RunningTask first, RunningTasks are correct, StoppedTasks are all wrong, and even the ID of StoppedTask1, which is originally 0, has become 2 here!

After repeated testing and observation of the result set, we had come to the conclusion that all the RunningTasks data is fine, and all the problems are with the StoppedTasks data!

To further formalize this conclusion: If the preceding data is a non-zero number, and the following data is a zero, then the following zero will be overwritten by the non-zero in front of it.

3 Explanation

Reviewing the program code again, I noticed this sentence:

// Use a temporary pointer as a carrier of objects in the file
// to save the cost of creating new objects.
tmpT := &task{}

In order to save the extra overhead of creating a new object, I used the same variable to recursively load the data in the file.

Combined with the bug pattern we found earlier, the answer seems to be close at hand: the so-called 0's are being overwritten by non-zero's in front of them, most likely because the same object is being used to load the file, which results in residual data in front of it.

The validation is also simple, just put that public object into the following for loop so that each loop recreates a new object for loading the file data to cut off the effects of the previous data.

Let's modify the code:

for _, t := range tasks {
    tmpT := &task{}
    // ...skip
}

// Output
{ID:1 Name:RunningTask1 State:1}
{ID:2 Name:RunningTask2 State:1}
{ID:0 Name:StoppedTask1 State:0}
{ID:3 Name:StoppedTask2 State:0}
{ID:4 Name:StoppedTask3 State:0}
Stopped: 3, Running: 2

Yes! It’s right!

The result is indeed as we thought, it is the reason of data residuals. But here's another question: Why is it that the old method reads everything correctly when there are 0s and then non-0s (Stopped first then Running tasks)? And why are all other numbers (e.g. non-zero IDs) not affected, except for the 0's?

All questions now seem to point to the special number 0!

It was only at this point that the zero-value property finally caught up with us. So I rushed to read the official documentation of the gob library and found this sentence:

If a field has the zero value for its type (except for arrays; see above), it is omitted from the transmission.

The words before and after this sentence are talking about struct, so field here also refers to a field in a structure, in line with the example in our text.

Based on the conclusions we got earlier, and the official documentation, we can now finally come to a complete conclusion:

The gob library ignores zero values outside of arrays when manipulating data. Whereas our code starts out using a public object to load the file data, since zero values are not transmitted, the fields in the original data that are zero-valued are not read, and we actually see the data from the last non-zero-valued object.

For the example in this article, the solution is also very simple, is what I did above, do not use a public object to load data.

4 Recall

In the project bug I described at the beginning of this article, I also used 0 and 1 to indicate the state of a timed task (stopped, running). As a result, different tasks were interfering with each other due to the zero value issue, which caused the task to execute abnormally, while the other fields that did not involve the zero value were fine. Although this is an online production environment, fortunately, the problem was found early and handled in time, and did not cause any production accidents. However, the whole process and the final answer were deeply imprinted in my mind.

Later I briefly discussed with my colleague why gob chose to ignore the zero value. From my perspective, it was probably to save space. The code we wrote at the beginning also created a public object to save space, and as a result, the two space-saving logics eventually collided to create a hidden bug.