Big chunk vs smaller chunks on transferring data

Jodhi — Fri, 29 May 2020 15:58:53 +0000

While working on online blake3 hasher, we want to support subjectively quite large file. Since default limit of grpc is around 4MB (on go lib at least), we got this idea : to chunk or not to chunk?

Then we are doing simple test to know what the impact of one very big sized data when transferred.

first thing first, we need to increase the limit of message size

const (
    maxMsgSize     = 900000000
)
...
conn, err := grpc.Dial(
        "127.0.0.1:7777",
    grpc.WithInsecure(),
    grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(maxMsgSize),grpc.MaxCallSendMsgSize(maxMsgSize)),
)

...
GRPCArgs := &pb.Request{
        Input:      []byte(strings.Repeat("A", 200000000)),
    }

r, err := pb.NewCallClient(conn).Handler(ctx, GRPCArgs)

Basically, the code send 200*1 million bytes and receive the base64 result with size of 266666668 bytes.

You can see that the first increase (around 320MB) and decrease on memory because of the transfer process, and the last decrease on the graph is when I close all rpc server and client (GC maybe?).

So, to chunk or not to chunk? Subjectively, If you have the time and short on resources, go for chunking.

Check out also the csv linter with big size file support

BLAKE3 Go Libraries simple comparison

Jodhi — Sat, 23 May 2020 19:34:03 +0000

Official Go Library for BLAKE3 is not available at the time of this post.

But, we have two alternative:
github.com/lukechampine/blake3
A pure Go implementation of the BLAKE3 cryptographic hash function

github.com/zeebo/blake3
Pure Go implementation of BLAKE3 with AVX2 and SSE4.1 acceleration

CPU used: intel i7-8550U

Simple benchmark code

package main

import (
    "bytes"
    "encoding/hex"
    "fmt"
    "testing"

    zeeboBLAKE3 "github.com/zeebo/blake3"
    lukechampineBLAKE3 "lukechampine.com/blake3"
)

var longBytes = bytes.Repeat([]byte("A"), 1000000)

func benchmarkLukechampine(b *testing.B) {
    for i := 0; i < b.N; i++ {
        hasher := lukechampineBLAKE3.New(32, nil)
        hasher.Write([]byte(longBytes))
        hex.EncodeToString(hasher.Sum(nil))
    }
}
func benchmarkZeebo(b *testing.B) {
    for i := 0; i < b.N; i++ {
        hasher := zeeboBLAKE3.New()
        hasher.Write([]byte(longBytes))
        hex.EncodeToString(hasher.Sum(nil))
    }
}

func main() {
    fmt.Println("lukechampine 1st test:", testing.Benchmark(benchmarkLukechampine))
    fmt.Println("zeebo 1st test:", testing.Benchmark(benchmarkZeebo))
    fmt.Println("lukechampine 2nd test:", testing.Benchmark(benchmarkLukechampine))
    fmt.Println("zeebo 2nd test:", testing.Benchmark(benchmarkZeebo))
}

The result

lukechampine 1st test:      577    2171834 ns/op
zeebo 1st test:     4146        302500 ns/op
lukechampine 2nd test:      532    2282167 ns/op
zeebo 2nd test:     3448        331177 ns/op

Performance wise, the zeebo lib is faster. This benchmark is still limited since other resources not taken into consideration. If you guys want to try BLAKE3 hasher online or generate fancy text. Thank you.

Forem: Jodhi

Big chunk vs smaller chunks on transferring data

BLAKE3 Go Libraries simple comparison