Forem: Taq Karim

You might not be using json.Decoder correctly in golang

Taq Karim — Mon, 25 Jan 2021 02:58:11 +0000

TL;DR: prevailing "secondary source" wisdom (ie: blog posts) about json.Decoder don't demonstrate the proper way to use it.

This post is a follow up to my (kinda lengthy) deep dive into what I thought was a bug in golang's json.Decoder pkg.

Instead, I realized that generally speaking, json.Decoder can be misunderstood - which may lead to unintended consequences. In this post, I will demonstrate a safer pattern that ought to be used instead of the prevailing wisdom.

Googling: "json.decoder example golang"

I ran a few google searches queries using some permutation of the following:

json.decoder example golang

The results were your standard mix of documentation from golang.org, blog posts on medium.com/random mom&pop devsites (like this one!), and a few stackoverflow threads.

Fortunately, results from golang.org are highly ranked - while it may be a bit harder to parse through golang's src code, the documentation is fairly thorough and more importantly: correct. (Personally, I would have preferred some additional context in the docs expounding on some of the gotchas I discuss on my other post but I digress)

Some of the threads I observed in Stack Overflow that referenced json.Decoder pulled directly from the docs (and therefore were also correct). Other's (probably) pulled from medium/other blog post sites similar to the ones I found googling around and were inaccurate/advocating incorrect usage.

For example, on Medium, et al - I saw a wide array of posts (such as this, this, this or this) that suggested using json.Decoder in some way, shape, or form similarly to this:



func main() {
    jsonData := `{
"email":"abhirockzz@gmail.com",
"username":"abhirockzz",
"blogs":[
    {"name":"devto","url":"https://dev.to/abhirockzz/"},
    {"name":"medium","url":"https://medium.com/@abhishek1987/"}
]}`

    jsonDataReader := strings.NewReader(jsonData)
    decoder := json.NewDecoder(jsonDataReader)
    var profile Profile
    err := decoder.Decode(&profile)
    if err != nil {
        panic(err)
    }
    // ...
}

(Example pulled from Tutorial: How to work with JSON data in Go)

The Problem

On the surfact, this code looks sound. I forklifted the src into go playground and ran it.



package main

import (
    "encoding/json"
    "strings"
)

func main() {
    jsonData := `{
"email":"abhirockzz@gmail.com",
"username":"abhirockzz",
"blogs":[
    {"name":"devto","url":"https://dev.to/abhirockzz/"},
    {"name":"medium","url":"https://medium.com/@abhishek1987/"}
]}`

    jsonDataReader := strings.NewReader(jsonData)
    decoder := json.NewDecoder(jsonDataReader)
    var profile map[string]interface{}
    err := decoder.Decode(&profile)
    if err != nil {
        panic(err)
    }
    // ...
}

(Note, changed profile to map[string]interface{} to make the code run)

...It works! Great. But, what happens if we fubar the JSON string?



package main

import (
    "encoding/json"
    "strings"
)

func main() {
    jsonData := `{
"email":"abhirockzz@gmail.com",
"username":"abhirockzz",
"blogs":[
    {"name":"devto","url":"https://dev.to/abhirockzz/"},
    {"name":"medium","url":"https://medium.com/@abhishek1987/"}
]}THIS IS INTENTIONALLY MALFORMED NOW`

    jsonDataReader := strings.NewReader(jsonData)
    decoder := json.NewDecoder(jsonDataReader)
    var profile map[string]interface{}
    err := decoder.Decode(&profile)
    if err != nil {
        panic(err)
    }
    // ...
}

(playground)

...It works!

Wait...WTF?!.

This code should not work at all! Our JSON string is clearly malformed and we expect - based on the logic - the code to panic.

This is the entire issue in a nutshell. I expound in detail in my other post but in one (kinda long) sentence:

json.Decoder.Decode was implemented for parsing streaming JSON data, meaning it will always traverse the JSON string until it finds a satisfactory, closing bracket (I use the term satisfactory here because it does use a stack to keep track of inner brackets).

So, in order to detect the malformed json, we must actually run this logic in a loop - like so:



package main

import (
    "encoding/json"
    "io"
    "strings"
)

func main() {
    jsonData := `{
"email":"abhirockzz@gmail.com",
"username":"abhirockzz",
"blogs":[
    {"name":"devto","url":"https://dev.to/abhirockzz/"},
    {"name":"medium","url":"https://medium.com/@abhishek1987/"}
]}THIS IS INTENTIONALLY MALFORMED NOW`

    jsonDataReader := strings.NewReader(jsonData)
    decoder := json.NewDecoder(jsonDataReader)
    var profile map[string]interface{}
    for {
        err := decoder.Decode(&profile)
        if err != nil {
            panic(err)
        }
        if err == io.EOF {
            break
        }
    }

    // ...
}

(playground)

Note the key diff here:



var profile map[string]interface{}
for {
    err := decoder.Decode(&profile)
    if err != nil {
        panic(err)
    }
    if err == io.EOF {
        break
    }
}
// ...

The Fix

From the golang docs, this example says it best:



// This example uses a Decoder to decode a stream of distinct JSON values.
func ExampleDecoder() {
    const jsonStream = `
    {"Name": "Ed", "Text": "Knock knock."}
    {"Name": "Sam", "Text": "Who's there?"}
    {"Name": "Ed", "Text": "Go fmt."}
    {"Name": "Sam", "Text": "Go fmt who?"}
    {"Name": "Ed", "Text": "Go fmt yourself!"}
`
    type Message struct {
        Name, Text string
    }
    dec := json.NewDecoder(strings.NewReader(jsonStream))
    for {
        var m Message
        if err := dec.Decode(&m); err == io.EOF {
            break
        } else if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%s: %s\n", m.Name, m.Text)
    }
    // Output:
    // Ed: Knock knock.
    // Sam: Who's there?
    // Ed: Go fmt.
    // Sam: Go fmt who?
    // Ed: Go fmt yourself!
}

(sauce)

In this usage, we create a new json.Decoder instance from the NewDecoder method and then continually loop and try to decode a chunk of our JSON string until we detect the end of the string (sucessfully breaking out of the loop) or an error.

I would take this a step further and only prefer to use json.Decoder when I am specifically working with streaming JSON data.

At any rate, this the The Way. Please keep this in mind going forward should you choose to use json.Decoder for your JSON string parsing needs.

A few notes

A big caveat I'd like to point out is that I did not perform any rigorous analysis of golang examples in the wild when I was inspired to write this post. My analysis comes largely from anecdotal examples I observed while researching my other deep dive article about the nature of json.scanner and json.Decoder.Decode.
I certianly am not being critical of the medium/mom&pop blogs I linked to earlier, instead I am merely pointing out that it is very easy to assume that the trivial usage of json.Decoder.Decode presented on those posts is generally correct when in actuality this is not the case.
I think someone (maybe me? perhaps you?) ought to open a PR against godocs to clarify gotchas/differences between json.Decoder usage vs json.Unmarshal

PS: this article is x-posted from my dev blog

Custom Args in Makefile

Taq Karim — Sun, 03 Jan 2021 05:59:04 +0000

TL;DR: use $(eval ARGS=${ARGS} [some additional arg]) within a make target to build custom argument sequences for commands wrapped by make targets - like make test

I expound further on the usecase and methodology below 👇

I like to wrap common tasks, such as running unit tests, around a make target.

This way, I can minimize the length of the command I need to run (ie: make test vs go test ./... -race -coverprofile=c.out)

However, as a project grows, it becomes necessary or just preferable to support a variety of permutations of the above.

(Note: I am not advocating that one ought to use make targets in this manner, just that if this route is chosen, there are patterns available to simplify things a bit).

For the sake of go, here are a few potential tests I'd like to be able to run:

Run tests w/race conditions check

go test ./... -race

Run tests in "short" mode

go test ./... -short

Run tests w/verbose results in terminal

go test ./... -v

Run tests for a specific package

go test ./my_awesome_pkg/...

Run a specific test func in a specific package

go test ./my_awesome_pkg/... -r=TestFoobar

A better way

Of course, the main issue that comes up here is: what if we wanted to mix and match some of these opts? (For instance, we might want to run tests on a specific package with verbose test output with race condition checks enabled.)

Again, assuming we'd like to reuse the make test interface, our make target can get really messy really quickly:

#   Usage:
#       make test-dev
#       make test-dev package=my_awesome_pkg
test:
ifdef package 
    go test -v ./${package}/... -coverprofile=c.out
else
    go test ./... -coverprofile=c.out
endif

In this example, we only support two use cases, run all tests in a single package and run all tests in all packages in current working directory. Even so, as you can probably see, it is easy to miss stuff (for instance, package tests have the verbose flag) and repetition can start to seep in (note the -coverprofile arg in both conditions).

An alternative (and personally, preferable approach might be):

#   Usage:
#       make test norace=1
#           Expl: run WITHOUT race conditions
#       make test ftest=1
#           Explt: run WITH "ftests", long lived non unit tests
#       make test v=1
#           Explt: run in verbose mode
#       make test package=my_awesome_pkg
#           Explt: run tests in a single package only
#           NOTE: if omitted, will run ALL tests
#       make test package=my_awesome_pkg func=TestViews
#           Explt: run a single test func (needs package as well)
test:
    $(eval ARGS=)
# by default, run with race conditions
ifndef norace
    $(eval ARGS=${ARGS} -race)
endif
# by default, run in "short" mode
ifndef ftest
    $(eval ARGS=${ARGS} -short)
endif
ifdef v
    $(eval ARGS=${ARGS} -v)
endif
# if package provided, run the package
ifdef package
    $(eval ARGS=${ARGS} ./${package}/...)
else
    $(eval ARGS=${ARGS} ./...)
endif
# if func provided, run the func only
ifdef func
    $(eval ARGS=${ARGS} -run=${func})
endif
    go test ${ARGS} -coverprofile=c.out

In this approach, we use $(eval ARGS=${ARGS} [append an arg] to build the actual args for our go test invocation based on make target args.

What's nice about this approach is we can choose to make certain things default vs non default by leveraging make's ifdef and ifndef (if not defined) conditionals.

The last line is the actual test invocation and if we wanted to, we could always echo ${ARGS} right before for debugging purposes, etc.

I really like this approach and have started using it in a lot of my makefile target patterns where I leverage make test for dev-ing or ci related tasks. While I've walked through a golang based usecase, this pattern can be used for more or less any make target usage (though personally, I really only use this for running tests during dev/ci).

Integrating Code Climate w/go pkgs

Taq Karim — Fri, 01 Jan 2021 14:28:32 +0000

TL; DR: When integrating golang code coverage with Code Climate, be sure to set the --prefix arg (set it to your pkg github prefix, ie: "github.com/mottaquikarim/esquerydsl") in the after-build stage in order for your src code to be recognized by the test reporter.

If ^^ that didn't make sense, read on below for the full background 👍

I've recently published a golang pkg, called esquerydsl, which facillitates safe construction of elasticsearch queries in golang.

In order to ensure future stability, I wanted to integrate Code Climate for tracking maintainability and test coverage.

I've used CC before on python projects but never for go and while integrating, I ran into a gotcha that I thought I ought to document / share.

Step 0: Write and run some tests!

I won't expound on the actual source code and instead focus on how to run and export test + results to code climate.

Running tests in go is super easy:

go test ./...

I prefer to work in docker so I usually run the above as:

docker run -w /app -v ${PWD}:/app golang:1.15 go test ./...

(All examples here on out will be run in docker)

Let's break this down before continuing:

docker run: within a docker container, let's run some stuff
-w /app: our "working directory" will be the app folder
-v ${PWD}:/app: we volume mount our current working directory into the "working directory" in the container. This is a nice trick for quickly running tests on source code you are writing during dev
golang:1.15: the image we want to run - in this case, golang v1.15
go test ./...: finally, the command we want to run in our docker container - which is of course our go test command

Track code coverage

Tracking code coverage in golang is also super easy. We just need to add a single arg to your go test command:

-coverprofile=c.out

The -coverprofile arg tells go where to publish the test coverage results, which can be parsed later on to help visualize your source code's coverage results.

Putting it all together:

docker run -w /app -v ${PWD}:/app golang:1.15 go test ./... -coverprofile=c.out

Now, there is a file called c.out which is available in your current working directory that encodes your line-by-line test coverage data. Great! All we have to do now is upload this to code climate and we are golden.

Sidebar: generating code coverage results as HTML

This is a proper tangent, so feel free to ignore.

I find it super useful to understand/view which lines I missed in my unit tests. While line by line test coverage does not necessarily guarantee stability, I find it to be a useful benchmark for knowing when to stop writing unit tests (somewhere around 90+% is my 👍 point)

To visualize this in go, run the following after your tests:

docker run -w /app -v ${PWD}:/app golang:1.15 go tool cover -html=c.out -o coverage.html

coverage.html can be opened in your browser to walk through your src code and it will visually indicate which lines have been covered by your tests and which lines have been missed.

Getting started with CodeClimate

I'm going to skip this because otherwise this post will get long. My assumption is folks reading this already have a CC Quality account and have a (golang!) repo set up for usage.

Step 1: Downloading the test reporter

Download and run the Code Climate test reporter. Of course, we want to do this in our docker container (there are plenty of examples available on how to do this "bare" around the internet). For me, this entailed:

# 1. download CC test reported
CC_URL=https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64
docker run -w /app -v ${PWD}:/app golang:1.15 \
    /bin/bash -c \
    "curl -L ${CC_URL} > ./cc-test-reporter"

# 2. update perms
docker run -w /app -v ${PWD}:/app golang:1.15 chmod +x ./cc-test-reporter

# 3. run before build
docker run -w /app -v ${PWD}:/app \
    -e CC_TEST_REPORTER_ID=${CC_TEST_REPORTER_ID} \
    golang:1.15 ./cc-test-reporter before-build

The three main parts here:

We need to download the test reporter, which is available to curl via the CC_URL. I run this in two steps in a single docker run command which is why I have the /bin/bash -c "...steps to run..." line (you can do it in two lines if needed as well)
Update perms to make this download executable. Notice that I run chmod on the renamed file cc-test-reporter (you can call this whatever you want) within the container itself.
Finally, run before-build which sets up the test reporter to send the coverage data. The CC_TEST_REPORTER_ID is significant - this is how CC authenticates your repo's code coverage as really from you. I have it as an envvar since it should not be committed to code anywhere

Step 2: Running the tests with code coverage

This step is easy, basically Step 0:

docker run -w /app -v ${PWD}:/app golang:1.15 go test ./... -coverprofile=c.out

Step 3: Running the after build

This is where I ran into my issue. The easy / documented step is this:

# upload data to CC
docker run -w /app -v ${PWD}:/app \
    -e CC_TEST_REPORTER_ID=${CC_TEST_REPORTER_ID} \
    golang:1.15 ./cc-test-reporter after-build

This is pretty standard at this point: we run the cc-test-reporter with the after-build arg as directed by CodeClimate. And, we pass along the CC_TEST_REPORTER_ID to property identify ourselves. However, the kicker is the command above alone will not work!

When I ran the line above for my esquerydsl package (heads up, I temporarily added the --debug flag too), I ended up with the following output:

time="2020-12-31T19:52:57Z" level=error msg="failed to read file github.com/mottaquikarim/esquerydsl/esquerydsl.go\nopen github.com/mottaquikarim/esquerydsl/esquerydsl.go: no such file or directory"
Error: open github.com/mottaquikarim/esquerydsl/esquerydsl.go: no such file or directory

To fix, it looks like the --prefix is required, as follows:

docker run -w /app -v ${PWD}:/app \
    -e CC_TEST_REPORTER_ID=${CC_TEST_REPORTER_ID} \
    golang:1.15 ./cc-test-reporter after-build --prefix=github.com/mottaquikarim/esquerydsl

Note the --prefix=github.com/mottaquikarim/esquerydsl. The full package prefix url is needed for this to work. Once I added this arg, I was able to upload code coverage stats for my pkg just fine. (Check it out here).

The Code

I have this thing tied up in a neat bow at this point and thought I'd share. I use make targets to handle running this in Github Actions. I'll share the recipes below in the hopes maybe it can help someone else in the future.

`Makefile`

# Definitions
ROOT                    := $(PWD)
GO_HTML_COV             := ./coverage.html
GO_TEST_OUTFILE         := ./c.out
GOLANG_DOCKER_IMAGE     := golang:1.15
GOLANG_DOCKER_CONTAINER := goesquerydsl-container
CC_TEST_REPORTER_ID := ${CC_TEST_REPORTER_ID}
CC_PREFIX       := github.com/mottaquikarim/esquerydsl

#   Usage:
#       make test
test:
    docker run -w /app -v ${ROOT}:/app ${GOLANG_DOCKER_IMAGE} go test ./... -coverprofile=${GO_TEST_OUTFILE}
    docker run -w /app -v ${ROOT}:/app ${GOLANG_DOCKER_IMAGE} go tool cover -html=${GO_TEST_OUTFILE} -o ${GO_HTML_COV}

# custom logic for code climate, gross but necessary
_before-cc:
    # download CC test reported
    docker run -w /app -v ${ROOT}:/app ${GOLANG_DOCKER_IMAGE} \
        /bin/bash -c \
        "curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter"

    # update perms
 docker run -w /app -v ${ROOT}:/app ${GOLANG_DOCKER_IMAGE} chmod +x ./cc-test-reporter

    # run before build
 docker run -w /app -v ${ROOT}:/app \
        -e CC_TEST_REPORTER_ID=${CC_TEST_REPORTER_ID} \
       ${GOLANG_DOCKER_IMAGE} ./cc-test-reporter before-build

_after-cc:
    # handle custom prefix
    $(eval PREFIX=${CC_PREFIX})
ifdef prefix
    $(eval PREFIX=${prefix})
endif
    # upload data to CC
    docker run -w /app -v ${ROOT}:/app \
        -e CC_TEST_REPORTER_ID=${CC_TEST_REPORTER_ID} \
        ${GOLANG_DOCKER_IMAGE} ./cc-test-reporter after-build --prefix ${PREFIX}

# this runs tests with cc reporting built in
test-ci: _before-cc test _after-cc

`Github Action`

I saved this file in: .github/workflows/run-build.yml and in my repo secrets, I saved my CC_TEST_REPORTER_ID so that I can export it as envvar on build time.

name: Build Status

on:
  push:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout master
      uses: actions/checkout@v2

    - name: Run unit tests
      run: |
        make clean test-ci
      env:
        CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}

Note that running make test-ci will run tests with the codeclimate reporter running and make test just runs normal unit tests for dev/debugging.

The results of the configs above should be viewable in the following:

Happy coding!

Generate Lambda Layer Artifacts w/Docker

Taq Karim — Mon, 21 Dec 2020 13:55:01 +0000

This post (and accompanying code) was inspired by this tutorial and a need to easily generate lambda layer artifacts for a grad course I teach at Baruch University.

Github

TL;DR: Generate a lambda layer artifact for python using this:

➜  cat >> requirements.txt
requests==2.24.0
➜  docker run -v $PWD:/data mottaquikarim/pkglambdalayer:latest

On completion, expect the following:

➜  ls -ahl
total 1792
drwxr-xr-x   5 tkarim  staff   160B Dec 21 08:36 .
drwxr-xr-x  23 tkarim  staff   736B Dec 21 08:31 ..
drwxr-xr-x  13 tkarim  staff   416B Dec 21 08:36 pkg
-rw-r--r--   1 tkarim  staff   879K Dec 21 08:36 pkg.zip
-rw-r--r--   1 tkarim  staff    17B Dec 21 08:36 requirements.txt

pkg.zip can be uploaded into your own Lambda Layer using the AWS Console 👍👍

The Backstory

I teach a course at Baruch University where I introduce folks to various AWS technologies in the context of "Big Data" processing.

So of course, Lambdas are a topic I cover. In order to keep the course load from being too technical, I try to lean hard on AWS console whenever possible vs going too deeply into the infra-as-code approach.

As such, most of our explorations in Lambda-land are done through the AWS Lambda UI.

The Problem

You can't actually load in dependencies into the AWS Lambda UI!

While I totally understand perhaps why this is so, it definitely throws a bit of a curveball in my class as I want folks to write useful lambda funcs but without having to delve too deeply into the packaging process.

Lambda Layers and Docker

I found Lambda Layers to be an acceptable compromise - last semester I just generated a single artifact similar to above which I shared via Google Drive and had folks go through the motions of uploading and leveraging the dependencies.

This process seemed to be a good candidate for automation so as a first step, I packaged it into a docker image. Folks in my class have basic working knowledge of pulling / running docker images so next semester I hope to leverage this without having to pre-gen a single build artifact on my own.

Plus, this will empower folks in class to perhaps do more with lambda if it is easier to create various combinations of dependencies in thier layers.

I figured maybe this would be interesting/useful to others so I wrapped into a Github repo and am publishing/sharing.

Safely Construct Elasticsearch Queries w/Golang

Taq Karim — Wed, 16 Dec 2020 14:28:28 +0000

TL; DR: ya boy wrote a golang elasticsearch query dsl utility. Find it here!

The Why

If you've used elasticsearch with golang, then you've probably used the official elasticsearch go client.

The es go client is exhaustive and generally, pretty great. However, it can be a bit...scary when having to deal with constructing search queries using the elasticsearch query dsl

Take for instance the following (from here in the docs):

GET /_search
{
  "query": { 
    "bool": { 
      "must": [
        { "match": { "title":   "Search"        }},
        { "match": { "content": "Elasticsearch" }}
      ],
      "filter": [ 
        { "term":  { "status": "published" }},
        { "range": { "publish_date": { "gte": "2015-01-01" }}}
      ]
    }
  }
}

Using strings

In my experience, the simplest/fastest way to construct this json string is with...well, a string:

elasticQuery := `
{
  "query": { 
    "bool": { 
      "must": [
        { "match": { "title":   "Search"        }},
        { "match": { "content": "Elasticsearch" }}
      ],
      "filter": [ 
        { "term":  { "status": "published" }},
        { "range": { "publish_date": { "gte": "2015-01-01" }}}
      ]
    }
  }
}
`

If we need to inject variable values, we just use fmt.Sprintf and move on with our lives. The primary issue here is that validating/formatting these json strings require additional work and can be error prone (ie: fat-fingering an additional comma somewhere, etc).

Using (use-case specific) structs

The other approach would be to make everything hyper specific and create structs / custom marshal-ers that would generate the query DSL json format when json.Marshal is called (on said custom struct(s)).

This approach requires creating custom structs and code for the sole purpose of building these queries. (This may work for certain usecases! But, it also means more code and therefore additional maintenance and more trouble translating to other projects).

What does Google say?

The best "documentation"/support I could find through google-fu was this article that also just suggested building a json string and crossing your fingers.

(excerpt from the blog post above):

func constructQuery(q string, size int) *strings.Reader {

    // Build a query string from string passed to function
    var query = `{"query": {`

    // Concatenate query string with string passed to method call
    query = query + q

    // Use the strconv.Itoa() method to convert int to string
    query = query + `}, "size": ` + strconv.Itoa(size) + `}`
    fmt.Println("\nquery:", query)

    // Check for JSON errors
    isValid := json.Valid([]byte(query)) // returns bool

    // Default query is "{}" if JSON is invalid
    if isValid == false {
        fmt.Println("constructQuery() ERROR: query string not valid:", query)
        fmt.Println("Using default match_all query")
        query = "{}"
    } else {
        fmt.Println("constructQuery() valid JSON:", isValid)
    }

    // Build a new string from JSON query
    var b strings.Builder
    b.WriteString(query)

    // Instantiate a *strings.Reader object from string
    read := strings.NewReader(b.String())

    // Return a *strings.Reader object
    return read
}

Finally, there was this issue on the go elasticsearch client from 2019:

This package is intentionally a low-level client, while olivere/elastic is a high-level client with extensions for building the requests and deserializing the responses. We are aiming for offering a more high-level API in the future, but — as I've indicated in other tickets — no sooner than a machine-readable, formal specification of the request and response bodies is available.

Moreover (and awesomely, that comment thread has a gem of a code snippet):

// BoolQuery Elastic bool query
type BoolQuery struct {
    Bool BoolQueryParams `json:"bool"`
}

// BoolQueryParams params for an Elastic bool query
type BoolQueryParams struct {
    Must               interface{} `json:"must,omitempty"`
    Should             interface{} `json:"should,omitempty"`
    Filter             interface{} `json:"filter,omitempty"`
    MinimumShouldMatch int         `json:"minimum_should_match,omitempty"`
}

that looks very similar to what I've ended up with (wish I had seen this first, heh) as I tackled this problem.

Regardless the main point is this:

Currently there isn't an easy way to define query DSL json strings for use with the elasticsearch go client.

And - for good reason perhaps - the official client looks like it will not support such a feature anytime soon.

`package esquerydsl`

For all these reasons, I decided to build a simple, dependency less (aside from go stdlib deps) utility that generically defines structs to build queryDSL json strings.

Here's an example (playground)

package main

import (
    "fmt"

    "github.com/mottaquikarim/esquerydsl"
)

func main() {
    _, body, _ := esquerydsl.GetQueryBlock(esquerydsl.QueryDoc{
        Index: "some_index",
        Sort:  []map[string]string{map[string]string{"id": "asc"}},
        And: []esquerydsl.QueryItem{
            esquerydsl.QueryItem{
                Field: "some_index_id",
                Value: "some-long-key-id-value",
                Type:  "match",
            },
        },
    })
    fmt.Println(body)
}

The output:

{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "some_index_id": "some-long-key-id-value"
          }
        }
      ]
    }
  },
  "sort": [
    {
      "id": "asc"
    }
  ]
}

(Find more examples in tests, including the initial queryDSL example referenced at the top of this post here)

PRs welcome! Especially re: unittests such that documentation coverage is increased. If you use this lib and it is useful, do let me know please!

Happy querying, fam 👍

Gotcha w/comparing base64 encoded strs

Taq Karim — Sun, 13 Dec 2020 09:29:57 +0000

Here's a fun experiment - consider two "numerical" strings such as:

a = "1004"
b = "1053"

Is `a < b`?

Why yes, of course it is!

Ok, let's try this again, this time b64encoding our two strings.

import base64

a_b64 = base64.b64encode(a.encode("utf8")).decode("utf8")
# MTAwNA==
b_b64 = base64.b64encode(b.encode("utf8")).decode("utf8")
# MTA1Mw==

How about now?

Is `a_b64 < b_b64`?

NOPE.

What gives?!

And more importantly, this observation implies that:

Given two strings a, b such that a < b it is not necessarily true that b64(a) < b64(b)!

(This is the key takeaway, the rest of this post explains why this is true for some cases)

But why though?

To answer this question, we must delve into the b64 encoding algorithm itself. Let's consider 1004 as an example.

Step 1: split the string by digit

1004
1
0
0
4

Step 2: convert each char to binary equivalent in ASCII

1004	ASCII value	Binary
1	49	00110001
0	48	00110000
0	48	00110000
4	52	00110100

Step 3: concatenate the binary representation

1004 => 00110001001100000011000000110100

Step 4: partition the above but now in groups `6`

1004 (in groups of 6)
001100
010011
000000
110000
001101
000000 (pad 4 to complete final group

Step 5: pad-left each partition to convert our 6 bit "byte" into an 8 bit byte

1004
00001100
00010011
00000000
00110000
00001101
00000000

Step 6: convert back to decimal (base 10) and translate by looking up each decimal value in the base64 characters table

1004	decimal	b64 character
00001100	12	M
00010011	19	T
00000000	0	A
00110000	48	w
00001101	13	N
00000000	0	A

So, 1004 => MTAwNA

Tada!

Now (do try this at home, if you want): repeat this exercise for 1053, resulting in:

1053 => MTA1Mw

Note that MTAwNA > MTA1Mw.

The key to understanding why b64(a) !< b64(b) has to do with Step 6.

We convert our new, transformed decimal values according to the base64 characters table - which places numbers such as 1-9 as higher than characters (in terms of index value).

Notice how 1 has decimal value 53 and w has decimal value 48. Contrast this to the ASCII table:

Note that "1" has a decimal value of 49.

But "w" has a decimal value of 77!

So, when comparing "w" > "1" as strings, the expression is evaluated as True!

In other words, when we convert our bits from Step 4 to characters, we splice the binary representation because our total # of bits is not divisible by 6 (see my PPS for more info).

Then we convert this spliced value in our b64 table where the numeric chars (1-9) have indices that are higher than the indices of the non-numeric chars (A-Z and a-z).

For this reason, when we attempt to perform a string compare, 1004 in b64 encoded form (MTAwNA) is indeed less than 1053 in b64 encoded form (MTA1Mw) according to the rules of b64 translation but not according to the ASCII translation (see my PS for more info).

In ASCII format, MTAwNA is indeed "greater" than MTA1Mw since char position 3 in MTAwNA (w) actually has a numerical index that is higher than char position 3 in MTA1Mw (1).

Womp womp.

Ok but...why care about this?

Well, long story short: friends don't let friends take fields in their elasticsearch index, b64 encode them into a key and then run queries sorting by these keys because well...it won't work.

I had one hell of a time figuring this out and even when I observed it, I had trouble justifying this to myself hence this post (which I used to work out my feelings on the matter).

I feel slightly better now.

PS: in py3 we compare unicode values not ascii when performing string comparison ops - but for the purposes of this post the decimal values are the same since we are only contending with A-Z, a-z,0-9,+,/
PPS: 8N mod 6 will always have remainder 2, 4, or 0. This problem seems to show up specifically when (8N mod 6) == 2 (ie: we pad 4 0s to the right of our last row). The mod 0 case is obvious but I've yet to explain why (8N mod 6) == 4 does not seem to this problem.

Extending SimpleNamespace for Nested Dictionaries

Taq Karim — Mon, 13 Apr 2020 08:35:33 +0000

I'm a huge fan of Python3's types.SimpleNamespace. Basically, it allows us to take a dict, like so:

my_dict = {
  "a": 1,
  "b": 2,
  "c": 3,
}

my_dict["a"] # 1
my_dict["b"] # 2, etc

and manage it like this:

from types import SimpleNamespace
my_namespace = SimpleNamespace(a=1, b=2, c=3)

my_namespace.a # 1
my_namespace.b # 2
my_namespace.c # 3

Alternatively, we could also do something like:

from types import SimpleNamespace

my_dict = {
  "a": 1,
  "b": 2,
  "c": 3,
}

my_namespace = SimpleNamespace(**my_dict)
my_namespace.a # 1
my_namespace.b # 2
my_namespace.c # 3

But - what happens if our my_dict is nested? Like so:

my_dict = {
  "a": {
    "d": 4,
  },
  "b": 2,
  "c": 3,
  "e": [5,6,7,{
    "f": 8,
  }]
}

my_namespace = SimpleNamespace(**my_dict)
my_namespace.a # {"d": 4} /womp womp 😭
my_namespace.a.d # raises Exception! 😭😭

In short, SimpleNamespace simply does not support this use case. But that's ok! We can extend SimpleNamespace and build this functionality for ourselves.

Defining `RecursiveNamespace`


from types import SimpleNamespace

# this is how SimpleNamespace looks when output
SimpleNamespace(**my_dict)
# namespace(a={'d': 4}, b=2, c=3, e=[5, 6, 7, {'f': 8}])

class RecursiveNamespace(SimpleNamespace):

  @staticmethod
  def map_entry(entry):
    if isinstance(entry, dict):
      return RecursiveNamespace(**entry)

    return entry

  def __init__(self, **kwargs):
    super().__init__(**kwargs)
    for key, val in kwargs.items():
      if type(val) == dict:
        setattr(self, key, RecursiveNamespace(**val))
      elif type(val) == list:
        setattr(self, key, list(map(self.map_entry, val)))

# this is how RecursiveNamespace looks when output
RecursiveNamespace(**my_dict)
# RecursiveNamespace(
#    a=RecursiveNamespace(d=4), 
#    b=2, 
#    c=3, 
#    e=[5, 6, 7, RecursiveNamespace(f=8)])

So, what's happening here? We establish a new class, RecursiveNamespace that extends SimpleNamespace. In the __init__ constructor method, we call SimpleNamespace's constructor. Then, we just walk through our dictionary and for value that is also a dictionary or list, we instantiate that with RecursiveNamespace. Ta da.

P.S.

Technically, we don't even really need types.SimpleNamespace here - we can implement this class without by just adding two lines of code:

class RecursiveNamespace2: # without extending SimpleNamespace!

  @staticmethod
  def map_entry(entry):
    if isinstance(entry, dict):
      return RecursiveNamespace(**entry)

    return entry

  def __init__(self, **kwargs):
    for key, val in kwargs.items():
      if type(val) == dict:
        setattr(self, key, RecursiveNamespace(**val))
      elif type(val) == list:
        setattr(self, key, list(map(self.map_entry, val)))
      else: # this is the only addition
        setattr(self, key, val)

Self-grading quizzes with Airtable

Taq Karim — Mon, 16 Mar 2020 13:37:13 +0000

This semester, I am teaching a course at Baruch University on Big Data Technologies.

I resolved to administer a small quiz at the start of class to get a feel for how the class is doing overall in terms of comprehension of key materials.

However, grading quizzes - especially at such a recurring cadence - will definitely be time consuming, even for a small class of say 15 people (my class has 47 students).

Requirements

I wanted a system that would:

Make it easy to create multi-question, multiple choice quizzes
Make grading automatic
Share individual grades automatically to students via email or some other manner
Require minimal infrastructure and coding (ie: maybe a lambda function somewhere? ideally no code at all)

TL;DR:

Using Airtable, there is a way to create multi-choice, self grading quizzes. I've created a set up that you can find and replicate here for your class's usecase:

AIRTABLE BASE
QUIZ FORM

Airtable based solution

Leveraging Airtable, I have come up with an approach that generally has worked for me thus far.

The key to this solution is to create a table that represents a quiz. Each row will include student name, email, created time, "questions", "checks" and a "grade".

The last two types of rows: "checks" and "grade" are both formula types.

Here's a screenshot:

The columns "1"..."5" represent questions and each are formatted to be "single select" types with 4 predefined selections.

Formula Columns

To assert correctness, each "question" column has an accompanying "solution" column that asserts for correctness.

As seen here, "Solution 1" is a formula {1} = "C" which is ensuring that the selection for column "1", if "C" is correct. This can be repeated with as many questions as necessary.

To compute grade, we simply find the average:

The "code" for this formula:

({Solution 1} + {Solution 2} + {Solution 3} + {Solution 4} + {Solution 5})/5

Where the label inside each {} corresponds to the column name Solution 1...Solution 5.

That's it! That's the internals of the quiz. Now, per student who completes this quiz, a corresponding row will be created with a "creation time" (for future filtering if needed) and "checks" for each question submitted and a "grade" that computes average.

If needed that "grade" column can also be updated to weight individual questions differently as well. 👍

Creating the quiz

For the "frontend", we need a form and the actual quiz questions! Airtable supports a concept of a "view" where it is possible to create a "form" that will populate the rows in the table.

Here's an example:

Essentially, each column from the table can be added (or removed!) as an input field. Also, it is possible to make certain fields required and to add arbitrary text (in my quiz, I even had a few coding questions).

Additionally, this form is by default not shared:

but can be toggled to shareable via a private link at any time (so it is easy to "publish" and "unpublish" a quiz)

The Verdict

For a non-custom coded system, this works "well enough"; looking back to the requirements:

✅Make it easy to create multi-question, multiple choice quizzes
✅Make grading automatic
❌Share individual grades automatically to students via email or some other manner
✅Require minimal infrastructure and coding (ie: maybe a lambda function somewhere? ideally no code at all)

Some happy additional features:

✅Tables and views are easy to duplicate, meaning multiple quizzes can be easily supported
✅Publishing/unpublishing forms are actually super helpful
✅At the bottom of each column, airtable calculates some metrics such as class average, etc which can be read in real time as students complete quiz

There are some definite drawbacks here as well:

❌lack of fine tuned controls in terms of formatting the questions
❌adding more than 4-5 questions becomes somewhat unreasonable due to the additional column that much be added
❌RATE LIMITS! If more that ~50 people open the form all at once (or refresh a lot), I've seen cases where the form is "disabled momentarily" (a few hours)
❌no automatic support for emailing grades to students

For that last item - I have come up with a (similarly codeless) solution! If there is sufficient interest (please let me know in the comments!) I will follow up this post with a tutorial for how I achieved near-real time email notification to students based on quiz submission 👍

Connecting to ElasticSearch configured via Docker

Taq Karim — Mon, 16 Mar 2020 05:39:13 +0000

Elastic Search and Docker

Setting up an elasticsearch + kibana runtime is very easy to do with docker-compose:

version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9200:9200"
  kibana:
    image: docker.elastic.co/kibana/kibana:6.3.2
    ports:
      - "5601:5601"

For more information on the defaults (ulimits or environment variables, please see this post)

To run, you simply:

docker-compose up -d

And then navigate over to http://localhost:9200 for elasticsearch or http://localhost:5601 for kibana.

Gotchas

As with all things, there may be some gotchas associated with this setup. Specifically, you may notice that sometimes navigating to the URLs referenced above will result in connection errors.

This can be for quite a few reasons, I'll list some common ones I've encountered below.

Windows: ElasticSearch doesn't work

A few things to check - run a docker ps -a to ensure your container is running and for how long.

if it only lives for a few seconds, it may be due to a default max_map_count issue. To fix, if using docker-machine:

docker-machine ssh
sudo sysctl -w vm.max_map_count=262144
exit

If not using docker-machine, try:

docker exec -it <container_id> /bin/bash
sysctl -w vm.max_map_count=262144

NOTE: this will not persist after container is killed. TO make it "permanent" you might want to:

vi /etc/sysctl.conf
vm.max_map_count=262144

(press :q to exit)

vi /etc/rc.local
echo 262144 > /proc/sys/vm/max_map_count

source

Even after changes above, not working

One bit to note is that the docker container is simply wrapping the elasticsearch service. As such, sometimes it just takes a bit of time to load.

To ensure that elasticsearch is running properly, consider running:

docker-compose logs elasticsearch

This will display the log messages emitted as elasticsearch is booted inside your container.

example_elasticsearch | [2020-03-16T05:32:12,704][INFO ][o.e.n.Node               ] [IzFG4tZ] initialized
example_elasticsearch | [2020-03-16T05:32:12,704][INFO ][o.e.n.Node               ] [IzFG4tZ] starting ...
example_elasticsearch | [2020-03-16T05:32:12,959][INFO ][o.e.t.TransportService   ] [IzFG4tZ] publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}

Once you see those three lines, you should be safe to attempt your "healthcheck" such as visiting http://localhost:9200 in the browser or curling for it.

Alternatively, if something is foobar'd you'll see it here which gives you further clues as to what went wrong.

PyTennessee 2020: Lessons Learned After a Year of Serverless in Production

Taq Karim — Wed, 11 Mar 2020 15:42:14 +0000

It was great speaking at PyTennessee 2020 this year, kicking off the first of 4 scheduled talks I have in Q2 around "serverless" technologies.

Docker, Code Coverage and Jenkins

Taq Karim — Wed, 11 Mar 2020 15:30:04 +0000

These posts are super quick and to the point - mainly, they are solutions that worked for me but were not anything I could easily find googling.

Problem

Trying to write to .coverage inside of docker fails. Here's the dump:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/_pytest/main.py", line 206, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/_pytest/main.py", line 250, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/pluggy/hooks.py", line 286, in __call__
INTERNALERROR>     return self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/pluggy/manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/pluggy/manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/pluggy/callers.py", line 203, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/pytest_cov/plugin.py", line 254, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/pytest_cov/engine.py", line 197, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/coverage/control.py", line 782, in save
INTERNALERROR>     self.data_files.write(self.data, suffix=self.data_suffix)
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/coverage/data.py", line 680, in write
INTERNALERROR>     data.write_file(filename)
INTERNALERROR>   File "/usr/local/lib/python2.7/site-packages/coverage/data.py", line 467, in write_file
INTERNALERROR>     with open(filename, 'w') as fdata:
INTERNALERROR> IOError: [Errno 13] Permission denied: '/app/.coverage'

Woof. The issue is definitely due to permissions issues. Here's my docker command:

docker run -v $(PWD):/app some_image pytest --cov=some_dir

The Fix

Adding --user=root solved my problem

docker run --user=root -v $(PWD):/app some_image pytest --cov=some_dir

In this case, as I am simply running some unit tests I don't think running as root user makes much of a difference. (Also note that if you are using docker-compose, there is also a supported method for specifying user either via cmd line or in docker-compose file)

I hope this helps!

Forem: Taq Karim

You might not be using json.Decoder correctly in golang

Googling: "json.decoder example golang"

The Problem

The Fix

A few notes

Custom Args in Makefile

Run tests w/race conditions check

Run tests in "short" mode

Run tests w/verbose results in terminal

Run tests for a specific package

Run a specific test func in a specific package

A better way

Integrating Code Climate w/go pkgs

Step 0: Write and run some tests!

Track code coverage

Sidebar: generating code coverage results as HTML

Getting started with CodeClimate

Step 1: Downloading the test reporter

Step 2: Running the tests with code coverage

Step 3: Running the after build

The Code

Makefile

Github Action

Generate Lambda Layer Artifacts w/Docker

The Backstory

The Problem

Lambda Layers and Docker

Safely Construct Elasticsearch Queries w/Golang

The Why

Using strings

Using (use-case specific) structs

What does Google say?

package esquerydsl

Gotcha w/comparing base64 encoded strs

Is a < b?

Is a_b64 < b_b64?

NOPE.

But why though?

Step 1: split the string by digit

Step 2: convert each char to binary equivalent in ASCII

Step 3: concatenate the binary representation

Step 4: partition the above but now in groups 6

Step 5: pad-left each partition to convert our 6 bit "byte" into an 8 bit byte

Step 6: convert back to decimal (base 10) and translate by looking up each decimal value in the base64 characters table

Ok but...why care about this?

Extending SimpleNamespace for Nested Dictionaries

Defining RecursiveNamespace

P.S.

Self-grading quizzes with Airtable

Requirements

TL;DR:

Airtable based solution

Formula Columns

Creating the quiz

The Verdict

Connecting to ElasticSearch configured via Docker

Elastic Search and Docker

Gotchas

Windows: ElasticSearch doesn't work

Even after changes above, not working

PyTennessee 2020: Lessons Learned After a Year of Serverless in Production

Docker, Code Coverage and Jenkins

Problem

The Fix

`Makefile`

`Github Action`

`package esquerydsl`

Is `a < b`?

Is `a_b64 < b_b64`?

Step 4: partition the above but now in groups `6`

Defining `RecursiveNamespace`