Forem: Daniel Oluojomu

Optimising Convoy’s Flatten Package

Daniel Oluojomu — Tue, 04 Feb 2025 11:53:34 +0000

Background

At convoy we started off using MongoDB as our primary data store and it was super great for a long time. However, as data complexities grew we quickly realised a relational database was superior for our use case. We detail our reasons for switching to PostgreSQL here. But there was just one minor (well not so minor) problem — we relied on MongoDB’s native queries heavily in the subscriptions filter engine and switching meant we needed to implement this ourselves in Go, as there was no readily available library to support this.

The filter engine determines if an event payload matches a given filter[1], if it matches, the event is routed to the subscription’s endpoint, otherwise the event will get dropped.

The filter engine determines if an event payload matches a given filter in two steps:

Flatten the event payload.
Compare the flattened payload to the pre-flattened[2] filter.

Going forward in this article, we’ll just focus on flatten library itself.

Let’s Dig In

The flatten lib uses Depth first search to find all entries in the json map and concatenate their keys separated by the dot notation in the result map. The only catch here is the library “steps over” any operators like $and, $in, $exist etc.
For example:

{
  "$and": [
    {
      "$or": [
        {
          "person": {
            "age": {
              "$in": [10, 11, 12]
            }
          }
        },
        {
          "places": {
            "temperatures": 39.9
          }
        }
      ]
    },
    {
      "city": "london"
    }
  ]
}

Becomes:

{
  "$and": [
    {
      "$or": [
        {
          "person.age": {
            "$in": [10, 11, 12]
          }
        },
        {
          "places.temperatures": 39.9
        }
      ]
    },
    {
      "city": "london"
    }
  ]
}

Analysing Initial Implementation

Flatten initially used the recursive version of DFS, meaning for every Map or array entry in the map, the function would call itself, see the full implementation here. While this approach works fine for small payloads, it leads to high memory and CPU consumption when dealing with larger payloads. This becomes particularly problematic when processing event payloads with multiple filters, as seen in BenchmarkFlattenLargeJson with a 17KB GitHub pull request webhook payload:

The approach was riddled with several compounding problems:

Heap memory allocation: The recursive calls create a new instance of the method, consuming stack space, though stack space itself is “cheap”, the function go through another round of heap memory allocations inside the function.

New key string concatenation: New map entry keys were being created with:

fmt.Sprintf("%s.%s", prefix, newKey)

While this makes sense for small strings, repeatedly doing this for potentially large strings can be disastrous for memory management. fmt.Sprintf has to do a lot of checks and casting to figure out where to place the given values in the format string, this consumes a lot of CPU time & memory.

Result Map Copies: When the recursion terminates and the stack starts to unwind, every item in the result map returned by a latter call will be copied into the result map of the former call.

Result Map resizing: Another problem not immediately obvious is map capacity when copying the items during unwinding the stack as depicted above. Because the result maps are not pre-allocated with enough space to hold all expected keys, with a sufficiently large input, the result map will resized by the runtime multiple times. This is an expensive operation managed by runtime.growWork_faststr & runtime.evacuate_faststr[3].

How about a for loop instead?

To address the inefficiency of the recursion, we opted to use a custom stack (stackFrame) to keep track of map entries iteratively.
In this approach, we keep a slice of stackFrames, this will serve as our self managed call stack. You can find the full implementation here.

func flatten(prefix string, nested interface{}) (M, error) {
  var stack []stackFrame
  stack = append(stack, stackFrame{prefix, nested})
}

This implementation removes the repeated heap memory allocations caused by recusrsion. A strings.Builder is now used to generate new result map keys efficiently. With a for loop, we can also avoid result map copy problem caused by stack unwinding, there is only one result map and all entries get stored there directly.

However the problem of map resizing still exists. Is it possible to make a pass over the input and determine how many entries will be required in the resukt map? Actually yes. For a sufficiently large input, it is cheaper to do this to avoid resizing operations. The countKeys function does this. With this we allocate a large enough result map once.

After the new implementation was profiled, we saw a nearly 70% performance improvement across CPU time & memory allocations. Here’s a side by side comparison:

Conclusion

Our initial implementation using the recursive version of DFS to flatten payloads was good enough for a small number filters to be processed. But at scale, it was too slow and costly causing a huge bottleneck for our customers. We revised the implementation and decided to get rid of the recursion in favour of a custom stack iterative approach and saw huge performance gains.

[1] — Please note; both event payloads & filters are json data marshalled into map[string]interface{}.

[2] — The filter is flattened at point of creation. This avoids the costly operation of flattening it for every event.

[3] — This operation is opaque to golang applications, it is managed entirely by the runtime.

P.S: sorry about the blurry screenshots, not sure what's wrong with dev.to image uploads, you can find clearer images in the medium post.

Publish An Event to Multiple Endpoints using Convoy

Daniel Oluojomu — Fri, 11 Mar 2022 13:05:50 +0000

One common scenario in publishing webhook events is enabling a user to provide multiple endpoints to receive events. One easy example of this is publishing an event that the user needs to process at more than one location. This location could be a no-code platform like zapier, a newly minted microservice or serverless function, or a good old slack notification. In this article, I’d like to explain how you can achieve this using Convoy.

Without Convoy, your users have to build in the fan-out mechanism themselves which is a lot more stressful.

Steps

Start Convoy Instance

To follow through with this article you’d need to run an instance of Convoy:

$ docker run \
    -p 5005:5005 \
    --name convoy-server \
    -v `pwd`/convoy.json:/convoy.json \
    ghcr.io/frain-dev/convoy:v0.4.18

Create an Application

Next, we have to create an application under this group.

Sample Payload

{
  "name": "test-app",
  "support_email": "test@gmail.com",
  "secret": "eyJhbGciOiJIUzI1NiJ9"
}

Bash

$ curl \
    --request POST \
    --data @app.json \
    -H "Content-Type: application/json" \
    http://localhost:5005/api/v1/applications

Response

{
    "status": true,
    "message": "App created successfully",
    "data": {
        "uid": "2b1e9973-ed03-403c-a8b0-341edd51fb14",
        "group_id": "f0a187f4-edaa-4f8e-adec-75b9a36b3c68",
        "name": "test-app",
        "support_email": "test@gmail.com",
        "endpoints": [],
        "created_at": "2022-03-09T14:17:51.111+01:00",
        "updated_at": "2022-03-09T14:17:51.111+01:00",
        "events": 0
    }
}

Create Two Endpoints

Now we can create multiple endpoints under this app. The first endpoint will take the * event type. Essentially this event type means all incoming events to that app will be published to that endpoint. The second endpoint will take the event type payment.created. Only incoming with the exact type payment.created will be published to that endpoint. Convoy tries to match the event type to all available endpoints under that app, the event is then published to all the matched endpoints.

For the first endpoint:

Sample Payload

{
  "description": "test-endpoint-1",
  "events": [
    "*"
  ],
  "secret": "12345",
  "url": "<your-endpoint>"
}

Bash

$ curl \
    --request POST \
    --data @endpoint-1.json \
    -H "Content-Type: application/json" \
    http://localhost:5005/api/v1/applications/{appID}/endpoints

Response

{
    "status": true,
    "message": "App endpoint created successfully",
    "data": {
        "uid": "2901bbc9-092e-4685-868d-a17298fe86ba",
        "target_url": "<your-endpoint>",
        "description": "test-endpoint-1",
        "status": "active",
        "secret": "12345",
        "events": [
            "*"
        ],
        "created_at": "2022-03-09T14:18:14.493+01:00",
        "updated_at": "2022-03-09T14:18:14.493+01:00"
    }
}

For the second endpoint:

Sample Payload

{
  "description": "test-endpoint-2",
  "events": [
    "payment.created"
  ],
  "secret": "12345",
  "url": "<your-endpoint>"
}

Bash

$ curl \
    --request POST \
    --data @endpoint-2.json \
    -H "Content-Type: application/json" \
    http://localhost:5005/api/v1/applications/{appID}/endpoints

Response

{
    "status": true,
    "message": "App endpoint created successfully",
    "data": {
        "uid": "2901bbc9-092e-4685-868d-a17298fe86ba",
        "target_url": "<your-endpoint>",
        "description": "test-endpoint-2",
        "status": "active",
        "secret": "12345",
        "events": [
            "payment.created"
        ],
        "created_at": "2022-03-09T14:18:14.493+01:00",
        "updated_at": "2022-03-09T14:18:14.493+01:00"
    }
}

Publish Event

Now let us publish an event with the type payment.created to this app. The payment.created will match both endpoints, since * will match all event types, and the payment.created type of the second endpoint matches exactly.

Sample Payload

{
  "app_id": "2b1e9973-ed03-403c-a8b0-341edd51fb14",
  "data": {
        "blog":"medium.com"
    },
  "event_type": "payment.created"
}

Bash

$ curl \
    --request POST \
    --data @event.json \
    -H "Content-Type: application/json" \
    http://localhost:5005/api/v1/events

Response

{
    "status": true,
    "message": "App event created successfully",
    "data": {
        "uid": "ddcd3928-1d7e-4527-901f-47673fd569ce",
        "event_type": "payment.created",
        "matched_endpoints": 2,
        "provider_id": "",
        "data": {
            "blog": "medium.com"
        },
        "app_metadata": {
            "uid": "2b1e9973-ed03-403c-a8b0-341edd51fb14",
            "title": "test-app",
            "group_id": "e6bbde4b-4c43-45a6-8d4c-c8eed1c2bb41",
            "support_email": "test@gmail.com"
        },
        "created_at": "2022-03-09T15:14:27.569+01:00",
        "updated_at": "2022-03-09T15:14:27.569+01:00"
    }
}

Show Endpoint Response

Illustration of the events coming through

Footnote

This article was originally published on our main blog. Check it out for more articles about convoy!

How to Setup PostgreSQL with SSL inside a Docker Container

Daniel Oluojomu — Tue, 02 Feb 2021 17:17:15 +0000

Hi guys, in this article I'll be sharing how to set up a PostgreSQL database that'll accept SSL connections only, inside a Docker container.

What's Docker?

Docker is a containerization engine, it allows you to bundle your app and its dependencies into a template file called an image, a running image is called a container. Learn more about Docker here.

What's PostgreSQL?

PostgreSQL is one of the most popular databases out there, it's a relational database. Learn more about PostgreSQL here.

Heads Up

Because OpenSSL is quite complex to use, we'll use certstrap for generation of certificates, install certstrap from here.

Follow the instructions here to install docker.

Structure

Create a folder postgres_ssl with a structure like so:

postgres_ssl
|   Dockerfile
│   certs
│   │   
|   └───out
|   ssl-conf.sh

Let's generate the SSL certificates

We're going to use a custom Certificate Authority(CA), open a terminal in the certs directory, and generate the CA using:

$ certstrap init --common-name myCA

This will create three files in the certs/out directory:

myCA.crt which is the CA certificate,
myCA.key which is the CA certificate key that will sign certificate requests,
myCA.crl which is the Certificate Revocation List(a list of revoked certificates).

Learn more about a Certificate Authority here.

Next, we'll request key pairs from our custom CA.

$ certstrap request-cert --common-name postgresdb  --domain localhost
$ certstrap sign postgresdb --CA myCA

The --domain option adds a list of domains(called Subject Alternative Names) that the generated certificate will be valid for. We set ours to localhost because the database will run on localhost, if yours is running remotely you can add the URL instead.

ssl-conf.sh is a simple bash script that'll clear all default network connection settings in /var/lib/postgresql/data/pg_hba.conf and set it to require SSL for each connection. Add the following content to the ssl-conf.sh:

# echo ssl setting into pg_hba.conf configuration file
echo 'hostssl all all all cert clientcert=verify-ca' > /var/lib/postgresql/data/pg_hba.conf

If you wish to keep the previous settings in the file change > to >>.
This script will be run inside the container.

Now we have our own CA and our server key pair & ssl-conf.sh. Let's write our Dockerfile.

Add the following content to the Dockerfile:

# This Dockerfile contains the image specification of our database
FROM postgres:13-alpine

COPY ./certs/out/postgresdb.key /var/lib/postgresql
COPY ./certs/out/postgresdb.crt /var/lib/postgresql

COPY ./certs/out/myCA.crt /var/lib/postgresql
COPY ./certs/out/myCA.crl /var/lib/postgresql

COPY ./ssl-conf.sh /usr/local/bin

RUN chown 0:70 /var/lib/postgresql/postgresdb.key && chmod 640 /var/lib/postgresql/postgresdb.key
RUN chown 0:70 /var/lib/postgresql/postgresdb.crt && chmod 640 /var/lib/postgresql/postgresdb.crt

RUN chown 0:70 /var/lib/postgresql/myCA.crt && chmod 640 /var/lib/postgresql/myCA.crt
RUN chown 0:70 /var/lib/postgresql/myCA.crl && chmod 640 /var/lib/postgresql/myCA.crl

ENTRYPOINT ["docker-entrypoint.sh"] 

CMD [ "-c", "ssl=on" , "-c", "ssl_cert_file=/var/lib/postgresql/postgresdb.crt", "-c",\
    "ssl_key_file=/var/lib/postgresql/postgresdb.key", "-c",\
    "ssl_ca_file=/var/lib/postgresql/myCA.crt", "-c", "ssl_crl_file=/var/lib/postgresql/myCA.crl" ]

What the Dockerfile entails

We're using postgres:13-alpine as our base image, this image will be pulled automatically if it isn't available locally.
Copies our SSL certificate files into the /var/lib/postgresql directory of the image.
Copy ssl-conf.sh into the /usr/local/bin directory of the image.
Change file permissions of the certificate files, so as to prevent malicious changes.
Specify Container Entrypoint, which is what to run on startup.
The CMD command adds a list of arguments to pass to the Entrypoint command. Learn more about Dockerfile syntax here.

Open a terminal in the postgres_ssl directory and build the Dockerfile:

$ docker build --rm -f "Dockerfile" -t postgres:ssl "."

Then run the image using:

$ docker run -d -p 5432:5432 --name postgres_db -e POSTGRES_PASSWORD=postgres postgres:ssl

The container should be running now, remember in the Dockerfile we copied ssl-conf.sh into the image's /usr/local/bin directory. We can run ssl-conf.sh like so:

$ docker exec -it postgres_db bash /usr/local/bin/ssl-conf.sh

This will run the bash script inside the container. Of course, you can avoid always typing these using an automation tool like Make.

What next?

To connect to the database in the container you'll need to connect with an SSL connection. To do this, you need to request another key-pair from the custom CA, using the same command we used above. Then use the generated key to connect, the method of doing this can vary with different tools/languages you use.

My Solution to Google's Hashcode 2019 final Problem

Daniel Oluojomu — Mon, 10 Feb 2020 23:53:07 +0000

Hi there my name is Daniel and I'm a passionate Gopher and Dart developer. About three weeks ago, i was going through the web i stumbled upon the Google Hashcode competition read about it and decided to join in this year.

It would seem proper to check the problems released in previous years, and that's exactly what i did. I chose the 2019 final round problem. Of Course, I got stuck at some point in solving the problem(at this point it was really frustrating), and so i decided to just look up the solution. Fortunately i found none. Hence, after solving the problem, i decided to write my first article on dev.to to give back to community, even in this small way. The solution is written in Go, and here's the link to the repo.

Okay, let's get to work, oh and, the question can be found here, i'll only explain the basic part of the problem in the article so take time to read it.

The problem states that:

Google has a large codebase, containing billions of lines of code across millions of source files. From these source files, many more compiled files are produced, and some compiled files are then, used to produce further compiled files, and so on. Given the huge number of files, compiling them on a single server would take a long time. To speed it up, Google distributes the compilation steps across multiple servers. In this problem, we will explore how to effectively use multiple compilation servers to optimise compilation time.

This basically means that multiple files will be compiled across multiple servers.It also says:

Given a description of dependencies between files in a massive codebase, a number of build servers, and deadlines for the compilation targets, schedule the compilation steps on the available servers.

Now we see that certain files have dependencies, dependencies are analogous to mandatory ingredients in a dish, without them, the dish cannot be called the name it would normally be called. For instance,consider Pepperoni Pizza without Pepperoni. Since files could have dependencies,waiting for each dependency to compile is imminent.
It also says:

Each compiled file is described by: name, dependencies (other compiled files that need to be ready before this file can be compiled),the time it take to replicate the file to all other servers, and the time it takes to compile it.

Okay that's a full description of each file. We now have all information we need.
Our main objective is to schedule compilation steps on each server to minimize the time needed to compile the target files as defined in the PDF.

Firstly we need to fully represent or model each file in code. So i use a struct named file:

//The file struct fully represents a file and all its provided properties
type file struct {
   Locker            sync.RWMutex
   Name              string
   CompileTime       int
   ReplicationTime   int
   Dependencies      []string
   CompiledOnServers []string //holds the distinct servers this file was compiled on
   IsCompiled        bool
   IsReplicated      bool
}

We do not have the number of servers that Google has and so we model this requirement using goroutines, we'll create as many goroutines as there are servers and compile files on each one of them, but we do not also have the luxury of time as specified in the input files so,we model using something like this in compile.go(in the repo):

time.Sleep((f.CompileTime)/1000000 * time.Duration) 
//this sleeps the caller goroutine for the amount of time specified

Here's a complete breakdown of the solution:

At birth, each goroutine picks a file from the allFiles slice and begins compiling it. If the the file has dependencies it'll check if each dependency is compiled on the current server(goroutine) or is replicated, that is, copied to all servers, if both conditions hold true, that particular dependecy isn't recompiled, else, it will be compiled on the current goroutine.

After going through all of it's dependencies, the chosen file is then compiled.
It will replicated to all servers only if the compilation time is greater than the replication time or if at least the ratio of replication time to compile time os less than or equal to 1.8:

(f.ReplicationTime / f.CompileTime) <= 1.8 //if this is true the file will be replicated

The reason for this is that, it isn't that much of a burden to replicate the file if the replication time isn't up to double the compilation time, this idea was spotted in the example output given in the PDF, file c1 is replicated even though
the Replication time is 18s and the Compile time is 10s.

After compiling it's chosen file each goroutine will come back to check if all files are compiled and if true execute the code below:

waitGroup.Done() //this will decrement the waitGroup counter for the main goroutine
runtime.Goexit() //the goroutine terminates itself

This is my first blog post. If you have any contributions or corrections or suggestions please leave them in the comments section, they'd be highly appreciated.
Cheers!