Forem: rasjonell

DynamoDB Workflows: stop rewriting incident scripts (with DynamoLens)

rasjonell — Mon, 26 Jan 2026 23:38:11 +0000

If you use DynamoDB on a real product, you probably have a hidden folder somewhere: incident scripts, debug scripts, one-off migrations, quick backfills.

It starts innocently:

You need to fetch a user by email, then find their recent orders, then look up a handful of related items. You run a Query, copy a key, paste it into another query, repeat… and eventually you give up and write a script.

The next time the same incident happens, you copy that script, tweak it, and hope you didn’t mix up Dev and Prod.

That’s the DynamoDB script tax.

What “Workflows” are (in DynamoLens)

DynamoLens is a free, open-source desktop client for DynamoDB (macOS / Windows / Linux).

Its core feature is Workflows: a way to turn repeatable DynamoDB operations into a first‑class object

you can:

build once (Query / Scan / PartiQL steps)
chain steps together (values flow forward automatically)
save and re-run without rewriting anything
export/import to share with teammates

Think of it as moving “incident scripts” from a private folder into a visible, repeatable, shareable UI.

The real win: sharing

Dev tools spread when they become team habits. Workflows are designed to be shared:

The person who debugs a nasty issue can turn the investigation into a workflow and hand it to the next on-call.
A backend team can standardize “safe” patterns (Query-first, guarded Scans) as reusable workflows.
Local dev flows can be reused in staging and production without rewriting.

Example: “User → Orders → Order Items” without copy/paste

A common DynamoDB debugging pattern is a chain of lookups. In a workflow you can:

Query the Users table by email
Extract the userId from the result
Query the Orders table using that userId
Extract an orderId
Fetch related items

The point isn’t that this is impossible with scripts, it’s that it becomes repeatable and shareable without constantly re-implementing the same glue.

Safety: fewer surprises

DynamoDB is powerful, but it’s easy to do expensive or dangerous things accidentally (especially with Scans). DynamoLens makes the

operation mode explicit and keeps environment context visible.

Try it

If you want to stop paying the script tax, start with one workflow you run repeatedly. When it saves you time, share it with your team.

Blog version: https://dynamolens.com/blog/dynamodb-workflows-stop-rewriting-incident-scripts
Download: https://dynamolens.com/download
GitHub: https://github.com/rasjonell/dynamo-lens

Adding RSS Feed To Serum Static Site Generator

rasjonell — Wed, 08 Jul 2020 15:33:45 +0000

About 4 minutes to read

1 Creating The Plugin
1.1 Serum.Plugin
1.2 Channels
1.3 Items
1.4 Build Succeeded

Creating The Plugin

I decided not to talk about the importance, simplicity, and necessity of having an RSS feed for your website. So I’m just going to demonstrate how I made a plugin for Serum, the static site generator tool(written in Elixir), that I use for this blog.

Serum has a plugin system. For example the Table Of Contents at the top of the article was generated with a plugin. They have a Plugin Behaviour which you can use to define your own plugin module.

Serum.Plugin

Serum exposes a number of events(full list can be found here) which you can handle. The one I find most useful for the RSS feed generator is build_succeeded/3. This particular event is triggered when, as the name suggests, the build step succeeds. After a successful build we want to update our RSS feed so the subscribers can get notified about new posts!

First we need some boilerplate to correctly implement the Serum.Plugin behavour.

defmodule Blog.Rss do
  @moduledoc """
  A Serum plugin that create an RSS feed.

  ## Using the Plugin

      # serum.exs:
      %{
        server_root: "https://example.io",
        plugins: [
          Rss
        ]
      }
  """

  @behaviour Serum.Plugin

  @title "Rasjonell's Blog"
  @url "https://www.rasjonell.tech"
  @blog_desc "Random rants about technology"

  @impl true
  def name, do: "RSS Feed Generator"

  @impl true
  def version, do: "0.1.0"

  @impl true
  def elixir, do: ">= 1.7.0"

  @impl true
  def serum, do: ">= 1.2.0"

  @impl true
  def description do
    "Generates an RSS feed for /posts"
  end

  @impl true
  def implements do
    [build_succeeded: 3]
  end
end

Here apart from the necessary function implementations, I’ve also defined some module attributes such as title and description which will be used latter for generating the RSS feed.

Now that we have our module defined we can start thinking about how we should generate the feed.

RSS, as the name suggests, is really simple. It’s an XML file so we should start with: <?xml version="1.0" encoding="utf-8"?>. Then we need to wrap all of our feed in an rss tag like so: <rss version="2.0"></rss>

So what are the things that need to be wrapped.

Channels

Remember the module attributes? We need them to define our channel. We can define a simple function that returns all the necessary tags with up-to-date information.

defp channel do
  """
    <title>#{@title}</title>
    <link>#{@url}</link>
    <description>#{@blog_desc}</description>
    <lastBuildDate>#{current_date()}</lastBuildDate>
    <language>en-us</language>
  """
end

See the call to current_date/0 there? This was the most painful part of developing this plugin. RSS requires dates in the RFC-822 format but Elixir’s date-related modules, understandably, don’t come with built-in formatting options. So I had to install a dependancy. Timex is the library I chose, as it also is a dependancy of Serum, so I wouldn’t add any more code to the final bundle.

The current_date/0 function has this definition:

defp current_date do
  {:ok, current_date} = Timex.now
    |> Timex.format("{RFC822}")
  
  current_date
end

Items

Now that we have defined our channel description, it’s time we add some items.

defp item(title, desc, link) do
  """
  <item>
    <title>#{title}</title>
    <description><![CDATA[#{desc}]]></description>
    <pubDate>#{current_date()}</pubDate>
    <link>#{link}</link>
    <guid>#{link}</guid>
  </item>
  """
end

Now we can just glue all of these together to generate a feed:

defp feed(channel, items) do
  """
  <?xml version="1.0" encoding="utf-8"?>
  <rss version="2.0">
  <channel>
    #{channel}
    #{Enum.join items, ""}
  </channel>
  </rss>
  """
end

This pretty much gives us all the building blocks that we need to complete this plugin by reading the posts, generating a feed, and writing it to a file accessable publicly on our website.

Build Succeeded

To handle the build_succeeded/3 event we need to implement that behavour:

@impl true
def build_succeeded(_src, dest, _args) do
  :all_posts
  |> Serum.GlobalBindings.get()
  |> build_feed(dest)
  |> create_file(dest)
  |> File.write()

  :ok
end

I’ll let Elixirs beautiful pipe operator do the explaining here.

If you want to use this plugin or just check the final code checkout the source code.

Using AWK To Process Huge CSV Files

rasjonell — Wed, 08 Jul 2020 15:33:45 +0000

1 Problem
2 Approach
3 Solution
4 Conclusion

Problem

I had 38 separate .csv files each consisted of at least 40K lines of data. Overall there were ≥ 2.6 million rows of information. The problem was that the data was not in the ideal form.

Each file had 5 columns. One of the columns was called birthday. It had a conventional(at least for the most of the world) format: DD/MM/YYYY. However I was going to put this data in a DB(more on that later!) and I needed to have 3 separate properties(ie. { day: DD, month: MM, year: YYYY }). Also there were some columns that were not useful so I had to get rid of them. In addition, I also wanted to reduce the number of files to one.

Approach

I know some python and from a couple of data science courses I was introduced to Pandas, a well-known tool for data analysis built on top of python. It had a straightforward way of reading, manipulating, and writing csv files. I was going to use that, but didn’t really want to download this tool to use only one of the features and never use it again. So I choose something that comes with nearly all modern unix-like systems. AWK is designed for text processing and much like other programs in the unix-like world it does one thing well and produces fascinating results in regards to simplicity and performance when composed with other unix tools.

Solution

Now we need to make AWK process .csv files. AWK uses field separators to split an input into fields. It may either be a character or a RegExp. So all we need to do to make AWK become a .csv processor is to set FS(field separator) to be ",".

We can achieve this using AWK’s Startup and Cleanup Actions.

# clean_csv.awk

BEGIN {
  FS = ","
}
{
  print $1
}
END {
}

Then if we pass an input such as one, two, three to the script, it will print one:

$ echo "one, two, three" | awk -f csv.awk

one

Now that we have this, we can finally solve our problems: splitting DD/MM/YYYY column into three separate columns and removing the useless column. AWK has a split function that takes a field, a new variable, and a separator and puts the separated values into the new variable.

The birthday column was #3 so to create the separate fields we need to make the following modifications:

# clean_csv.awk

BEGIN {
  FS = ","
}
{
  # split DD/MM/YYY into an array of values
  split($4, dob, "/")

  # turn these values into separate columns
  total = dob[1] "," dob[2] "," dob[3]

  # finalize the row, omitting the useless #4 column.
  row = $1 "," $2 "," total "," $5

  print row
}
END {
}

This mainly solves most of our problems, except for the first row of the input. .csv files have headers as a set of labels on the first row, thus we need to skip every first line of every .csv file.

In AWK NR and FNR represent the record number. The latter shows the record number of the current file, while the first holds the total count. Since we are going to use this script on more then one file, we’ll stick with FNR.

So the final script looks like so:

# clean_csv.awk

BEGIN {
  FS = ","

  # create a new header with correct labels
  print "last_name,first_name,birth_day,birth_month,birth_year,country"
}
{
  if (FNR > 1) { # omit every first line
    # split DD/MM/YYY into an array of values
    split($4, dob, "/")

    # turn these values into separate columns
    total = dob[1] "," dob[2] "," dob[3]

    # finalize the row, omitting the useless #4 column.
    row = $1 "," $2 "," total "," $5

    print row
  }
}
END {
}

At last, to execute this on our 38 .csv files we need to run this command:

awk -f clean_csv *.csv > clean.csv

Conclusion

We read and manipulated over 30 .csv files with more than 2.6 million rows of data combined and reduced them into a single clean file without installing any software. So if you’re faced with a problem, you may not need to install a bloated software to solve it when the solution comes pre-installed with your system.

Separate Data Access Layer With TypeScript Powered MicroServices

rasjonell — Mon, 03 Feb 2020 09:05:00 +0000

TL;DR

ArangoDB is a multi-model database supporting Key/Value, Document, and Graph models with one unifying query language.
Foxx Microservices are Arango's way of writing separate data access and domain logic code running directly within the database with native access to in-memory data.
Foxx TypeScript is an ArangoDB Foxx service template that lets you write testable, typescript powered microservices.
foxx_ts_demo is the source code of the tutorial in this article.

Graph Databases

Graph databases caught my interest and I've been experimenting with them for a couple of months now. This a very broad topic that does not really belong to this article so I'm just going to demonstrate why I choose to work with this model with a couple of key points:

Better Mental Modeling

As an example let’s think about a basic social media interaction. At the core all we need is user follows user. With traditional "relational" databases you'd need to create a through table that holds foreign-keys to represent this many-to-many relationship. This itself is already unnecessarily complicated, but what if we introduce posts, reactions, etc. In order to fetch posts only from users that the current user follows you'd need to do expensive multiple joins and possibly sub-queries.

However with graphs it is as simple as stating the relationship.

This image is the actual representation of this relationship. Having this simplicity is essential both for modeling your business logic and communicating it across different teams in your company.

Graph Algorithms

Graph Theory is the study of graphs. There are a number of well-known algorithms and techniques that help us conduct efficient data analysis and enable us to create personalized suggestions.
This example from Amazon Neptune shows how having your data model in such form makes it really intuitive and straightforward to generate suggestions and make your platform more personalized.

Separate Data Access Layer (DAL)

For projects that are relatively big and/or heavily data-reliant it makes sense to separate your Data Access Layer(DAL) from your API. It is also widely accepted to have a separate Business Logic Layer(BLL).

ArangoDB is a relatively new database and is being rapidly developed. Recently it was communally voted as the number 1 graph database. Currently there are not many ORMs(or OGMs) supporting Arango. Because of this I tried finding the right way of accessing my data layer. This is when I came across Foxx Microservices.

Foxx Microservices are Arango's way of writing separate data access and domain logic code running directly within the database with native access to in-memory data.

Foxx services consist of JavaScript code running in the V8 JavaScript runtime embedded inside ArangoDB. Each service is mounted in each available V8 context (the number of contexts can be adjusted in the server configuration). Incoming requests are distributed across these contexts automatically.

Essentially ArangoDB offers a REST API to communicate with the database. So what ORMs(or OGMs) do is introduce an abstraction over these APIs to be used in your client. Foxx Microservices can be used to extend this API. Your Business Logic Layer then can directly call the extended database API instead of a separate service that does not have direct access to the database. Thus, using Foxx as your DAL not only offers modularity and long-term maintainability but also ensures the best possible performance.

Our First TypeScript Foxx Microservice

We've created an ArangoDB 🥑 Foxx Microservice template supporting TypeScript and Yarn >= 2.0. Navigate to this link https://github.com/SquashConsulting/foxx_typescript/generate to generate your first Foxx Microservice with our TypeScript template.

In order to follow the rest of the article make sure you have arangodb running locally and have foxx-cli set up.

Project Structure

├── LICENSE
├── README.md
├── manifest.json
├── package.json
├── src
│   ├── entry.ts
|   └── scripts
│       ├── setup.ts
│   └── routes
│       ├── hello.ts
│       └── index.ts
├── test
├── tsconfig.json
└── yarn.lock

If you have your foxx-cli installed and set up, then in order to use the typescript service in ArangoDB, you need to run yarn build and then do foxx install <mount> --server <server_name> --database <db_name>

*NOTE: * this template uses yarn >= 2.0, so please make sure you go through the instructions in the template README to complete your setup.

This will extend your database API by adding a /hello route that simply returns hello world.

Setting Up The Service

The template project comes with @types/arangodb package. So you can already use them in your project.

Let's write another route that inserts user data in the database.

First of all we need to create a setup script that creates the necessary collections for us.

inside src/scripts/setup.ts write this:

import { db } from "@arangodb";

const documents: [string] = ["users"];

documents.forEach((collection: string): void => {
  if (!db._collection(collection)) {
    db._createDocumentCollection(collection);
  } else if (module.context.isProduction) {
    console.debug(
      `collection ${collection} already exists. Leaving it untouched.`
    );
  }
});

Then update your manifest.json to run this script.

/* ... */
"scripts": {
  "setup": "dist/scripts/setup.js"
},
/* ... */

Notice how we use dist instead of src to tell aranago to run the bundled commonjs code instead of typescript.

After this you can finally install your service using foxx-cli

$ yarn build
...

$ foxx install /foxx_ts_demo --server squash --database squash

Installed service at "/foxx_ts_demo".

Notice that I installed the service on /foxx_ts_demo using the server squash and the homonym database that I configured using the foxx-cli

Now if you navigate to ArangoDB's WebView(default is http://localhost:8529) choose the database you configured you will see users collection.

This happened because setup is one of Foxx lifecycle scripts. For more info refer to the official docs.

Creating A Custom Route

Now that we have everything set up, we can move forward and create our first ever typescript API!

You can see the example route in src/routes/hello.ts.
Before we extend this with a POST request that creates a user, let's update our setup script to add unique index on the username field.

// src/scripts/setup.ts
/* ... */

const users: ArangoDB.Collection = db.users;

users.ensureIndex({
  unique: true,
  type: "hash",
  fields: ["username"]
});

Now we can add the route:

// src/routes/hello.ts
import joi from "joi";
import { db } from "@arangodb";

interface IUser extends ArangoDB.Document {
  email?: string;
  username?: string;
}

router
  .post("/user", function(req: Foxx.Request, res: Foxx.Response) {
    try {
      const user: IUser = req.body;

      const users: ArangoDB.Collection = db.users;

      const meta: IUser = users.save(user);
      const data: IUser = { ...user, ...meta };

      res.send({ data });
    } catch (e) {
      // Failed to save the user
      // We'll assume the uniqueness constraint has been violated
      res.throw("bad request", "Username already taken", e);
    }
  })
  .body(
    joi
      .object({
        email: joi.string().required(),
        username: joi.string().required()
      })
      .required(),
    "User Info"
  )
  .description("Creates a new user.");

The combination of TypeScript and joi validations makes this code pretty self-explanatory. We take the user data from the request save it in the collection and return it with additional data.

If you navigate to Arango WebView's Services tab you will see your service. Click on the service. There are three main sections:

Info - Your README.md file.
API - We used joi to define our APIs this helps Arango to generate Swagger API definitions for us.
Settings - Your service configurations(Will talk about this later)

Now If you go the the API tab, you will see the newly created POST /user API, expanding it will reveal the API options with an additional option to try the API directly from the web view.

Give it a try and create a user. After successfully creating the user, the API will respond with 200 OK and send the user data back. If you try to create a user with the same username the API should respond with 400 Bad Request and error massage that states that the username is already in use.

Next Steps

I hope I've succeeded in convincing you to give ArangoDB and Foxx Microservices a try. If you do so, you'll discover a lot more than I've managed to cover in this article. My favorite Foxx features are Linking Services together and making them configurable.

You can checkout https://github.com/SquashConsulting/foxx_services to see how we used both of these(and other cool) features to create a separate Data Access Layer with shared utils where you can define your configurations and have all the other linked services be generic.

Conclusion

To wrap things up, separating your Data Access Layer(DAL) from your Business Logic Layer(BLL) is a technique that results in maintainable and modular code-base. Using ArangoDB Foxx with Typescript and joi validations to accomplish this provides additional benefits such as configurable, performant, and type-safe microservices.

If you liked this article consider following me on Mastodon or Twitter @irasjonell and maybe starring the repos used in this article.

References

foxx_ts_demo - The source code of the tutorial in this article.
Foxx TypeScript - The typescript template project
Foxx Services - Foxx services meta repo
ArangoDB - ArangoDB source code