Forem: jorin

Getting Started Building a Data Platform

jorin — Fri, 17 Oct 2025 08:42:30 +0000

Ever wonder what a data platform is and if your company needs one?

If the idea of hiring a data team to build and manage an enterprise data platform feels overwhelming, you're not alone.

Let me break down how you can get started from zero and build up data capabilities at your company, one step at a time.

Who Needs a Data Platform?

Nowadays every company is a data company. From marketing and sales to product usage and customer support, all aspects of your business generate data.

And that data is waiting to be activated. Turn it into reports to drive decisions. Build dashboards to support operations. Offer new product features and services to your customers that are directly driven by data and automation.

Getting started is much more a cultural challenge than a technical one.

Start small. Make sure you see first successes by doing the work manually without worrying about big investments in making it scalable.

But once you see concrete value and it becomes painfully clear that technology is holding you back, you know it's time to build out your data capabilities.

Where do you go from here? Can you buy an off-the-shelf solution? Do you hire a data engineer? Do you need a dedicated data team or can your existing engineers handle it?

Breaking It Down

You can break down data infrastructure into four main layers:

Ingestion: Connect data sources, extract data and load it into a central repository
Storage: Store data in a structured format that is optimized for analytical workloads
Transformation: Clean, enrich, and transform data to make it practical to work with and ensure consistent definitions of key metrics across the organization
Business Intelligence: Create dashboards, reports, and alerts to share insights internally and with your customers and partners

One Step at a Time

There are many different tools to address all of these layers.

Focus on solving concrete problems you are experiencing and add new tools only when they directly solve a problem.

Ingestion

Start with querying your data directly where it is. Introduce tools to load data into a central repository only when the complexity and volume make this impractical.

You don't need to address all data sources at once. Focus on the ones creating problems. Accept manual workarounds when practical.

Your data tooling should be able to query data across different data sources. You don't need to worry about ingestion if directly querying a Postgres database and a Google Sheets gets the job done.

Storage

Chances are you already store your data in a database such as PostgreSQL or MySQL. If you're not having performance issues, there's no need to introduce a separate database for analytical workloads.

Only if performance or cost becomes an issue should you start addressing it.

Storage is a critical component since it's where the actual data lives. Data outlives applications built on top of it. Pick an established and open standard to store data.

Keep in mind that there is no one-size-fits-all solution. You might need multiple data stores optimized for different use cases. You'll know what you're looking for when you act on concrete problems instead of trying to find a solution for hypothetical future problems.

Transformation

Start delivering value before adding a separate data transformation step. Introduce a dedicated data transformation layer when queries start taking too long, or metrics become unreliable and hard to maintain because the same logic is repeated many places.

A few materialized views in your database can take you a long way.

You'll know it's time to look into real-time stream processing, data lineage and orchestration tools once you experience the issues that these tools are designed to solve.

Business Intelligence

Many software companies start out by building custom analytics features. As you use data to drive operations and user-facing functionality, building custom solutions for every new workflow and view on the data becomes slow and expensive.

Introduce a data visualization tool to quickly build analytics dashboards and reports. This is a great first step and enables a single data analyst to deliver a lot of value without introducing any other data infrastructure.

Getting Started

I built Shaper to help companies in exactly this situation.

Shaper is a simple interface on top of DuckDB that allows you to build analytics dashboards and automate data workflows with only SQL.

Thanks to DuckDB, it's easy to query data across various sources ranging from databases to CSV files and Google Sheets.

You can go a long way before having to add more layers to your data stack.

Give it a try and let me know what you think.

Clickhouse for Embedded Analytics: First Impressions and Unexpected Challenges

jorin — Mon, 30 Sep 2024 09:40:38 +0000

I implemented Clickhouse for an embedded analytics project handling millions of data points, and here I share my first-hand experience with its impressive performance benefits and the unexpected challenges I encountered along the way.

The project

For a recent project, I got to help a client with the implementation of an embedded analytics system.

The system collects data from multiple internal services. The data is then prepared to build various dashboards on top of it. The dashboards will be embedded in the main application to provide customers with insights into their operations. The system needs to be flexible enough to support new use cases for using the data in the future.

In the past, I built similar systems on top of Postgres. Postgres is very flexible and it's a great default option for any data storage needs.

For this new project, we expect the number of data points to grow into the millions quickly.

It is certainly possible to handle millions of rows with Postgres. But doing analytical queries over millions of rows gets unfeasibly slow with Postgres. You end up having to deploy different caching strategies such as maintaining different pre-aggregated versions of the same data. This quickly increases the complexity of the system and makes it expensive to evolve it as new needs come up.

Analytical workloads are not the primary use case of Postgres and that's okay.

We started to look for alternatives and quickly landed at Clickhouse.

Many options were ruled out quickly because the client operates in regulated industries which require us to run the system in their own data centers.

What we were hoping to get out of Clickhouse

Raw performance: The main benefit we get from choosing Clickhouse is the raw performance its design enables for analytical workloads. And with how fast it is by default, we can save a ton of time and keep the system simpler. We can do complex queries over millions of rows and they are fast enough without further optimization.
Materialized views: Once we do want to cache aggregated data, we can make use of incremental materialized views that are built into Clickhouse. No separate system needed for processing the data.
Made for analytics: Clickhouse comes with a ton of built-in functionality to simplify analytical queries. From supported data formats to helper functions - the available tools make our job simpler and quicker. For example, ArgsMaxIf simplifies our aggregate queries by a lot.
Integrations: Clickhouse comes with many integrations out of the box. In our case, we are streaming JSON data via NATS. But Kafka, S3 and many more common systems are supported out of the box. This reduces the number of moving parts needed to build a complete data platform.

First impressions

I always feel a bit intimidated by technology that is intended to operate at the scale of millions of events per second and petabytes of data.

But it was surprisingly simple to get started with the Clickhouse Docker image. It's well documented. The whole documentation of Clickhouse is simple to read and navigate. And the CLI tools feel modern and intuitive to interact with.

Surprises we ran into

When first reading about how fast Clickhouse is for analytical workloads, it feels like magic. Of course, they are not hiding that there are tradeoffs to get the system to be this performant. But reading about a system not having transactional guarantees and encountering it in practice are very different things. The following are what I wish I knew before getting started:

Table Engines: You have to select an engine for each table you create. And this is not only an afterthought. If you want to replace old values with new ones, you need a different engine than if you only append new data. And if you aggregate or summarize values, there are specific engines for that. I can't say I understand all available engines yet. It takes some time to understand which one to pick.
Encountering duplicate data: The ReplacingMergeTree table engine replaces existing rows and allows you to basically do UPSERT. But the old data will not be removed directly. Duplicate rows will only be removed asynchronously. Also, setting a primary key only affects how data is stored on disk, but doesn't guarantee unique values. There are ways to get unique query results, but it takes some extra care. You can ensure unique rows using GROUP BY at query time and there are also settings like FINAL and optimize_on_insert that remove duplicates for a performance penalty.
Incremental Materialized Views are not always updated: In Clickhouse, Materialized Views are incrementally updated automatically. This was a main selling point that convinced me that it can simplify the system by a lot. Materialized Views behave like insert triggers. The database client that does an INSERT on a parent table receives an error if updating any materialized view failed. But there are actually no transactional guarantees around this and the data is still written to the parent table. We work around this by using a ReplacingMergeTree as parent table to deduplicate data and resend failed events using NATS to retrigger the Materialized View.
Joins in Materialized Views are hard to get right: Materialized Views support joins and sub-queries. But Incremental Materialized Views are implemented as triggers on a single parent table. If data in a joined table changes, the view is not updated. This is not directly mentioned in the documentation. I only later found a blog post mentioning this. While the docs don't mention this limitation directly, they do mention a way to work around it: You can create multiple Materialized Views that write to the same table. If you use the AggregatingMergeTree table engine, you can combine data from multiple parent tables by writing into separate columns. It's not the most intuitive to work with and think through the performance implications of it though.
No NATS Jetstream support: I was amazed that we can integrate data from NATS without any additional infrastructure. After wondering why it doesn't work as expected, I found out that Jetstream is not yet supported. So for the time being, we have a small ingest service to get the data guarantees we need.

The surprises we encountered are understandable tradeoffs that come with the chosen system design behind Clickhouse and luckily we can work around all of them.

Overall, getting started with Clickhouse was a smooth experience. Let's see how the system is holding up a few months into the future.

Advance of AI

jorin — Sat, 14 Sep 2024 14:20:34 +0000

As someone working in tech, it's hard to talk about anything other than AI these days. The progress we're seeing is intense. While a lot of it is hype, AI is already impacting many of our jobs and lives. These changes are here to stay.

The core ideas of AI are rooted in simple mathematics for identifying patterns in sample data. However, with sufficient data, these algorithms develop logical reasoning and creative expressions that can match —and often surpass— the best human experts, especially in terms of speed.

These algorithms are continuously improving their ability to understand and generate various forms of media, including text, images, video, and audio. Working with images and audio is crucial for interacting with the physical world. This capability will play a vital role in advancing robotics.

For me, AI’s advancements around working with natural language are the most fascinating. Text and language are direct representations of human reasoning. It feels as if machines can think just like a human.

The direct consequence is that many jobs will look very different — and this transformation has already started. AI can now assist and automate various forms of knowledge work. As a software developer, I must drastically shift my approach to stay relevant. What we've seen so far is just the tip of the iceberg.

AI is here to stay, whether we like it or not, and the times ahead will bring many changes. The world has always been changing, but the pace of change is now accelerating more than ever.

It's not my place to judge whether the changes are good or bad. However, I know that trying to stop change is futile. We cannot influence what happens but how it happens.

We must raise awareness about the challenges accompanying AI technology advancements. It's crucial to educate people on both the promises and risks of AI. We should also urge governments and organizations to support those at risk of being left behind and to keep potential bad actors in check.

The most crucial task now isn't to hastily improve and integrate AI into every aspect of life. Instead, we should focus on humans rather than machines.

The economic and political interests in leading the AI revolution are irresistible. These changes will happen too quickly for most of humanity to keep up. What we truly need are individuals who ensure these advancements happen with everyone's interests in mind.

In the face of AI, we will gain a completely new understanding of what it means to be human. We need to treat each other with the respect and dignity we deserve, rather than allowing billions of people to become small cogs in machines that only benefit a select few.

Moving the Blog to Elixir

jorin — Mon, 16 Oct 2023 08:47:22 +0000

After running my blog with Hugo hosted on Netlify for nearly a decade, I decided to migrate to a custom solution. I think I resisted the urge to rewrite my personal site for long enough and I deserve to have some fun.
I am not going to find an excuse. I wanted to do this in Elixir. I wanted to build all the details of a static website from scratch once. It feels empowering to understand everything. And everything I learned along the way is generally useful Elixir knowledge. There are no concepts specific to a single static side generator.

With that out of the way, the reason why I am doing this now is that I plan to build more with Elixir in the near future and there will be more static sites too.
I also wanted to learn the details of generating SEO-friendly meta tags, getting a high pagespeed score and everything else involved in the progress.

<meta charset="utf-8" />
<title><%= @title %></title>
<meta name="description" content={@description} />
<meta name="author" content={Content.site_author()} />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<link href="/index.xml" rel="alternate" type="application/rss+xml" title={Content.site_title()} />
<meta name="ROBOTS" content="INDEX, FOLLOW" />
<meta property="og:title" content={@title} />
<meta property="og:description" content={@description} />
<meta property="og:type" content={@og_type} />
<meta property="og:url" content={"#{Content.site_url()}#{@route}"}>
<meta name="twitter:card" content="summary" />
<meta name="twitter:title" content={@title} />
<meta name="twitter:description" content={@description} />
<meta itemprop="name" content={@title} />
<meta itemprop="description" content={@description} />
<%= if @og_type == "article" do %>
  <meta itemprop="datePublished" content={format_iso_date(@date)} />
  <meta itemprop="dateModified" content={format_iso_date(@date)} />
  <meta itemprop="wordCount" content={@wordcount} />
  <meta itemprop="keywords" content={Enum.join(@keywords, ",")} />
  <meta property="article:author" content={Content.site_author()} />
  <meta property="article:section" content="Software" />
  <meta :for={keyword <- @keywords} property="article:tag" content={keyword} />
  <meta property="article:published_time" content={format_iso_date(@date)} />
  <meta property="article:modified_time" content={format_iso_date(@date)} />
<% end %>

What the site does

You can find all the code on Github.
Elixir renders static HTML pages from Markdown and YAML content. Most pages are blog posts. There are additional pages such as the about page. And XML files are generated for the Sitemap and RSS feed.

def sitemap(pages) do
  {:urlset,
   %{
     xmlns: "http://www.sitemaps.org/schemas/sitemap/0.9",
     "xmlns:xhtml": "http://www.w3.org/1999/xhtml"
   },
   [
     {:url, [{:loc, Content.site_url()}, {:lastmod, format_sitemap_date(DateTime.utc_now())}]}
     | for page <- pages do
         {:url,
          [{:loc, Content.site_url() <> page.route}, {:lastmod, format_sitemap_date(page.date)}]}
       end
   ]}
  |> XmlBuilder.document()
  |> XmlBuilder.generate()
end

How it works

I have to give a huge shout out Jason Stiebs' post on the fly.io blog. I followed the tutorial to use NimblePublisher to read markdown files, render them to HTML using Phoenix LiveView HEEx templates and integrate Tailwind into the setup.
Having Tailwind available is something I really wanted. I plan to use it to simplify my hardly maintainable CSS.

I also have to thank Hugo again. Having the original output generated by Hugo helped me to learn about setting proper meta tags, rendering the sitemap and generating the RSS feed.

The setup includes a tiny Plug server to serve files during development and ExSync auto-compiles the site when files change.

Finally, the site is now deployed to Github pages using a Github action. This means one less separate tool to understand and maintain.

def build_pages() do
  pages = Content.all_pages()
  all_posts = Content.all_posts()
  about_page = Content.about_page()
  reads = Content.get_reads()
  assert_uniq_page_ids!(pages)
  render_file("index.html", Render.index(%{posts: Content.active_posts()}))
  render_file("404.html", Render.page(Content.not_found_page()))
  render_file(about_page.html_path, Render.page(about_page))
  render_file("archive/index.html", Render.archive(%{posts: all_posts}))
  render_file("reads/index.html", Render.reads_index(%{pages: reads}))
  write_file("index.xml", Render.rss(all_posts))
  write_file("sitemap.xml", Render.sitemap(pages))
  render_posts(all_posts)
  render_redirects(Content.redirects())
  render_reads(reads)
  :ok
end

Note that the code can be as simple as it is because it is catered to my use case. It doesn't have to support thousands of pages or flexible content hierarchies. It only needs to do exactly what I want, nothing more, nothing less.

Problem Solving

jorin — Sun, 25 Jun 2023 11:19:25 +0000

Software creates value by solving problems. But effectively solving problems is hard. How can we get better at it?

Motivation

I have seen it many times in different teams and projects:
We are certain our users need a new dashboard view. It will solve all their problems.
A month later we are sure the users need 3 more filter options on the dashboard, then all their problems will go away.
A year later users are still loading data into their custom Excel file and barely anyone touches the fancy dashboard we built.

We create solutions without actually knowing what the problem is.

We all want to create value for our users. We want to create solutions that truly solve their problems. We want happy users. We want to build features that remain useful and do not become a burden down the line.
So how do we create solutions that effectively solve problems?

It all starts with really understanding what the problem is.

The problem is not always what it seems and the most obvious solution might not be the best.

Spend more time understanding the problem, its scope and the potential solutions.
It will pay off many times over.

Not only do you significantly increase the chance of building the right thing to start with - you enable yourself and your team to improve the product sustainably in the future.

Capturing your thought also process creates a powerful artifact to work with in the future:

No one gets everything right the first time. And that's fine.

But you want to be able to look back and know which problem you tried to solve. What was the known context at the time? Which solutions did you consider? What was the reason to go with the chosen option?

With answers to these questions at hand you don't have to guess why things are the way they are and you can improve the product with confidence.

Framework

I like to follow a simple framework to guide me through the problem solving process:

Problem Statement: State the problem in a clear and concise sentence.
Context: Describe the context. What do you know? What do you need to know? What matters? What can you exclude?
Options: List possible solutions. What options can you think of? What are the pros and cons?
Decision: Decide for a solution. Write down the decision and reasoning behind it.

Make sure you talk to actual users when establishing the context.
Spent some extra time on listing options. Discuss options with others. Be creative. Think of combinations of solutions. Doing nothing is also always an option.
Sleep over it for at least a night before making a decision.

After you made a decision and implemented the solution, make sure to come back to it.
It's fine to get things wrong. You are one step further in any case. You gained new knowledge about what doesn't work.

Problem solving is an iterative process.

Repeat until you get it right.

Mindset

A framework can be a helpful tool to guide our thinking.
But to become really good at problems solving we need to change our mindset.

Create a culture that encourages people to ask questions, challenge assumptions and explore alternatives.
Learning, reflection and active listening need to become part of your daily work.
Establish a team of people with diverse backgrounds and perspectives.
Encourage people to think of interdependencies, feedback loops and long-term implications when choosing a solution.

All this is not a simple undertaking and takes time.

Start by making time for it and lead by example.

Summary

Software development is problem solving. Focus on understanding problems.

Define the problem and the context.

List multiple solution options and document your decision.

If you like to learn more, I can highly recommend watching the talk Design in Practice by Rich Hickey.

Migrating Data When You Never Erase History

jorin — Wed, 28 Oct 2020 18:05:08 +0000

Let's explore the considerations necessary when evolving temporal data over time.

Almost all actively used software evolves over time. Most software also includes certain data that is stored long-term. The database behind such applications tends to be slower-changing than any particular application logic. However, the underlying data still evolves over time as requirements change. Migrations of data are already a challenge. If you now persist the history of the data changes over time and your requirements evolve in time, you now have to consider these changes not only at a specific point in time but along the whole temporal dimension.

To make any decision about migration strategies in a particular system, you have to understand the context the data is used in.

Feel free to skip sections and jump to the ones relevant to you.

Why keep history?

Many things become possible once you store change events and thus the previous state of a software system.

One of the most obvious uses of historic data is implementing requirements for auditing and version control in critical applications.

Having historic data also becomes invaluable for understanding and debuging complex processes. It is not only helpful for developers but also for users trying to understand the processes of the bigger system the software is used in.

From a technical perspective building solid, fault-tolerant, distributed software that interacts with external systems becomes more feasible once you have a log of change events. You can replay the events at any time to recreate the state of different sub-systems which is a big help keeping everything in sync without running into race conditions.

The value of storing historic data is huge and often the need for it will only arise later on. I argue that all software should be built this way by default. Ignoring history should be merely a possible performance optimisation for scenarios required that.

How to store history?

Storing historic data can be achieved in many ways. In a large-scale architecture it might be interesting to work with raw events. Apache Kafka is a great store for raw events. Having raw events directly accessible as interface for interactions provides a lot of flexibility and control necessary to work at scale. However, it also requires building a lot of logic on top to recreate a view of the current state of the world.

There are databases such as Datomic which is specifically designed for keeping all history of the data around. With Datomic the database takes care of keeping history while you can work with the current state of the system. The same queries you use to query the current state can then be used to query any previous point in time. Datomic also allows you to look at time ranges. It also provides functionality to follow the log of changes to work with the change events continuously.

Crux is a temporal database that allows you to work with historic data in similar ways to Datomic, but Crux is not only a temporal database but a bitemporal database. Crux not only stores the time you inserted data into the database, also referred to as transaction time, but also allows you to specify another time dimension called valid time. By differentiating what the system knew at a certain transaction time and what was the actual state of the world at a given valid time, you can accurately update not only the current state but also knowledge about the state in the past. This can be a crucial tool when you need to be able to correct historic data and at the same time have the requirement of knowing precisely what change was done at what time. I would argue that this is the only way to accurately represent historic data when you want to work at the level of entities and attributes and not at the level of raw change events.

The SQL standard also specifies the concept of temporal tables. SQL Server implements this and allows for automatic versioning of tables in what they call system time, which we described before as transaction time. This is internally implemented by having a separate history table, plus providing additional syntax to query data at a point in the past. MariaDB is another SQL database that supports temporal tables. MariaDB goes even further and also supports bitemporal data using a second time dimension here referred to as application time, which matches the earlier introduced valid time.

Apart from existing implementations of temporal and bitemporal datastores, there are also use cases with needs for scale and operational characteristics, which might be best served by implementing a custom solution on top of another storage.

Keep history forever?

Different use cases of historic data demand different strategies for data retention. Ideally you can always keep all data around forever. Unfortunately that can become unfeasible in certain situations. In that case you have different options to reducing the number of data points you have to retain:

Completely erase old data after a certain expiration time or through other expiration policies.
Delete the data itself but keep meta data around.
Keep old data in lower granularity. For example, only keep one data point per day per entity for data older than a week.
Merge multiple data points into one using a merge logic specific to your needs.

An important property to consider when planning a retention strategy is the ability to recreate the current state from the log of historic data.

How do you use historic data?

Do you control the code that receives the data? Can you update it in sync with changes to the data schema?
Are the use cases of the current view of the data and historic data separate from each other?
Often most parts of an application use only the current state and historic data is only used in a small number of places. Can you require a stricter schema for the current state and a looser one for historic data?
In case you have two time axis, such as valid time plus transaction time, there might be different features building on each of them. Likely there are more use cases of valid time than transaction time.
Do you know the data schema version that downstream consumers support? Can you remove support for deprecated data features once all consumers are on a recent version?

What changes to the data do you need to make?

The most important thing to consider when changing data is that you want to avoid breaking changes.

When the logic of your system evolves it sometimes becomes necessary to change the shape and thus the schema of the data the system works with. This can come in different forms:

Adding a new optional attribute is a non-breaking change. Explicit, maybe even namespaced, attribute names help to keep clarity and avoid conflict.
Adding a required attribute or making a previously optional attribute required, is a breaking change.
Can you introduce a new attribute instead of renaming an existing one? Mark the old one as deprecated, but leave it there for consumers of historic data.
If you must change the type of an attribute, can you extend it instead? Instead of changing from string to integer, can you use a union type of string or integer?
Removing data will break downstream code that depends on it.

Sometimes we don't have to change the shape of data but the data itself. This is especially common if there has been an error in the system and we have to correct or even delete erroneous data.
Changes to the data itself can be seen as normal updates to the current state of the sytem. A temporal storage then gives you a record of everything for later reference. A bitemporal system also allows you to correct errors in historic data while keeping a record of everything.

Do data changes only apply to current state of data or also historic data?

If you introduce a new attribute, do you only need it in the current view of the data or do you need to generate it also for historic data? Is it even possible to reconstruct the attribute for past data points?
If you work with valid and transaction time, is it enough to introduce a new attribute for historic data in valid time or do you also need to rewrite transaction time?
When removing data, can you keep meta data or do you need to remove every trace that a certain event ever happened?
For deletion or modification of historic data, how does that propagate to downstream, maybe external, consumers?
Do you explicitly need to remove a specific attribute from all of history or can you model data in a way that you can delete whole documents instead? Depending on the system, deleting whole documents might be simpler to implement. How does that affect the schema of the documents in your system and the relations between them?
Before a data type schema becomes too complex, can you introduce a new data type instead? Or maybe you want a concept of versioning for types? You can save a specific version attribute with every entity and use that to handle different types through this. Or can you even version attributes themselves?

Two Kinds of Software

jorin — Thu, 20 Feb 2020 21:54:11 +0000

Let's talk about the difference between two kinds of software systems.

This might be controversial but for me it helps to use the terms Software Engineering and Software Development to differentiate between two fundamental modes of creating software.

That said, neither of the two directions is right or wrong and both are necessary to create actually useful software systems. Basically every software system is made up of software of both kinds. The systems are often highly entangled which makes it so easy to ignore the fundamental different modes of thinking they require.

Software Engineering

Engineering is about designing, building and working with machines.

Software engineering should strive for the same rigorousness as other engineering disciplines. It should use scientific methods and formalism to ensure correctness of software.

Engineering is the right approach for creating a solid foundation:

If you implement internet protocols, build a database, write a compiler, architect a distributed key-value store, create a file system - these systems require an engineering approach and we should strive for perfecting correctness.

Software Development

The concept of development is about bringing something into existence and it entails the notion of growing something into a more advanced form.

I like to see software development as the discipline we use to create software systems that interact with the rest of the world.

Whenever we can encapsulate and precisely define a problem, an engineering approach will be very effective for producing software of high quality.

However, once a system interacts with humans or other external systems in the world, things suddenly are much less fixed. It becomes hard to foresee all possible interactions and inputs our system will receive from the outside world. External systems can exhibit faulty behaviour or can even become malicious. Being situated in an ever-evolving world, the requirement we expect our systems to fulfil are in constant flux and change fundamentally over time.

Rigorous planning a correct solution is not a viable option when requirements are extremely dynamic and it is not something that can be achieved with any reasonable amount of effort.

We need different methods when working on such systems:

The first step is to accept that there will be mistakes and optimise your system to adapt and learn from errors.
Keeping your system flexible becomes one of the top priorities when decisions become invalid at any moment.
Being able to empirically try out different experiments is a useful tool when you cannot foresee the consequences of your system's behaviour upfront.
Having visibility through monitoring your software in production becomes even more crucial when you are unable to verify correctness upfront.

Creating social networks and online shops, but also building resource planning systems or working in financial markets - these systems keep on changing and should be development dynamically over time.

Once we accept that there is such a difference between the kinds of software systems we build, then we can making better decisions on how to build them.

For the first category, putting effort into upfront planning and all sort of efforts towards proving correctness can be very effective.

For the second category, instead of finding a perfect solution upfront we start with our best guess. The most important property of such a system is to build it in away that allows us to adapt it quickly and to observe it over time.

If we take apart our work, identify different sub-systems and judge them separately, then we can make more educated decisions for selecting the most effective methods and tools.

You need a different mindset to build a bridge than you do for planning a city.

Crux as General-Purpose Database

jorin — Thu, 12 Dec 2019 21:53:32 +0000

Crux is an innovative database in the Clojure world. What follows is an experience report.

Crux is open source, bitemporal, highly scalable and built on solid foundations with Kafka and RocksDB as default storage.
It is very flexible and provides low-level building blocks to allow you to build exactly the database you need for your use-case.

I believe that having time as first class concept in your database should be the default.
It is much easier to have the concept of time built-in from the beginning and only build specialized solutions for the parts where performance becomes a problem.
Nowadays computers are powerful enough that in most cases performance is not the issue and the focus should be on simplicity and correctness first.

By coincidence I started a toy project, letsdo.events, to write more Clojure in the spring of this year.
Just after I started, Crux announced its public beta. Naturally I had to give it a try.

Since letsdo.events is mostly a toy project to play with different ideas.
I took this opportunity to test Crux for a normal, standard application.
The goal was to see if Crux hides the overhead of bitemporal storage well enough
to make it viable as a default approach even when it is not an initial requirement of the application.

To start out I chose to only persist using RocksDB which allows to pack the whole system in a single .jar file and deploy it as a single process.
Once the storage needs to scale, it would be trivial to switch to Kafka as storage backend.
Since all the data is simply a log of transactions and documents, replaying them into a new storage would be straight forward.

Because of the flexibility Crux gives you, it does not necessarily have all the tools you would expect from a database built-in.
Surely there will be more tools and patterns in the ecosystem of Crux once it has been around for a bit longer.
But for now, here are a few things I built in the application's database layer:

Single Transactor

Enforcing constraints such as uniqueness is built into many databases.
The simplest way to achieve this outside the database is to only allow a single write at a time.
Reads can still happen in parallel.

I created a tx helper which is used for all writes to the database:

(let [ctx (init {:config config})]
  (tx ctx (if (exists-by-attribute :user/email mail)
              :email-taken
              (create! {:user/email mail} ctx)))
  (close ctx))

`:id` key

Crux expects the ID it uses internally under the namespaced key :crux.db/id.
This is a good pattern to not conflict with any domain-specific data.
However in my business logic I always ended up wanting to use this same ID without coupling the business logic to Crux.

So in letsdo.events all database functions convert the ID to :id transparently using these helpers:

(defn- crux->id [x] (rename-keys x {:crux.db/id :id}))
(defn- id->crux [x] (rename-keys x {:id :crux.db/id}))

Working with document time

It is totally fine in Crux to have normal :created and :updated as part of your documents.
And in most cases that is probably a good idea for performance.
But you don't need to have these timestamps since Crux also tracks time in its history.

It might not always be a good idea to do this but I created a helper which takes a list of document IDs
and returns the documents with additional :created and :updated keys:

(defn list-by-ids-with-timestamps [{:keys [::crux]} ids]
  (let [db (crux/db crux)]
    (->> ids
         (map #(let [h (crux/history crux %)]
                 (assoc (crux->id (crux/entity db %))
                        :created (-> h last :crux.db/valid-time)
                        :updated (-> h first :crux.db/valid-time)))))))

Common database operations

The primitives Crux provides for transactions and the datalog language for querying are simple yet powerful.
Still, for an application to interact with the database I prefer to hide the details of the how the database works behind plain functions.

The lde.core.db namespace contains helpers for operations such as create, update, delete, list, get by id, get by attribute, count, exists.

However, the database layer still allows to do plain datalog queries which is very useful for join-like queries likes this one:

(defn get-organizer-names-by-event-id [ctx event-id]
  (->> (db/q ctx {:find ['?name]
                  :where '[[o :organizer/event e]
                           [o :organizer/user u]
                           [u :user/name ?name]]
                  :args [{'e event-id}]})
       (map first)
       not-empty))

Global queries

Another great development feature of Crux has been the ability to easily query across all your data.

In relational databases this would involve writing out all tables and joining them together in a gigantic query,
but with Crux and datalog there is no strict separation between data types.

You can easily query attributes across all documents which makes it very simple in development to understand the data in your system:

; All data in db
(->> (crux/q (crux/db (:lde.core.db/crux @ctx))
             {:find '[id]
              :where '[[id :crux.db/id _]]})
      (map first)
      (map #(crux/entity (crux/db (:lde.core.db/crux @ctx)) %)))

This is very handy for all sorts of global statistics:

; All attributes in db
(->> (crux/q (crux/db (:lde.core.db/crux @ctx))
             {:find '[id]
              :where '[[id :crux.db/id _]]})
      (map first)
      (map #(crux/entity (crux/db (:lde.core.db/crux @ctx)) %))
      (mapcat keys)
      set
      sort)

Wrap up

It has been fun to work with Crux for this project.

Getting started with a new application with a schemaless database also made things easier.
I can imagine that for mature projects it would be helpful to enforce more constraints on write in the database or via clojure.spec.

So far I haven't tested out Crux at scale but the existing examples look very promising.

The pluggable architecture is one of biggest strong points of Crux and makes it very flexible to switch to a different storage backend or index implementation.

The biggest challenge will be to integrate with the world people are familiar with since Clojure, EDN and datalog are still far from mainstream at this point.
But with the HTTP API, integrations such as Kafka Connect, potential plans for an SQL-like frontend and detailed documentation, the future looks bright.

I do believe having time built-in from the start is a huge advantage for any new project and will become the norm in the future.

image credit

Fixing the World With Software

jorin — Sun, 30 Jun 2019 15:29:35 +0000

There are so many big problems the world faces today. We hear about them in the news every day.
You want to do something good in the world. You want to make a change. But you like building software.
What can a software developer do to make the world a better place?

Probably most of humanity had big problems in their time. In some way or another we might always fight for survival, for our existence. Uniquely nowadays is the global scale of our problems. The challenges we are facing have never been more complex and as part of our information-driven society the media constantly keeps reminding us of the threads we are facing. At the same time our abilities and the reach of our actions has never been greater.

I mostly see three obvious challenges nowadays:

Keeping the global human society in order to prevent fatal conflicts between different groups of humans. Even with free and diverse societies we need to find ways to solve conflicts peacefully. Modern warfare has advanced to a point where any wrong action could make the whole planet uninhabitable.
Finding ways to control human health and prevent epidemics. We still understand surprisingly little about the biology we are made of. A single virus could wipe out all of humanity.
Managing to live in harmony with our environment. The impact humans have on the planet has become so big that we have to find ways to keep our planet's ecosystem in balance. We need to find sustainable ways to deal with carbon emissions, air pollution, energy consumption, waste, water pollution, food production and many other issues.

The most frightening thing with these issues is that they seem too big for anyone to solve and it is difficult to even know where to start.

John Kay has some great advice for achieving complex goals:

But how can software help approaching these problems?

These issues seem to be only solvable by politicians and policy makers, scientists and engineers, company managers and journalists.

And this, I think, is the key:

I don't believe software can change the world. Humans can. I think our role as software developers is to support the ones that can make a change. We build tools and platforms to enable others.

If you are building software and want to tackle critical issues, ask yourself who you are building software for. Think about how you can enable others.

(cover photo by NASA)

Trends of the 21st Century

jorin — Sat, 02 Mar 2019 16:06:00 +0000

To define our own role and to know what to do next, it helps to sometimes take a break and get an idea of the bigger picture of what's happening around us.

I collected a few observations of patterns I see happening around me. Nothing here is novel but how often do we stop and question things? Noticing is the first step. Then it is up to us to decide what all this means.

Cultures mix more and more. Air travel is common. International tourism keeps growing. Communication is easier than ever thanks to digital technology. Cultures become more similar to each other through global media. Moving to another country is not a big. Cuisine has never had this much variation. Global markets give us access to goods from all across the world.
Urbanisation is still happening. It is not a new trend. People move to cities. Cities grow. Cities have very different culture to country side. Cities become more international. Major cities of the world become quite similar while culture in countryside still has bigger differences. Some mega cities have bigger political and economic power than whole countries.
Language consolidation accelerates. Everyone has to learn English nowadays. The internet is very dominated by the English language. Sciences and lots of other knowledge is published primarily and often solely in English. Events use English as default language. Other languages like Chinese, Spanish, French, Arabian dominate some local niches, but smaller languages don't even have the vocabulary for many things anymore since they simply borrow from English; they keep being languages for communication in everyday life but not for business and science.
Economy is global by default. More and more businesses don't depend on location. Teams can work distributed from any location. The default for companies is to target a global market. Competition also becomes global. Economies of scale make many businesses only feasible when targeting a global audience.
Innovation is accelerating at speeds we can't comprehend. Computers, the internet and so much other technology has been invented in the last couple of years. And those tools have enabled us to innovate at ever increasing speed. Being connected to all knowledge globally gives us so much leverage and power. We can't even imagine how the world will look like ten or twenty years from now.
Diversity is increasing. Even if we destroy cultures, traditions and whole species, the human-created variety is ever increasing. People with very different lives, cultures and levels wealth are living together. Old and new culture and technology are part of our lives. We need to create an environment that can accommodate all those differences.
Right wing parties are gaining power. With accelerating change and global competition comes a lot of uncertainty. People are afraid of loosing their dominant position in economy and culture and want to feel safe. They start voting for whoever promises to protect them. Big changes don't happen without resistance.
The whole globe is completely dominated by humans at this point. Humanity keeps growing and expanding. Space is getting more scars. Territorial conflicts are happing. Humans transform the environment to their own will. We already altered our environment up to a point that there is no turning back. We have to figure out how to control the effects of our doings. For us humans to survive on this planet, we have to redefine our relationship with the planet.

Not a Craftsmanship

jorin — Thu, 17 Jan 2019 19:31:23 +0000

There is absolutely nothing new in here. It is simply me slowly understanding the things people have been saying all along. Maybe sharing this does not even help others. But writing things down helps myself. After all we cannot really learn through theory alone. We need to experience for ourselves. There are many ideas mixed up in here. But they are all related. Feel free to get in touch to elaborate on some part.

The longer I am in this profession the more I understand what all those old people are talking about.
One thing I realize more and more, is how much we overvalue code as engineers.

Code feels very valuable since it is our creation. At some point we need to write code to make something become real. It feels very powerful.

But code is cheap in terms of the time and energy you have to put into writing code. If you compare actually writing code to planning what to do, understanding a business domain, understanding the technology options, understanding the process we have to go through while developing a product - writing code will take the least of your time.

If you are working on a product that will be around for a while and not dead right away, chances are that the code you write today is more a liability than an asset to you in the future. Much of the code we write will not be around for too long. Rather soon we will have better understanding of the domain, have found a more suited solutions, requirements changed or what we were creating was not that useful to the world in the first place.

When starting out programming I felt like it is about perfection. I was intrigued by creating this little perfect world where everything is logical. I felt like when I just search and experiment long enough I will distill the perfect solution. It was a beautiful world to live in.
I guess that others had similar experiences. It is very easy to be tricked by the nature of our site projects, student assignments and so on. They are very different to working on a living, evolving product that interacts with the real world.

The reality I experienced after is that all choices are compromises. Some part of your project will always be legacy code which you would like to rewrite and improve but you don't have the time to do so since priorities matter and this old, imperfect code still works for now and it is cheaper in terms of time spend to wait before changing it.
How did this code end up in this imperfect state? Didn't you come up with the best solution you could back when you wrote it? I'm sure I tried my best. But requirements might have changed, I might have learned new things about the problem we were solving back then or maybe when writing the code we did a trade-off to invest not too much time here.
There is something very fundamental to this process: As you solve a problem, you will learn from the solution. The first solution is very likely to not be the most suitable one. Sometimes - or rather, too many times, you also only realise after that you solved the wrong problem in the first place.

The most important measurement of “good” software design is to make it easy to change in the future for yet unknown requirements. Rarely any software is something static we can perfect like creating a sculpture or building a guitar. Creating software is a process. I have the suspicion that by using craftsmanship as metaphor for thinking about software development we might do us more harm than good.
A more fitting image would be politics or economics:
We saw in the soviet union what happens if you try to plan a perfect system. Think about all the failed planned cities out there. Politics and software are too complex to plan ahead. The only way to build solid systems of this complexity is to grow them organically. So instead of focusing on perfecting the result (the code) we should focus on optimizing the process.
Another image I like is a botanists taking care of a garden:
Take inspiration from organic systems out there. Think about how evolution allows nature to come up with such complex organic systems: Create new experiments, keep the good ones, ignore the bad ones.
We must not care too much about each of these experiments. Don't get attached to the code.
Build it in a loosely coupled manner so changing parts is simple. Make throwing away parts simple.

We need to focus more on understanding and optimizing the process of how we create software.
Literal code, code comments and so on are not enough documentation if the very structure of the code is very temporarily.
We need to learn documenting the process we are going through.
Version control for our code is one step in the right direction. But it is only on the lowest level of granularity.
We need to document what we are doing. If we look at it again in the future, we need to understand the context for why and how we did something. What where our assumptions back then? What where the constrains? How do they differ from where we are now?

Nowadays programming education is very much focused on the code. And it's not just universities. Education is a life-long process. The education we have in the industry in form of articles, books, conferences - it's all very focused on code. We need more tools for organizing processes. Teach programmers how to write down their reasoning, teach us how to draw diagrams, teach us how to experiment and throw away code, teach us how to call the right trade-offs for decisions. I realize that these things are very difficult to teach in an academy environment. Maybe that's why traditional education is so utterly useless for a job as a programmer in the industry. These trades are best learned in a realistic surrounding where we can gather experience and learn from each other. Share your stories with each other. Don't just show me what you build. Way more interesting is the how. Why did you end up with this solution? What was the process you went through to get there? What else did you consider in the process? These are my favorite kinds of exchange of thoughts.

Make it easier for us to observe what our systems are doing. Be aware that systems exist over time. Make our systems observable through monitoring, logging, metrics. Collect data to understand changes over time. See how your system evolves. Catch up with the trends. Remove or update outdated parts. What is great today might be utterly useless tomorrow. Focus on what is important at the moment: If you realize an API endpoint is not in use anymore, you know that next time you would have to touch it, you might simply delete it instead of bothering to update it. If you realize a feature is under heavy use, invest in it. Optimize performance when necessary, improve the feature when people are struggling with it.

Be aware that your processes and your product emerge from your organizational structure. If you have a hierarchical organization where decisions are made top-down, you will be planning systems instead of letting them evolve naturally. Individual actors need to have freedom to make their own decisions independent from each other. However they still need to communicate with each other and understand each other. Build organizations with flat hierarchies and loosely coupled teams. Build mechanisms for communicating with each other.

Do this on an organizational level by sharing stories about your work with each other. You can do this privately in your company: Internal blogs and talks are great.

On a technical level, to enable two parties to build software in a loosely coupled way that works together, the parties need to come together and define a contract for the communication of their software. As long as they stick to that contract, they are free. Have good tools for defining and enforcing contracts.

All these things are where the complexity in software comes from. Stop caring so much about the code you write.

The Cloud Is the New OS - A Developer's Perspective

jorin — Thu, 29 Nov 2018 18:30:57 +0000

It already happened a few times in the history of computing that the level of abstraction the majority of us work on has been raised.

There are, and probably always will be, people that write and understand assembler. But most developers don't worry about the exact instruction they send to their CPU on a day to day basis. Most developers nowadays don't even worry about allocating memory manually. Syscalls to talk to our Operation System is the lowest level the majority of developers has in mind. Mostly it's not even syscalls but an API which the language runtime provides.

I think we are close to a point where the general level of abstraction of computing will be raised once more:

My prediction is that in the near future it will be normal to not think about which physical machine the program we are writing is running on.

We are moving to the cloud.

Of course this is not a new idea. I never have new ideas. But it took me a while to realise the implications of moving to the cloud.

Also please forgive me for using such a buzzword - Cloud is the best description for software with invisible hardware that I have heard of.

From a consumer's point of view this doesn't sound like a new idea. It has become totally normal to have all your files, all your data somewhere in the internet and access it from all your devices whenever you want.

People stopped caring where the cloud actually physically is already a long time ago (this is not completely true: here in Germany everyone is still worried and trying hard to find the cloud).

Our social and working tools have basically all moved already. The laptops and phones we own don't need to offer much. They are becoming commodity. They simply provide a window to interact with the cloud.

In some cases such as gaming the hardware you own is still critical but personal, high-end gaming PCs are also a thing of the past.

For consumers and workplaces the convenience of not having to worry about hardware is such a huge gain, it was an easy move. People just don't care about how technology works. They don't want to know. They don't want to bother. They want their services and tools to simply work and do their job. Let someone else do the maintenance.

Having control over your computer is simply a burden for most people.

But we developers like technology. We want to be in full control over our system and our data, right? I doubt it. Not the majority at least. We are also simply humans. We own or work for companies that try to run a business. If there is a more efficient way and the gain is big enough, we also move.

Our production systems are largely running in the cloud already. We rent virtual servers from Amazon, Google, Microsoft. We push static content to CDN services. More and more of the functionality we need is now available as a service and we don't have to manage them ourselves anymore:
databases, search engines, firewalls, message queues, file storage, load balancers, web servers, build servers, test runners, registries, secret management, user authentication, …
There is a service for basically every generic piece of software out there (if you see one missing, make sure you are the first one to build it!).

The only thing that makes our software unique is our own, custom business logic of how we connect the pieces together.

Surprisingly it is generally still the case that we express business logic as software in the same programming languages and runtimes we used when we were thinking about a physical machines. We put great efforts into taking the existing environments we have in the form of operation systems such as Windows and Linux, faking a virtual environment that is identical to the OS we have from the past and executing our business logic in there nested in unnecessary layers of indirection with OSes in OSes in OSes, ten layers deep.

What if we let go of the past, let go of the control and create efficient platforms suited for expressing our business logic?

If we want to tackle more complex problems, we need a solid foundation. We need to raise the level of abstraction. More complex business domains require us to be able to focus on them exclusively. We need to separate the work of building the foundation from the next layer.

The foundation is important and critical to get right. There will always be a need for great people to work on the foundation of computing. But the majority of problems developers are trying to solve today and an even bigger percentage of what we will have to solve in the future is not about technology in itself - it is all the problems in the world where technology has the potential to help us solve them. And there are plenty. We better get good at using technology effectively instead of fighting against the complexity we created through layers of unhelpful abstractions. Let's admit that it's time for a new abstraction. Let's use our existing languages, platforms and tools for what they are good at and build another solid, efficient layer as next abstraction.

A programming platform for this level of expression has different characteristics than lower-level platforms:
It is mostly glue connecting lower-level components. There needs to be a good API of primitives to talk to available components. The concept of starting and stopping the system is handled by the underlying platform which allows this layer to be much more dynamic, loading only what is needed at this time. Performance critical tasks are most likely handled by lower-level primitives.
There are many more properties to define for such as system.

Did I mention before that we are giving up control? And you know that software is eating the world?

If we don't want all power to end up in the hands of a selected few that run these new platforms, we better make sure we get this right.

Let’s not wait for the big infrastructure companies to come up with a platform. Let’s not wait for them to bind us to their specific ideas and their products. Let’s make sure we create a healthy system with a multitude of options and diversity of ideas.

It is great to see alternatives to the big cloud providers showing up. Shout out to netlify, fastly, DigitalOcean and friends! A healthy market needs competition.

Similar to the variety of operating systems and programming languages out there, we should aim for a variety of cloud platforms that are compatible with each other and share standards and protocols.

This is how technology like email and the web became this widely adopted.

The greatest efforts I see at the moment is the work happening under the umbrella of the Cloud Native Computing Foundation. We need more initiatives like cloudevents and openmetrics!

One of the next big steps I see is in developer tooling. Developers like to and need to be in control of their systems to test, monitor and debug them. It cannot be the right way that developers have to emulate a cloud on their laptop. Instead, let's move development environments into the cloud. And developers need good tools to be productive. A web interface is not enough. We should have realtime APIs on which we can build tooling on top of.
There is still a lot of room in this space waiting to be filled. Moving IDEs into the cloud is only a very first step.

Fellow developers, beware of the level of abstraction the problem you are trying to solve is on. Be ready to development in the cloud. Let's work together to make our next platforms a great place to work in.

Thank you.