Forem: Christian Himpe

DatAasee 0.5 Released!

Christian Himpe — Thu, 13 Nov 2025 15:48:16 +0000

DatAasee - A Metadata-Lake for Libraries - Version 0.5 has been released:

Using declarative programming all the way down:

A Compose-based web service,
combining the ArcadeDB database,
with a Benthos back-end,
and a Lowdefy front-end.

DatAasee - A Metadata-Lake

Christian Himpe — Tue, 06 May 2025 15:42:54 +0000

DatAasee is an open-source metadata-lake, functioning as a metadata aggregator of distributed data-sources in libraries. It instantiates a data space and provides a FAIR-compliant metadata-layer for repositories. Its data architecture is a data-lake which is its own metadata catalog, hence a metadata-lake, with an underlying graph data model.

Internally, DatAasee is realized as a three tier software architecture, where each tier is encapsulated in its own container. The inner data-tier is a multi-model NoSQL database ArcadeDB. The middle logic-tier is an (HTTP-JSON-REST) API-server and semantic-layer around the database realized via Connect (aka Benthos). The outer presentation-layer uses Lowdefy as a prototype frontend which exclusively uses the API. This multi-container application is deployed and orchestrated via Compose.

For in-depth explanations, see the open-access white paper: "DatAasee -- A Metadata-Lake as Metadata Catalog for a Virtual Data-Lake".

Scheming About Clojure

Christian Himpe — Wed, 13 Nov 2024 20:53:52 +0000

Clojure is a LISP for the Java Virtual Machine (JVM). As a schemer, I wondered if I should give Clojure a go professionally. After all, I enjoy Rich Hickey's talks and even Uncle Bob is a Clojure fan. So I considered strength and weaknesses from my point of view:

Pros

S-Expressions
Makes functional programming easy
Schemy naming with ? and ! suffixes
Integrated testing framework
Platform independence due to JVM
Simple Java interoperability
Clojure map type corresponds to JSON
Web-server abstraction with extensions (Ring)
Dedicated Ubuntu-based Docker image

Cons

Too many core functions
Too many concurrency concepts
Having collection functions and the sequence API is confusing
Keywords feels unnecessary, given symbols
Unwieldy default project structure
Leiningen feels forced upon you
Clojure is not just a single jar (anymore)
No integrated JSON parser

Insight

Clojure seems good enough. It is not flawless and somewhat overloaded, but far, far ahead of Javascript, Python, Go, or Rust. Of course, I would always prefer CHICKEN Scheme for any passion project. But in an environment that already runs databases written in Java, the JVM has street cred, and a large community hints at sustainability, Clojure presents itself as well balanced in novelty and stability. All in all, Clojure seems to be the enterprise Lisp.

References

One Minute: DatAasee

Christian Himpe — Tue, 10 Sep 2024 15:43:52 +0000

DatAasee is a metadata-lake to aggregate and centralize bibliographic and research-data metadata from distributed data and metadata sources to improve discoverability and FAIRness.

Features: REST-like CQRS HTTP-API, Faceted Search, Full-text Search
Interface: OpenAPI, JSON, JSON API, JSON Schema
Query Languages: SQL dialect, Cypher, Gremlin, MQL, GraphQL
Ingest Protocols: OAI-PMH, S3
Ingest Encoding: XML
Ingest Formats: DataCite, DublinCore, MARC, MODS
Deployment: Compose, Docker-Compose, Podman-Compose, K8s (via Kompose)
Components: ArcadeDB, Connect, Lowdefy
License: MIT

Website: https://ulbmuenster.github.io/dataasee
Repository: https://github.com/ulbmuenster/dataasee
Companion Paper: https://arxiv.org/abs/2409.05512

One Minute: DuckDB

Christian Himpe — Tue, 20 Aug 2024 17:16:46 +0000

DuckDB is an open-source, in-memory, column-oriented relational database management system inspired by SQLite.

Data Model: Relational
Interfaces: C, C++, Go, Java, Julia, Node, Python, R, Rust, Swift, WASM, ADBC, ODBC
Query Language: SQL super-set
Features: Console, ACID
Programming Language: C++
License: MIT

Website: https://duckdb.org
Documentation: https://duckdb.org/docs
Repository: https://github.com/duckdb/duckdb
Mastodon: https://mastodon.social/@duckdb
Twitter: https://twitter.com/duckdb

Database of Databases: https://dbdb.io/db/duckdb
DB-Engines: https://db-engines.com/en/system/DuckDB

One Minute: Compose

Christian Himpe — Sat, 30 Mar 2024 18:51:46 +0000

Compose is a single-host container orchestrator for single and multi-container applications.

Specification: https://compose-spec.io
Definition: Compose File (YAML)
Elements:
- Services,
- Network,
- Volumes,
- Configs,
- Secrets
Supports:
- .env files,
- Interpolation
Features:
- Dependency resolution,
- Resource limits (CPU & RAM),
- Remote deploy (SSH)
Compatibility:

My Data Engineering Library

Christian Himpe — Tue, 19 Mar 2024 14:08:20 +0000

This is neither an "Ultimate List of Data Engineering Books" nor the "Data Engineering Must-Read List", but rather my subjective recommendation for a minimal data engineering library; particularly for data engineers coming from software development, as I used to.

General

Reis, Housley: "Fundamentals of Data Engineering"; Oreilly Media, 2022.
- Discusses all aspects of data engineering and interlinks topics through "under currents".

Data Modeling

Hoberman: "Data Modeling Made Simple"; Technics Publication, 2016.
- Must-read for anybody doing data modeling.
Serra: "Deciphering Data Architecture"; Oreilly Media, 2024.
- Description and comparison of major data architectures.

Databases

Forta: "SQL in 10 Minutes"; Pearson, 2010.
- The best minimal SQL guide I found, yet.
Kaufmann, Meier: "SQL and NoSQL Databases"; Springer, 2019.
- Beyond touching all the database basics, the relational (SQL), document (Mongo) and graph model (Cypher) are recurringly contrasted.

Containerization

Öggl, Kofler: "Docker"; Rheinwerk Computing, 2023.
- Touches on all facets of Docker alongside with typical application examples.
Gkatziouras: "A Developer's Essential Guide to Docker Compose"; Packt, 2022.
- In deep description of the Compose orchestrator.

Extra

Densmore: "Data Pipelines Pocket Reference"; Oreilly Media, 2021.
- Good example-driven overview; original source of EtLT (extract, partial transform, load, transform).
Chromatic: "Extreme Programming Pocket Guide"; Oreilly Media, 2003.
- Extreme programming (XP) is the sanest agile method with some good sustainability practices like the 40h week.

One Minute: Lowdefy

Christian Himpe — Sun, 17 Mar 2024 17:03:07 +0000

Lowdefy is a declarative low-code front-end framework for internal tools or CRUD apps defined via YAML or JSON.

Block Types: Input, Display, Container, List
Connections: Redshift, S3, HTTP, ElasticSearch, Google Sheets, Knex, MariaDB, MongoDB, SQL Server, MySQL, PostgreSQL, Redis, SendGrid, SQLite, Stripe
Features: AgGrid, ECharts, Markdown (Display Blocks)
Modes: Development, Production
Written In: Javascript
Layout: Flexgrid
Build On: Next.js
Requires: PNpM
Deploy Via: Docker
License: Apache-2.0
Website: https://lowdefy.com
Documentation: https://docs.lowdefy.com/introduction
Repository: https://github.com/lowdefy/lowdefy
NPM Package: https://www.npmjs.com/package/lowdefy

YAML-based Database Docu

Christian Himpe — Fri, 08 Mar 2024 13:36:15 +0000

I recently came across the modest yasql standard to document database schemas of SQL databases, which I now use to document my rather involved ArcadeDB schema. My favorite yasql features are:

Very readable,
Rich metadata fields,
A definitions object for custom "macros".

Here is a small example of a yasql docu:

database:
  name: "My DB"
  source: "PostgreSQL"
  project: "My Project"
  version: 1.0
  license: "CC-BY"
  authors:
    - "Me"

definitions:
  tiny: 255

tables:
  myTable:
    myCol: "VARCHAR(tiny)"

NoSQL

Generally, NoSQL databases do not use tables to organize data. For example, ArcadeDB - a multi-model DBMS - provides a document model and a graph model. Hence, instead of a tables object, I use documents, vertexes, and edges objects:

documents:
  myDoc:
    myStr: "STRING (mandatory, notnull, readonly)"

vertexes:
  myVtx:
    myNum: "LONG (default 0)"

edges:
  myEdg:
    myLst: "LIST (max 255)"

Alternatives

There are (open) alternatives available which provide the benefit of visualization as entity-relationship diagram. So why not use ...

... DBML ? Because:
- Column settings have to be from a very limited vocabulary.
- C-Style comments // and /* */.
- Insufficients metadata fields (in the project object).
... D2 sql_table ? Because:
- Every table needs a shape: sql_table, meaning a column cannot be named shape?
- The constraint keyword is superfluous since a constraint is already delimited by curly braces { }.
- No metadata fields.

Visualization

You can visualize a yasql-file using PlantUML, which provides a YAML renderer. For instance, you can find the rendering of the yasql sample here

Practically, I embed a plantuml verbatim block wrapping the yasql YAML, in a Markdown document:

```plantuml
@startyaml

# your yasql goes here ...

@endyaml
```

This way, using the VScode PlantUML Plugin I can get live visualizations of my schema.

One Minute: Lightwhale

Christian Himpe — Sun, 28 Jan 2024 22:17:39 +0000

Lightwhale is a lightweight, immutable Linux for hosting Docker containers.

Based On: Buildroot and Docker
Concept: Immutability and Persistence
Advantages: Small attack surface, no redundant apps, no installation
Compatibility: Docker-Compose
Connectivity: SSH
Platform: x86-64
License: GPL-2.0
Website: https://lightwhale.asklandd.dk
Repository: https://bitbucket.org/asklandd/lightwhale

About Functional Software Architecture

Christian Himpe — Sun, 10 Dec 2023 13:56:34 +0000

Software architecture is the practice of modularizing and organizing software development from large to small. Functional programming on the other hand is a style of programming centered around functions and, among others, concepts like Pure functions or avoiding State and Side-Effects as far as possible. Functional software architecture is thus the transfer of ideas from functional programming to software architecting. In the following I will describe my point of view of such functional architecture.

I note, that there are already quite a few contributions to this topic, for example:

https://increment.com/software-architecture/primer-on-functional-architecture/
https://www.infoq.com/news/2023/04/late-arch-functional-programming
https://www.functional-architecture.org
as well as various talks available on Youtube.

However, I often miss the practical correspondence to functional programming. So, here is my take on functional architecture.
To avoid a lengthy introduction to functional ideas, I assume you have some very basic understanding.

Software Architecture

Every software is in essence a transformation of data, and software architecture is, to me, first of all a bird's eye view of this transformation. Thus the most basic depiction of software can be given by:

Inputs
Outputs
State
Side-Effects

which describe the core functional requirements of a software project (or component).

Inputs and Outputs

Inputs control a process, while outputs result from a process. Jointly, inputs and outputs are called ports, and are specified by a:

Protocol
Media type
Structure
Format
Content

which need to be defined on the architecture level to allow interoperability. An example for a web-service is:

HTTP as protocol
JSON as media type
OpenAPI to define the structure of a HTTP API
JSON:API to format each message
JSON-schema to define templates for request and response contents.

Generally, HTTP and JSON are a widespread choice, facilitating interfacing. HTTP is easily testable with a browser or standard tools like wget or curl. JSON is more compact than XML, not white-space sensitive like YAML, and natively parsable in Javascript, which has relevance for frontends.

On a lower level, a definition could be linking type (protocol), data types (media type), header file (structure), function signatures (format), argument and return value constraints (content).

State vs Side-Effects

State is a remberance of past effects, while side-effects may alter state outside the current scope. Hence, it is important if an effect affects inside or outside the project's (or component's) scope.

Component View

Every non-trivial software project consists of components. Each component again is defined by a bird's eye view consisting of inputs, outputs, state, and side-effects.

In my opinion, in functional architecture, functions in programming generalize to components in architecture. Hence, the characteristic of components one should aim for is purity, which means in component terms idempotence or statelessness. If a component has to carry state to be useful, such as a database, then it should be isolated from other components.

To correctly architect a component's statefulness and side-effects, dependency management is paramount. This refers to dependent components as well as external dependencies, ie libraries, services, etc.

Components and Containers

As a practical approach to isolate top-level components, I think, containers (process virtualization via Docker or Podman) are particularly suitable for functional architectures, since typically containers do not retain state after restarts. Using containers to isolate components has the additional benefits of reproducible deployments, and in case of stateless containers, easier replication and scaling.

Note that state, in terms of containers, can be obvious in case permanent storage in form of an external volume is needed, but state can also come in terms of temporary files or memory-based. However, state can be discardable, for example certain types of lock files are only relevant to a currently running instance.

Furthermore in larger projects, components may have sub-components; here the same characterization of a sub-component, ie inputs, outputs, state, and side-effects apply, maybe not in terms of containers but in terms of libraries.

Side-Effect or Not?

The more self-sufficent components are, the more care needs to be taken on their self-management. For example each component likely contributes to a log. Writing to a log can be considered state in case of an internal (local) log or a side-effect in case of a central (global) log. Importance of such an effect determines if it induces state, side-effect or can be discarded.

At some level of granularity one has to disregard effects, otherwise one could consider just running the software causing heating of the environment surrounding the host system computer as a side-effect.

Technology Stack

Classically, a technology stack is defined for a whole project to have some degree of homgeneity across components. Using isolated components, like containers, technology stacks can be set up individually per component while retaining homogenous ports. Individuality can refer practically to the container image, programming languages, and dependencies.

Functional Languages?

A software architecture can be functional even though no functional language or functional programming paradigm is explicitly used. This is due to the focus on the ports and effects of a project and its components. However, strict discipline is required from architects with regard to those ports. Furthermore, developers need to take care, that no (non-discardable) disk- or memory-based state is introduced by using non-functional means of programming or utilized external dependencies; not to forget dependencies' dependencies.

Summary

To use this flavor of functional architecture, internalize that its functions all the way down: A software project (main function) and each of its components correspond to functions and have inputs, outputs, state and side-effects. And while additional views maybe needed to document component interaction, these can be based off the presented component characterization.

Bonus: Documenting Functional Architecture

There a various standards for documenting software architecture, like arc42 or C4. While useful and somewhat well-known (there is certainly a correlation here), here architecture documentation can be further simplified, particularly due to the self-similarity of project and component. Following is a small template, that can also serve as a project's and component's README:

# {Project/Component name}

What: {One-line executive summary of project/component}

Lead: {Main responsible person(s) for this project/component}

This: {URL, version pinpointing project/component change}

Repo: {URL to project/component repository}

Docu: {URL to project/component documentation}

Ports:
  - {Summarize an input, output or pair of input and output}
  - ...

State:
  - {Summarize how if at all state is carried}
  - ...

Side Effects:
  - {Summarize how a side effect is caused}
  - ...

Tech:
  - Image
  - Software
  - Language
  - Library
  - ...

Internal Deps:
  - {Name/URL of component dependency}

External Deps:
  - {Name/URL to an external software depedency}

References:
  - {Title/URL to a relevant reference}

Note, that this file is a Markdown and YAML file at the same time, and as such human- and machine-readable, if the fields are filled carefully.

Lastly, I note using such a README.md for each component (and sub-component) induces a hierarchical structure (for example of directories) in a project.

One Minute: BurmillaOS

Christian Himpe — Sun, 26 Nov 2023 11:29:42 +0000

BurmillaOS is a minimal Linux distribution for hosting Docker containers on virtual machines. BurmillaOS is pure containers!

Based On: RancherOS
Concept: Everything is Docker
Architecture: System Docker and user Docker layers
Advantages: Small attack surface, fast boot, fast container startup
Compatibility: Docker Compose (Optional)
Platform: x86-64
License: Apache-2.0
Website: https://github.com/burmilla/os
Documentation: https://burmillaos.org

PS: Testable on Apple Silicon MacOS via UTM (Deactivate UEFI Boot and use virtio-vga display card emulation)