Forem: Martin

What Git Doesn’t Solve: Designing a Transform Command for Data

Martin — Mon, 30 Mar 2026 18:08:10 +0000

In my previous post, I wrote about DataTracker's storage architecture (hashes, objects, and SQLite metadata). This follow-up is about what I think is the most technically interesting command: transform.

If you have not read the first post, quick context: DataTracker is a local CLI tool for versioning datasets (files or directories) with git-like commands (add, update, history, compare, diff, export, etc.).

This article focuses on one question:

How do you run a data transformation in Docker and still keep version history useful instead of chaotic?

Why `transform` Exists at All

Most data versioning tools stop at "store versions". That is useful, but in real workflows the interesting part is what happens between versions:

cleaning
reshaping
converting formats
running scripts in reproducible environments

I wanted this to be one command, not three manual steps each time.

Without dt transform, the flow looks like this:

Run some custom Docker command manually
Hope the output is written where you expected
Remember to call dt update afterward
Pick a version number and message

That works until you forget step 3 once and history becomes incomplete.

So the design target became: run transform + apply versioning rules in one place.

The Command Surface

At a high level:

dt transform --input-data <path> --output-data <path> [options]

The key options are:

--image, --command (required unless using a preset)
--auto-track
--no-track
--dataset-id
--version
--message
--preset
--force

The command validates Docker, validates tracker initialization, resolves paths, decides whether output should be versioned, runs the transformation in a container, and then possibly versions the output or even creates a new dataset.

Design Principle: Separate "Run" from "Track Decision"

One thing I changed early was separating concerns:

Docker execution stays in docker_manager.py
tracking/versioning policy stays in transform.py
Click argument parsing stays in commands.py

That split made the logic easier to test and refactor. It also prevented the CLI layer from becoming an unmaintainable if-else block.

Mount Contract and Safety Checks

Internally the command mounts:

input at /input (read-only)
output at /output

and runs:

docker run --rm -v <input>:/input:ro -v <output>:/output <image> /bin/sh -c "<command>"

By default, I validate that the command references both /input and /output. This catches a surprisingly common user error: the transformation technically succeeds, but writes data to a path that is not mounted back to the host.

If someone knows exactly what they are doing and wants to bypass this, --force disables that check.

This is one of the recurring themes in the CLI design: safe default, escape hatch available.

Auto-Versioning: The Real Core

After Docker runs, the command decides what to do with output history.

The policy is:

If --no-track is set: do not version output.
Else if input path matches a tracked dataset: version output into that dataset.
Else if input is untracked and --auto-track is set:
- add input as a new dataset
- then version output into that new dataset
Else: run transform only, no versioning.

This gives predictable behavior for both cautious users and exploratory users.

Decision table

Once I had the rules written down, the behavior became much easier to reason about. In practice, the command reduces to this table:

Input already tracked?	`--auto-track`	`--no-track`	Result
Yes	No	No	Run transform and version output into the existing dataset
Yes	Yes	No	Same result as above — input is already tracked, so `--auto-track` has nothing extra to do
Yes	No	Yes	Run transform only, do not version output
No	No	No	Run transform only, do not version output
No	Yes	No	Add input as a new dataset, then version output into that dataset
No	No	Yes	Run transform only, do not version output
Any	Yes	Yes	Invalid combination, exit with usage error

Why path-based dataset lookup?

I use original dataset paths to infer identity when --dataset-id is not provided. It keeps the command convenient for everyday usage, while still allowing explicit control when needed.

Version increment behavior

For transform-generated versions, I intentionally use fractional increments (+0.1 by default) rather than forcing integer bumps.

Reasoning: many transforms are intermediate processing steps, not "major new source snapshot" events. Keeping the increments smaller prevents version numbers from ballooning unnecessarily and makes the history easier to scan.

Conflict Rules and Flags

Two flags are mutually exclusive by design:

--auto-track
--no-track

If both are provided, command exits with a usage error.

This might look obvious, but explicit validation here matters because these conflicting options can otherwise produce silent, confusing behavior.

I would rather fail fast than invent precedence rules users have to memorize.

Presets: Turning Repetition into Reuse

The obviously annoying part about transform is the length of the command and the repetition: image, command, and tracking flags are very often the same.

That led to transform presets in .data_tracker/presets_config.json.

A preset stores things like:

image
command
auto-track/no-track
force
message

Then you can run:

dt transform --input-data ./raw --output-data ./processed --preset clean-sales

and optionally override any preset field from CLI.

Override hierarchy

The rule is simple and explicit:

CLI value > preset value > default value

That gives reusable defaults while keeping one-off runs easy.

Preset management commands

I added a small CRUD interface:

# list presets
dt preset ls
dt preset ls --detailed
# add/remove presets
dt preset add <name> --image ... --command ... [flags]
dt preset remove <name>

The detailed listing is intentionally human-readable, so you can quickly review what a preset actually does before using it.

Failure Cases I Handled Explicitly

The most important ones:

Docker not installed
tracker not initialized
input path does not exist
transform command succeeds but output directory is empty
preset missing or malformed
tracking/version update fails after successful transform

I also added rollback behavior for one edge case: if --auto-track adds a dataset but the transform fails immediately after, the command removes the auto-added dataset to avoid leaving junk history.

This is not truly transactional, but it keeps state cleaner than a naive implementation.

Example Workflow

# initialize
dt init

# track raw input
dt add ./data/raw.csv --title sales --message "raw export"

# run transform and auto-version output
dt transform \
  --input-data ./data/raw.csv \
  --output-data ./data/cleaned \
  --image python:3.11-slim \
  --command "python /input/clean.py --output /output/clean.csv" \
  --message "normalize + remove nulls"

# inspect what changed
dt history --name sales
dt compare --name sales # auto-compares latest two versions by default

What Is Still Imperfect

transform works well for my scope, but a few things are intentionally out of scope for now:

no remote execution (local Docker only)
no pipeline DAG orchestration (single command execution)
no built-in preset edit command (remove + add is currently enough)

I prefer this to overbuilding features before there are real users.

Lessons Learned from Building It

Command design is mostly policy design. The hard part is not running Docker; it is defining clear, deterministic rules for when to version and how.
Safety checks are worth a few extra lines. Validation around mount paths and conflicting flags prevented multiple confusing runs.
Defaults should be opinionated, not rigid. Good defaults (/input, /output, auto behavior) plus escape hatches (--force, explicit --dataset-id, custom --version) make the tool usable for both normal and advanced scenarios.

Repo

Source code: github.com/martin-iflap/DataTracker

This project is open source under the MIT License. Contributions and feedback are welcome.

If you have ideas for improving transform (or the overall CLI design), feel free to open an issue or submit a pull request. The main goal of this project for me is to learn more about CLI design and data versioning, so I am very open to suggestions.

I Built a Dataset Version Control Tool — and Accidentally Reimplemented Git's Core

Martin — Wed, 04 Mar 2026 19:40:32 +0000

A few months ago I started a side project mostly to get hands-on experience with three things I hadn't used seriously before: Docker, SQLite, and Python CLI tools. The plan was to build something small for educational purposes. But the project turned into DataTracker — a local version control system for data files.

This article is about the architecture, specifically the part that surprised me most once I started designing how to actually store versioned files, I kept arriving at the same solutions git already uses. Not because I copied them, but because they're probably the best answers to the problem.

The Problem

The use case is simple. You have a CSV, a set of images, or any data file really. You run some processing, the file changes. Later you want to know what it looked like before, compare the two versions, etc. You want this without manually copying files into data_v1/, data_v2/, data_final/, data_final_REAL/.

Git solves this for source code. It does not solve it well for binary files or large datasets, and it was never designed to. So the question becomes: what does a minimal version of git look like if you build it specifically for data files?

The Core Idea: Content-Addressed Storage

The most important design decision, and the one that everything else follows from, is how you store files.

The naive approach is to copy the file into some storage directory and name it after the dataset and version: sales-data-v1.csv, sales-data-v2.csv, and so on. This works until two versions of a dataset happen to contain identical data — then you've stored the same bytes twice for no reason.

Git's answer to this is content-addressed storage: don't name files after what they are, name them after what they contain. Specifically, hash the file contents and use the hash as the filename.

DataTracker does exactly this. When you run dt add ./sales.csv, this happens:

# file_utils.py
def hash_file(file_path: str) -> str:
    sha256_hash = hashlib.sha256()
    with open(file_path, "rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            sha256_hash.update(chunk)
    return sha256_hash.hexdigest()

def copy_file_to_objects(tracker_path: str, data_path: str, file_hash: str) -> None:
    save_path = os.path.join(tracker_path, "objects", file_hash)
    shutil.copy2(data_path, save_path)

The file ends up stored as .data_tracker/objects/a3f8c2... — a 64-character hex string. The original filename is recorded separately in the database. The storage layer has no concept of names at all.

Git calls these "blob objects". The structure is identical:

git:          .git/objects/a3/f8c2...   ← raw file contents
DataTracker:  .data_tracker/objects/a3f8c2...  ← raw file contents

The immediate practical benefit is automatic deduplication. If you add two datasets with identical contents, or update a dataset without actually changing the file, the hash is the same, the object file already exists, and no second copy is written. This is handled by a single database check before the copy:

# db_manager.py — INSERT OR IGNORE means a collision is silently a no-op
conn.execute("INSERT OR IGNORE INTO objects (hash, size) VALUES (?, ?)", (file_hash, size))

No other deduplication logic is needed.

One important thing to note is that duplicates are allowed in DataTracker — you can add new versions with the same contents if you want, DataTracker will only provide a warning.

The Database as the Index

Storing files by hash solves the storage problem, but it creates a new one: how do you know which hash belongs to which dataset version? Git uses a combination of tree objects and refs (branches, tags) stored as small files in .git/. I used SQLite, which gives you the same thing with foreign keys and transactions.

The database schema has four tables:

datasets      — one row per tracked dataset (id, name, message, created_at)
    │
    └── versions  — one row per version (dataset_id → datasets, object_hash, version number, original_path)
            │
            └── files     — one row per file in a version (version_id → versions, object_hash → objects, relative_path)
                    │
objects       — one row per unique file stored (hash, size)  ←──────────────┘

The files table is the key one. It's what lets DataTracker reconstruct a full directory version: given a version_id, you get back every file's hash and its relative path within the original directory. That's enough to recreate the original structure anywhere.

datasets and versions are roughly equivalent to git's refs and commits. objects is the object store manifest — it tracks sizes and prevents orphaned files, but the actual content lives in the filesystem.

Why SQLite over a JSON file? Three reasons:

Atomic transactions. When adding a dataset with 50 files, either all 50 succeed or none do. A JSON file gives you no such guarantee.
Foreign keys. They enforce referential integrity — you can't have a version row pointing to a non-existent dataset. SQLite has them, but notably they're off by default. You have to enable them per connection: conn.execute("PRAGMA foreign_keys = ON").
Queries. Finding the latest version of a dataset, checking for hash collisions, listing all datasets — these are one-liners in SQL and would be loops with a JSON file.

Directory Versioning: Where It Diverges from Git

Git's unit of storage is always a single file (a blob). Directory structure is captured separately as tree objects that reference blobs. DataTracker takes a slightly different approach because the use case is different.

When you add a directory, DataTracker stores each file individually in objects/ (same as git), but it also computes a single primary hash for the whole directory. This primary hash is used for one specific purpose: duplicate detection at the version level, so you can warn users if needed.

# file_utils.py
def hash_directory(dir_path: str) -> str:
    sha256_hash = hashlib.sha256()
    for root, dirs, files in os.walk(dir_path):
        dirs.sort()
        files.sort()
        for filename in files:
            filepath = os.path.join(root, filename)
            rel_path = os.path.relpath(filepath, dir_path)
            sha256_hash.update(rel_path.encode('utf-8'))  # include structure
            with open(filepath, 'rb') as f:
                for chunk in iter(lambda: f.read(8192), b""):
                    sha256_hash.update(chunk)
    return sha256_hash.hexdigest()

The directory hash covers both file contents and relative paths — so renaming a file in the directory produces a different hash even if the contents are unchanged.

So the two-hash approach is:

Directory hash → stored in versions.object_hash, used to warn about duplicate versions
Individual file hashes → stored in objects and files, used for actual storage and retrieval

Git doesn't need this distinction because it always works at the file level. DataTracker needs it because the primary user-facing unit is a dataset version, not a file.

A Concrete Walk-Through

Here's what happens end to end when you run dt add ./data/ --title "experiment-1":

1. Walk the directory, collect all file paths
2. For each file:
   a. SHA-256 hash the contents
   b. Copy to .data_tracker/objects/<hash>  (INSERT OR IGNORE — dedup is free)
   c. Record (version_id, hash, relative_path) in the files table
3. SHA-256 hash the entire directory → primary_hash
4. Check if primary_hash already exists in versions → warn if so
5. Insert a row into datasets (or reuse existing for dt update)
6. Insert a row into versions (dataset_id, primary_hash, version number, original_path)
7. conn.commit() — all of the above is one transaction

Step 7 is important. If anything fails between steps 1 and 6, the commit never happens and the database sees none of it. The object files that were already copied to disk are then cleaned up explicitly. This is the same reason git's staging area exists — operations on the object store and operations on the index need to be kept consistent.

What's Missing Compared to git

Pointing out the gaps is more useful than pretending the tool is complete.

No delta storage. Git stores diffs for text files rather than full copies, which is why a git repository with hundreds of commits doesn't grow linearly with the number of commits. DataTracker stores a full copy of every file in every version. For small datasets this is fine. For large ones it becomes expensive quickly.

No garbage collection. Git has git gc. DataTracker cleans up orphaned objects when you remove a dataset or version, but there's no general-purpose GC pass. If something goes wrong mid-operation and leaves orphaned objects, they stay there until you notice the dt storage numbers look wrong.

Linear history only. Git has branching. DataTracker has a single version number per dataset, incrementing linearly. There's no concept of parallel versions or merging.

No remote. Everything is local. There's no push/pull, no sharing between machines.

Some of these are limitations by design — the tool is meant to be simple. Some of them (delta storage in particular) are things I would perhaps like to add in the future, but they require a lot of new logic and complexity, which I didn't want to deal with initially.

The Lesson I Didn't Expect

I started this project thinking Docker and SQL would be the only interesting parts. The Docker integration (the dt transform command, which runs a transformation inside a container and auto-versions the output) did turn out to be very interesting. But the part that surprised me the most was how much of git's core architecture I ended up reimplementing without even trying — and thinking about it turned out to be a great way to understand git itself better.

Content-addressed storage is a 30-year-old idea. It shows up in git, in IPFS, in container image layers, in package managers. The reason it keeps appearing is that it solves a hard problem — identity and deduplication — with almost no code. The hash is the identity. Two files that are identical are automatically the same object. You don't have to write that logic; the data model expresses it.

A lot of developers use git every day without thinking much about how it actually works under the hood. I think building something similar, even something much simpler, is one of the better ways to change that.

Demo

# Install
git clone https://github.com/martin-iflap/DataTracker.git
cd DataTracker
pip install -e .

# Track a dataset
dt init
dt add ./data.csv --title "sales" -m "Raw export"

# Update it
dt update ./data_cleaned.csv --name sales -m "Removed nulls"

# See what changed
dt compare 1.0 2.0 --name sales

# Go back to v1
dt export ./recovered --name sales -v 1.0

Output from dt compare:

Comparison between version 1.0 and version 2.0:
...
Modified files:
  ~ data.csv | Size: 48.20 KB → 45.10 KB = -3.10 KB
    Similarity: 94.30%
    Lines added: 0, Lines removed: 47

The Repo

The full source is at github.com/martin-iflap/DataTracker. The project is still actively developed — the transform/presets system is not finished, there's no GC command, and no status command yet. But I plan to add them in the coming weeks.

If you've built something similar, ran into one of the same problems, or have a strong opinion about delta storage — I'd genuinely like to hear it in the comments.