Forem: Paderich

Versioning Your AI Workflow with a Custom Claude Code Marketplace

Paderich — Sat, 21 Mar 2026 08:36:15 +0000

TL;DR: You can host a Git repository that acts as a personal marketplace for Claude Code skills. Any project you work on can point to it, pin a version, and get your curated skills installed automatically. This is how you move from "chatting with an AI" to "engineering a workflow."

Why Build a Skill Marketplace?

In my previous logs, I talked about the frustration of "black boxes" and the realization that I was only "scratching the surface" with basic setups. If you use Claude Code regularly, you’ve probably noticed a pattern: you end up reexplaining the same architectural standards, user story formats, or code review checklists for every new project.

A personal marketplace treats your AI prompts like managed infrastructure:

Portable: Your skills live in one standalone Git repo. Every project you work on can reference it.
Versioned: You pin to a Git tag (e.g., v1.0.2). If you update a skill, it won't break your older projects until you're ready to upgrade.
Shareable: Push the repo to GitHub and your entire team gets the same "Senior Engineer" level skills.
Zero Copy-Paste: Claude Code handles the fetching and installation. No manual file management required.

Architecture at a Glance

The setup is straightforward. You have the Marketplace Repo (the source of truth) and your Client Repos (the projects where you actually work).

Step 1: Create the Marketplace Structure

Claude Code expects a specific folder hierarchy to discover plugins. Your marketplace repository should look like this:

claude-skills/
├── .claude-plugin/
│   └── marketplace.json       # The "App Store" index
├── plugins/
│   └── user-story-skill/      # A folder for each specific tool
│       ├── .claude-plugin/
│       │   └── plugin.json    # Metadata for this specific plugin
│       ├── commands/
│       │   └── user-story.md  # Defines the /user-story slash command
│       └── skills/
│           └── user-story/
│               └── SKILL.md   # The "Brain" (The actual logic)

Step 2: Define the Manifests

The Marketplace Index

At the root, .claude-plugin/marketplace.json tells Claude what plugins are available in this repo:

{
  "name": "pat-skills",
  "owner": { "name": "Pat" },
  "metadata": {
    "description": "Personal Claude Code skill library",
    "version": "1.0.0"
  },
  "plugins": [
    {
      "name": "user-story-skill",
      "source": "./plugins/user-story-skill",
      "description": "Generate agile stories from plain text",
      "version": "1.0.0"
    }
  ]
}

The Plugin Metadata

Inside plugins/user-story-skill/.claude-plugin/plugin.json, define the individual plugin:

{
  "name": "user-story-skill",
  "version": "1.0.0",
  "description": "Reference skill for agile user stories",
  "author": { "name": "Pat" }
}

Step 3: Writing the "Brain" (SKILL.md)

This is the most critical file. As I noted in Log 003, vague instructions lead to hallucinations. A professional skill file needs a strict contract, including few-shot examples.

Location: plugins/user-story-skill/skills/user-story/SKILL.md

# Purpose
Generate a well-formed agile user story from a plain-language feature request.

## Context
This skill follows standard agile coaching patterns. Stories must be independent, 
negotiable, and small enough to fit in a single sprint.

## Instructions
1. Identify the role from the input. Infer the most likely role if not stated.
2. Format the story: **As a [role], I want [goal], so that [benefit].**
3. Generate 3–5 acceptance criteria using the **Given / When / Then** pattern.
4. If the input is ambiguous, use a `[CLARIFY: ...]` marker rather than guessing.
5. Output must be concise: no preamble, just the story and criteria.

## Examples (Few-Shot)

### Example 1
**Input:** Users should be able to reset their password via email.
**Output:** 
As a registered user, I want to reset my password via email, so that I can 
regain access to my account if I forget my credentials.

Acceptance criteria:
- Given I am on the login page, when I click "Forgot password", then I am 
  prompted to enter my email address.
- Given I submit a valid email, when the system processes it, then I receive 
  a reset link within 2 minutes.

## Quality Criteria
- The role is specific (not "the system" or "a person").
- The benefit explains *why* the user wants this, not just what they want.
- Every acceptance criterion is testable by a QA engineer.
- Ambiguous inputs are flagged with `[CLARIFY]`.

Step 4: Connecting a Project

Once you’ve pushed your marketplace to GitHub (or a local Git path), head to your project directory and start Claude Code:

Type /plugin.
Select Add marketplace.
Enter your Git URL (e.g., https://github.com/your-username/claude-skills.git).

Claude handles the rest, updating your .claude/settings.json. Commit that file so your whole team shares the same skill set.

Pro-tip: In your settings.json, always ensure the ref field points to a specific tag (like v1.0.0) rather than main. This is how you ensure an update to your library doesn't silently change the behavior of an older project mid-flight.

The Reality Check: Testing Skills

As I learned with my "RAG in a box", the magic is in the plumbing. Testing a skill isn't about string matching; it's about behavioral validation.

You should build a tests/ folder in your plugin with JSON files defining your expectations. If a skill doesn't catch a "vague" input or misses a quality criterion, your SKILL.md instructions aren't sharp enough. Fix the instructions, don't just "hope" the AI gets it right next time.

Wrapping Up

Building a marketplace turns your accumulated prompting knowledge into a reusable, versioned, and shareable library. It moves you from being a "prompt user" to an AI Systems Engineer. It takes about 20 minutes to set up, but the compounding value across every new project is massive.

Cheers,
Pat

Log Entry 004 - Certified (The Hard Way)

Paderich — Mon, 02 Mar 2026 12:13:40 +0000

The original plan was February 26th. Today is March 2nd, and I just closed the exam window with a "Pass" on the screen. I had to take an extra week because I realized I had underestimated the fundamentals.
I thought I could breeze through them, but the deeper I dug, the more

I realized that "knowing of" a service is not the same as understanding its architectural purpose.

The Reality Check on Study Material

In my first log, I mentioned using Whizlabs. To be honest: I wasn't happy with it this time. The questions felt slightly off-target for the AI-900.

I had to pivot my strategy:

Microsoft Practice Assessment: This was the real game-changer. It was much more aligned with the actual exam logic and helped me identify the gaps I had previously overlooked.
NotebookLM: I used this heavily to "interrogate" specific sections of the documentation. It’s one thing to read a paragraph; it’s another to have a focused tool help you deconstruct complex concepts through targeted Q&A.

The Bigger Picture: Breadth vs. Depth

The AI-102 remains my North Star because it aligns my private interests with my current professional environment. However, passing the AI-900 made me realize that I want to be more than just an "Azure implementer". I want to be a well-rounded AI Engineer. To achieve that, I’m considering broadening my horizon by working through DataCamp’s AI Fundamentals. While Microsoft gives me the tools, I want to ensure my foundation is platform-agnostic and theoretically sound.

What’s next?

The "Fundamentals" badge is on the digital shelf. For now, the "RAG-in-a-box" project is going on the back burner. I have some pressing job-related tasks that require my full attention, which will take some time. Once the dust settles, I’ll dive into the DataCamp curriculum. From there, I’ll reassess my path and decide on the next concrete steps toward becoming an AI Engineer.

Cheers,
Pat

Log Entry 003 - My Lag in RAG-in-a-box

Paderich — Sun, 22 Feb 2026 14:03:59 +0000

In my recent blog post, I was pretty happy about setting up a simple RAG system. Well, it turns out it wasn't that big of a deal. After reading more on the topic, I realized my approach was just scratching the surface. That is why I added a quick P.S. to the original post with a reality check.

Reflecting on my "RAG in a box", I realized the following issues:

My AI factory pattern is essentially a hardcoded if/else block, lacking dynamic registration.
My Docker configuration is perfectly functional but very basic; there is nothing highly app-specific about it.
My approach to chunking documents relied purely on LlamaIndex's built-in functionality, which I now know is a black box that can lead to retrieval failures and hallucinations when data gets messy.
My response payload is blind: The API currently just spits out a text string. Because I am not returning any source nodes, citations, or similarity scores, I have no way to prove if the system actually retrieved the answer from my PDF, or if the LLM is just confidently hallucinating.
In general, I orchestrated the tools well, but I really only hit the tip of the iceberg.
My Python skills could also use some polishing.

What is next? I think I will stay with this project for now so I can do it right. I am not quite there yet on a theoretical level, and I need to play around with the different parts of the problem scope. However, for now, my priority is getting ready for my AI-900 exam this coming Thursday. I am using NotebookLM heavily to study properly. I added some videos, documentation, and the MS Learn path into the system, and I am getting a lot of fantastic learning tools out of it.

Well... that’s my update. Not much, but something honest to share.

Cheers
Pat

Log Entry 002 - RAG in a box

Paderich — Mon, 16 Feb 2026 18:13:56 +0000

Today I decided to refresh my Python skills. Nothing fancy, just the basics that carried me through the past years. I installed VS Code, added some helpful extensions, set up a venv and wrote a few test lines. Simple steps, but it felt good to get back into it.

My main goal was to build a small RAG application. I wanted a simple retrieval augmented generation setup that I could take with me and run anywhere, as long as Docker is installed. That is why I started calling it Rag in a Box.

Before writing any code, I watched some YouTube videos to get an idea of where things stand today. It was a bit underwhelming. Most videos focused on clicking things together with n8n or Zapier. Nice tools, but not what I wanted right now. I wanted to get my hands dirty :D
Spoiler, nothing got dirty here. My expectations were probably too high.

So I approached the problem like I would approach a new feature at work. I broke it down into smaller parts. Things that, together, would make this tiny project work.

Here is what I noted down:

must be portable, so a Docker container is needed
must be exchangeable, so instead of a container I could plug in an Azure service
must read files from a directory and load them into a vector database
must ingest this data properly
must retrieve data properly
must work fully locally, for example with LM Studio but also offer the option to switch to another LLM provider if needed

With that list in mind, I looked around for tools that could make this doable. I found several posts and examples using qdrant. It is open source, runs locally and seems perfect for a project like this. It also pulls an LLM model at startup, which is fine for now.

To orchestrate the entire workflow, I picked LlamaIndex. I wanted as much flexibility as possible and LlamaIndex makes it easy to connect my data to any LLM backend. But the real heavy lifting happens before that: to turn my documents into searchable numbers (embeddings), I am running a local Hugging Face model. This brings torch and transformers into the picture, allowing me to process data privately without sending it to an external API.

To ensure the "exchangeability" I listed in my requirements, I implemented a simple Factory Pattern. I didn't want to hardcode the connection to LM Studio. If I want to switch to GPT-4 or Azure tomorrow, I want to do it by changing a single environment variable, not by rewriting code.

Since LM Studio mimics the OpenAI API, I can actually use the standard OpenAI driver, just pointing to my local machine instead of their servers:

def get_llm(provider: str):
    """Factory to switch between Local, OpenAI, or Azure drivers"""
    if provider == "local":
        # LM Studio mimics the OpenAI API, so we just point the base_url locally.
        # No real API key is needed, but the library expects a string.
        return OpenAI(
            api_base="http://host.docker.internal:1234/v1",
            api_key="not-needed",
            model="local-model" 
        )
    elif provider == "openai":
        return OpenAI(model="gpt-4-turbo")
    elif provider == "azure":
        return AzureOpenAI(
            engine="my-deployment", 
            api_version="2023-05-15"
        )
    else:
        raise ValueError(f"Unknown provider: {provider}")

Next steps were straightforward. I wrote a docker compose file and defined the containers. One for the database, one for ingestion and one for retrieval via API. The rest was mostly following the documentation of each module. Preparation took me around two to three hours. The actual implementation maybe thirty to forty minutes. It is not an enterprise grade RAG system, but it worked.

I added a PDF file to the designated folder, ran:

docker compose up --build

and then used curl to ask my system about the document.

curl -X POST "http://localhost:8000/query" \
     -H "Content-Type: application/json" \
     -d '{"query": "gibe me 5 key facts of the document. shouldnt ne longer than a normal sentence"}'

And it answered:

{
  "response": "1. The document discusses an algorithm for generating test cases from sequence diagrams.\n2. It presents a method to transform sequence diagrams into tree representations for analysis.\n3. The system was evaluated using a login feature's sequence diagram and test cases.\n4. The results showed that the generated test cases matched the sequence of messages in the diagram.\n5. Sequence Dependency Tables (SDT) are used to analyze message dependencies in sequence diagrams."
}

That moment made me happy.

But then I thought, this cannot be everything that AI engineering is about.

So I started reading about chunking and clustering. It is impressive how the tools split the work: LlamaIndex handles the chunking automatically, while Qdrant takes care of the vector indexing. All this logic is hidden from me. I am just a handyman with a set of tools. I know how to replace a light bulb, but I do not need to understand how energy travels through the entire grid to make it shine. I know where the fuses are and that is enough for now.

This was just the first tiny dip into the water with a very tiny toe.
But it is a start, and I like where this is going.

Cheers
Pat

P.S.

The Reality Check

After the initial high of getting a response, I’ve quickly realized that building the "box" was just the tip of the iceberg. It’s one thing to get a clean summary from a simple PDF, but it’s another thing entirely to ensure the system doesn't confidently hallucinate when the data gets messy. I’ve learned that the "magic" is actually in the plumbing, specifically how you chunk the data and how you verify the source of the answer.

By letting LlamaIndex handle everything "automatically," I’ve essentially handed over the keys to a black box. If the chunking slices a paragraph in the wrong place, the retrieval fails; if the retrieval fails, the LLM just starts guessing. Moving forward, my focus has to shift from just "orchestrating" tools to actually validating the data pipeline. The infrastructure is solid, but the real engineering challenge is turning "it looks right" into "I can prove this is right."

Log Entry 001 - My first steps toward AI‑102

Paderich — Mon, 16 Feb 2026 16:05:32 +0000

I like the idea of treating this series as a personal log. A bit nerdy, a bit structured, something that marks progress. So here it is: Log Entry 001.

The year is still young, and so is my path toward becoming an AI engineer. Time to take the first real step.

Recently I decided to go for the AI‑900 certification. It is the most basic AI certification on Azure, but that is exactly why it feels right. I want to build a solid foundation instead of jumping straight into the deep end. Understanding the principles behind AI services, machine learning basics and how Microsoft structures its ecosystem will help me later when things get more complex.

Learning for AI‑900 also forces me to zoom out. I get to revisit ideas that I have seen many times over the years, but with a fresh perspective. Things like supervised learning, responsible AI, cognitive services, classification and prediction. Nothing too complex, but enough to get my brain into AI mode again.

My plan for the next days is simple. Read, try things out and write down what I discover. As with everything in tech, theory alone does not stick. So I want to get hands‑on as early as possible. Azure has plenty of small playgrounds for this exam, and I want to use them.

In addition to the official Microsoft learning path, I bought a book called Machine Learning and AI by Sebastian Raschka. This book is completely out of my comfort zone, but I like it. It opens up things I never thought about. Sure, I learned linear algebra back in my math classes, but for the past 15 years I didn’t even think about it. I am at least happy that I remembered parts of it, which reminds me that you don’t just learn useless stuff when studying.

So yes, this book will keep me busy for a while. It is packed with information, way too dense to read in one go.

To prepare myself better for the exam I bought an exam prep course on Whizlabs, just to study with as much focus as possible. This exam prep consists of a lot of mock questions that are pretty close to the actual exam. I used a similar prep for my AZ‑900, and it helped a lot. The built‑in Microsoft assessment is just a bunch of checkboxes, but the real exam had drag and drop, sentence completion and many other formats.

Anyway, I just scheduled the exam: 26 February.

Yes, right, next Thursday...

So ... this is Log Entry 001.

This log will follow my progress. Not perfect, not structured for a textbook, but honest and curious. If I learn something useful, I will bring it here. If I hit a wall, I will write about that too.

Cheers
Pat

Starting My Journey Into AI

Paderich — Mon, 16 Feb 2026 15:34:57 +0000

I’ve been working in the industry for quite some time. I worked as a sys admin, test engineer, software developer and later as an agile coach and product owner. I’ve seen a lot and I’ve learned a lot. During all these years, something kept growing in the background and slowly turned into something big: AI Engineering.

I want to use this space as a learning log where I share my thoughts and ideas around this domain. I want to become an AI Engineer, not today, not tomorrow, but maybe the day after. I’m eager to learn new things, just like I have for the past 15 years.

So, what’s next?

I don’t want to pressure myself with a lot of structure right from the beginning. I just want to use this blog as a companion. Right now I’m experimenting with new technologies, probably connected to AI engineering, obviously.

I also want to strengthen my path by getting the AI‑102 certificate from Microsoft.

Why AI‑102, you ask?

Because it’s the common tech stack at the place where I’m working.
So yes, this is the plan.

Let’s see how far I’ll come.

This is my starting point.

Cheers
Pat