Forem: Matteo Bortolazzo

Real-Time JSON Parsing from Semantic Kernel Streams in .NET

Matteo Bortolazzo — Thu, 13 Mar 2025 21:42:16 +0000

The problem

The repo has been updated after this article to use NDJSON HTTP stream instead of events.

LLMs like GPT-4o are now
good at generating JSON, which opens up many possibilities.

Most of the time, we can wait for the LLM to complete the generation, parse the answer and return to the UI.

However, given the speed of LLMs, it can be frustrating for users to wait for the generation to be complete.

The best solution would be to display the generated content incrementally as soon as possible. This is relatively easy with
text, but it's a bit more complicated with JSON as we need to ensure the content is valid at each step.
We need to parse the JSON as it's generated, understand its structure, and act accordingly.

We'll build a Life’s to-do list generator. The LLM will generate a list of tasks, and we'll display them as soon as they are generated.

We'll make it more complex with the following schema:

{
  "listName": "Bucket List",
  "items": [
    {
      "recommendedAge": 30,
      "description": "Skydiving"
    },
    {
      "recommendedAge": 50,
      "description": "Visit all seven continents"
    }
  ]
}

There are two ready-to-use tools we can use:

Semantic Kernel: an SDK to interact with AI models.
Utf8JsonReader: a high-performance, low-allocation, forward-only reader for JSON.

It should be straightforward to combine these two tools to achieve our goal. There's even a section in the
documentation: Read from a stream using Utf8JsonReader!

The actual problem

There are multiple challenges:

The reader example uses a MemoryStream while Semantic Kernel uses IAsyncEnumerable<StreamingTextContent>.
Utf8JsonReader is a ref struct, so:
- It doesn't work with streams anyway, only with ReadOnlySpan<byte> in the constructor.
- It can't be passed as a parameter to an async method.
- It can't be used across await or yield boundaries.
- It's a lexer/tokenizer, not a parser, so we need to handle the JSON structure ourselves.

The solution

We need to solve two problems:

How to use Utf8JsonReader with IAsyncEnumerable<StreamingTextContent>.
How to parse the JSON structure incrementally.

Let's start with the latter, as it's simpler.

The parser

The primary method of Utf8JsonReader is Read(). A simple JSON like { "name": "test" } will generate the
following tokens:

StartObject
PropertyName
String
EndObject

Each time we call Read(), the reader moves forward by a token, and we use:

TokenType to know the type.
ValueSpan and other methods are used to get its value
The bool returned to know if more tokens exist to read.

The interface for this is quite simple:

public interface IIncrementalJsonStreamParser
{
    void ContinueParsing(ref Utf8JsonReader reader);
}

Once we have buffered enough data from the response, we try to parse it.

State machine

The easiest way I found to parse the JSON with this setup is a state machine.
We can update the machine's state with each token and act accordingly, for example, by triggering an event.

Here is the state machine for the TODO list:

In this case, we trigger two events:

onListNameParsed when we find a String token while in the ReadingListName state.
onItemParsed when we find an ObjectEnded token while in the ReadingItem state.

Visitor pattern

We can use the visitor pattern to hide the complexity of Uft8JsonReader.

The base abstract base looks something like this:

private T State { get; set; } = initialState;

public void ContinueParsing(ref Utf8JsonReader reader)
{
    while (reader.Read())
    {
        switch (reader.TokenType)
        {
            case JsonTokenType.PropertyName:
                State = VisitProperty(State, reader.GetString()!);
                break;
            case JsonTokenType.String:
                State = VisitStringValue(State, reader.GetString()!);
                break;

            // etc.
        }
    }
}    

protected virtual T VisitProperty(T state, string propertyName) => state;

protected virtual T VisitStringValue(T state, string value) => state;

// etc.

We can then implement only what we need, changing the state and triggering events:

public enum TodoStateMachineState
{
    None,
    ReadingListName
    // etc.
}


public class TodoStateMachineJsonTokenParser(
    Action<string> onListNameParsed) : StateMachineJsonTokenParser<TodoStateMachineState>(TodoStateMachineState.None)
{
    protected override TodoStateMachineState VisitProperty(TodoStateMachineState stateMachineState, string propertyName)
    {
        return propertyName switch
        {
            "listName" => TodoStateMachineState.ReadingListName,
            // etc.
        };
    }

    protected override TodoStateMachineState VisitStringValue(TodoStateMachineState stateMachineState, string value)
    {
        switch (stateMachineState)
        {
            case TodoStateMachineState.ReadingListName:
                onListNameParsed(value); // Trigger event
                return TodoStateMachineState.None;
            // etc.
        }
    }

    // etc.
}

The feeder

Let's now see how we can keep feeding the parser. Below the full implementation:

/// <summary>
/// Provides a way to feed an <see cref="IIncrementalJsonStreamParser"/> with an <see cref="IAsyncEnumerable{T}"/> of <see cref="StreamingTextContent"/>
/// </summary>
/// <param name="incrementalParser">The parser.</param>
/// <param name="chunkBufferSize">The number of chunks to read before feeding the parser.</param>
public class JsonAsyncStreamTokenFeeder(IIncrementalJsonStreamParser incrementalParser, int chunkBufferSize)
{
    /// <summary>
    /// Start feeding the parser with the text content stream
    /// </summary>
    /// <param name="textContentStream">The source.</param>
    public async Task FeedAsync(IAsyncEnumerable<StreamingTextContent> textContentStream)
    {
        // Control the pace of the stream by reading in chunks
        var e = textContentStream.GetAsyncEnumerator();

        var completed = false;
        var buffer = new ArrayBufferWriter<byte>();

        try
        {
            JsonReaderState jsonReaderState = new();
            while (!completed)
            {
                // Load the buffer with the next chunk of text
                for (var i = 0; i < chunkBufferSize; i++)
                {
                    var readSuccess = await e.MoveNextAsync();
                    // Reached the end of the stream
                    if (!readSuccess)
                    {
                        completed = true;
                        break;
                    }

                    if (e.Current.Text == null) continue;
                    var bytes = Encoding.UTF8.GetBytes(e.Current.Text);
                    buffer.Write(bytes);
                }

                // Load the reader with the buffer
                var reader = new Utf8JsonReader(
                    buffer.WrittenSpan,
                    isFinalBlock: false,
                    state: jsonReaderState);

                // Parse as much as possible
                incrementalParser.ContinueParsing(ref reader);

                // Save the parsing state
                jsonReaderState = reader.CurrentState;

                // Create a new buffer with the leftover bytes that were not consumed by the parser
                // This happens when the parser is in the middle of a token
                var remainingBytes = buffer.WrittenSpan[(int)reader.BytesConsumed..];
                buffer = new ArrayBufferWriter<byte>();
                buffer.Write(remainingBytes);
            }
        }
        finally
        {
            await e.DisposeAsync();
        }
    }
}

Explanation:

We manually load a given number of chunks into a buffer.
We create a Utf8JsonReader with the buffer.
- IsFinalBlock is false as we don't know if we have reached the end of the stream.
- We pass the JsonReaderState to keep track of the parsing state.
Call ContinueParsing on the parser. The parser returns once there are no more tokens to read.
We save the state of the reader.
We create a new buffer with the remaining bytes not consumed by the parser.
We start again until we reach the end of the stream.

Usage

Here's an example of the usage of everything we created:

OpenAIPromptExecutionSettings openAiPromptExecutionSettings = new()
{
    ResponseFormat = "json_object",
};
var textContentStream = textGenerationService
    .GetStreamingTextContentsAsync(prompt, openAiPromptExecutionSettings);

var parser = new TodoStateMachineJsonTokenParser(PrintName, PrintItem); // Callbacks
var feeder = new JsonAsyncStreamTokenFeeder(parser, chunkBufferSize: 32);
await feeder.FeedAsync(textContentStream);

FeedAsync waits till the stream is completed; at that point, all events have been fired.

Repository

The repo has been updated after this article to use NDJSON HTTP stream instead of events.

matteobortolazzo/semantic-kernel-json-streaming-parser

I wrote a programming language with ChatGTP. Our jobs are safe.

Matteo Bortolazzo — Sun, 25 Jun 2023 22:38:43 +0000

Intro

In this article, I'll talk about how I used ChatGPT to write a new programming language to compete with Figma called Stylo.

I'll not go deep into the project as I wrote everything in the GitHub repository where you can find the app, the syntax, and the project's future. In this article, I'll focus on what I learned instead.

Context

A few months ago, I read a tweet on how the industry focuses on design-to-code instead of design-as-code. At first, I couldn't understand. That's what we do every day, I thought.

Then I started thinking of a more accessible language, faster to write than HTML and CSS. My brain clicked: "Somethinging like Microsoft's Bicep could work!". I took my notepad and opened VS Code.

That was only one problem. I never wrote a language or a compiler before.

Syntax

Luckily, ChatGPT was released a few days prior, and the internet said you can build new business without effort. We'll see how that went...

Disclosure
I don't have conversation screenshots as OpenAI updated the app and ordered chats by the last message instead of the first one. I deleted every conversation on top, thinking they were the newest ones... Sorry, I suffer if I see more than three tabs opened in a browser.

I started by submitting my ideal language, a mix of Bicep and Angular. I was proud of myself. ChatGPT, whom we'll call Gipi from now on, replied, "Great job, buddy! You are awesome! But here's a completely different language that's much better than yours".

I took my pride aside and refined its language with a few more messages until I was happy.

The syntax changes constantly during development as more cases arise.

Branding

As you know, naming is the most challenging part of programming. And I couldn't live knowing my project folder was called something like "design-language-test". I needed a name.

Gipi helped here too. We went back and forth a few times. I wanted something short, easy to remember, and that made sense.

We ended up with Stylo. Which I later found means pen in French. That is good enough for me. Now I can start coding.

First code

What I knew about compilers was that we needed at least three things:

A lexer: to split the string into building blocks (tokens)
A parser: to create an Abstract Syntax Tree (AST) representing the code
A renderer: to converte the AST to HTML and CSS

I asked Gipi to create a lexer in Typescript, and few seconds I got it. I was amazed by how fast it was.

I then asked it to write the parser and, finally, the renderer. Job done. Ready to get rich.

Not really. Nothing was working. Actually, it was not even compiling.

Minimum Viable Product

Now the gory journey to make anything work starts. I wanted to use as less brain as possible, letting Gipi think on my behalf. I pay for it after all!

I started with the lexer, pasting the code and the error in the chat, applying the fix, running, repeat. This went on for several cycles until I did it myself. Gipi couldn't understand the code, like, "I am sorry, now I get it, here's the solution". But they were just lies.

Fixing the lexer was easy. Fixing, aka building-from-scratch, the parser was a different story. That took a couple of days in my spare time.

The renderer was relatively easy. Once you have the AST, you just convert the nodes to HTML, which is also a tree.

New features

Of course, I was unhappy with some basic features. I wanted the syntax to support anything I needed. I did everything on my own. At this point, the code was so complex that it was not worth trying with Gipi.

However, I still needed a web editor to use easily use the renderer. Here AI helped a lot as a glorified search engine. It told me which code editor to use for React, how to enable syntax highlight and create a pan and zoom frame.

However, when I asked for the actual code, again, nothing was working, and it was stuck in a loop. I ended up doing everything myself again.

component Footer(primaryText = "Ok", secondaryText = "Cancel") {
  block(class="p-4 flex justify-end gap-2 border-t-2") { 
    SecondaryButton(secondaryText)
    PrimaryButton(primaryText)
  }
}

Conclusions

If you read this far, you know what I am going to say. For a few more years, our jobs are safe. Gipi is great for boilerplate code, while Copilot is a faster autocomplete that I greatly appreciate.

However, we still need to put effort into what we do, know how to code and how things work. Most importantly, be ready to maintain what AI wrote for years to come.

Check the GitHub repository too!

Cosmos DB server-side with TypeScript

Matteo Bortolazzo — Sun, 06 Nov 2022 20:15:24 +0000

Server-side programming

An underestimated aspect of Cosmos DB is its server-side components:

Stored procedures
Triggers
User-defined functions (UDFs)

These are all JavaScript functions that run natively in Cosmos DB's query engine (close to Edge 18).

Stored procedures

These are close to the ones we find in RDMSs run in a transactional context with a logical partition. We can read and write documents within the partition. If the operation throws any exception, the engine rolls back the changes.

Triggers

Triggers are functions that we can run before or after specific document operations.

For example, we can:

Add properties to a request before the document is created.
Only allow the update of a document if the request is correct.
Update a second document (within the same partition) after the document is updated.
Rollback document changes if a post-trigger fails some checks.

User-defined functions

UDFs are functions we can use in queries with clauses like SELECT or WHERE. We can take full advantage of a modern JavaScript engine and its APIs.

The JavaScript experience

Most of us are used to types. Types self-document the code, and with the help of IDEs, they make development quicker.

Cosmos DB has no official TypeScript type definitions. So, when I started working on the stored procedures, I quickly felt the need for them. Mainly because there are just a bunch of examples and the official JDocs takes a lot of work to consume.

function createToDoItems(items) {
    var collection = getContext().getCollection();
    var collectionLink = collection.getSelfLink();
    var count = 0;

    if (!items) throw new Error("The array is undefined or null.");

    var numItems = items.length;

    if (numItems == 0) {
        getContext().getResponse().setBody(0);
        return;
    }

    tryCreate(items[count], callback);

    function tryCreate(item, callback) {
        var options = { disableAutomaticIdGeneration: false };

        var isAccepted = collection.createDocument(collectionLink, item, options, callback);

        if (!isAccepted) getContext().getResponse().setBody(count);
    }

    function callback(err, item, options) {
        if (err) throw err;
        count++;
        if (count >= numItems) {
            getContext().getResponse().setBody(count);
        } else {
            tryCreate(items[count], callback);
        }
    }
}

For these reasons, I converted the official JDocs to TypeScript type definitions and published an NPM package, azure-cosmosdb-js-server-types

Use types

Cosmos DB requires functions to be self-contained. We cannot reference external packages and import types from them. We need to import types during builds.

Run npm install azure-cosmosdb-js-server-types, then update the tsconfig.json file to include the new types:

{
  ...
  "compilerOptions": {
    ...
    "types": [ "azure-cosmosdb-js-server-types" ]
  }
}

Now we can use getContext or __ without any import:

interface User {
    name: string;
    age: number;
    addresses: Address[];
}

interface Address {
    city: string;
}

function runQuery() {
    const result = __.chain<User>()
        .filter(doc => doc.age > 30)
        .sortBy(user => user.age)
        .map(user => user.addresses)
        .flatten<Address>()
        .value(null, callback)
    if (!result.isAccepted)
        throw new Error("The call was not accepted");

    function callback(err: Error, items: Address[]) {
        if (err) throw err;

        // or getContext().getResponse().setBody({    
        __.response.setBody({
            result: items
        })
    }
}

Compile it in .NET builds

If we are a .NET developer, depending on the use case, we might want to compile scripts when building APIs or console applications. Luckily, it's pretty easy.

Add a package.json file in the project folder with dependencies to TypeScript and the type definitions:

{
    "scripts": {
        "build": "tsc"
    },
    "dependencies": {
        "azure-cosmosdb-js-server-types": "^1.0.0",
        "typescript": "^4.8.4"
    }
}

We also add a script to compile TypeScript with the tsc command:

Add a tsconfig.json file in the project folder. Here we tell TypeScript how it should compile.

{
    "compileOnSave": true,
    "compilerOptions": {
        "noImplicitAny": false,
        "noEmitOnError": true,
        "removeComments": false,
        "sourceMap": false,
        "target": "es5",
        "types": [ "azure-cosmosdb-js-server-types" ]
    },
    "include": [
        "./Scripts/**/*"
    ]
}

In the include section we specify the folder where the TypeScript files are located.

In the csproj file we need to add a step to install the dependencies:

   <Target Name="NpmInstall" Inputs="package.json" Outputs="node_modules/.install-stamp">
        <Exec Command="npm ci" Condition="'$(RestorePackagesWithLockFile)' == 'true'" />
        <Exec Command="npm install" Condition="'$(RestorePackagesWithLockFile)' != 'true'" />
        <Touch Files="node_modules/.install-stamp" AlwaysCreate="true" />
    </Target>

This code installs packages only if needed.

Finally, we add a step to compile the TypeScript files:

   <Target Name="NpmRunBuild" DependsOnTargets="NpmInstall" BeforeTargets="BeforeBuild">
        <Exec Command="npm run build -- --outDir $(OutDir)Scripts" />
    </Target>

The JavaScript files will be generated in a Scripts folder in the project's output folder, e.g. bin\Debug\net6.0. We use the --outDir option to specify the output folder and $(OutDir) to get the project's output folder.

Final words

You can find a working example on GitHub https://github.com/matteobortolazzo/azure-cosmosdb-js-server-types-example.

I hope this article has been helpful to you. If you have any questions or suggestions, please leave a comment below.

The C# Source Generators revolution

Matteo Bortolazzo — Sat, 14 Aug 2021 00:07:20 +0000

Along with .NET 5, Microsoft released C# Source Generators.

And after creating my first generator, I consider it one of the coolest features .NET developers can use.

But what are Source Generators?

They are C# programs that can analyze code, dynamically generate files and inject them into the build pipeline.

Use cases?!

Generate C# classes from JSON
Extend a POCO class with INotifyPropertyChanged
Generate builders from POCO classes
Generate mapping extension methods to replace AutoMapper

Super basic example

Each generator has 2 methods: one to analyze the existing code and one to generate new code.

[Generator]
public class MyGenerator : ISourceGenerator
{
    public void Initialize(GeneratorInitializationContext context) { }

    public void Execute(GeneratorExecutionContext context) { }
}

To generate a C# class:

var source = @"
namespace Dev.To {{
    public class Post {{
        public string Title {{ get; set; }}
    }}
}}";
context.AddSource("Post.cs", SourceText.From(source, Encoding.UTF8));

It might not seem much; however, think about what you can do with partial classes or extensions methods.

How I tried to use them

If you work with .NET Core you might be familiar with appsettings.json.
We often need to create POCO classes representing the JSON and then register them in Startup to be injected later with IOptions<>.

Why don't we let source generators do the heavy lifting for us?

So, I create a preview NuGet that does exactly that!

It reads appsettings content
For each object property it creates a new C# class
It creates a partial Startup class with a private RegisterOptions method that register the new classes with services.Configure<PropertyName>(Configuration.GetSection(nameof(PropertyName)));
The only thing people need to do is making their Startup partial and call RegisterOptions in ConfigureServices method.

Given this JSON:

{
  "MyOtherOptions": {
    "MyString": "any",
    "MyInt": 1,
    "MyDouble": 1.1,
    "MyBool": true,
    "MyObject": {
      "MyObjectString": "any"
    },
    "MyArray": [ "any" ]
  }
}

It generates these classes:

Generated content:

// MyOtherOptions.cs
public class MyOtherOptions
{
    public string MyString { get; set; }
    public int MyInt { get; set; }
    public double MyDouble { get; set; }
    public bool MyBool { get; set; }
    public MyObject MyObject { get; set; }
    public string[] MyArray { get; set; }
}

// Startup.Generated.cs
public partial class Startup
{
    private void RegisterOptions(IServiceCollection services)
    {
        services.Configure<MyOtherOptions>(Configuration.GetSection(nameof(MyOtherOptions)));
    }
}

Final words

As you can see, there's a lot of potential behind this new feature.

It will not replace Reflection completely but it can drastically improve performance for a lot of libraries!

However, since it's in preview, there are limitations, and things might change, so keep that in mind!

The code for my example can be found here on GitHub.

Useful links

CouchDB and C#. EFCore-like queries!

Matteo Bortolazzo — Tue, 09 Mar 2021 21:29:16 +0000

In this post, we'll see how to query CouchDB from C# using LINQ!

I am going to use my library that after few years, today it reached v3.0.0: CouchDB.NET.

What's CouchDB

Apache CouchDB is a distributed document database and its replication protocol is available from servers to mobile phones and web browsers.

It speaks JSON natively, and that's great when using Javascript, but when using .NET it might be painful because of JSON's dynamic nature.

How CouchDB.NET can help

If you worked with Entity Framework before, this library might look familiar.

The setup

Create your CouchContext with the list of databases you have and configure authentication.

Rebel is of type CouchDocument as it contains things like Id, Rev, Attachments, and others.

public class MyDeathStarContext : CouchContext
{
    public CouchDatabase<Rebel> Rebels { get; set; }

    protected override void OnConfiguring(CouchOptionsBuilder optionsBuilder)
    {
      optionsBuilder
        .UseEndpoint("http://localhost:5984/")
        .EnsureDatabaseExists()
        .UseBasicAuthentication(username: "anakin", password: "empirerule");
    }
}

Use LINQ

The library supports native .NET LINQ, ven the SQL-like syntax.

await using var context = new MyDeathStarContext();
var skywalkers = await context.Rebels
    .Where(r => 
        r.Surname == "Skywalker" && 
        (
            r.Battles.All(b => b.Planet == "Naboo") ||
            r.Battles.Any(b => b.Planet == "Death Star")
        )
    )
    .OrderByDescending(r => r.Name)
    .ThenByDescending(r => r.Age)
    .Take(2)
    .Select(
        r => r.Name,
        r => r.Age
    })
    .ToListAsync();

What is sent to the database

{
  "selector": {
    "$and": [
      {
        "surname": "Skywalker"
      },
      {
        "$or": [
          {
            "battles": {
              "$allMatch": {
                "planet": "Naboo"
              }
            }
          },
          {
            "battles": {
              "$elemMatch": {
                "planet": "Death Star"
              }
            }
          }
        ]
      }
    ]
  },
  "sort": [
    { "name": "desc" },
    { "age": "desc" }
  ],
  "limit": 2,
  "fields": [
    "name",
    "age"
  ]
}

It will automatically optimize the query.

Just to name a few:

Merge multiple Where calls;
Remove useless condition from the filter (like Where(r => true || r.Age == 19));
Convert SingleOrDefault(d => condition) to Where(d => condition).Take(2).SingleOrDefault();
Convert Max(d => d.Property) to OrderByDescending(d => d.Property).Take(1).Select(d => d.Property).Max()

Other features

In the year, more and more features have been added.

For example:

Adding/Updating/Deleting documents;
Adding/Updating/Deleting indexes;
Database spitting;
Views;
Local documents;
Users management;

Dependency Injection

If you work with Asp.NET Core a DI package is available.

It just as easy as register the dependency in the Startup:

services.AddCouchContext<MyDeathStarContext>(builder => builder
    .UseEndpoint("http://localhost:5984")
    .UseBasicAuthentication(username: "admin", password: "admin"));

Then just inject the context:

public class RebelsController : Controller
{
    private readonly MyDeathStarContext _context;

    public RebelsController(MyDeathStarContext context)
    {
        _context = context;
    }
}

Finally

If you work with CouchDB and C# I highly suggest checking the library (even if it might result in more bugs to fix for me).

You can see all that it offers in the README and let me know what you think.