Forem: Christian Zink

How to Integrate Endoflife.Date in Dependency-Track EoL

Christian Zink — Sun, 29 Mar 2026 10:19:48 +0000

Keeping your software up-to-date is crucial — but what happens when a library reaches end-of-life (EoL)? It stops receiving security updates, leaving your applications exposed to hidden risks.

OWASP Dependency-Track is great for scanning SBOMs (Software Bill of Materials) for vulnerabilities/CVEs. But EoL dependencies, but EoL software may have unpatched vulnerabilities that aren’t reported — creating hidden risks.

In this guide, I’ll show you how to set up my experimental integration for Dependency-Track and start detecting EoL dependencies from endoflife.date in your projects.

Steps of this tutorial

Dependency-Track installation (You can skip this if you already have a running installation)
Import SBOM (You can skip this if you already have a running installation)
Get the Dependency-Track API key from the Web UI
Install and run the integration

Step 1: Install Dependency-Track

If you already have a running Dependency-Track installation, skip this step. Otherwise, the easiest way is via Docker:

# Download Docker Compose file
curl -LO https://raw.githubusercontent.com/DependencyTrack/dependency-track/main/docker-compose.yml

# Start Dependency-Track stack
docker compose up -d

Once started, navigate to http://localhost:8080 to access the web UI.

Step 2: Import Your SBOM

You need at least one SBOM loaded in Dependency-Track to analyze dependencies. In the web UI:

Go to Projects → Add Project
Upload your SBOM (Or use the example sbom)
Wait for the components to be processed

If you already have a project in Dependency-Track, you can skip this step.

Step 3: Get Your API Key

The EoL integration uses the Dependency-Track API. To get your key:

Log in to the Web UI at http://localhost:8080 (That's the default for Dependency-Track Installations)
Click on your Administration → Access Management → Teams
Generate a new key
Copy it for later use

Step 4: Install and Run the Integration

You can choose Linux or Windows depending on your environment.

Linux

# Download the latest binary
curl -LO https://github.com/Chriz76/endoflife-dependencytrack/releases/download/v0.1.0-alpha/eol-dt-linux-x64.tar.gz

# Extract the archive
tar -xzvf eol-dt-linux-x64.tar.gz

# Make it executable
chmod +x eol-dt

# Run it
./eol-dt --apikey YOUR_DEPENDENCY_TRACK_API_KEY

Windows

Download the latest Windows binary
Unzip it
Run it with your api key

eol-dt --apikey YOUR_DEPENDENCY_TRACK_API_KEY

Step 5: Review Results

In the Dependency-Track UI, search for "INT" to see flagged EoL components
Check the program output for details about matched components
Optionally, provide your own EoL dataset using the --eoldata option

Notes & Tips

This project is experimental — use in test environments first
Matching relies on PURL, CPE, and name heuristics — some results may be incomplete
Future improvements: integrate more package repositories, combine EoL with CVE data, allow manual overrides
For further options or feedback see the endoflife-dependencytrack project on github

Feedback & Contributions

I’d love your feedback:

Does this help detect hidden vulnerabilities?
Ideas to improve matching accuracy?
Found bugs or missing components?

How to use Redis and Lua Scripts in a C# ASP.NET Core Microservice Architecture

Christian Zink — Thu, 22 Jul 2021 20:23:33 +0000

Use StackExchange.Redis and run Redis in Docker to Cache Aggregated Database Data for a Scaled C# Application

Redis has many great use-cases like session or full page caching, queues, pub/sub and leaderboards/counting, etc. Usable in your applications and microservice architectures.

In this article, I show you how to use StackExchange.Redis in ASP.NET Core to access a Redis server running in Docker.

The example application lets users write posts in categories. It uses Redis to cache aggregated data for top categories and users while the database stores the item/line data as “source of truth”.

This article starts with a simple first use-case and then advances to a more complex one with Lua scripting and the inbox pattern.

Setup Redis in Docker and Prepare the .NET Core Project
Implement the Top Categories
Top Users, the Inbox Pattern, and Lua Scripting for Atomicity
Final Thoughts and Outlook

1. Setup Redis in Docker and Prepare the .NET Core Project

Install Docker Desktop

Create the Redis container:

C:\dev>docker run --name redis -d redis

Use Visual Studio Community (it’s free) with the ASP.NET and web development workload. Open an ASP.NET Core 5 Web API project with EntityFramework.

Install the following NuGet packages:

Microsoft.EntityFrameworkCore.Tools
StackExchange.Redis

Create the following entities:

2. Implement the Top Categories

Add Top Categories

Increment the count for a category when inserting a new post.

Line 1: Connect a Redis multiplexer
Lines 7–11: Insert the post and update the category count in a transaction
Lines 13–15: Read the latest count for the category and add/update the entry in the Redis sorted set “CategoriesByPostCount”

The ZADD command has an optional GT parameter: The value is only updated if it is greater than the current value. This prevents race conditions where another thread also increased and updated the count.

Read Top Categories

SortedSetRangeByRankWithScore is a wrapper for the Redis ZRANGE command. It reads the category and the count. Use the start and end parameters for paging.

3. Top Users, the Inbox Pattern, and Lua Scripting for Atomicity

For the users, I show you how to count the posts in Redis instead of the DB. This is e.g. needed in a sharded environment where the user’s posts are scattered over all shards.

Atomically Add Posts with Lua Scripting

The PublishOutstandingPosts method sends new posts to Redis. It uses the inbox pattern and a Lua script to make the operation “idempotent”.

Further reading: Outbox, Inbox patterns and delivery guarantees explained

Line 3–8: Prepare the Lua script
Line 13: Read the last sent post ID from Redis
Line 15: Load the unsent posts from the DB
Line 17–21: Execute the Lua script to add the post to the user
Line 22: Set the post count for the user in UsersByPostCount
Line 25: Update the last sent ID after all operations are finished

The Lua script tries to add the post ID and timestamp to the PostsByTimestamp sorted set of the user.

If it can add the key, it also increments the PostCount of the user. The script makes the ZADD and ZINCRBY commands atomic. If the post ID already exists, it returns the existing PostCount.

An extra counter per user is needed so that all key parameters map to the same Redis hash tag. And therefore would be placed on the same Redis shard. The curly braces like in the key “{User:5}:PostsByTimestamp” are signifiers for a Redis hash tag.

Read the Top Users

Use SortedSetRangeByRankWithScore to read the IDs of the top users. Then read the names:

4. Final Thoughts and Outlook

You used StackExchange.Redis to access Redis from C#. See my previous article if you want to manually access Redis via redis-cli or want more information about the example data model and use-cases.

The code in this article focuses on the Redis-related part. The basis for the code can be found in the solution from my article about database sharding if you want to build and run the service.

There are many other use-cases for you to use Redis in your applications!

Please contact me if you have any questions, ideas, or suggestions.

How to Cache Aggregated Data with Redis and Lua Scripts for a Scaled Microservice Architecture

Christian Zink — Wed, 30 Jun 2021 20:22:54 +0000

A scaled microservice-based application with a huge amount of growing data has a challenge to effectively deliver aggregated data like top lists.

In this article, I show you how to use Redis to cache the aggregated data. While the databases store the item/line data as “source of truth” and use sharding to scale.

A single Redis instance can handle some 100,000 operations per second

My example data model with users, posts, and categories can be a basis for your own usecases.

Example Use-Cases and Data Model
Setup Redis and Implement the Top Categories
Top Users, Latest User Posts, and the Inbox Pattern
Lua Scripting for Atomicity
Final Thoughts and Outlook

1. Example Use-Cases and Data Model

In the example microservice application users can write posts in categories. They can also read the posts by category including the author name. The newest posts are on top. The categories are fixed and change seldom.

See my previous post “How to use Database Sharding and Scale an ASP.NET Core Microservice Architecture” if you are interested in source code and more details about the example application.

Logical Data Model:

Currently, one million users exist. Every day each user writes about ten posts.

Top 10 Categories

The top 10 categories will be displayed on the main page. This would require a statement like this for MySql:

SELECT CategoryId, COUNT(PostId) FROM Post GROUP BY CategoryId ORDER BY COUNT(PostId) LIMIT 10;

Executing this statement for millions of lines would be very slow. And on every page visit, it would be impossible.

Because of the large amount of data, I also decided to shard by category. So it would require merging the top lists from multiple databases:

2. Setup Redis and Implement the Top Categories

Install Docker Desktop

Create the Redis container:

C:\dev>docker run --name redis -d redis

Connect to the container and start the redis-cli:

C:\dev>docker exec -it redis redis-cli

Add Top Categories

The the top categories (“CategoriesByPostCount”) use a Redis sorted set (ZSET).

Add the first entry with ZADD and 99 posts for the category “Category5”:

127.0.0.1:6379> ZADD CategoriesByPostCount GT 99 "Category5"

It adds one entry:

(integer) 1

Add some more entries:

> ZADD CategoriesByPostCount GT 1 "Category1"

(integer) 1

> ZADD CategoriesByPostCount GT 10 "Category2"

(integer) 1

Update Category5:

> ZADD CategoriesByPostCount GT 100 "Category5"

(integer) 1

> ZADD CategoriesByPostCount GT 98 "Category5"

(integer) 0

The last command gives a result of zero. This happens because of the GT parameter. The parameter helps to handle situations where updates arrive out-of-order (post counts don’t decrease).

Read Top Categories

Use ZRANGE and read the top 10 categories with count of posts:

> ZRANGE CategoriesByPostCount 0 9 WITHSCORES REV

1) "Category5"
2) "100"
3) "Category2"
4) "10"
5) "Category1"
6) "1"

Easily retrieve the second page (entries 11–20), etc:

ZRANGE CategoriesByPostCount 10 19 WITHSCORES REV

Prerequisites

The posts per category can be counted in SQL when a new post is created:

BEGIN TRANSACTION
INSERT INTO Post (...)
UPDATE Categories SET PostCount = PostCount + 1
COMMIT TRANSACTION

This is possible because the database is sharded by category. All posts of one category are in the same database.

3. Top Users, Latest User Posts, and the Inbox Pattern

The user’s posts are scattered over all shards. It is not possible to use UPDATE User SET PostCount = PostCount + 1 and then update Redis.

The operation in Redis has to be “idempotent”. The inbox pattern makes this possible.

Further reading: Outbox, Inbox patterns and delivery guarantees explained

Add Posts (with a race condition)

On every new post add add an entry to the *PostsByTimestamp *sorted set of the user:

> ZADD {User:5}:PostsByTimestamp 3455667878 '{Title: "MyPostTitle", Category: "Category5", PostId: 13}'

(integer) 1

Then increment the post count in UsersByPostCount:

> ZINCRBY UsersByPostCount 1 "5"

To make it idempotent check the result of adding the post to the inbox. Issuing the command again gives a result of zero (the entry already existed):

> ZADD {User:5}:PostsByTimestamp 3455667878 '{Title: "MyPostTitle", Category: "Category5", PostId: 13}'

(integer) 0

Then don’t increment UsersByPostCount.

The command ZADD to PostsByTimestamp and the command ZINCRBY to UsersByPostCount have to be atomic. I will show you how to use a Redis Lua-Script to make it atomic. But first, let’s read the top users and latest user posts.

Read the Top Users and the Latest User Posts

Top 10 users:

> ZRANGE UsersByPostCount 0 9 WITHSCORES REV

1) "6"
2) "10"
3) "5"
4) "8"
5) "3"
6) "4"
7) "1"
8) "3"

The user with ID 6 has 10 posts, ID 5 has 8 posts, etc.

Top posts of the user with ID 5:

> ZRANGE {User:5}:PostsByTimestamp 0 9 WITHSCORES REV

1) "{Title: \"MyPostTitle2\", Category: \"Category1\", PostId: 14}"
2) "3455667999"
3) "{Title: \"MyPostTitle\", Category: \"Category5\", PostId: 13}"
4) "3455667878"

4. Lua Scripting for Atomicity

Atomically Add Posts with Lua Scripting

A Redis Lua script can make the command ZADD to PostsByTimestamp and the command ZINCRBY to UsersByPostCount atomic. But an extra counter per user is needed so that all key parameters map to the same Redis hash tag.

The curly braces like in the key “{User:5}:PostsByTimestamp” are signifiers for a Redis hash tag.

This Lua script tries to add a key to a sorted set. If it can add the key, it also increments a counter. If the key already exists, it returns the value of the key:

Use EVAL to call the Lua script and pass “{User:8}:PostsByTimestamp” and “{User:8}:PostCount” as keys (one line on the command line):

> EVAL "if tonumber(redis.call('ZADD', KEYS[1], ARGV[1], ARGV[2])) == 1 then return redis.call('INCR', KEYS[2]) else return redis.call('GET', KEYS[2]) end" 2 {User:8}:PostsByTimestamp {User:8}:PostCount 3455667999 "{Title: \"MyPostTitle2\", Category: \"Category1\", PostId: 14}"

(integer) 1

Then set the count for user 8 in UsersByPostCount:

ZADD UsersByPostCount GT 1 "8"

Store the Script in Redis

For performance reason you can store the sript in Redis:

> SCRIPT LOAD "if tonumber(redis.call('ZADD', KEYS[1], ARGV[1], ARGV[2])) == 1 then return redis.call('INCR', KEYS[2]) else return redis.call('GET', KEYS[2]) end"

"cd9222afab5eb8d579942016a8c22427eff99429"

Use the hash to call the script:

> EVALSHA "cd9222afab5eb8d579942016a8c22427eff99429" 2 {User:8}:PostsByTimestamp {User:8}:PostCount 4455667999 "{Title: \"MyPostTitle3\", Category: \"Category1\", PostId: 20}"

(integer) 2

5. Final Thoughts and Outlook

In this article, you set up Redis and started with a simple use-case to cache aggregated data. Then you used the inbox pattern and Lua scripting for atomicity.

In one of my next articles, I will show you how to implement it in a C# ASP.NET Core microservice application.

Redis offers much more than I showed you in this article. You can explore the other commands and use-cases how they solve problems in your application. In a real-life application, you might have to use TTL to automatically expire entries so that the cache does not grow unlimited. Maybe you also need to scale Redis.

Please contact me if you have any questions, ideas, or suggestions.

How to Scale an ASP.NET Core Microservice and Sharded Database. Load Test with JMeter

Christian Zink — Sun, 20 Jun 2021 20:31:17 +0000

Run multiple DBMS and C# Containers Behind an HAProxy Load Balancer with Docker Compose. Test Scaling with Different Number of Instances

In the previous article, you created a microservice architecture and manually implemented application-layer database sharding.

Now, you will scale the application and run multiple container instances of the microservice and databases. You will use Docker Compose and an HAProxy load balancer:

Then you run JMeter load tests to see how the application scales when using a different number of instances. Finally, you will also publish and receive messages from RabbitMQ.

1. Run Multiple Database and Microservice Instances

Dockerize the Microservice

Use the code and environment from the previous article as the basis.

Rename the file “Dockerfile” in the Visual Studio Project-Explorer to “dockerfile” (first character lowercase). Then right-click the dockerfile and select “Create Docker Image”. This will also push the image to docker.

Run the Application in Docker

Create the file docker-compose.yml:

The docker file configures 3 database containers and 4 instances of the Post service. Currently, only 2 databases are used by the Post service instances. You can later remove the comment to use the third database. An HAProxy load balancer exposes the Post service containers on port 5001.

There is a 0.5 CPU limit for each container to help with a realistic load test on a local machine. On my 12 core notebook, there are still unused resources so that adding more service and database instances can bring a benefit.

Start the Application

C:\dev>docker-compose up -d

Init the Databases

Open your browser at http://localhost:5001/swagger/index.html

Init the database with at least 100 users and 10 categories.

You could create more users and categories, but it will take some time because of the CPU limits.

2. Load Test the Scaled Application with JMeter

Create the JMeter Test Plan

Install and open JMeter.

Create a test plan and a thread group:

32 threads is a good number to start. On each loop of a thread, it adds one post and reads 10 posts.

Add an HTTP request to create a post:

Servername: localhost
Port: 5001
HTTP-Request: POST
Path: /api/Posts
Body Data:

    {
     "title": "MyTitle",
     "content": "MyContent",
     "userId": ${__Random(1,100)},
     "categoryId": "Category${__Random(1,10)}"
    }

It creates a post for a random user (ID 1–100) and category (1–10).

Add a Content-Type application/json HTTP header to the request:

Read 10 posts of a random category:

Servername: localhost
Port: 5001
HTTP-Request: GET
Path: /api/Posts
Send Parameters with the Request:

    NAME | VALUE | CONTENT-TYPE
    category | Category${__Random(1,10)} | text/plain
    count | 10 | text/plain

Run the Test

Take a look at the Summary Report while the test is running:

There should be no errors.

Wait some time until the values for the average (response time) and throughput get stable.

Modify the Test Parameters

Stop the test in JMeter.

You can change the threads in the test plan. Rise them to e.g. 64 or 128 threads. Or decrease the threads to 16 or even 1.

Shutdown the application before you edit the docker-compose.yml:

C:\dev>docker-compose down

You can change the number of post service instances via the “scale” property. Change the “environment” property for the number of databases (add/remove the comments):

Start the application after your changes:

C:\dev>docker-compose up -d

It takes some time until the database servers are running. And remember to initialize the database:

3. Example Test Results

A ratio of two post services to one database gives good results on my computer. I can scale it up to six services and three databases until I reach the limits of my hardware. The average time stays below 500ms. Increasing the threads higher than 64 produces errors.

The results depend on my environment and the CPU limitations. They will be different on your machine.

The throughput per second is proportional to the number of instances:

305 requests per second with the CPU limited containers are about 25 million requests per day. That would allow 1 million users to write 10 posts a day and read posts. Of course, a real-life application will be more complex, but I hope the example shows the basic ideas.

4. Inter Microservice Communication and Replicating User Changes

The Post service receives changes to the users from the User microservice via Rabbitmq messages:

You will simulate that with JMeter, too.

Create the RabbitMQ Container

Issue the following command (in one line in a console window) to start a RabbitMQ container with admin UI:

C:\dev>docker run -d  -p 15672:15672 -p 5672:5672 -e RABBITMQ_DEFAULT_USER=test -e RABBITMQ_DEFAULT_PASS=test --hostname my-rabbit --name some-rabbit rabbitmq:3-management

The command configures “test" as user and password instead of the default Guest credentials. Guest is limited to localhost, but inside docker, the containers are on different hosts.

Modify the Post Microservice

Install the RabbitMQ.Client NuGet package in Visual Studio.

Add the IntegrationEventListenerService class:

The background service uses the “test" account to access RabbitMQ and host.docker.internal and localhost as hosts. This allows connections from inside a container and the Visual Studio debugger.

Single active consumer guarantees that only one Post service instance receives messages.

If the active instance crashes then the next instance will take over. You can later try it by stopping the currently receiving instance in docker.

The code creates the exchange and pipe if they are not already existing. It uses manual acknowledgments.

Modify Startup.cs to run the IntegrationEventListenerService:

Run the Changed Microservice in Docker

Shutdown the application in docker:

C:\dev>docker-compose down

Build the Post service, dockerize it and publish it to docker.

Start the application after your changes:

C:\dev>docker-compose up -d

It takes some time until the database servers are running. And remember to initialize the database:

Modify the JMeter Tests

In the JMeter test plan add a thread group with only one thread:

Add a constant timer:

The constant timer limits the test to one message per second.

One message per second is still very often. Even if 1,000,000 users sign up in half a year it would be only 4 new users per minute.

Add the HTTP Request to publish the message to RabbitMQ:

Servername: localhost
Port: 15672
HTTP-Request: POST
Path: /api/exchanges/%2F/userloadtest/publish
Body Data:

    {
     "properties":{},
     "routing_key":"user.update",
     "payload":"
     {
      id:1,name:\"NewName${__counter(TRUE,)}\",version:${__counter(TRUE,)}
     }",
     "payload_encoding":"string"
    }

The user entity has a version field to handle out-of-order messages. To keep the test simple, it only updates a single user and increments the version field. The performance impact should stay the same.

Add an HTTP Header Manager for the Content-Type and the authorization to RabbitMQ.

Content-Type | application/json
Authorization | Basic dGVzdDp0ZXN0

“dGVzdDp0ZXN0" is the base64 encoded user and password “test".

Run the Tests

The throughput should be similar to the previous tests.

You can also identify the active RabbitMQ consumer in docker by looking at the console output. You can stop the container and another instance will take over.

5. Final Thoughts and Outlook

You created a microservice architecture and implemented application-layer database sharding. Then you scaled the application with multiple container instances and load tested it. You also processed user change events from RabbitMQ.

This is only an example application. You will have to adjust the code to use it in a production environment.

What’s next? Getting data like the top 10 categories with the most posts requires querying multiple database instances. This might cause delays and kill performance. Redis can be a solution for querying aggregated data. I will show it in one of my next posts.

You can also run the application in Kubernetes. See my other articles:

Please contact me if you have any questions, ideas, or suggestions.

How to use Database Sharding and Scale an ASP.NET Core Microservice Architecture

Christian Zink — Sun, 13 Jun 2021 19:37:25 +0000

Load Balance a C# ASP.NET Core Service and Use MySql App-Layer Sharding. Shows the Concepts, Which Also Apply to MongoDB, etc.

One of the big advantages of microservices is, that they can be scaled independently. This article shows the benefits and challenges of scaling one microservice and its database.

You will create a working example application and manually implement application-layer sharding. It shows how to choose a shard key based on the use-cases and data model. This helps to apply the same principles to DBMS with integrated scaling like MongoDB, etc.

Further reading: Microservices vs. Monolith Architecture and Database Sharding Explained- 2021 Database Scaling Tutorial

This is the first of two parts. You will implement the microservice and use a sharded DB.

In the second part, you will scale and run multiple container instances of the microservice and databases. You will use docker compose and a load balancer. Finally, you run JMeter load tests to see how the application scales when using a different number of instances.

1. Usecases and Datamodell

The example application consists of a user and a post microservice. They communicate via messages:

See also my previous article How to Build an Event-Driven ASP.NET Core Microservice Architecture

The User microservice handles adding and modifying users. The Post microservices handles viewing and adding posts. There is far more interaction with the Post microservice. So when the load to the app increases the Post microservice will be the first microservice that needs to scale.

The name of the author is part of the PostService bounded context and therefore the Post microservice. Adding and modifying authors is done in the User microservice. The User microservice sends events when a new user is added or a username changes.

Logical Data Model of the PostService

Users can write posts in categories. They can also read the posts by category including the author name. Newest posts are on top. The categories are fixed and change seldom.

Based on these use-cases I decided to shard by category:

2. Implement the Microservice

Install Visual Studio Community (it’s free) with the ASP.NET and web development workload.

Create a solution and add an ASP.NET Core 5 Web API project with the name “PostService”. Disable HTTPS and activate OpenAPI Support.

Install the following NuGet packages:

Microsoft.EntityFrameworkCore.Tools
MySql.EntityFrameworkCore
Newtonsoft.Json

Create the Entities

The index of the Post entity should speed up the retrieval of the latest posts in a category:

The Version in the User entity will later help to handle out-of-order messages:

Create the PostServiceContext

Add connection strings for the shards in appsettings.Development.json (you will use two shards during debugging)

Add the DataAccess Code

GetConnectionString(string category) calculates the hash of the CategoryId. The first part of the hash modulo the number of configured shards (connection strings) determines the shard for the given category.

InitDatabase drops and recreates all tables in all shards and inserts dummy users and categories.

The other methods create and load posts.

Register DataAccess as a singleton in Startup.cs

Create the PostController

It uses the DataAccess class

3. Access a Database from the PostService

Install Docker Desktop

Create two MySql Containers (each command as one line)

C:\dev>docker run -p 3310:3306 --name=mysql1 -e MYSQL_ROOT_PASSWORD=pw -d mysql:5.6

C:\dev>docker run -p 3311:3306 --name=mysql2 -e MYSQL_ROOT_PASSWORD=pw -d mysql:5.6

Start the Post service in Visual Studio. The browser opens at http://localhost:5001/swagger/index.html

Use the swagger UI to interact with the service:

Init the Databases with 100 users and 10 categories:

Add a post to “Category1”:

{
  "title": "MyTitle",
  "content": "MyContent",
  "userId": 1,
  "categoryId": "Category1"
}

Read the top 10 posts in “Category1” to see your new post:

Connect to the database containers and verify which database contains the new post.

C:\dev>docker container exec -it mysql1 /bin/sh

The second instance does not contain any post:

C:\dev>docker container exec -it mysql2 /bin/sh

4. Final Thoughts and Outlook

You created a working application and implemented application-layer sharding and used the concept of shard keys.

In the second part, you will scale and run multiple container instances of the microservice and database. You will use docker compose and a load balancer. You will then run JMeter load tests to see how the application scales when using a different number of instances. Finally, you will simulate user events from the User microservice via RabbitMQ.

Please contact me if you have any questions, ideas, or suggestions.

The Outbox Pattern in Event-Driven ASP.NET Core Microservice Architectures

Christian Zink — Wed, 26 May 2021 19:34:49 +0000

How to Build a Resilient Architecture with RabbitMQ, C#, Entity Framework, and the Transactional Outbox Pattern

In the first step, you will create two microservices. Each microservice has its own database. They use events to publish changes to a RabbitMQ event bus. (You can skip this part if you already implemented it in my last article). Next, you will see how messages get lost e.g. when the message bus is down.

In the second part, you apply the transactional outbox pattern and see how it prevents losing messages.

In the last step, you add publisher and subscriber acknowledgments and duplicate/out-of-order message handling.

Create the .NET Core Microservices and Exchange Messages
Implement the Transactional Outbox Pattern
(Optional) Test the Implementation
Add Publisher Notify, Acknowledgments, and Resilient Message Handling
Final Thoughts and Outlook

1. Create the .NET Core Microservices and Exchange Messages

See my previous article for how to create the Microservices and configure RabbitMQ.

This will give you the following components and workflow:

Now both microservices and the RabbitMQ container “some-rabbit” should be running. Use the REST API of the User service to create and modify users and make sure everything is working. The User service should be sending events to the event bus and the Post service handles them. And the users in the user database are in sync with the users in the post database.

Messages Getting Lost

Stop the RabbitMQ container:

C:\dev>docker stop some-rabbit

Use the Swagger UI to create a user in the UserService:

{
 "name": "Chris2",
 "mail": "chris2@chris2.com",
 "otherData": "Some other data"
}

The RabbitMQ server can’t be reached:

The user is in the user database:

The event is lost and therefore the user is not in the post database and the microservices are now inconsistent. In the next step of this guide, you will see how to solve this problem.

2. Implement the Transactional Outbox Pattern

In this part of the guide, you will add the transactional outbox pattern to the UserService project to prevent lost messages.

If you want to read about the details and concepts of the transactional outbox pattern, you can get more information about it at: https://microservices.io/patterns/data/transactional-outbox.html

Create the IntegrationEvent entity:

Add it to the UserServiceContext:

public DbSet<UserService.Entities.IntegrationEvent> IntegrationEventOutbox { get; set; }

Modify the UserController

The code in PutUser and PostUser starts a transaction and updates/inserts the User entity. In the same transaction it inserts the IntegrationEvent in the database instead of directly publishing the event:

Create the publisher as BackgroundService

The IntegrationEventSenderService polls the database and sends all outstanding events to RabbitMQ:

Continuously polling the database is not good. You will improve the polling in the next step.

Add the IntegrationEventSenderService as HostedService to Startup.cs:

Further reading: Implement background tasks in microservices with IHostedService and the BackgroundService class

3. (Optional) Test the Implementation

Now it’s a good time to test if everything is working like expected:

Delete the user.db so that the database schema is created including the outbox table

Delete the post.db so both databases are empty

Start the RabbitMQ container:

C:\dev>docker start some-rabbit

Start the User Service in Visual Studio

Use the Swagger UI to create a user in the UserService:

{
 "name": "Chris",
 "mail": "chris@chris.com",
 "otherData": "Some other data"
}

The sending of the event is logged to the console:

Stop the RabbitMQ container:

C:\dev>docker stop some-rabbit

Use the Swagger UI to create another user in the UserService:

{
 "name": "Chris2",
 "mail": "chris2@chris2.com",
 "otherData": "Some other data"
}

The IntegrationEventSender can’t send the message and logs a RabbitMQ exception every 5 seconds:

Start the PostService (it has no automatic restarting logic)

Start the RabbitMQ container:

C:\dev>docker stop some-rabbit

The IntegrationEventSender sends the message:

Congratulations: Your outbox implementation is working like expected!

3. Add Publisher Notify, Acknowledgments, and Resilient Message Handling

Inserting or updating the user entity notifies the publisher so it loads data from the outbox table only if there are new entries. A version field for the user entity enables handling of duplicated or out-of-order messages in the consumer. RabbitMQ publisher confirms, persistent messages and acknowledgments handle situations where the publisher, subscriber, or the event-bus don’t work.

Detailed workflow of the outbox pattern, publisher notify, acknowledgments, and version control:

Modify the User entity to include the version field:

You could use RowVersion with a production SQL database. But SQLite doesn’t support it.

Modify the IntegrationEventSender

It uses a CancelationToken to wake up when there are new entries in the outbox. See the comments in lines 47, 49, and 65 for how persistent messages and publisher confirms are implemented.

Synchronously waiting for publisher confirms is very inefficient. See the official RabbitMQ guide for how to improve it.

Make sure, you created the queue as durable:

Durable queues will be recovered on node boot, including messages in them published as persistent. Messages published as transient will be discarded during recovery, even if they were stored in durable queues. (https://www.rabbitmq.com/queues.html)

Modify the UserController

Set/increment the version field in PostUser/PutUser.

Get the IntegrationEventSender in the constructor and call it after committing the transactions.

Modify program.cs in the PostService project

Line 78 disables automatic acknowledgments
Line 75 sends the acknowledgment
Lines 45 and 64 check for duplicate messages

Using Program.Main is not optimal. You could move the code to a BackgroundService.

You can now delete the user and post DB and test your implementation as you did in part three of this guide.

4. Final Thoughts and Outlook

You added the outbox pattern to your event-driven architecture and made it more resilient.

You would have to adjust the code to use it in a production environment: Clean up the code and apply security best practices. Apply .NET Core design patterns, error handling, etc. You should also optimize the publisher acknowledgments and use a BackgroundService in the post service.

You could move the outbox logic to a central location like the DbContext or use a framework like MassTransit. (These frameworks, SAGAs, and scaling/sharding might be good candidates for my next articles.)

See my other articles on how to deploy your ASP.NET Core application to Kubernetes, use Angular for the UI, and add MySql and MongoDB databases.

Please contact me if you have any questions, ideas, or suggestions.

How to Build an Event-Driven ASP.NET Core Microservice Architecture with RabbitMQ and Entity Framework

Christian Zink — Sun, 16 May 2021 19:49:59 +0000

Use RabbitMQ, C#, REST-API and Entity Framework for asynchronous decoupled communication and eventually consistency with integration events and publish-subscribe

In this guide, you will create two C# ASP.NET Core Microservices. Both microservices have their own bounded context and domain model. Each microservice has its own database and REST API. One microservice publishes integration events, that the other microservice consumes.

Decoupled Microservices — A Real-World Example With Code

The application uses a real-world example with users that can write posts. The user microservice allows creating and editing users. In the user domain, the user entity has several properties like name, mail, etc. In the post domain, there is also a user so the post microservice can load posts and display the writers without accessing the user microservice. The user entity in the post domain is much simpler:

The microservices are decoupled and the asynchronous communication leads to eventual consistency. This kind of architecture is the basis for loosely coupled services and supports high scalability. The microservices access their example Sqlite databases via Entity Framework and exchange messages via RabbitMQ (e.g. on Docker Desktop).

Overview diagram of the workflow, components, and technologies:

The code and configurations in this article are not suitable for production use. This guide focuses on the concepts and how the components interact. For this purpose error handling, etc. is omitted.

Steps of this Guide

Create the .NET Core Microservices
Use RabbitMQ and Configure Exchanges and Pipelines
Publish and Consume Integration Events in the Microservices
Test the Workflow
Final Thoughts and Outlook

1. Create the .NET Core Microservices

In the first part of this guide, you will create the User and Post Microservice. You will add the Entities and basic Web APIs. The entities will be stored and retrieved via Entity Framework from Sqlite DBs. Optionally you can test the User Microservice with the Swagger UI in the browser.

Let’s get started.

Install Visual Studio Community (it’s free) with the ASP.NET and web development workload.

Create a solution and add the two ASP.NET Core 5 Web API projects “UserService” and “PostService”. Disable HTTPS and activate OpenAPI Support.

For both projects install the following NuGet packages:

Microsoft.EntityFrameworkCore.Tools
Microsoft.EntityFrameworkCore.Sqlite
RabbitMQ.Client

Implement the UserService

Create the User Entity:

Create the UserServiceContext:

Edit Startup.cs to configure the UserServiceContext to use Sqlite and call Database.EnsureCreated() to make sure the database contains the entity schema:

Create the UserController (It implements only the methods necessary for this demo):

Debug the UserService project and it will start your browser. You can use the swagger UI to test if creating and reading users is working:

Implement the PostService

Create the User and Post entities:

Create the PostServiceContext:

Edit startup.cs to configure the UserServiceContext to use Sqlite and call Database.EnsureCreated() to make sure the database contains the entity schema:

Create the PostController:

Currently, you can’t insert posts, because there are no users in the PostService database.

2. Use RabbitMQ and Configure Exchanges and Pipelines

In the second part of this guide, you will get RabbitMQ running. Then you will use the RabbitMQ admin web UI to configure the exchanges and pipelines for the application. Optionally you can use the admin UI to send messages to RabbitMQ.

This graphic shows how the UserService publishes messages to RabbitMQ and the PostService and a potential other service consume those messages:

The easiest way to get RabbitMQ running is to install Docker Desktop. Then issue the following command (in one line in a console window) to start a RabbitMQ container with admin UI :

C:\dev>docker run -d  -p 15672:15672 -p 5672:5672 --hostname my-rabbit --name some-rabbit rabbitmq:3-management

Open your browser on port 15672 and log in with the username “guest” and the password “guest”. Use the web UI to create an Exchange with the name “user” of type “Fanout” and two queues “user.postservice” and “user.otherservice”.

It is important to use the type “Fanout” so that the exchange copies the message to all connected queues.

You can also use the web UI to publish messages to the exchange and see how they get queued:

3. Publish and Consume Integration Events in the Microservices

In this part of the guide, you will bring the .NET microservices and RabbitMQ together. The UserService publishes events. The PostService consumes the events and adds/updates the users in its database.

Modify UserService.UserController to publish the integration events for user creation and update to RabbitMQ:

The connection and other RabbitMQ objects are not correctly closed in these examples. They should also be reused. See the official RabbitMQ .NET tutorial

Modify (and misuse) PostService.Program to subscribe to the integration events and apply the changes to the PostService database:

4. Test the Workflow

In the final part of this guide you will test the whole workflow:

Summary of the steps in the last part of this guide (you can access the services with the Swagger UI):

Call the UserService REST API and add a user to the user DB
The UserService will create an event that the PostService consumes and adds the user to the post DB
Access the PostService REST API and add a post for the user.
Call the PostService REST API and load the post and user from the post DB
Call the UserService REST API and rename the user
The UserService will create an event that the PostService consumes and updates the user’s name in the post DB
Call the PostService REST API and load the post and renamed user from the post DB

The user DB must be empty. You can delete the user.db (in the Visual Studio explorer) if you created users in previous steps of this guide. The calls to Database.EnsureCreated() will recreate the DBs on startup.

Configure both projects to run as service:

Change the App-URL of the PostService to another port (e.g. http://localhost:5001) so that both projects can be run in parallel. Configure the solution to start both projects and start debugging:

Use the Swagger UI to create a user in the UserService:

{
 "name": "Chris",
 "mail": "chris@chris.com",
 "otherData": "Some other data"
}

The generated userId might be different in your environment:

The integration event replicates the user to the PostService:

Now you can create a post in the PostServive Swagger UI (use your userId):

{
  "title": "MyFirst Post",
  "content": "Some interesting text",
  "userId": 1
}

Read all posts. The username is included in the result:

Change the username in the UserService Swagger UI:

{
  "id": 1,
  "name": "My new name"
}

Then read the posts again and see the changed username:

5. Final Thoughts and Outlook

You created the working basis for an event-driven microservice architecture. Besides data replication, you can also use it for classic producer-consumer workflows, SAGAs, etc.

Please make sure to adjust the code to use it in a production environment: Clean up the code and apply security best practices. Apply .NET Core design patterns, error handling, etc.

Currently messages could be lost in edge cases when RabbitMQ or the microservices crash. See my follow-up article on how to apply the transactional outbox pattern and make the application more resilient.

See my other articles on how to:

Please contact me if you have any questions, ideas, or suggestions.

How to Add MySql & MongoDB to a Kubernetes .Net Core Microservice Architecture

Christian Zink — Tue, 04 May 2021 20:24:53 +0000

How to add a MySQL DB and a MongoDB Replica Set in K8S on Docker Desktop using Persistent Volumes and Access the Databases from ASP.NET Core C# and Angular

Overview diagram of the components

In this guide, you will use databases in a raw microservice-based cloud architecture. It starts with a single MySQL instance and continues to a MongoDB replica set with a headless Kubernetes service. Both databases use persistent volumes. The databases are accessed by ASP.NET Core backend services and use Angular as the frontend. The application is exposed via an Ingress controller.

MySQL in Kubernetes with Persistent Volume
MongoDB Replica Set in Kubernetes as StatefulSet
Access the Databases in ASP.NET Core REST-API Services
Call the ASP.NET Core APIs from Angular
Deploy the Angular App and the ASP.NET Core Services to Kubernetes
Final Thoughts and Outlook

1. MySQL in Kubernetes with Persistent Volume

You will deploy a single MySQL pod in Kubernetes on Docker Desktop. A service will make it accessible on port 3306. You will use a Kubernetes persistent volume so the MySQL data survives restarts or deletions of the pod.

MySQL Deployment in Kubernetes on Docker Desktop:

Preparation

Install Docker Desktop for Windows and enable Kubernetes:

Create the Deployment

Create the file mysql.yaml:

Apply the yaml:

C:\dev\demo>kubectl apply -f mysql.yaml

(Optional) Use a Temporary MySQL Client Container to Access and Test the MySQL Deployment

Create the container and connect to the MySQL database server (enter the command as one line):

C:\dev\demo>kubectl run -it — rm — image=mysql:5.6 — restart=Never mysql-client — mysql -h mysql -ppassword

Create a new database:

mysql>create database testdb;

You can delete the MySQL pod:

C:\dev\demo>kubectl delete pod -l app=mysql

Kubernetes will automatically recreate the pod. The persistent volume guarantees that the sql data survives the deletion and recreation of pods.

You will have to secure your MySQL configuration to use it in a production environment. Use Kubernetes secrets, etc.

Further reading in the official Kubernetes documentation: Run a Single-Instance Stateful Application (MySQL)

2. MongoDB Replica Set in Kubernetes as StatefulSet

You will use a StatefulSet to deploy two MongoDB pods to Kubernetes. Each pod uses its own persistent storage. Then you configure the two instances as a MongoDB replica set.

A headless service makes the instances accessible on port 27017 and the deterministic names “mongod-0.mongodb-service” and “mongod-1.mongodb-service”:

Create the StatefulSet

Create the file mongodb.yaml:

Apply the yaml:

C:\dev\demo>kubectl apply -f mongodb.yaml

Configure the MongoDB Replica Set

You will manually configure the replica set. See this article for how to uses a sidecar container for an automatic configuration: Running MongoDB on Kubernetes with StatefulSets

Connect to the mongod-0 pod:

C:\dev\demo>kubectl exec -it mongod-0 -c mongod-container bash

Start the mongo client:

root@mongod-0:/# mongo

Configure the replica set (enter the command as one line):

MainRepSet:PRIMARY> rs.initiate({ _id: “MainRepSet”, version: 1, members: [{ _id: 0, host: “mongod-0.mongodb-service.default.svc.cluster.local:27017” }, { _id: 1, host: “mongod-1.mongodb-service.default.svc.cluster.local:27017” } ]});

You will have to secure your MongoDB configuration to use it in a production environment. Use Kubernetes secrets, etc.

Further reading: MongoDB StatefulSet in Kubernetes by Deeptiman Pattnaik

3. Access the Databases in ASP.NET Core REST-API Services

You will create one ASP.NET Core web API to access the MySQL database and a second ASP.NET Core web API to connect to the MongoDB replica set. Finally, you will create a docker image for each API.

This post focuses on database access. So I keep the other aspects short. See my previous post for details on how to use K8S on Docker Desktop with Ingress to develop locally for the cloud using an ASP.NET Core REST API and Angular.

Install Visual Studio Community (it’s free) with the ASP.NET and web development workload.

Create an API Using the MySQL Service

Create an ASP.NET Core 5.0 Web API project and call it “MySqlApi”. Activate Docker and use the “Linux” setting. Disable HTTPS.

Install the “MySqlConnector” NuGet package.

Further reading: https://mysqlconnector.net/tutorials/connect-to-mysql/

Add a new controller “MySqlController”:

The code will query and return all databases in the MySQL server.

For easier understanding, all code is placed in one method. Please make sure to use proper configuration, etc. when running in a production environment.

Dockerize the API

In the Visual Studio MySqlApi project rename the file “Dockerfile” in the Project-Explorer to “dockerfile” (first character lowercase). Then right-click the dockerfile and select “Create Docker Image”. This will also push the image to docker.

Create an API Using the MongoDB Service

Create a second Web API project and call it “MongoDbApi”. Use the same settings as for the “MySqlApi” project.

Install the “MongoDB.Driver” NuGet package.

Add a new controller “MongoDbController”:

The code will query and return all collections in the MongoDB replica set.

Create the docker image as you did for the MySqlApi project.

4. Call the ASP.NET Core APIs from Angular

You will create an Angular app to load and display the data from the ASP.NET APIs. Finally, you create a docker image that uses NGINX to serve the app.

Download and install Node.js/NPM.

Install Angular (I used Angular 9 when writing this guide):

C:\dev\demo>npm install -g @angular/cli

Create the app:

C:\dev\demo>ng new DemoApp

Adjust the code in DemoApp\src\app

Add the HttpClientModule to app.module.ts:

Call the REST-APIs in app.component.ts:

Display the loaded data in app.component.html:

Dockerize the Angular App

Create the file DemoApp\src\app\dockerfile

Make sure the file “dockerfile” has no file extension. This will use NGINX to serve the app:

Create the file “.dockerignore” (with a dot at the beginning):

Open a command prompt and build the docker image (the dot at the end is important):

C:\dev\demo\DemoApp>docker build -t demoapp .

5. Deploy the Angular App and the ASP.NET Core Services to Kubernetes

Install the NGINX Ingress controller (in one line on the command line):

C:\dev\demo>kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/cloud/deploy.yaml

Create kubernetes.yaml:

Apply the yaml:

C:\dev\demo>kubectl apply -f kubernetes.yaml

You are done! Open your browser on http port 80 and see how your Kubernetes microservices are working:

6. Final Thoughts and Outlook

You created the working basis for a microservice-based cloud architecture and used two different types of databases.

Please make sure to adjust the code to use it in a production environment: Clean up the code and apply security best practices, use Kubernetes secrets and the latest secure images. Apply Angular and .NET Core design patterns, error handling, etc.

See my follow-up posts how to add MySql and MongoDB to your architecture and how to use events for inter-microservice communication.

I will show you more in further posts: Kubernetes secrets, security aspects like SSL, logging, debugging, CI/CD, Helm charts, (code) quality, (auto) scaling and self-healing, etc.

See my other stories how to:

Please contact me if you have any questions, ideas, or suggestions.

How to Build an ASP.NET Core C# Kubernetes Microservice Architecture with Angular on Local Docker Desktop using Ingress

Christian Zink — Sat, 10 Apr 2021 18:51:43 +0000

In this guide, you will create a raw microservice-based cloud architecture. It uses .NET Core with REST-APIs in the backend services and Angular as the frontend. All components are dockerized and Kubernetes orchestrates the containers. The application is exposed via an Ingress controller.

Kubernetes runs in a local environment with docker desktop. It is similar to a cloud environment. You can use it for developing and testing. There are no extra costs. You can always easily deploy it later to the cloud.

Overview diagram of the components:

Build the ASP.NET Core REST-API Backend
Create the Angular Frontend App
Dockerize the .NET Core API and the Angular App
Deploy the Containers to Kubernetes on Docker Desktop
Final Thoughts and Outlook

1. Build the ASP.NET Core REST-API Backend

Install Visual Studio Community (it’s free) with the ASP.NET and web development workload.

Create an ASP.NET Core 5.0 Web API project and call it “DemoApi”. Activate Docker and use the “Linux” setting. Disable HTTPS:

Add a new controller “DemoController”. It just returns “Hello World”:

Enable CORS for the Angular app on port 4200 (develop) and 80:

Edit launchsettings.json and change the URL of the “DemoApi” configuration to Port 80. Then choose the DemoApi configuration and start it:

Test it in the browser:

2. Create the Angular Frontend App

Download and install Node.js/NPM.

Create the folder C:\dev\demo and open a command prompt.

Install Angular (I used Angular 9 when writing this guide):

C:\dev\demo>npm install -g @angular/cli

Create the app:

C:\dev\demo>ng new DemoApp

Further reading: Understanding Angular and Creating Your First Application by gravity well (Rob Tomlin)

Call the REST-API

Adjust the code in C:\dev\demo\DemoApp\src\app

Add the HttpClientModule to app.module.ts:

Call the REST-API in app.component.ts:

Display the loaded data in app.component.html:

Test the app

Make sure the .NET Core API is running. Start the Angular app and open it in your browser to see if everything is working:

C:\dev\demo\DemoApp>ng serve

It should load and display the data “Hello World”:

3. Dockerize the .NET Core API and the Angular App

I will show you two different ways to create and run a docker image. For the .NET Core API, you will use the UI. For the Angular App the command line.

Install Docker Desktop for Windows

Dockerize the .NET Core API

In the Visual Studio DemoApi project rename the file “Dockerfile” in the Project-Explorer to “dockerfile” (first character lowercase). Then right-click the dockerfile and select “Create Docker Image”.

Check the build output if everything worked:

Stop your API project in Visual Studio if it is still running, because docker and Visual Studio can’t bind to port 80 at the same time.

Run the “latest” “demoapi” image in the Docker Desktop UI and map port 80:

Further reading: How to deploy a .NET 5 API in a Kubernetes cluster by Ivan Porta

Dockerize the Angular App

Create the file C:\dev\demo\DemoApp\dockerfile

Make sure the file “dockerfile” has no file extension

This will use NGINX to serve the app:

Create the file “.dockerignore” (with a dot at the beginning):

Open a command prompt and build the docker image (the dot at the end is important):

C:\dev\demo\DemoApp>docker build -t demoapp .

Stop your Angular app if it still running. Use the command line (the whole command in one line) to start the app in the container and publish it on port 4200:

C:\dev\demo\DemoApp>docker run --publish 4200:80 --name demoappcontainer demoapp

Further reading: Build and run Angular application in a Docker container by Wojciech Krzywiec

Test Both Containers

You can list your containers (your container IDs and timestamps will be different):

Start the app in the browser:

Stop both containers so the ports are not blocked for the next step of this guide:

C:\dev\demo\DemoApp>docker stop demoappcontainer
C:\dev\demo\DemoApp>docker stop demoapi

4. Deploy to Kubernetes on Docker Desktop

Enable Kubernetes in Docker Desktop:

Install the NGINX Ingress controller (in one line on the command line):

C:\dev\demo>kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/cloud/deploy.yaml

See the NGINX Ingress Installation Guide for details

Create C:\dev\demo\kubernetes.yaml:

Apply the yaml:

C:\dev\demo>kubectl apply -f kubernetes.yaml

This will create two Kubernetes “Deployments” and for each deployment a “Service”. Both services are then exposed via an Ingress on port 80.

See the Kubernetes documentation for details

The first deployment uses the DemoAPI image and two replicas to show how it is possible to scale. The second deployment uses a single replica of the DemoApp. The Ingress routes to the service based on the path prefix:

Test the Deployments

“get deployments” shows the deployments and number of pods:

Open your browser on port 80(!) and test your Kubernetes deployments:

Further reading: How to deploy a .NET 5 API in a Kubernetes cluster by Ivan Porta

5. Final Thoughts and Outlook

You created the working basis for a microservice-based cloud architecture. But there is still a lot missing.

See my follow-up posts how to add a MySql database and MongoDB replica set to your architecture and how to use events for inter-microservice communication.

I will show you more in further posts: Kubernetes secrets, security aspects like SSL, communication, logging, debugging, CI/CD, Helm charts, (code) quality, (auto) scaling and self-healing, etc.

See my other stories how to:

Please contact me if you have any questions, ideas, or suggestions.

How to JWT Authenticate with Angular to an ASP.NET Core C# API (OpenAPI/Swagger) using NSwag TypeScript

Christian Zink — Tue, 30 Mar 2021 14:15:32 +0000

This guide shows you how to use NSwag to automatically add an OpenAPI specification to an ASP.NET Core REST-API. It also serves the Swagger UI to the browser. You will use NSwag Studio to generate a TypeScript client and add the strongly typed client to your Angular app. Finally you authenticate to the backend via a JWT bearer token.

Workflow and involved components:

Background and Structure of this Guide

When I wanted to create this kind of workflow I found descriptions showing some part of it. This guide brings the parts together.

This guide focuses on the key aspects. I link to other articles for details and download locations of the tools. I use Visual Studio Community for ASP.NET Core 3.1 development.

The guide uses the environment from my previous article “How to Sign-In with Google in Angular and use JWT based .NET Core API Authentication (RSA)”. You can easily apply it to your own Angular project with ASP.NET Core backend.

Use NSwag in the ASP.NET Core Backend
Generate the TypeScript Client with NSwag Studio
Authenticate and Access the API from Angular
Final Thoughts and Outlook

1. Use NSwag in the ASP.NET Core Backend

Use the Visual Studio package manager to install the NSwag middleware for the API project:
Install-Package NSwag.AspNetCore

Edit Startup.cs (lines 18, 34, 35) and configure NSwag to create the Swagger UI and OpenAPI specification:

Test the Result in the Browser

Start the API project and open the Swagger UI in your browser. In my project this is https://localhost:5002/swagger:

My API is a bit boring. Since it is only used to test the authentication, it has no parameters and no return values.

Load the OpenAPI Specification

You can also access the OpenAPI specification (you will need the URL in the next step for NSwag Studio):

Further reading for this step and the next step of this guide: Using NSwag to Generate Angular Client for an ASP.NET Core 3 API by Eric Anderson

2. Generate the TypeScript Client with NSwag Studio

Download NSwag Studio and install it.

Start NSwag Studio and enter the “Specification URL” from the previous step of this guide (e.q. “https://localhost:5002/swagger/v1/swagger.json”). Select “TypeScript Client” as “Outputs”:

Make sure your API project from the previous step is running. NSwag Studio needs to access the OpenAPI sepcification.

Select the “Angular” template and add “ApiBase” as “Base Class Name”:

Select “transformOptions” so later you can add the authentication header in the base class:

Create the file apibase.ts (e.g. at “C:\dev\apibase.ts”). The code sets the authentication header. ApiBase is the base class of the file that NSwag will generate for you:

Then configure apibase.ts as extension code file in NSwag Studio:

Hit the “Generate Outputs” button and copy the output:

3. Authenticate and Access the API from Angular

Create the file api.ts in the app folder and paste the output from the previous step of this guide:

Use the generated code in app.component.ts to access the API. Import the file (line 4) and call the API (line 24–26):

Start the app

c:\dev>ng serve

Login and use the debugger and logging in the chrome browser to verify that calling and authenticating at the backend is working:

4. Final Thoughts and Outlook

You used NSwag to automatically create the OpenAPI/Swagger specification for your API. Then NSwag Studio created the TypeScript client and you used it in Angular.

The app is fully working! But it is only a sample application and you will have to clean up the code, use Angular and .NET Core design patterns, error handling, etc. to use it in production.

You need to handle logging off and timeouts. Maybe you want API paths that don’t require authentication, etc. You can use the NSwag command line tool to automate the build pipeline.

Please contact me if you have any questions, ideas, or suggestions.

How to Sign-In with Google in Angular and use JWT based ASP.NET Core C# API Authentication (RSA)

Christian Zink — Wed, 24 Mar 2021 20:33:55 +0000

This guide shows you all the steps to build an Angular SPA with a focus on authentication. The single-page web application uses Sign-In with google and angularx-social-login. The .NET Core authentication backend creates asymmetrically signed tokens to access another REST-API. The sample is fully working and can be the basis for a microservice-like architecture.

Authentication workflow and involved components:

Background and Structure of this Guide

When I wanted to create this kind of application I found many descriptions showing some part of it. This guide brings the parts together. It is structured so you can optionally test every step before continuing to the next.

This guide focuses on the key aspects. I link to other articles for details and download locations of the tools. I use Visual Studio Community for ASP.NET Core 3.1 development.

Create the Angular App
Add the Login Functionality
Authentication API in the .NET Core Backend
Use the Authentication API
Use the Access Token to Call the Service API
Final Thoughts and Outlook

1. Create the Angular App

Download and install Node.js/NPM.

Create the folder C:\dev and open a command prompt.

Install Angular (I used Angular 9 when writing this guide):

C:\dev>npm install -g @angular/cli

Create the app:

C:\dev>ng new SampleAuthentication

Install the angularx social login module

C:\dev>npm install angularx-social-login

(Optional) Test the app

Start the app and open it in your browser to see if everything is working:

C:\dev>ng serve

Further reading: Understanding Angular and Creating Your First Application

2. Add the Login Functionality

Get a client ID from google

Create a configuration for your app in the google developer console. And add the URL of the Angular debug server (http://localhost:4200) as “Authorised JavaScript origin”:

Get the app’s Client ID that google created for you. Copy the client ID. You will need it in further steps of this guide.

Configure the social login module

Use your google client ID and activate the social login module in the providers section of the app.module.ts:

The code above also adds the HttpClient module, which will be needed later in this guide.

Further reading: Google authentication in Angular

Create the Login Code

Edit app.component.ts

Edit app.component.html

(Optional) Test the login

Start the app

c:\dev>ng serve

Open the app URL in your browser, click “Sign in with google” to open the login dialogue. Sign in with your google credentials:

You are now logged in:

Use the debugger of the chrome browser to get the idToken and other information:

Google creates a new idToken on every login. The token is only valid for a short time.

Copy the idToken. You will need it in the next step of this guide if you want to manually test the API.

3. Authentication API in the .NET Core Backend

The authentication API will use the idToken from google and verify it. Then it creates an access token that grants access to the other APIs of your app.

You will have to increase security before you run this code in a production environment: Shorter token lifetime and refreshs, maybe use sessions instead of tokens, etc.

Install Visual Studio Community (it’s free) with the ASP.NET and web development workload. Create an ASP.NET Core Web API project.

Verification of the idToken

Use the Visual Studio package manager to install the google Auth Package:
Install-Package Google.Apis.Auth

Create an API to verify the idToken.

Change the value in settings.Audience to your google client ID, that you created in the first step of this guide:

The call to GoogleJsonWebSignature.ValidateAsync will throw in case of an error. The Authenticate method uses the JwtGeneratorclass to create and return a custom access token.

Create Custom Access Tokens in .NET Core Using Asymmetric Signatures

With asymmetric JWT signing, only the authentication service knows the private key. The authentication service uses the private key to sign the access token and other APIs use only the public key to verify the access token.

Add the JwtGenerator class, that generates the token:

The JwtGenerator uses the private key, that it gets passed from the UserController. Add the key to appsettings.json, where the UserControler reads it from.

You can use my private key for testing but you will have to create your own private-public key pair for production use. You will also have to adjust token lifetime, etc. for production use.

*Further reading: JWT Authentication with Asymmetric Encryption using certificates in ASP.NET Core. *This story also shows how to create your own public-private key pair.

Edit Startup.cs, which configures the ASP.NET Core backend to use the config, allow access from the Angular client, etc.:

4. Use the Authentication API

Configure the .NET Core API project in debug mode on HTTPS port 5001:

Start the authentication API in Visual Studio:

(Optional) Manually access the authentication API

Use postman to access the API at: https://localhost:5001/user/authenticate

Post the idToken, that you copied from the Chrome debugger in the second step of this guide:

Call the authentication API from the app

Use the Angular HttpClient to post the idToken to the authentication API. Modify app.component.ts:

Line 5: Import the HttpClient
Lines 21: Send the idToken

(Optional) Test the workflow in the app

Do the same as in the first step of this guide.

Start the app

c:\dev>ng serve

Login and use the debugger and logging in the chrome browser to verify that calling the backend and receiving the generated access token is working:

Copy the access token if you want to do a manual test in the next step of this guide.

5. Use the Access Token to Call the Service API

The access token from the authentication API will grant access to this new service API.

Create an ASP.NET Core Web API project and configure it in debug mode on HTTPS port 5002:

Add a controller:

Edit Startup.cs. It took me a while to configure the validation of the token. The public key to the corresponding private key of the authentication API is used to validate the token. The hard part was to inject the public key. See the comments in the source code for details:

(Optional) Use postman to access the service API

Use postman to access the service API via HTTP GET at: https://localhost:5002/secured

In the previous part of this guide, you copied the access token in the Chrome debugger. Add it as bearer token to the call:

The Final step of this guide: Use the app to access the service API

Modify app.component.ts to post the access key to the service API:

You successfully accessed the protected service API:

6. Final Thoughts and Outlook

Your app is now fully working! But it is only a sample application and you will have to clean up the code and apply security best practices, maybe use a more secure OAuth flow, use Angular and .NET Core design patterns, error handling, etc. to use it in production. And you need your own private-public key pair (see JWT Authentication with Asymmetric Encryption using certificates in ASP.NET Core by
Eduard Stefanescu).

You can add other providers like facebook or your own account database.
See also my follow-up stories:

How to use NSwag to autogenerate the TypeScript client to access the .NET Core service API and use OpenAPI/Swagger to describe the API
How to deploy the .NET Core API and Angular App as Microservices to Kubernetes using Ingress and develop local using Docker Desktop

Please contact me if you have any questions, ideas, or suggestions.

Forem: Christian Zink

How to Integrate Endoflife.Date in Dependency-Track EoL

Steps of this tutorial

Step 1: Install Dependency-Track

Step 2: Import Your SBOM

Step 3: Get Your API Key

Step 4: Install and Run the Integration

Linux

Windows

Step 5: Review Results

Notes & Tips

Feedback & Contributions

How to use Redis and Lua Scripts in a C# ASP.NET Core Microservice Architecture

Contents

1. Setup Redis in Docker and Prepare the .NET Core Project

2. Implement the Top Categories

Add Top Categories

Read Top Categories

3. Top Users, the Inbox Pattern, and Lua Scripting for Atomicity

Atomically Add Posts with Lua Scripting

Read the Top Users

4. Final Thoughts and Outlook

How to Cache Aggregated Data with Redis and Lua Scripts for a Scaled Microservice Architecture

Contents

1. Example Use-Cases and Data Model

Top 10 Categories

2. Setup Redis and Implement the Top Categories

Add Top Categories

Read Top Categories

Prerequisites

3. Top Users, Latest User Posts, and the Inbox Pattern

Add Posts (with a race condition)

Read the Top Users and the Latest User Posts

4. Lua Scripting for Atomicity

Atomically Add Posts with Lua Scripting

Store the Script in Redis

5. Final Thoughts and Outlook

How to Scale an ASP.NET Core Microservice and Sharded Database. Load Test with JMeter

Run multiple DBMS and C# Containers Behind an HAProxy Load Balancer with Docker Compose. Test Scaling with Different Number of Instances

1. Run Multiple Database and Microservice Instances

Dockerize the Microservice

Run the Application in Docker

Start the Application

Init the Databases

2. Load Test the Scaled Application with JMeter

Create the JMeter Test Plan

Run the Test

Modify the Test Parameters

3. Example Test Results

4. Inter Microservice Communication and Replicating User Changes

Create the RabbitMQ Container

Modify the Post Microservice

Run the Changed Microservice in Docker

Modify the JMeter Tests

Run the Tests

5. Final Thoughts and Outlook

How to use Database Sharding and Scale an ASP.NET Core Microservice Architecture

1. Usecases and Datamodell

Logical Data Model of the PostService

2. Implement the Microservice

3. Access a Database from the PostService

4. Final Thoughts and Outlook

The Outbox Pattern in Event-Driven ASP.NET Core Microservice Architectures

Contents

1. Create the .NET Core Microservices and Exchange Messages

Messages Getting Lost

2. Implement the Transactional Outbox Pattern

3. (Optional) Test the Implementation

3. Add Publisher Notify, Acknowledgments, and Resilient Message Handling

4. Final Thoughts and Outlook

How to Build an Event-Driven ASP.NET Core Microservice Architecture with RabbitMQ and Entity Framework

Decoupled Microservices — A Real-World Example With Code

Steps of this Guide

1. Create the .NET Core Microservices

Let’s get started.

Implement the UserService

Implement the PostService

2. Use RabbitMQ and Configure Exchanges and Pipelines

3. Publish and Consume Integration Events in the Microservices

4. Test the Workflow