Forem: James Hickey

Databases For Web Developers: Indexing

James Hickey — Fri, 04 Oct 2024 19:37:13 +0000

You're a web developer helping to build a growing SaaS application. You've got your back-end programming language and framework under control. You know about building models, controllers, and views.

Your web framework of choice handles all the database stuff behind the scenes. You're most likely using a relational database like PostgreSQL. It's all good.

However, your user base has grown. Your product has landed larger customers with more users.

Now you're facing some new issues with your application:

Certain pages on your site take forever to load
That global search feature in your app is totally useless because it takes too long to load
Those gnarly multi-filter reports run fine for simple combinations, but add a few more filters and your report crawls to a halt
Customer-facing analytics/reports for your biggest customers is near impossible to query against without taking minutes to run

These are all usually symptoms of database issues. Some common causes include:

Poorly written queries
Lack of or misplaced indexes
Data structures that don't lend themselves to decent performance
Using the wrong database tool for the job
Approaches that were fine for expected smaller loads but have outlived their usefulness

I want to take you through some of the database techniques, tricks and solutions I wish I knew years ago as a developer helping build multiple SaaS applications. I'll focus on using PostgreSQL as my database of choice - but many of these ideas are transferable.

We'll start this series with indexing.

Indexing

I remember some of my earlier jobs trying to get certain queries to perform well.

"Maybe if I just change how I'm joining tables it'll go faster?"

"Maybe if I don't select as much data it'll be faster?"

It was like black magic where you had to use trial-and-error to figure out if a query could make things faster.

I remember specific analytical reports that I couldn't get to work very efficiently. Now that I understand how to use indexes well I know what I should/could have done 🥲.

One of the learnings that propelled my ability to create efficient SQL queries and build pages/reports that load quickly is learning why and when to apply indexes.

Why Indexes Work

I don't think I need to dig into this too much: just like an index at the back of a book - database indexes are a structure that allow searching through many records really fast.

If I want to find which pages in a book talk about "gravity", then instead of searching through every single page, I can look at the index to see if this topic is listed and which pages I should turn to!

Databases have many types of indexes: but understanding the basic index types like b-tree indexes can get you a long way. These are called "b-tree" in PostgreSQL. In SQL Server most indexes are b-tree structures.

Index Basics

In an index for a book, you have generally two different pieces to consider:

The book to index
The term to index

The pages that you need to turn to are the "values" of the index that tell you where to look.

In a database, the parts of an index that you need to know are similar:

The table to index
The column to index

The "pages" that the query engine needs to look up are found using the values of those indexes.

Note: Yes, in some databases "pages" are ironically the same term for where our rows/records are stored.

In reality, the index doesn't look like the above. It might look like a b-tree. Or, it might be stored some other way as there are many different types of indexes.

For example, hash indexes work well for looking up a record by a unique string (like an email address):

The value stored is a pointer to where on the database's file storage the row/tuple/record with that specific username is. The database engine can "look-up" that disk location super quick.

Otherwise, the database engine would have to loop through all the records in the table to figure out which one has that specific username.

Multi-Column Indexes

Usually, in a book's index, each index is for one term. But imagine (and it probably exists) a book where there are multiple terms that are related to each other quite often: like "space" and "time".

There might be a "multi-term index" where each index helps you to find sentences where both terms are used.

Similarly, you can index multiple database columns in the same index.

create index my_index on orders (user_id, completed);

This type of index can help to support more complex querying scenarios.

How And When To Use An Index

Here are the common cases when it probably makes sense to add an index to a column on a relational table with some tips sprinkled throughout.

Looking up specific records by id

Consider this query:

select * from users where id = 5;

If id is a primary key, in all relational databases an index on the id column is already created for you. This one's easy - you're already covered!

If an index didn't exist on the id column, then this query would have to loop through every single record in the table until it finds the correct record.

What many developers miss - which will help with later tips - is the same principle applies to all queries including delete and update.

Consider these two statements:

delete from users 
where id = 5;

update users
set age = 22
where id = 5;

Both of these statements have to first find the row that has id 5.

Because id is a primary key, it's already indexed.

The kicker is this: if there wasn't an index on id then these statements would have to loop through the entire table until it finds the row with id 5.

Without an index, the select, delete and update would be really slow on any non-trivially sized table.

This idea is foundational to the following tips and will unlock your database skills because it applies to all types of queries and statements.

Selecting a subset of records

Consider this query:

select * from orders 
where user_id = 5;

Just like querying against the id column, creating an index on the user_id column will greatly speed this up in any non-trivially sized table.

create index my_index orders (user_id);

Remember the principle that indexes don't just apply to select queries but to any time that the query engine needs to find records?

This index would apply to the following delete and update statements:

delete from orders 
where user_id = 5;

update orders
set price = 5.33
where user_id = 5;

Note: This is why answering the question "should I add an index" is not always straightforward. The answer depends on whether use said column in delete, update or select statements. As you'll read later, impacts on write performance, disk usage, etc. also impact this decision!

Conditions On Multiple Columns

select *
from users
where work_remote = true 
and tenant_id = 5;

In this query, we're looking for all users that are owned by a specific SaaS tenant where each user is working remotely.

Now that the query is filtering by more than one column the possibilities of what you can/should do become more "interesting".

What are your options?

No indexes
Create an index on tenant_id
Create an index on work_remote
Create both of the indexes above
Create a multi-column index on (tenant_id, work_remote)
Create a multi-column index on (work_remote, tenant_id)

That's a lot of options! And it grows even more if we add another column to the mix!

This is where the trade-offs of using indexes play into your decision. Here are some general guidelines that might be helpful as a starting point:

Is your table only few thousand records in length?
- It might be best to not use any indexes
- Or just index tenant_id as a high-value lower impact change and monitor
Do you have many other queries that use these columns in different combinations?
- Might be best to consider using two separate indexes
Do you need this query to perform as well as it possibly could (and these tables are more than a few thousand records)?
- Create an index with the order (tenant_id, work_remote)

Note: Some databases are really poor at applying multiple indexes in the same query. It's good to benchmark these things vs. assuming things and understand what indexing optimizations your database supports. For example, PostgreSQL supports using bitmap index scans where many separate indexes can be combined and used by the same query. Not every database supports this.

Order Matters?

The order of columns defined in a multi-column index make all the difference.

That is, order matters in a b-tree index. For other specialized index types, it might not.

Why? The purpose of an index is to eliminate as many non-matching records as quickly as possible. You always want the first column in an index to be the most specific of the columns. This will whittle down the pool of possible records that match your criteria faster.

In the image above, we want to whittle down our possible records as quickly as possible.

The first option with work_remote as the leading indexed column doesn't really help us. Best case, we still have 1 million records left to search through. We've only split the possible records to search by two.

The second option using tenant_id will more quickly reduce the number of possible records. We're able to quickly grab a very selective subset of all the records: only the records belonging to tenant_id 5 will be fetched from the table.

Index columns used in joins

Usually, queries with join will join on columns that are foreign keys.

Consider this query:

select *
from users u
inner join addresses a
on u.id = a.user_id;

This is a silly query to run because it grabs data for every single user.

What does the query engine need to do?

Loop through every record in the users table.
For each user record, find the address record who's user_id is the same value as the id column on the users table.
Return the data to the database client.

For no. 2 above, each time the query engine needs to find a specific record in the address table it will loop through all the records in that table.

Let me repeat: the query will loop through the entire addresses table for every single user record.

If both tables are large: this could take a while. You'll have a really slow query.

Joins With Foreign Keys

Luckily for you, some databases will automatically create an index on all foreign keys that you create! So no worries, right?

Oh. But some databases won't do this.

We're all looking at you PostgreSQL:

... it is often a good idea to index the referencing columns too. Because this is not always needed, and there are many choices available on how to index, the declaration of a foreign key constraint does not automatically create an index on the referencing columns.

Oh ya. SQL Server too:

Creating a foreign key constraint doesn't automatically create a corresponding index.

In contrast, MySQL will create an index on a foreign key column for you:

Such an index is created on the referencing table automatically if it does not exist.

Foreign Key Joins In Update And Delete Statements

Again, this idea of adding an index to foreign keys affects update and delete statements. Consider the following:

delete
from addresses a
inner join users u
on u.id = a.user_id
where u.work_remote = true;

This statement would delete all addresses where the corresponding user works remotely. If there's no index on foreign key column addresses.user_id then that table would need to do a full scan (e.g. full loop through the table) for every single user record that works remotely.

In PostgreSQL and SQL Server, you'd definitely want to be aware of this.

Note: It's also worth mentioning that some ORMs will automatically create an index for all foreign keys you create, like Entity Framework! I pays to understand the tools that you're working with 🥲.

Joins Using Non-Foreign Key Columns

I'd say this is rare in many SaaS applications. But, it does happen in analytical queries, unstructured data, etc.

Consider the following:

select *
from account_codes codes
inner join imported_codes imported
on codes.code = imported.code
where imported.tenant_id = 4;

Imagine that you have some account codes (whatever that is) in our system. We've also imported codes from a 3rd party system that doesn't match up with our data by any IDs.

In this join, you could (most likely) significantly speed up the query by indexing both sides:

create index on account_codes (code);
create index on imported_codes (code);

Cascading Operations

Consider this schema:

create table orders (
  id bigint primary key,
  placed_at timestamp with time zone not null,  
  user_id bigint references users (id) on delete cascade
)

Let's focus on the foreign key definition.

on delete cascade means that whenever a user record is deleted any referencing records in orders will automatically be deleted too.

Imagine user with id 5 is deleted. The database has to find all orders where user_id = 5 to delete them.

If there's no index on user_id then this will require a full table scan. If you tried to delete many user records, then this means many full tables scans against the orders table!

Note: If you ever have a page in your application that deletes records and it takes longer than expected - it could be there are some cascading deletes that might benefit from some indexes!

Index Trade-Offs

Indexes aren't magic. As with all things in software development: there are trade-offs.

Let's shoot through some of the important trade-offs:

Storage

Indexes use a lot of disk space. In many cases, one index can use the same space as the table it's indexing. Yup.

Once you start using multi-column indexes and specialized index types then your indexes can be even larger than your tables!

The benefit is that you can significantly speed up certain queries or operations like deleting and updating records. But if you aren't careful and considerate of this point, you can quickly cause your database size to expand significantly.

This can lead to slow backups, slow performance due to memory pressure, etc.

Memory Pressure

Databases will load the most often used objects into available RAM to speed things up (tables, indexes, etc.). However, if you create too many indexes then the amount of data that your database needs to keep in memory can start to exceed available RAM.

Not entire indexes are kept in memory - only the most often used disk pages, to be specific. But the point still applies: you're forcing the database to require storing too much "stuff" in RAM. This can start to cause severe issues if you're not careful!

Write Performance

Indexes have to be in sync with the table that they index. This is one way that relational databases keep the data consistent and referential integrity in place.

Every time a new record is inserted, updated or deleted, the corresponding index entries across all indexes have to be modified/added.

The more indexes you have the more work has to be done to insert, update or delete a record.

This is why guidance around bulk loading data usually recommends that you turn off all indexes, load your data and then turn the indexes back on again.

If you have tables where you need data modifications to be super fast then consider keeping your usage of indexes at a minimum.

More Index Types

You don't need to understand how all the different index types work.

However, it's really useful to have a cursory knowledge of what problems specialized indexes solve. If you come across these problems then you'll remember that there is a solution. Then, you can get into the specifics.

Here's a quick rundown of some more specialized indexes available in PostgreSQL and what problems they solve:

Hash index
- Fast equality checks. Really useful for scanning against large unique string/text values.
- create index user_email on users using hash (email);
GIN index (Generalized Inverted Index)
- Useful for full-text search and indexing JSON data.
- create index gin_docs on documents using gin (to_tsvector('english', text));
Expression index
- Supports indexing usage of functions within queries
- create index lower_email on users (lower(email));
Covering index
- Supports squeezing as much performance as possible out of very specific queries.
- create index covering_orders on orders using btree (user_id, ordered_at desc) include (special_instructions, coupon_code);
Bloom index
- You know those pesky reports that have a million different filters the user can choose? Ya - this is for that 🤣.
- create index my_bloom on report_data using bloom (index1, index2, index3, index4, index5);
BRIN index
- Useful when you have a table that is append-only
- create index my_brin on facts using brin (created_at);

My Final Tip

There's much to say about standards and how they give you transferable knowledge as a developer.

But, the fact is: learning in-depth details about how your specific database works does have a massive impact on how you think, design, code and troubleshoot databases.

You don't need to become a database expert. But... I mean... all of your slow pages, reports, slow backups, large disk usage, downtime, etc. are caused by your database!

For real applications with a legit user base: databases are important. Critical.

That doesn't mean you can't get by with a general purpose database using general design and coding guidelines. But if you ever land those big customers that push your application - your database is going to be the bottleneck.

Learn how it works.

A great place to start are the PostgreSQL official docs. It's some of the best written documentation in the industry. The docs really get into the nitty-gritty and help you to understand what's happening at a lower level. Much of this knowledge, in my experience, is transferable.

Start with the documentation on indexes. Take what you've learned here and dig into some more!

15 Common .NET & C# Blunders Based On My Experience

James Hickey — Thu, 01 Aug 2024 17:37:17 +0000

I’ve been using C# and .NET for over a decade now 🙄. I’ve worked with developers professionally in private organizations and out "in the open" within the open-source world.

I thought I would compile some of the most common ".NET blunders" that I personally have seen time and time again.

And just to make it clear: these are all things I had to learn too and keep up to date on too!

#1: Async/Await…

I’ll start by saying that I see C# async misunderstood and misused all the time. I’ve seen very senior, experienced, skilled and knowledgeable C# developers do it. I’ve seen myself do it too!

Based on my experience, I say that C# does not lead developers into the "pit of success" on asynchronous tooling and syntax.

The official docs say:

C# has a language-level asynchronous programming model, which allows for easily writing asynchronous code without having to juggle callbacks or conform to a library that supports asynchrony.

Sure.

There are great resources on this topic 1, 2.

I think the fact that these exist highlight that it’s not so straightforward.

Take a look at the comments on this recent HackerNews thread about C# async/await.

In any event, here’s a quick-fire list of specific questions I’ve often had from other developers when digging into C# asynchronous code. I’ll try to address each one very briefly.

`async void`: doesn’t that just support asynchronous code and the caller can "fire-and-forget"?

No. It’s really bad. Never do it.

There are all sorts of issues. An exception in an async void method can crash your entire application, for example.

Can’t I just put a synchronous call inside `Task.Run` to make it asynchronous?

var myVar = await Task.Run(async () =>
{
    doSynchronousThing();
    return Task.CompletedTask;
});

No. It doesn’t.

This doesn’t do any asynchronous work and could force the use of an extra thread depending on your context, system load, etc.

Note: This code will usually trigger a warning in your IDE because there’s no await keyword used . But you can do this, and I keep seeing it.

Can’t I use `Task.Run` to make really performant parallelized code?

No.

Again, asynchronous code is not the same as parallelized code.

A bit of a tangent maybe, but related… I’ve seen the following code many times:

var myTasks = new List<Task>();

myTasks.Add(Task.Run(async () => {
    doSynchronousThing1();
}));

myTasks.Add(Task.Run(async () => {
    doSynchronousThing2();
}));

await Task.WhenAll(myTasks);

There are a few issues here:

Notice that inside the delegate for each Task.Run we aren’t returning Task.CompletedTask. This makes that delegate’s signature async void (see point above on async void).
This has the same issue as the point above about using Task.Run to force asynchronicity (e.g. it doesn’t).

In the end, this code doesn’t do any asynchronous work and can cause some thread context switching (e.g. there’s unnecessary overhead).

If I don’t `await` a `Task` doesn’t that make my method way more performant?

public Task DoAsynchronousThing()
{
   var myTask = someOtherAsyncStuff();
   return myTask;
}

To keep this simple – just use async on all your asynchronous methods. You benefit more than trying to be smart with a feature that developers keep getting wrong.

David Fowler’s Async Guidance:

There are benefits to using the async/await keyword instead of directly returning the Task…

Note: Depending on where/how you do this, you might get some IDE warnings. Heed the warnings.

Isn’t `Task.Factory.StartNew` the same as `Task.Run`?

Create a new empty .NET console application using dotnet new console my-test.

Replace Program.cs with the following:

using System.Diagnostics;

var watch = new Stopwatch();
watch.Start();

var tasks = new List<Task>();

for (var i  = 0; i < 5; i++)
{
    tasks.Add(Task.Factory.StartNew(async () =>
    {
        await DoDelay();
    }));
}

await Task.WhenAll(tasks);

Console.WriteLine($"The program is closing... it took {watch.ElapsedMilliseconds} ms to run!!");

async Task DoDelay()
{
    await Task.Delay(10000);
}

How long will this program take to execute: more than 10 seconds? Try it.

Spoiler: It takes milliseconds to run. Did you expect that?

Here are a couple articles addressing this in detail (1, 2).

#2: Parallel Processing While Handling HTTP Requests

Here’s a silly example of an HTTP endpoint to show what I’m getting at:

IActionResult PostUpdateUserName(string userName)
{
    int existingNameFound = 0;
    var existingNames = GetExistingUserNames();

    Parallel.ForEach(existingNames, name =>
    {
        if (name == userName)
        {
            Interlocked.Increment(ref existingNameFound);
        }
    });

    // Do something with existingNameFound > 0
}

Using an asynchronous model to process HTTP requests is not the same as parallel processing.

With asynchronous code, you are trying to make your .NET threads do as little work as possible. For web applications specifically, this will keep your applications responsive, capable to serve many requests concurrently and therefore scale well.

Parallel code is the opposite: it uses and keeps a hold on threads. This is to do CPU-intensive work like heavy calculations. This is usually performed using the Parallel static class.

I’ve seen parallel code in web applications being performed during an HTTP request. Usually, the code performing these heavy calculations wasn’t inside MVC controllers but farther down the chain of execution.

This is bad because parallel processing hogs threads that otherwise should be used sparingly to respond to HTTP requests.

If you need to use parallel processing, don’t do it during an HTTP request. This will lead to reducing the scalability and throughput of your overall web server. It could also lead to increasing overall memory pressure on your web process/server too.

#3: Trying To Self-Manage Database Connections

Have you ever seen code where an SqlConnection is passed around as an argument and then used like this?

SomeData DoDatabaseThing(SqlConnection connection)
{
    if (connection.State != ConnectionState.Open)
    {
        connection.Open();
    }

    // Do a bunch of stuff, calling more methods
    // and passing the connection as an argument...

    if (connection.State != ConnectionState.Closed)
    {
        connection.Close();
    }

    // Return data.
}

Some say that being defensive and making sure you manage your database connections this way is "good". In my experience, this type of code is an easy source of bugs.

One time, an application I worked on was using this pattern. I had mentioned a few times that this kind of code could lead to issues if we aren’t careful…

One day, we found that after a number of hours our web server would crash. We had to manually restart the web server in production every couple of hours… Eventually, I found the source:

The developer in question introduced database code that caused C# code to throw an exception – before trying to close the connection. That exception was then silently caught up the stack! 💣

E.g. The connection was never closed and caused a memory/connection leak.

The more you try to be "smart" with database connections, the more chances you have to mess things up.

There are 2 things that come to mind on this topic :

Always put your SqlConnection inside of a using block. It will be automatically closed once it’s out of scope.
Under the covers, SqlConnection is really a pool of connections. It’s already managed for you so you’re not gaining anything trying to manage things yourself.

#4: HttpClient

HttpClient is a class that does what you think it does. However, this class is not as straightforward as it could (should?) be.

The normal thinking in C# is that any class which implements IDisposable should be used within a using block. Since HttpClient isn’t disposable, we should be able to create a new instance any time we need to use it, right?

If you check out the current example of using HttpClient in the official documentation, you’ll see this comment:

HttpClient is intended to be instantiated once per application, rather than per-use. See Remarks.

In the "Remarks" section, there’s a link to this article with more information about how this class works.

I’ve seen systems that had weird network issues and socket exceptions because of misusing HttpClient.

The general advice with using HttpClient is to use IHttpClientFactory to "create" instances for you.

public MyClass
{
    private IHttpClientFactory _factory;

    public MyClass(IHttpClientFactory factory)
    {
        _factory = factory;
    }

    public async Task DoRequest()
    {
        using var client = _factory.CreateHttpClient();

        // Use `client` now to perform http requests.
    }
}

#5: Calling ToList() Unnecessarily

I’ve seen this one a lot. In most situations it’s not a big deal and the side-effects are not felt.

This can happen for many reasons. For example, a developer doesn’t realize that they’ve already called ToList() before from some other method acting on the same collection.

var users = await GetUsers();

Thing1(users);
Thing2(users);

private void Thing1(IEnumerable<User> users)
{
    var list = users.ToList();
    // Do something stuff on list.
}

private void Thing2(IEnumerable<User> users)
{
    var list = users.ToList();
    // Do something stuff on list.
}

Again, this is usually not an issue… until it is.

When working with larger collections, performance sensitive applications or sensitive hot paths of an application can be negatively affected.

Calling ToList() will cause extra memory allocations. With large collections or when creating a large numbers of collections overall (or both), this can lead to:

Using more memory than necessary
Allocating memory is CPU intensive – so it uses extra CPU resources
Additional GC pressure – which can cause large slow downs and pauses

One example from my experience was improving the performance of an application that processed/aggregated millions of data points as quickly as possible. By finding a couple of places where I could remove redundant calls to ToList(), I decreased processing time by 10%.

In this case, the processes could take 6 days to run. Just this one change would have an improvement of a half-day.

Again, this isn’t necessarily about reducing memory usage – it’s about conserving CPU cycles used to allocate new memory and how often the .NET garbage collector has to run.

If you want HTTP requests that are blazing fast too, then this is one thing to keep an eye out for!

#6: Ignoring Database Fundamentals Because The ORM Will "Take Care Of It"

ORMs are great. They can make working with the 90% of database usage in application code straightforward and easy. 90% of database migrations that you’ll need can be autogenerated too!

But then you have some aspect of your system that needs something special:

A reporting query that used to be fast but that now takes down the database when running with that 1 massive customer
That batch insert/update code path that used to be fine but is now abysmally slow
A set of dashboard queries that customers have been complaining about
That new full-text search feature customers have been asking for

What do you do? The more you know about databases – the better.

There’s a place for simplicity, which ORMs are great at. But some situations call for more nuanced solutions.

Here are some areas where it might help you to have some extra knowledge:

Indexing:
- Which tables? Which columns? Clustered vs. non-clustered? B-tree vs. GIN? Etc.
Beyond basic SQL operators:
- Merge
- Cross Apply
- CTEs
PostgreSQL Hot Updates
Full-text Search
Query performance analysis
- SQL Server Query Store
- Explain Analyze

#7: Long-running Or Overnight Jobs That Keep Running Out Of Memory!

I’ve seen this a number of times. It’s not something complicated, but a mistake usually made when the dataset being used in production is small. Then, over time it grows larger and larger and bad data access patterns begin to cause problems.

Imagine an overnight processing job that grabs all the records from a database table and processes them all in memory.

With 10 items to process, this might be fine. But with 10,000 items you’ll start getting memory issues.

My advice: Paginate when dealing with a collection of items where you aren’t sure that it’s going to remain the same size.

#8: Creating A Ton Of Different Separate Projects For One Business/Domain Context

I’ve written about this a few times (1, 2, 3). I’m not the biggest fan of clean architecture. I think many of the principles found in the typical presentation of it are solid (no pun intended!), but I tend to disagree with the application of those principles.

In short, we software developers like to complicate things because it’s "more SOLID" or "more X".

For me simplicity (usually) wins.

Do you need 5 different .NET projects like Web, Core, Infrastructure, DataAccess, Shared, etc.?

What if instead we focus how our folders & projects are structured by the capabilities of our business/system?

Incoming heresy: Maybe it’s okay to put database related code inside the same folder as business logic code? They are still in separate files. But now you don’t need to context switch to find your domain code vs. database code. You can still use interfaces. No one is stopping you.

For my more nuanced thoughts, check out ".NET Architecture: How To Structure Your Solutions."

#9: Allowing Unlimited Network Requests To Your APIs

If you build any HTTP API: it will be abused. Sonner or later. Even by those within the same organization!

It could be a rogue/poorly written script or even a malicious attacker.

Not protecting your API with rate limits leaves the door open for unnecessary outages.

Again, don’t trust that consumers will not abuse your API.

For modern .NET developers, there are newer built-in tools to easily add rate limiting to your APIs. Use them!

#10: Using `PostAsync` Instead Of `PostAsJsonAsync`

The newer PostAsJsonAsync extension method on HttpClient should be your "go to" for making typical HTTP POST requests.

Behind the scenes, this method uses some fancy memory pooling, etc. to make it’s memory allocations minimal.

A much better performance profile can mean things like using a smaller cloud instance size – which reduces cost and complains from the DevOps people 🤣. If you are building a sharable library, then your code is less of a performance burden on the consumer code.

All-in-all, it’s a quick win!

#11: Building Proprietary "Frameworks" And Wrappers Around Built-in .NET Functionality

Anytime I see the name "framework" inside source code that’s not part of .NET I raise an eyebrow.

I’ve seen so many custom frameworks: custom UI frameworks, custom HTTP routing, custom HTML renderers, custom data access frameworks, etc.

Internal frameworks that make working with HTTP requests, routing, data binders, etc. "easier" are bound to cause awkward situations:

The framework wasn’t supposed to be used everywhere. But now it is.
The framework overrides built-in .NET functionality, leading to a net restriction of available functionality instead of a net increase.
New (to the org) developers have to (re)learn all this stuff
There’s usually no documentation. If there is, it’s probably terrible.
What if the 1 person who wrote most of the framework leaves the company?

In my experience, most things that deviate from a standard .NET functionality that’s existed for years is (usually) a can of worms. Or a pandora’s box. Something like that.

#12: Using Stored Procedures Everywhere

Back in the day (like over 10 years ago…) stored procedures were considered by many as a silver bullet.

They are more secure than parameterized SQL statements from application code, more performant, easier to understand, easier to test, etc.

So they said.

Today, I think that the consensus (and truth?) is that none of these points are true:

Parameterized SQL statements are just as safe.
99% of queries you need to do are just as performant.
Stored procedures are harder to (source) version.
The size and scope of stored procedures (in my experience) have the tendency to grow out of control quickly. This is because they are difficult to modularize and test.
Parameterized SQL statements can be tested as part of your .NET test suite if you’ve organized things well. No need to run separate database tests.

This is a great article on the topic.

#13: Dogmatically Never Using Stored Procedures

With all things in our industry, it’s easy to be dogmatic. This article is somewhat dogmatic 🤔.

Specifically with stored procedures, they do have a place.

In my mind, their main benefit is when you have some bulk-ish operations that you need to perform. Sending multiple SQL statements over-the-wire (like hundreds) can be slow. Especially if the database is not on the same machine as your .NET application (this is probably the case for most readers).

The latency between your application and database is usually negligible. But when you are doing bulk-ish stuff, then it can be a source of bottleneck.

Stored procedures have the benefit of storing the SQL statements on the database. You can send over the data you want to work on once, and then let the database do the bulk of the work. Instead of doing a loop in your application code, you put the loop in the stored procedure.

You can send all your data via 1 database call, or in chunks. Either way, this can be a great use case for using a stored procedure. I’ve seen massive improvements to critical operations with this approach, when used sparingly.

#14: Microservices

"Microservice all the things" sounds great.

The idea of decoupling teams and having high-performing, modular and isolated services sounds great. But there can be huge costs to over-leaning into microservices.

If you work in an organization who’s leaned into microservice heavily, you often find issues that appear as questions like:

"How the heck do I integrate with team X?"
"There are 5 versions of service X. Which one should I use since they are all being deployed to production?"
"I’ve found 5 different ways that our organization solves [insert common software design problem]? What should I do?"
"Which repository is that in? Which team owns that? The guy who wrote this isn’t around anymore. Does anyone know how this thing works?"
"Our UI needs data from teams X, Y and Z. Which services and APIs should I use? Oh… team Y has 5 different services that I need to stitch together? Does anyone understand the data points for any of these? [10 meetings later]: So I guess there is a new service that combines these data points that no one else knew about."

All of these questions / musings are signs of costs: You need really good coordination, documentation, planning, etc. to pull it off well.

And so… all the things I said about internal "frameworks" I think applies to microservices:

"It wasn’t supposed to be used everywhere"
"This service doesn’t do functionality F that the previous version of the service did…"
New developers have a lot of extra stuff to learn just to get started
Terrible or no documentation

For more of my thoughts, I’d suggest checking out ".NET Architecture: How To Structure Your Solutions" and "What Is A Microservice Architecture?"

#15: Building Your Own Background Job Scheduler

This is just a plug for my open-source project 🤣.

But, there are costs to using open-source projects that I find developers rarely consider when they add a new library to their projects:

No guarantee of support
Using more libraries expands potential security profile
Poorly abstracted code can cause your own code to introduce hacks or poor design patterns to fit the library into your code

Conclusion

We’re all still learning. There’s always more to learn. But, it’s okay to not learn something new every single day.

More important than learning everyday, I think, is to learn from our mistakes.

That is all.

High-Performance .NET CRON Jobs

James Hickey — Tue, 16 Jul 2024 14:54:23 +0000

CRON jobs are a staple for many software systems:

Nightly reporting jobs
Processing a backlog of queued long-running work
Kicking off database clean-up scripts
Periodic calculations of customer subscription renewals and payments
etc.

See a recent article by Slack, for example, where they talk about their own CRON setup:

Over the years, both the number of cron scripts and the amount of data these scripts process have increased. While generally these cron scripts executed as expected, over time the reliability of their execution has occasionally faltered, and maintaining and scaling their execution environment became increasingly burdensome.
https://slack.engineering/executing-cron-scripts-reliably-at-scale/

In the .NET ecosystem, there are a few great libraries for scheduling or queuing background work. I created Coravel as an easy way to build .NET applications with more advanced web application features. But it's mostly known as a background job scheduling library.

I thought it would be fun to play around with the idea of building a basic CRON job system and progressively building it into a more high-performance CRON job processing system.

We'll start by learning how to use Coravel in a simple scenario. Then, we'll further configure and leverage Coravel's features to squeeze more performance out of a single .NET process. Finally, you'll learn a few advanced techniques to build a high-performance background job processing system.

Note: I'm using a copy of the Wide World Importers database for this exercise using an SQL Server docker container as my backing sample data.

Code/Repository

You can see the sample code repository on GitHub.

Building A Basic CRON Job Process

First, I've installed some packages like Coravel, Dapper and the usual stuff to get a basic .NET console application up and running.

Here's Program.cs from my project named Basic:

using Basic;
using Coravel;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddScheduler();
builder.Services.AddTransient<string>(p => "Server=127.0.0.1,1433;Database=WideWorldImporters-Standard;User Id=sa;Password=P@assword;TrustServerCertificate=True;");
builder.Services.AddTransient<ProcessAllOrdersInvocable>();

var host = builder.Build();

host.Services.UseScheduler(scheduler =>
{
    scheduler.Schedule<ProcessAllOrdersInvocable>()
        .EverySeconds(5)
        .PreventOverlapping(nameof(ProcessAllOrdersInvocable));
});

await host.RunAsync();

We're scheduling one job, ProcessAllOrdersInvocable, to run every 5 seconds. If another instance of this job is running at a given due time, we skip that run. This way it doesn't step all over its own feet 😅.

Here's the Coravel invocable ProcessAllOrdersInvocable:

using System.Diagnostics;
using Coravel.Invocable;
using Dapper;
using Microsoft.Data.SqlClient;

namespace Basic;

public class ProcessAllOrdersInvocable : IInvocable
{
    private string connectionString;

    public ProcessAllOrdersInvocable(string connectionString)
    {
        this.connectionString = connectionString;
    }

    public async Task Invoke()
    {
        var lastIdProcessed = 0;
        var watch = new Stopwatch();
        watch.Start();

        await using var connection = new SqlConnection(this.connectionString);

        while (true)
        {          
            var items = (await connection.QueryAsync<(int OrderId, string OrderDate)>
                (SQL, new { LastIdProcessed = lastIdProcessed })).AsList();

            if (!items.Any())
            {
                break;
            }

            var tasks = new List<Task>(items.Count);
            foreach (var item in items)
            {
                tasks.Add(SimulateProcessOrderAsync(item));
            }

            await Task.WhenAll(tasks);

            lastIdProcessed = items.Last().OrderId;
        }

        watch.Stop();
        Console.WriteLine($"### {nameof(ProcessAllOrdersInvocable)} took {watch.ElapsedMilliseconds} ms");
    }

    private static async Task SimulateProcessOrderAsync(object order)
    {
        await Task.Delay(10);
    }

    private const string SQL = @"
SELECT TOP 100
    *
FROM Sales.Orders 
WHERE 
    OrderID > @LastIdProcessed
ORDER BY OrderID";
}

About this job:

Loads all orders from the database into memory in chunks of 100
For each item in a chunk/batch, it does some fake processing that takes 10 milliseconds
The code stores each Task from the processing method and awaits them all at the end for increased performance

Note: You might have noticed that the SQL query I'm executing is grabbing data from all the columns. This is to perform a more realistic query, even though our code doesn't use any column except OrderId.

For this job to execute once, on my machine (Ryzen 7 4800), takes about 12 seconds when profiling.

Profiling the memory usage we get about 32.9 MB of usage:

Increasing Batch Sizes

12 seconds is too long! We need this to be within our SLA of 5 seconds to process all pending orders (we're a busy business).

The next step we can take is to increase our batching size from 100 to let's say 5000 👀. This should reduce the amount of database calls we need to make.

Doing this brings our processing time down to about 2 seconds.

In the real world, if we are sending 5000 emails at the same time over the network a few "bad" things can occur:

We start getting an increase in overall latency in our network
Our email provider starts rate-limiting our process

However, our (fake) email provider is pretty super so we don't get any issues here.

The results:

Processing time is between 500 to 1000ms
RAM usage is up to about 45MB

Adding More CRON Jobs

Over time more CRON jobs have been added to the application.

host.Services.UseScheduler(scheduler =>
{
    scheduler.Schedule<ProcessAllOrdersInvocable>()
        .EverySeconds(5)
        .PreventOverlapping(nameof(ProcessAllOrdersInvocable));

    scheduler.Schedule<ProcessAllCitiesInvocable>()
        .EverySeconds(5)
        .PreventOverlapping(nameof(ProcessAllCitiesInvocable));

    scheduler.Schedule<ProcessAllInvoicesInvocable>()
        .EverySeconds(5)
        .PreventOverlapping(nameof(ProcessAllInvoicesInvocable));

    scheduler.Schedule<ProcessAllStockItemTransactionsInvocable>()
        .EverySeconds(5)
        .PreventOverlapping(nameof(ProcessAllStockItemTransactionsInvocable));
});

Some of these are hitting bigger database tables now:

Let's try to run all CRON jobs each with a batch of 5000 items assuming each record or item takes 10ms to "process" (whether that's sending emails or whatever).

Not bad. But, I'm running on a powerful laptop. What if we ran this in a docker container with limited resources?

Real-World Scenario: Running In Limited Docker Containers

I created a basic docker file:

FROM mcr.microsoft.com/dotnet/runtime:6.0 AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["Basic/Basic.csproj", "Basic/"]
RUN dotnet restore "Basic/Basic.csproj"
COPY . .
WORKDIR "/src/Basic"
RUN dotnet build "Basic.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "Basic.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Basic.dll"]

I've configured my application to run with run options --memory="200m" --cpus="1". This is probably a more realistic scenario.

To enforce this, I created a docker-compose.yaml file to set resource limits:

version: "3.3"
services:
  dotnet:
    build:
      context: ..
      dockerfile: ./Basic/Dockerfile
    deploy:
      resources:
        limits:
          cpus: "1"
          memory: 200M

Note: My connection string also needs to reference "host.docker.internal" instead of "localhost" now.

So... what are the results?

In Coravel's default configuration, jobs are executed one after the other. So, the total time it takes Coravel to process these 4 is the sum of all the logged values: over 6 seconds.

This isn't meeting our SLA!

Coravel Schedule Workers

The reason why Coravel runs CRON jobs one by one is so that it doesn't hog extra threads. In Web applications, this is a really good thing!

Coravel has a feature to dedicate more threads in order to isolate and parallelize jobs. This is called Schedule Workers:

scheduler.Schedule<ProcessAllOrdersInvocable>()
    .EverySeconds(5);

scheduler.Schedule<ProcessAllCitiesInvocable>()
    .EverySeconds(5);

// Dedicating a separate thread for this job.
scheduler.OnWorker(nameof(ProcessAllStockItemTransactionsInvocable));
scheduler.Schedule<ProcessAllStockItemTransactionsInvocable>()
    .EverySeconds(5);

// Dedicating a separate thread for this job too.
scheduler.OnWorker(nameof(ProcessAllInvoicesInvocable));
scheduler.Schedule<ProcessAllInvoicesInvocable>()
    .EverySeconds(5);

Typically, schedule workers are useful for isolating longer-running tasks so they don't hold up other shorter-running tasks. But, does this help if we give one of the jobs a dedicated thread when it runs?

Keeping in mind that most of these jobs are running in parallel, the total time it took to run the 4 jobs is the job that took the longest: ~3.7 seconds.

However, we're seeing the time it takes individual jobs actually go up a bit. It might be that the overhead of having to manage multiple threads to do work is being throttled/limited by our container's measly 1 CPU allocation.

More Power!

Bumping the configuration to 2 CPUs gives us:

The 3 first jobs are definitely running faster. However, that last job isn't. It might be limited by database resourcing or it still might not have enough CPU power.

To verify, let's bump the CPU limit to 16 (my machine's total logical CPU count):

No material change here.

If we were okay with the results we have, the next steps might be to play around with the code of the ProcessAllInvoicesInvocable. We could try to change how large the batches of data it fetches from the database are, verify and optimize database indexing, etc.

Distributed Processing

We need more processing power. But, our organization only allows us to configure up to 1 docker CPU per running container! (Pesky governance!)

What if we could split the processing of scheduled jobs across multiple running processes though?

There's a great library for using distributed locks called DistributedLock. Go figure.

There's a specific package that supports locking with SQLServer that we'll start with.

Our invocables now look like this:

public async Task Invoke()
{
    var @lock = new SqlDistributedLock(nameof(ProcessAllCitiesInvocable), this.connectionString);
    await using var handle = await @lock.TryAcquireAsync();
    if (handle != null)
    {
        // do stuff
    }
}

I also changed docker-compose-yaml to run 3 instances of this project. Here are the results where you can see that jobs aren't overlapping and are handled by different instances:

The benefit of this approach, much like using Coravel's schedule workers, is that if one job is taking too long to process it won't cause other jobs to wait for it. Another instance/process will pick up those other jobs.

It also looks like we have a bottleneck with ProcessAllInvoices.

This bottleneck on my machine could be due to disk speed, memory speed, CPU, or something else. This might just be another reminder that the database is such an important piece of your performance profile!

A Potential Issue

There's one issue to consider though. If you absolutely need to ensure that a specific job only runs once every 5 seconds, this approach won't work. Look at the screenshot above closely to see this - some jobs are executed multiple times within a span of 5 seconds even though different instances are running the job.

Let's say your job X takes 500 ms to run. On Instance 1 it runs at 12:00:00 p.m.
Instance 2 also sees that job X is due at the same time. So instance 2 starts to run a batch of all the jobs that are due.
Instance 2 runs a job Y first, which takes about 600 ms.
Then, at 12:00:60 p.m., it tries to get a lock on job X.
It gets a lock since job X on instance 1 finished running 100 ms ago.

If you don't need this kind of exact timing guarantee within a small time span of work, then this distributed lock approach might work.

Decoupling Distributed CRON Jobs

The way that we've architected our CRON jobs might work for most teams. But, one of the other downsides is that we've coupled our scheduling logic with our job logic. Sometimes that's okay.

It also has the concurrency issue mentioned in the section above.

If we wanted to scale this solution so that other teams in our organization could use it, it wouldn't work. Other teams shouldn't be able to add their own invocable/job logic in our code. Our code would become a dumping ground!

Separate Scheduling Logic From Job Logic

The next step is to separate the scheduling logic from the CRON job logic. If we were using Coravel, then we'd make Coravel push an asynchronous message to a message broker.

We'd have to determine what the lowest interval or "tick" is. Every second? 5 seconds? Minute (like normal CRON in an OS)?

Coravel would send a message every interval X with the exact time that it ran.

Every team/consumer would get a message each interval that they should execute their job logic based on what time it is. But each team now owns that code.

Alternately, this approach could also help with the issue we saw where different instances executed the same job multiple times. Now, we're only triggering "ticks" once across all distributed systems and could leverage the messaging technology to only send each tick to one consumer in a round-robin fashion (or some other load-balancing technique).

An Alternate Way To Measure Performance

Let's go back to the beginning. Our assumption was that our entire batch of jobs had to run within 5 seconds.

One other way to measure the performance of the 3 different approaches we looked at (serial, multi-threads & multi-instance) is to calculate performance based on "database records processed per second".

What if our SLA wasn't "process 1 batch in 5 seconds" but "process X number of records every X seconds"?

So, I made the changes needed. For example, here's the extra job to output the value:

scheduler.Schedule(() =>
    {
        Console.WriteLine("### Total records processed: " + TotalRecordsProcessed.Value);
    })
    .EverySecond();

Each job adds the number of saved records to the database using Interlocked.

I ran these using the same docker-compose files created across each project with a CPU limit of "2" for each instance.

Basic Scheduler Results

### Total records processed: 10000
### Total records processed: 30000
### Total records processed: 50000
### Total records processed: 85510
### Total records processed: 415772
### Total records processed: 438712
### Total records processed: 453712
### Total records processed: 478712
### Total records processed: 574222
### Total records processed: 847424

We have an average of 84,742 records per second (847,424 records / 10 runs).

Schedule Workers Results

### Total records processed: 80000
### Total records processed: 273595
### Total records processed: 281535
### Total records processed: 393202
### Total records processed: 413202
### Total records processed: 637307
### Total records processed: 685247
### Total records processed: 801914
### Total records processed: 821914
### Total records processed: 837424

We have an average of 83,742 records per second (837,424 records / 10 runs).

Distributed Instances

For these results, we have to take 10 seconds of activity across 3 nodes.

distributed-dotnet3-1  | ### Total records processed: 15000
distributed-dotnet1-1  | ### Total records processed: 20000
distributed-dotnet2-1  | ### Total records processed: 15000
distributed-dotnet3-1  | ### Total records processed: 92940
distributed-dotnet1-1  | ### Total records processed: 93595
distributed-dotnet1-1  | ### Total records processed: 111535
distributed-dotnet2-1  | ### Total records processed: 30000
distributed-dotnet1-1  | ### Total records processed: 111535
distributed-dotnet3-1  | ### Total records processed: 227940
distributed-dotnet2-1  | ### Total records processed: 50000
distributed-dotnet3-1  | ### Total records processed: 334607
distributed-dotnet1-1  | ### Total records processed: 111535
distributed-dotnet3-1  | ### Total records processed: 348202
distributed-dotnet2-1  | ### Total records processed: 95510
distributed-dotnet2-1  | ### Total records processed: 108450
distributed-dotnet2-1  | ### Total records processed: 108450
distributed-dotnet3-1  | ### Total records processed: 448202
distributed-dotnet1-1  | ### Total records processed: 195130
distributed-dotnet1-1  | ### Total records processed: 223070
distributed-dotnet2-1  | ### Total records processed: 108450
distributed-dotnet1-1  | ### Total records processed: 228070
distributed-dotnet3-1  | ### Total records processed: 548202
distributed-dotnet2-1  | ### Total records processed: 108450
distributed-dotnet1-1  | ### Total records processed: 243070
distributed-dotnet3-1  | ### Total records processed: 654869
distributed-dotnet3-1  | ### Total records processed: 658464
distributed-dotnet2-1  | ### Total records processed: 108450
distributed-dotnet3-1  | ### Total records processed: 658464
distributed-dotnet1-1  | ### Total records processed: 263070
distributed-dotnet2-1  | ### Total records processed: 146390

We have an average of 106,810 records per second.

Cool! But is that because we are actually running more tasks than we should? (Remember the locking issue? The answer turns out to be "yes" 😅)

I tried again by changing the code so that each instance only runs explicitly 1 or 2 jobs (e.g. no sharing across instances):

distributed-dotnet1-1  | ### Total records processed: 0
distributed-dotnet2-1  | ### Total records processed: 45000
distributed-dotnet3-1  | ### Total records processed: 5000
distributed-dotnet1-1  | ### Total records processed: 20000
distributed-dotnet2-1  | ### Total records processed: 155000
distributed-dotnet3-1  | ### Total records processed: 15000
distributed-dotnet1-1  | ### Total records processed: 87940
distributed-dotnet2-1  | ### Total records processed: 236667
distributed-dotnet3-1  | ### Total records processed: 30000
distributed-dotnet1-1  | ### Total records processed: 111535
distributed-dotnet2-1  | ### Total records processed: 236667
distributed-dotnet3-1  | ### Total records processed: 45000
distributed-dotnet1-1  | ### Total records processed: 111535
distributed-dotnet2-1  | ### Total records processed: 236667
distributed-dotnet3-1  | ### Total records processed: 60000
distributed-dotnet1-1  | ### Total records processed: 111535
distributed-dotnet2-1  | ### Total records processed: 351667
distributed-dotnet3-1  | ### Total records processed: 70510
distributed-dotnet1-1  | ### Total records processed: 214475
distributed-dotnet2-1  | ### Total records processed: 473334
distributed-dotnet3-1  | ### Total records processed: 70510
distributed-dotnet1-1  | ### Total records processed: 223070
distributed-dotnet2-1  | ### Total records processed: 473334
distributed-dotnet3-1  | ### Total records processed: 70510
distributed-dotnet1-1  | ### Total records processed: 223070
distributed-dotnet2-1  | ### Total records processed: 473334
distributed-dotnet3-1  | ### Total records processed: 70510
distributed-dotnet1-1  | ### Total records processed: 223070
distributed-dotnet2-1  | ### Total records processed: 473334
distributed-dotnet3-1  | ### Total records processed: 70510

That's roughly about 76,690 records per second.

While not super scientific, using 1 .NET instance with Coravel was able to perform just as well as using multiple instances.

Keep in mind that we are measuring overall throughput across our entire processing with this measurement vs. the performance of an entire batch that we expect to run within 5 seconds.

Conclusion

The conclusions of our not-super-scientific-but-fun experiment are:

Changing the size of batches that you fetch from the database can dramatically improve the performance of an individual CRON job 💰
Even though we didn't 100% verify (in my experience this is often true): database performance is often a bottleneck 🍾
1 process using Coravel to run your CRON jobs is pretty efficient compared to running multiple processes 🚀
Only distribute work if you know it's going to help as it can introduce unseen issues. Make sure you understand what's going on. Distributed systems are hard 🙄
For high-performance CRON processing across teams, you should decouple scheduling logic from job logic 💻

Check out the repository with code that you can play around with. Give Coravel a try in your own projects if you haven't yet!

What Are Aggregates In Domain-Driven Design?

James Hickey — Wed, 03 May 2023 03:12:20 +0000

Aggregates are one of the most misunderstood concepts in domain-driven design.

What is an aggregate? Sure, it's a pattern that's central to domain-driven design... but is it just a collection of objects?

Martin Fowler explains:

Aggregates are the basic element of transfer of data storage - you request to load or save whole aggregates. Transactions should not cross aggregate boundaries.
https://www.martinfowler.com/bliki/DDD_Aggregate.html

Those with experience in DDD might understand what that means and why it applies.

But for those starting to get familiar with aggregates, such an explanation might still be too detailed and nuanced.

Let's start looking at what an aggregate is not.

What An Aggregate Is Not

An aggregate is not:

Just a graph of entities
Merely a behaviour rich object
An entity or collection of entities that you can dump into your database tables

So... what is it?

This is usually where people start talking about consistency boundaries, transactional consistency, eventual consistency, aggregate boundaries, invariants, aggregate roots, etc.

When learning about these things, it's natural to grab onto a familiar term or idea when all this jargon is thrown at us. From there, we (falsely) form an idea of what this is all about.

Let's try to keep things simple and practical.

Bubbles

I like to use a very simple idea to help people understand the essence of what aggregates are.

Bubbles.

Imagine your software project was not one massive codebase - but a collection of small bubbles. Each bubble can be worked on independently. That means, you only need to think about what's in the bubble at any given moment - not the entire system all-at-once.

Aggregates are the same. They are bubbles. Just on a smaller scale.

Use Case: Teams

Imagine we are building a new feature for our system. This new feature includes the concept of projects, teams and team members.

Each team can have multiple staff members associated with it.

Staff members can be a part of multiple teams.

Each team can have multiple projects.

Well, that's simple enough.

Let's add the fields that might exist on each object:

Database All The Things

Doesn't that look just like an entity diagram? Don't the database tables scream out at you?

"Obviously", we need to have a composite table linking each team with each team member.

Just hold on.

Let's think about behaviour instead and not treat these as code objects. Let's treat them as business objects or concepts. What can these objects do?

Well, isn't it clear? You can create a team. Edit a team. Delete a team.

Obviously, deleting a team means that all the associated projects ought to cascade and be deleted too.

Note: Let's put aside the fact that these are not the real behaviours of our system. Anytime you see CRUDy language, it should be a red flag!

Real Business Is Not So Simple

But wait. You just found out from your users that this won't work. The assumptions you made about the business were wrong...

There are times when projects are moved from one team to another.

There are times when projects are orphaned for a period of time.

So now, other questions arise:

What should our model look like now?
When we write our code, should we load the entire graph of objects into memory?
What happens when a project is orphaned? Will the teams just have a reference to a null project object?

More Requirements That Complicate Things

Now the business has a new requirement: a team member's role can change per project.

So... do we just create another composite table to match each team member with each project they are on and the role of each project?

That's what we usually do. Developers naturally think about systems in terms of database design first 🤦‍♂️.

Note: Yes, that's a huge problem that domain-driven design tries to help avoid!

With each new requirement, our model gets more bloated. Over time, this might consume lots of memory in our system too.

Imagine a project whose team has 500 members. Yes, these are large projects we're talking about.

We need to load all the staff members into memory, and all their data too!

That will lead to performance issues around memory usage, etc. Is there a better way?

Aggregates

Aggregates are what solve these kind of problems.

They help:

Simplify our models when they start getting out of hand
Isolate complex business rules
Deal with performance issues when loading large object graphs into memory
Allow flexibility to more easily deal with future unexpected business requirements

That's what aggregates are for.

Ok. But what are they?

Instead of telling you, I'll show you what one might look like in this case (otherwise, we need to start talking about consistency, transactional boundaries, concurrency, etc!).

Notice that I split the original Member model into three?

The Team and the Project have their own dedicated "version" or "view" of the Member that only has the exact data it needs to make decisions about business rules and behaviours within it's "bubble".

For example, the Member's role is not needed by the team bubble. Why keep it there when it doesn't belong?

Instead, we have split our model into two "branches". Two bubbles/aggregates, in this case.

To support being able to assign the same Member to multiple teams, we then have to create a dedicated authoritative model of a Team Member and link the other aggregate's Member entity as a foreign key-like reference (again, we aren't talking about databases).

Note: Notice I added specific Ids to the non-authoritative "Member" models (like "TeamMemberId")

There's much more to discuss. And many more improvements we could make to this model around using value objects, etc.

I think this is enough in order to help you see that aggregates are more than simply creating a graph of entities.

It's all about allowing the domain rules to guide you.

Many times, the aggregates we discover are not the aggregates we thought we would need!

Securing Your API With Long-Lived Authentication Keys (With A Deep-Dive!)

James Hickey — Thu, 24 Feb 2022 19:34:29 +0000

There comes a time in the life of a multi-tenant SAAS company when its larger customers need the automation and power of an API. The multitude of reasons includes scaling user provisioning, integrating with other products, facilitating advanced reporting and more.

If you're in this position, one of the most important areas you need to think about carefully is authentication. There are many different factors to consider:

Is your API responding to server-to-server requests? Or will you need to support other types of clients like web applications, mobile, etc.?
Do you need long-lived or short-lived tokens?
What about OAuth? Does that make sense right now?
Should you jump on the JWT bandwagon?
How will your customers configure authentication? How does your choice on this affect their user and developer experience?

At ProcedureFlow, we've spent a fair amount of time researching how the best APIs out there have implemented API authentication: Stripe, Slack, GitHub, Twilio and others. We want to make sure our API is really easy to use, gives superpowers to our customers and is rock-solid secure.

Needless to say, this one area can get overwhelming quickly! To figure out what works best for your company's needs, you have to start by thinking about the requirements of your API.

Minimum Viable API

If you're a startup like us, then you won't have the time and resources to build a complete API that will integrate with "all the things" immediately. You have to identify what API products make sense as a priority. Then, you can build your foundational API infrastructure, that first API product and iterate based on your customer's feedback.

For example, we decided to build a SCIM API first.

Authentication is one of the areas where SCIM is flexible - so we can use the mechanism that works best for our custom API and use it for SCIM too 👍.

Based on our requirements, there were a few smaller decisions we had to make first:

We'd be focusing on server-to-server APIs for the first few API products so this influences what kind of authentication mechanism makes sense for us.
We wanted to retain the ability to implement other authentication mechanisms in the future.
TTFC (Time-to-first-call) is considered one of the most important API metrics. Configuring authentication should be as simple as possible for our customers to facilitate a low TTFC.
How do well-designed APIs like Stripe, Twilio, Slack, Twitter and GitHub do things? What can we learn or implement ourselves?

The Big Decision

Your overall decision will be based on a balance between valuing ease of use for your customers and enforcing solid security.

At ProcedureFlow, we decided to use a long-lived API key that our customers can generate from within our product. This is similar to how Stripe and Twitter V2 do API authentication. We generate both a client_id and a client_secret (more on these later 😉).

Some of the qualities and behaviors of using a long-lived API key includes:

API keys can be tied to individual tenants or specific users like Slack does.
Can be scoped to specific products or features like Stripe does, with read-only or read/write access per feature.
Also inspired by Stripe's approach, customers can have a user-friendly UI to generate new keys, revoke them immediately or revoke them with a grace period.
API secrets should be hidden on the UI, except when first generated. This is mainly inspired by Stripe's approach and is a best practice. You don't want those secrets getting leaked!

Another thing we chose to do is prefix our key values. This makes them easier to identify and makes the type of key/token more explicit. This technique is inspired by Stripe and GitHub - who have recently switched to this format too.

Deep-Dive: Long-lived API Token Authentication

Let's deep-dive into some of the more detailed decisions we had to make and why they might make sense for you too.

The Anatomy Of An API Key

Services like Stripe, Slack and Twilio use two values for their long-lived API keys. Slack, for example, calls these the Client Id and Client Secret. Stripe calls them the Publishable and Secret Keys. Other vendors have used other terms like "Access Key" and "Access Key Secret".

We have chosen to call ours client_id and client_secret. This helps to set us up for future use-cases around delegated authorization (e.g. OAuth) and is consistent with the standard OAuth naming terminology. Also, each token can be used by a specific client - web, mobile, server, etc. so client_id seems more semantically accurate to us than other options like app_id, access_key or public_key.

Public Identifier

The client_id is used to identify the user or owner of the API key.

This is helpful when you may have multiple types of clients that use your API keys - web applications, mobile applications, back-end systems, etc. It also helps when tokens can be assigned to specific users in your system. You can use the client_id to peg rate limits per client, logging metrics that are tied to the identifier, etc.

Secret Token

The client_secret is a random cryptographically generated token.

When thinking about the technical aspects of implementing API key authentication, one of the most important issues to make sure you consider is that API secrets are just like passwords.

It is inappropriate to store these tokens as plain-text for the same reasons you wouldn't store passwords as plain-text. Even worse, API secrets are present for every HTTP request, so you want to make sure to protect them well.

Like passwords, API secrets ought to be hashed when stored in a database or some other tool like a cache. If an attacker ever managed to get access to your database, then they could start impersonating your customers by using plain-text API secrets.

Other Security Considerations

One of the reasons you may not want to only use a "secret" value, even if hashed, is that it becomes both the secret and the lookup value.

When an API request comes into your system, and if all you have is the secret value, then you would have to hash the secret and check if it exists in your database. If it does exist, then you know who owns the API key and therefore who's allowed to perform the API request.

Your SQL query would look similar to the following:

select * from api_auth_keys where secret_hash = $secret_hash

One potential problem with this approach is that it could be susceptible to timing attacks. While in practice there are multiple ways to mitigate these kinds of security vulnerabilities, the issue simply disappears if you use a different value as the identifier for the lookup - assuming you compare the hashes in memory by using a safe constant time comparison function.

Bearer Token

At the moment, we are only using API keys for server-to-server communication. This means our customer's servers will need to communicate directly with our API as opposed to via some delegated third-party system (which would require something like OAuth).

Because of this, we can combine both the client_id and client_secret and display that to our users as one value. This helps with ease of use and user experience, yet allows us to still gain the benefits of using a split token approach.

Best Hashing Algorithm?

PBKDF2 is one of the most popular and reliable hashing algorithms today. It's a great algorithm for password hashing because it's slow and cryptographically sound. You don't want attackers to brute-force guess passwords quickly, so PBKFD2 makes that process much slower and therefore harder to attack.

That's great - but one of the hallmarks of any high-quality API is having a short response time. This affects developer experience, can cause more load on servers (which means paying money for more servers), can affect contractual issues due to not meeting certain SLAs and can affect the overall perception of your products and company.

Is PBKFD2 the best option then?

SHA256 is another common hashing algorithm that is solid - but it's quick to compute hashes. But, is there a way to use SHA256 so that hashes are still computed quickly but have high entropy so that attackers can't brute force API tokens within a reasonable amount of time?

Two core problems with user-supplied passwords are that they are generally short and predictable. So, what you need are API tokens that are long and random!

This is what we did - we opted to generate very long random values for our client_secret - and we hash this using SHA256.
The code for this might look like the following:

Storing API Tokens

You already know that you should store the hash of an API secret. However, you might also want to store the last few digits of your API secret.

Like Stripe, you should only show the raw API secret immediately after it is created. From that point forward, show the last few digits of the token on the UI. This helps customers identify which key they want to revoke, change scopes for, etc.

We opted to follow suit with Stripe and keep 4 digits from our non-hashed API secrets.

The client_id column would also benefit from using an index so that your database lookups performed during incoming API request authentication are fast 🚀.

API Token Pipeline

Whatever technology stack you are using, you'll want to verify any incoming API tokens early in the lifetime of an HTTP request.

While .NET can sometimes get a bad rap, we use it for reasons like being one of the most secure web development platforms out there, having most of the basic tools you'll need for web development out of the box and having a robust type system. We might write more on that topic another time 😅.

.NET's MVC framework has a flexible middleware system that's easy to hook into. We use the built-in middleware (a.k.a. "filters") to process API tokens early in the HTTP pipeline.

Here's the flow of how we process bearer tokens when authenticating against our API:

A client issues an HTTP request with an API token as a bearer token.
We intercept the bearer token early in the HTTP pipeline.
We split the token into client_id and client_secret.
Using client_id we check if a row exists in the database.
We hash the user input client_secret and use a constant-time comparison function to verify that it matches the stored hash.
We create a "session" object that only exists in memory for the lifetime of the HTTP request.
Other events occur downstream that use the API key - like rate limiting.

The core of our API authentication middleware looks roughly like the following:

Conclusion

There's much more you'll have to consider when implementing API authentication and API tokens in particular:

Resource/feature scopes
UI elements
Logging
Tracking token usage for troubleshooting
Database indexing
Rate limiting
Etc.

For example, we had to extend our existing Redis-based rate-limiting to support limiting based on API keys. Originally, limit data was not being made available to the caller. We had to modify our LUA scripts to return rate limit data and then assign values to the appropriate HTTP headers.

We do our best at working fast and smart so that we can deliver value to our customers quickly. But we also care about the details, especially when decisions we make now can have an impact on the future of our products. For example, choosing to prefix our API tokens may seem insignificant at face value, but it affects user experience, the overall security of our API, and is not trivial to change in the future.

If you like solving problems like the ones we've discussed and you're interested in the possibility of joining our team then check out our open positions!

Building a Scalable E-Commerce Data Model

James Hickey — Tue, 29 Dec 2020 21:31:07 +0000

If selling products online is a core part of your business, then you need to build an e-commerce data model that’s scalable, flexible, and fast. Most off-the-shelf providers like Shopify and BigCommerce are built for small stores selling a few million dollars in orders per month, so many e-commerce retailers working at scale start to investigate creating a bespoke solution.

This article will look at what it takes to start building this infrastructure on your own. What are some of the areas to consider? What might the data model look like? How much work is involved?

Along the way, we’ll explore an alternative: API-based commerce platforms that manage data for you across product catalogs, pricing, and orders—without locking you into a monolith, and without requiring you to replatform.

Note: A full summary diagram of the e-commerce data model is at the end of the article.

Who Are Your Customers?

First, you need to consider who will be purchasing items from your e-commerce application. How might you model customer information in a database as a result? You’ll probably want to have basic information like your customer's name, email address, etc. Do you want your customers to be able to create a profile in your system? Or just fill out a form each time they want to purchase something?

Just starting out, a basic model might look like this:

If you want your customers to have a persistent profile, then you need to build some way for them to log in to your application. Moving forward with more real-world requirements, you might also want to keep track of their login attempt history and password history.

You might also want to consider whether your customers are part of a large organization; and, if so, how would they like to handle password resets? Do they need single sign-on or OAuth support?

Deep Dive: Addresses

Did you notice there’s no address tied to a customer in any of the data models shown so far? It might be your first inclination to include a customer’s address as part of those models. However, most customers will have multiple addresses and multiple kinds of addresses, like billing and shipping. B2B retailers might also have to consider multiple delivery locations based on the number of warehouses and offices they support.

What happens if the billing and shipping address are different? Well, you’ll need to do more than just add extra columns to the Customer table! It’s not that simple.

So how does storing a billing address affect the scalability of your application?

If you were to split the payment and shipping areas into separate (micro)services each having their own database, then putting billing and payment addresses into the Customer area would lead to having “chatty” services. This is a well-known design smell when building microservices.

To avoid this issue, you’re better off putting the addresses within the appropriate area/service that requires them, but with that, your data model becomes more complex.

One way to avoid much of this complexity is to consider an order management system (OMS) by an API-first software provider. With this software, you can integrate the OMS into your data model without spending months of engineering time.

How Do You Organize Products And Catalog?

The first thing you see when you enter a store (either in-person or digitally) are products ready for you to purchase, and usually displayed with some thought for how you might be likely to shop.

For an e-commerce web application, you will probably want to highlight things like:

Best selling products
Trending products
New products
The ability to browse products by search criteria
Providing customers with that information means you first need to keep track of a lot of data about your products: their prices, historical purchase data, and so on.

Let’s see what a “first shot” at creating a data model for a product catalog might look like:

Here’s a Product table with some basic information, like a product’s name, SKU, and price. The product is also linked to another table representing various categories that product is associated with. You might also strategically add indexes and full-text search to the Product table to enable site visitors to efficiently search for various products.

This is a decent first attempt. However, to get an even more realistic and useful e-commerce product catalog, you’ll need to support more requirements such as:

Tracking pricing history so site administrators can analyze trends in product pricing
Supporting related products to display on a product’s page
Incorporating product vendors so customers can view all products sold by an individual vendor/company

To address those extra requirements, you might end up with the following data model:

This model still isn’t perfect as it embeds your prices into the product itself, but at least it lets you maintain a previous pricing history.

Another option is to integrate your e-commerce store with a pricing and promotions engine from an API-first software provider that handles pricing for you. This will let you roll out different prices to different users based on their intent, location, cart, or order history.

Deep Dive: Pricing

While the more complex product data model still has a product’s price in the same table, this may not be the best thing to do in a real large-scale application.

Consider that your organization has various departments, such as inventory/warehousing, sales, marketing, customer support, etc. You might have dedicated systems that allow merchandisers to change the price of an item since they are the experts in determining how much a product should sell for. Similar to the considerations with a customer’s billing and shipping addresses, this would lead to cross-boundary/service communication if we left the price in the core Product table.

Therefore, you might want to store product prices under the data stores that the sales department owns. But don’t forget, there are many different kinds of “prices” that haven’t been taken into consideration yet, including:

Price (cost) when purchasing stock from vendors
Customer sale price
Discounted sale prices
Manufacturer’s suggested retail price

Handling all these in context of your organizational structure would require even more exploration and complexity in your data model. While your engineering team could likely accomplish this task, it’s going to take time. Using ready-made solutions can shave weeks or months off your e-commerce data modeling timeline.

How Do You Streamline Orders?

Now that you have customers in your database and products available to purchase, you’ll need to think about how to design the order-taking process and data model.

The process of placing an order might look something like this:

A customer places products into their cart while browsing.
The customer decides they want to purchase the products that are in their cart.
They proceed to purchase the order.
The customer gets an emailed receipt or confirmation number.

However, it’s rarely so simple. Placing orders can be deceptively tricky as there are many moving parts:

Products
An active cart
Cart converted into an order
A finalized order with confirmation

If you were to look at a simple data model for an order placement, it might look something like this:

Notice that each row in the ShoppingCartItem table contains the “captured” price of the product. When the customer puts an item into their shopping cart should the price at that moment be “locked-in”? If so, for how long?

Note: How the price functions is a business requirement that would need to be discussed with your product owners, and so on, as mentioned in the "Deep Dive: Pricing" section earlier.

The same question applies to an unpaid order. If a customer has ordered a discounted item, should they be able to keep the promise of that discounted price forever until they pay? Or does it expire?

Other questions to consider for an orders data model might include:

Are you tracking analytics on orders?
What happens if a customer returns a defective item?
Should you handle shipping within the same data model or have a dedicated shipping context/schema?

With some of these concerns in mind, you might end up with a data model that looks more like this:

Some things to take note of in this more complex orders model:

ShoppingCartItem now supports an expiration date for a locked-in price.
ShoppingCartHistory tracks when items are added, removed, etc.
An order item may be returned (this still does not handle cases where 1 out of X items of the same product are returned).
An order may have multiple shipments (e.g. how Amazon will sometimes split an order up into multiple packages/shipments).

This article also hasn’t even touched the surface of using alternative data storage methods like JSON documents or event sourcing!

Conclusion

To help you see how all the pieces fit together, here are all the diagrams shown together. I’ve removed a number of links/lines to the Customer table to increase readability:

As I mentioned above, this article still doesn’t even cover many of the basics like payment processing and invoicing. Beyond the features covered here, you might eventually require more advanced features like:

Coupon codes
Taxes
Third-party integrations with OAuth providers, other retailers, or partners
Shipment tracking notifications

Building a data model for an e-commerce application, as you can see, is not so simple. What looks upfront to be a straightforward set of database tables is not so simple once you dig into real-world requirements.

There’s Another Way

What if you could have more of these abilities out-of-the-box?

Fabric is an all-in-one commerce platform that helps you do everything this article talked about, like manage customers, orders, and shipments. Most importantly, it is a microservices-based and API-first platform. This means you can choose the services you need and integrate them seamlessly with any other internal or external service.

At a high level, Fabric's platform includes:

A full suite of e-commerce APIs to help you manage customers, orders, and so on
Tools to help you bulk load existing SKUs
An analytics tool that uses customer behaviors to offer customized promotions and pricing
An order management tool that helps you forecast inventory fulfillment and automate payment processing
And more!

Check out Fabric if those sound like abilities your e-commerce application may need.

Using Micro UIs To Extend Legacy Web Applications

James Hickey — Fri, 10 Jul 2020 03:25:53 +0000

Many are coming to learn about micro UIs or micro frontends (which is not a new thing).

In my article about Microservices Architecture, it highlights a big issue with how many think about and design microservices. In most microservice designs, the UI layer isn’t considered as a part of the service.

With this design, you end up coupling the UI layer with all the services.

Team Coupling

By coupling the services to a single monolithic UI, you’ve essentially eliminated the largest benefit for using a microservice design:

The ability for teams to be autonomous and deploy their work independently of other teams.

In other words, a microservice architecture enables loose coupling between teams.

However, teams will conflict at the UI layer when UI concerns aren’t addressed.

Instead, we want to use what the SOA world has been doing for years: give each team the ability to own their UI.

Micro UIs

Micro UIs is an approach where each team or service will create the UI parts that they need for their features and functionality.

Eventually, teams will have to co-ordinate their efforts when integrating their work.

Especially on user-facing web pages, we will have the need to display functionality from multiple teams at the same time. This is un-avoidable.

However, the amount of coupling is minimal.

The approach is to create a collection of individual components per service. These components are then composed together in the UI layer.

The Trouble With Legacy

Many developers have to maintain existing legacy systems. On top of that, they have to continue to add new functionality too!

These systems may have grown out-of-control due to a lack of design & architecture, planning, technical strategy or skillset of the team throughout the years.

In any event, adding new functionality into the system can be very hard.

There are little to no automated tests: how can you be sure you didn’t break any existing code?

Should you change the code that already exists?

Where do you even put the new code?

Chances are, those are difficult questions to answer since most of these legacy systems are very hard to reason about.

Using Micro UIs To Extend Legacy Applications

I’ve been using a technique that comes right out of the SOA world to enhance legacy systems.

It does depend on the kind of change you need to add.

If you need to add, let’s say, a new section of a screen or a new web page altogether then this might work for you.

Case Study

Here’s one example of where I’ve done this successfully.

I was working on a system that had a combination of:

User-facing VB.NET web application
Internal VB.NET web application
Public-facing product marketing web application
.NET Framework web app used to render initial web pages
100% custom-built JS framework with 100% custom-built components (using JQuery)
.NET Web API that would interact with the custom-built JS framework to return JSON, HTML and everything in-between
Various shared libraries between them
One massive shared database behind everything

There are other systems that are missing. But this should give you an idea.

Oh ya, there were no automated tests. Anywhere.

New Features Requested!

As you can imagine, some existing features had logic spread across all of these systems and much of it in the database too.

The custom-built JS framework was also incredibly difficult to work with. What should have taken days to build was taking weeks and even up-to months of work.

I decided that a new approach to building new features was worth trying.

Forming A Strategy

In my mind, there were many issues to address. Relevant to this article, here are the solutions to the most pressing problems:

Stop using the custom-built JS framework because it’s dragging everything down with it
Enable automated testing for all new features
Encapsulate business/domain logic in a common area that can be shared by all the different parts of the system

Slowly, over time, I started applying domain-driven design approaches and techniques to improve things. This involved a lot of learning and having to train other developers.

Here’s a rough sketch of how new features are designed from a conceptual look:

Of course, that’s step one in a massive process of improving things.

This is all excluding a new mobile application and API that I’ve been leading 😅.

How To Integrate?

Some existing features needed to be enhanced. Using this design, we can, as much as possible, direct all new code into the domain project. Existing features will just call into the domain project and use it as needed.

By using refactoring techniques, we have been able to extract existing logic into the domain layer and improve existing functionality and add tests to them!

But what about times when you have entirely new features or products to build?

This is where micro UIs come in.

Instead of integrating directly into the existing legacy codebase, we can build as much as possible into:

The domain project (which includes using DDD approaches and testable code)
Dedicated UI components that interact directly with the domain logic

Here’s a diagram of what this logically looks like:

In practice, it technically looks like this:

In this case, we tried to minimize adding more infrastructure and complexity.

The data flow of each feature is:

Vue.js components are on a shared CDN.
These components send HTTP messages to the existing .NET MVC application.
The MVC application is split up into feature folders. Each feature is essentially a “mini-app” like structure.
Each HTTP handler simply passes the request through to a domain level command or query handler.

About Bounded Contexts

In this case, the core of each of the new features was placed into the same .NET library. This is working with a small team that manages all the systems.

However, it’s worth noting that this is essentially an SOA-like approach, logically.

Some of these new features can potentially be bounded contexts. If so, then they are perfect candidates for becoming a microservice (if ever warranted) or a module in a modular monolith.

The diagram above looks very similar to the one at the beginning of this article about microservices that own their own UI. It’s logically the same thing.

There’s No Recipe

Every team is different. Every situation is different.

Right now, I’m helping this team separate some of these concerns into separate modules acting as bounded contexts.

But, with these large legacy systems, it’s a long process of iterative improvements step-by-step!

You have to work with what you have and face the reality of:

Infrastructural constraints
Technological constraints
Business constraints
Developer skill constraints
etc.

This takes some thinking and design skills.

The Benefits

There’s no doubt though that everyone, especially the business folk, sees the benefit.

Being able to build new features that took weeks/months in days is a huge win
Able to apply automated testing to domain logic produces more predictable behaviours and allows safe refactoring
Able to isolate changes to a cohesive area means less chance of errors breaking unrelated parts of the software
Ability to integrate new technologies like Vue.js into ancient technologies enables a richer user experience and selling point when hiring new developers

Next Steps

Are you working on a legacy application that is really hard to deal with?

Is the business wanting to build new features but, so far, the existing code and design are causing tons of drag?

The techniques in this article might help.

Sometimes, it can be immensely helpful and time-saving to have someone with expertise to help you build a strategy around these kinds of changes and approaches.

If you could use help, then you can get in touch with me.

Till next time!

Interview With Nate Barbettini: OAuth, GraphQL, Event Sourcing

James Hickey — Wed, 25 Mar 2020 14:18:08 +0000

Derek Comartin (@codeopinion) and I interviewed Nate Barbettini about topics like OAuth and why it can be confusing, domain-driven design, graphQL, event sourcing and more!

Nate is the co-founder and CTO of Cobbler and a former Microsoft MVP. He previously worked at Okta as a developer advocate and is the author of the Little ASP.NET Core Book.

We had lots of fun just chatting it up! Check it out if those topics interest you 👍.

Should I Use A Microservices Architecture?

James Hickey — Tue, 25 Feb 2020 19:22:38 +0000

Microservices is a huge topic. However, it’s not a very precise topic.

Martin Fowler agrees:

While there is no precise definition of this architectural style...

https://martinfowler.com/articles/microservices.html

A Sea Of Confusion

Diving into the topic of microservices, you’ll quickly discover that there are many differing opinions.

For example, Martin Fowler suggests that you shouldn’t start a new system using a microservices architecture.

However, his peer Stefan Tilkov suggests that you should always start with a microservices architecture. 🤔

Udi Dahan has believed that microservices, as defined, are still too fine-grained and potentially lack critical concerns around real boundaries and UI considerations.

Who should we believe?

Shifting Sands

Let’s face it: opinions in tech are always changing and are often not as black-and-white as we’d like.

“Best practices” are often shifting every few years.

Once we thought DRY was the best thing since sliced bread. Now we are told that sharing code can be one of the most damaging architectural decisions one could make.

What about microservices?

Microservices And Distributed Systems

Most coverage about microservices is shallow, vague and contradictory.

Sadly, however, there’s not much information that outlines what the microservice style is and how to do it.

https://martinfowler.com/articles/microservices.html

To highlight this fact (it’s a very important fact), let’s look at InfoQ’s Architecture And Design Trends Report for 2019.

Notice that microservices is in the “late majority” section. This means microservices are fairly common in the industry.

Now, look at the 2nd section labelled “Early Adopters.” See that one called Correctly built distributed systems?

As an industry, in general, we’ve been doing microservices, but not well.

When In Doubt, Clarify

However, let’s look at whether you should use a microservices architecture as a general concept.

We’ll begin by asking some fundamental questions about microservices:

What is a microservice?
What is a microservices architecture?
How many microservices should be in my architecture?
How big should they be?
Should my entire architecture be comprised of microservices in order to have a microservices architecture?
What about macro-services? Is there such a thing?
If there is, are they better or worse than microservices? Why?
What about my UI? Should that still be one massive single-page application?
Can I use an MVC based web framework and still be using microservices?
How autonomous should they be?
How does this all relate to the cloud?
Is serverless a kind of microservice architecture?
If I do have a server can I still have a microservices architecture?

These might seem like simple questions. But once you dig a little further you’ll find that the answers are not so straightforward.

What Is A Microservice?

Martin Fowler recognizes that microservices don’t have a precise definition. However, he tries to clarify what a microservice might look like:

...services are independently deployable and scalable, each service also provides a firm module boundary, even allowing for different services to be written in different programming languages. They can also be managed by different teams.

https://martinfowler.com/articles/microservices.html

Given this definition, the typical microservices architecture you’ll see will look like this:

Each microservice can be deployed and scaled on its own. They are running as independent processes and communicate over the network. Each has its own database.

Ok. Now, take this example:

This is a common architecture you’ll find in real companies who’ve been making money for a while.

Imagine that each of these monolithic applications handles separate areas like shipping, billing, etc.

The glaring problem here is that the shared database can lead to all kinds of issues that are typically avoided via proper encapsulation.

What some companies have done is not re-architect their system to a collection of microservices, but applied proper encapsulation to their monoliths (via bounded contexts from domain-driven design).

What’s The Point?

I have a question that exposes the general notion of what microservices are supposed to be:

What’s the difference between the microservices diagram and the properly encapsulated monoliths diagram?

The only difference is that the microservices architecture diagram is sharing the user interface across the entire system.

What I find strange is that in most articles and discussions about microservices the issue of UI is hardly ever mentioned.

Given the typical definition and explanation of microservices, the difference between the encapsulated monolith and “microservice” is not immediately evident.

(What if we removed the UI from some of our monoliths, added a REST API to each and gave them all a shared UI? That sounds awfully similar to the typical microservices architecture diagram you’ll see…)

After all, the biggest reason for moving to microservices is to enable different cross-functional teams to fully own their own products.

This works very well given the properly bounded monolithic approach:

But, the general “microservices architecture” falls short on this:

Who owns the UI? Everyone? No one? Create a new team?

Putting On Our Big Boy/Girl Architectural Pants

The reality is that software systems are much more fluid and dynamic than we would like to think.

It’s harder than we’d like it to be.

Software architecture involves taking many smaller ideas and putting them all together in a way that makes sense for your use case.

It’s not about taking one general idea and applying that to our entire system.

A Practical Example

Imagine a business that has an existing legacy system.

This monolithic system is generally pretty risky to change. The business demands a faster time-to-market for some new features.

Should we rewrite the entire system as a collection of microservices?

It would make more sense to add new functionality via focused and encapsulated services that own dedicated business functions.

This would be more of an evolutionary approach to software architecture (keep reading).

Some Reminders And Side Notes

Again, it’s just not so cut-and-dry. We have to think about our existing systems, where we want the business to be in the near future and apply techniques and approaches that allow our business to keep growing and making money.

Wasting time rewriting your entire system as microservices might not work out so well (1, 2, 3, 4).

A poor team will always create a poor system – it’s very hard to tell if microservices reduce the mess in this case or make it worse.

https://martinfowler.com/articles/microservices.html

And don’t forget that most of us aren’t even building distributed systems correctly anyways:

Awareness of the “microservices” architectural style may be moving into the late majority, but the related themes of “correctly designing distributed systems”, and designing for reactivity and fault tolerance are not so far along the adoption curve.

https://www.infoq.com/articles/architecture-trends-2019/

Evolutionary Architecture

This fluid and more adaptive approach to software architecture is sometimes called evolutionary architecture. This is, in my opinion, just a plain responsible approach to software architecture.

InfoQ has this approach listed in the “early adopters” phase. This indicates that most companies are still just “hammering away” with microservices.

Evolutionary architecture is a broad principle that architecture is constantly changing. While related to Microservices, it’s not Microservices by another name. You could evolve towards or away from Microservices for example.

https://www.infoq.com/podcasts/refactoring-evolutionary-architecture/

Since 2011, evolutionary architecture has been in the “adopt” category of ThoughtWorks’ technology radar.

It’s not really that new.

In contrast to traditional up-front, heavy-weight enterprise architectural designs, we recommend adopting evolutionary architecture. It provides the benefits of enterprise architecture without the problems caused by trying to accurately predict the future… We advocate delaying decisions to the latest responsible moment, which might in fact be up-front for some decisions.

https://www.thoughtworks.com/radar/techniques/evolutionary-architecture

That sounds like doing responsible, reasonable and intelligent software architecture to me 😅.

If this interests you, I’d recommend watching this 16-minute discussion by Randy Shoup.

Step zero is to take a real customer problem, something with real customer benefit, maybe a reimplementation of something you have or ideally some new piece of functionality that was hard to do in the monolith.

Do that in the new way, first, and what you are trying to do there is to learn from mistakes you inevitably will make going through that first transition, right? You do it in a relatively safe way, but at the same time you do with real benefit at the end.

https://www.infoq.com/interviews/randy-shoup-microservices/

As Randy later points out, it’s perfectly fine to leave certain parts of a system untouched if they have no real reason to change.

There were still pages on this site that were on V2 architecture, simply because they continue to work, they got 100,000 hits a day, no big deal, and they were neither painful enough to migrate nor having the sufficient ROI to migrate. So, they just stayed and they happily stayed.

https://www.infoq.com/interviews/randy-shoup-microservices/

About That UI Issue…

You might be wondering about the shared UI that appears in most microservices architecture diagrams.

Is there a way to fix that?

Yes, there is. But it’s not so simple as we’d like.

It boils down to allowing each microservice to own it’s UI components.

This allows individual teams to own the entire stack.

Composite UIs

What happens when we have a UI that needs to display data from multiple services?

The solution is to compose multiple UI components owned by those services.

In many cases, composite UIs are much more complicated than the image above. But this is a starting point to help you understand where this approach leads.

Not The Microservices I Knew

This isn’t what most people think of when they say “microservices.” Microservices usually don’t have the notion of having UI components that are owned by it.

In discussing this issue, Udi Dahan uses an example of a product’s price. Udi points out the fact that specific services need to own their UI. Otherwise, we break encapsulation.

In this manner, no code outside the service boundary would know about the concept of the product price and thus could not end up coupled to it.

...I don’t really see how a [microservice] could end up having this level of ownership of data.

http://udidahan.com/2014/03/31/on-that-microservices-thing/

Size Doesn’t Matter… Or Does It?

You’ll hear that microservices need to be around 1000 lines of code or less. At least, I’ve heard that one before.

However, microservices need to own the data, behaviour and UI of a given business context.

We, as developers or architects, shouldn’t get to magically decide how big our services should be. It’s 100% totally dependent on the business context we are modelling.

In relation to the issues surrounding UI ownership, Udi Dahan points out:

You’ll tend to see services that aren’t all that small, and probably not so many of them.

http://udidahan.com/2014/03/31/on-that-microservices-thing/

Udi is considered one of the most highly respected experts in building distributed systems. And he’s told us – years ago – that we’re doing it wrong.

Microservices are potentially too micro because they aren’t respecting the boundaries of real business contexts. Instead of focusing on size we ought to focus on proper boundaries (which is a topic for another day).

Conclusion

Should you use a microservices architecture?

No.

You should do what makes sense for your business.

For example, your business might want to build new products that require a quick time-to-market.

You could add those new products as entirely separate monoliths that communicate with each other via event messaging to keep them loosely coupled.

They could be built as microservices that integrate back into the existing system via HTTP REST calls or event messaging?

Perhaps there’s an existing off-the-shelf solution that exists which your company can purchase? You could build a custom API on top that will allow integration with the existing system.

There are cases that call for splitting an existing system up into isolated components. They could be microservices.

Or, you could use a plug-in component-like system within a monolithic system.

(I’ve written about an approach to building modular monoliths using .NET Core.)

We haven’t even talked about factors around team skill/competency, project management styles, organizational influences, etc. These are all factors that affect your software architecture!

In conclusion:

It depends.

But, are there guidelines for smaller organizations? Absolutely. And again: the meta-point with all these things is “only solve problems that you actually have”.

https://www.infoq.com/interviews/randy-shoup-microservices/

Need Help?

I help Saas businesses improve their overall agility and time-to-market of new products and features.

I also offer guidance for teams using .NET Core who need help with building web APIs, framework migrations, improving development processes, etc.

Get in touch!

Also, check out my book about keeping your code healthy!

Keep In Touch

Don't forget to connect with me on:

Introducing The Loosely Coupled Show: Architecture And Design Video Chat / Podcast

James Hickey — Fri, 21 Feb 2020 16:03:23 +0000

Are you an intermediate or senior web developer?

Do you recognize that learning about more higher-level and advanced design topics around structuring your software systems, software design, distributed systems, etc. are important and are required if you want to level-up in your career?

For example, our industry is quickly migrating all web-based software to "the cloud."

Do you know how to take advantage of the cloud? How would you even begin to start thinking about how to migrate an existing system to the cloud? Or design a new system well?

Derek Comartin (@codeopinion) and I have started a new video chat / podcast that will help introduce these topics in a conversational format.

We'll have friendly chats about these topics, talk about our own experiences, maybe do some mock design use-cases and, of course, will chat with recognized industry experts from time-to-time.

Check out our first introductory episode, like it and subscribe if you're interested in joining us!

Check it out here!

Modular Monoliths And Composite UIs With .NET Core Razor Class Libraries

James Hickey — Tue, 10 Dec 2019 05:06:10 +0000

Originally published on my blog.

Microservices are all the rage now.

But, many are (finally) realizing that it's not for everyone. Most of us aren't at the scale of Netflix, LinkedIn, etc. and therefore don't have the organizational manpower to offset the overhead of a full-blown microservices architecture.

An alternative to a full-blown microservices architecture that's been getting a lot of press lately is called "modular monoliths."

Shouldn't well-written monoliths be modular anyways?

Sure. But modular monoliths are done in a very intentional way that usually follows a domain-driven approach.

Summary / Table Of Contents

This is part of the 2019 C# Advent! Take a look at all the other awesome articles for this year!

Here are the main sections in the article in case you would like to skip certain parts:

Introduction / Why Modular Monoliths?
Building A Modular Monolith With .NET Core Razor Class Libraries
Building A Composite UI With Blazor/Razor Components

The GitHub repo for everything in this article is here.

Why Modular Monoliths?

The biggest benefits that microservices give us (loose coupling, code ownership, etc.) can be had in a well-designed modular monolith. By leveraging the domain-driven concept of bounded contexts, we can treat each context as an isolated application.

But, instead of hosting each context as an independent process (like with microservices), we can extract each bounded context as a module within a larger system or web of modules.

For example, each module might be a .NET project/assembly. These assemblies would be combined at run-time and hosted within one main process.

Each module would own everything from UI components, back-end business/domain logic to persistence.

Compared to microservices, the benefits of modular monoliths include:

In-memory communication between contexts are more performant and reliable than doing it on the network
A much simpler deployment process for smaller teams
Retain boundaries and ownership around specific contexts
Simplified upgrades to contexts/services
Modules are "ready" to be extracted as services if needed

I see modular monoliths as a step within the potential evolution of a system's architecture:

For domain-driven approaches, starting with a modular monolith might make the most sense and is definitely worth considering.

Here's a more in-depth primer on modular monoliths by Kamil Grzybek if you're interested.

How Can I Build Them?

There are many ways to build modular monoliths!

For example, Kamil Grzybek has created a production-ready modular monolith sample.

I personally prefer less separation within each bounded context / module, but his example is super detailed and worth looking at!

And, of course, Kamil says it well in his repo's disclaimer:

The architecture and implementation presented in this repository is one of the many ways to solve some problems

In this article, we'll be looking at a much simpler approach to building modular monoliths. Again, the direction and implementation depends on your needs (business requirements, team experience, time-to-market required, etc.)

We'll be looking at one way that's unique to .NET Core by using a new-ish feature of .NET Core called razor class libraries. Razor class libraries allow you to build entire UI pages, controllers and components inside of a sharable library! It's the approach I've used for Coravel Pro and enables some exciting possibilities.

Life Insurance Application

I recently wrote an article about using some domain-driven approaches to think about and re-design an insurance selling platform I once worked on.

To keep things simple for now, let's imagine we've determined two bounded contexts from this domain:

Insurance Application
Medical Questionnaire

The specific details are not so important since the remainder of this article will look at implementation and code.

The structure of our solution will roughly look like the following - with a .NET Core web application as the host:

Creating Our Skeleton

Let's implement the skeleton for our modular monolith and use razor class libraries as a way to implement our bounded contexts.

First, create a new root host process:

dotnet new webapp -o HostApp

Next, we'll create our two modules as razor class libraries:

dotnet new razorclasslib -o Modules/InsuranceApplication

dotnet new razorclasslib -o Modules/MedicalQuestions

Then, we'll reference our modules from the host project.

From within the host project:

dotnet add reference ../Modules/InsuranceApplication/InsuranceApplication.csproj

dotnet add reference ../Modules/MedicalQuestions/MedicalQuestions.csproj

Building Our First Module

Navigate to the Insurance Application module at Modules/InsuranceApplication.

Your razor class library will have some sample Blazor files, www folder, etc. You can remove all those generated files.

Let's build a couple of Razor Pages that will be used as this bounded context's UI:

dotnet new page -o Areas/InsuranceApplication/Pages -n ContactInfo

dotnet new page -o Areas/InsuranceApplication/Pages -n InsuranceSelection

You might have to go into your razor pages and adjust the generated namespaces. In my example, I changed them to InsuranceApplication.Areas.InsuranceApplication.Pages.

Oops! It Doesn't Work!

Navigate back to the HostApp web project and try to build it.

It will fail!

Our razor class libraries are trying to use razor pages - which is a web function. This requires referencing the appropriate reference assemblies.

Before .NET Core 3.0, we would have added a reference to some extra NuGet packages like Microsoft.AspNetCore.Mvc, etc. As of .NET Core 3.0, many of these ASP related packages are actually included in the .NET Core SDK.

For some projects, this breaking change can cause issues and confusion! For more details, check out Andrew Lock's in-depth look at this issue.

So, let's change the project files of our two razor class libraries to the following:

<Project Sdk="Microsoft.NET.Sdk.Razor">

 <PropertyGroup>
 <TargetFramework>netcoreapp3.0</TargetFramework>
 <AddRazorSupportForMvc>True</AddRazorSupportForMvc>
 </PropertyGroup>

 <ItemGroup>
 <FrameworkReference Include="Microsoft.AspNetCore.App" />
 </ItemGroup>

</Project>

Yes, I removed all the boilerplate Blazor code since the appropriate references are included in the FrameworkReference.

It's Alive!

Go into the razor pages you created and add some dummy HTML.

For example:

@page
@model InsuranceApplication.Areas.InsuranceApplication.Pages.ContactInfoModel
@{
}

<h1>Insurance Contact Info</h1>

Try running your host application and navigate to /InsuranceApplication/ContactInfo and /InsuranceApplication/InsuranceSelection.

Both pages should display. Cool!

UI Flow: Beginning The Insurance Application

Let's look at building out a basic flow between our two modules.

First, in the HostApp project, in your Pages/Index.cshtml file add the following HTML:

<a href="/InsuranceApplication/ContactInfo">Begin Your Application!</a>

That will give us a link to click from our home screen to begin the flow through our application logic.

UI Flow: Inside The Insurance Application Module

In your ContactInfo.cshtml file within the InsuranceApplication project, insert the following:

@page
@model InsuranceApplication.Areas.InsuranceApplication.Pages.ContactInfoModel
@addTagHelper *, Microsoft.AspNetCore.Mvc.TagHelpers

<h1>Insurance Contact Info</h1>

<form method="post">
 <label>Email:</label>
 <input asp-for="EmailAddress" required type="email" />

 <button type="submit">submit</button>
</form>

In ContactInfo.cshtml.cs, add the following:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;

namespace InsuranceApplication.Areas.InsuranceApplication.Pages
{
  public class ContactInfoModel : PageModel
  {
    [BindProperty]
    public string EmailAddress { get; set; }

    public IActionResult OnPostAsync()
    {
      Console.WriteLine($"Email Address is {this.EmailAddress}.");
      return RedirectToPage("./InsuranceSelection");
    }
  }
}

This will allow us to enter an email address and move to the next page in our flow.

In the InsuranceSelection.cshtml page - just add a link to keep things simple:

<a href="/MedicalQuestionnaire/Questions">Next<a>

You can imagine that this page would allow the user to select a specific insurance plan they want to apply for.

UI Flow: Medical Questionnaire Module

You might have noticed that we never created any razor pages in the Medical Questionnaire module. Let's do that now.

Navigate to the root of the MedicalQuestions project and enter the following from your terminal:

dotnet new page -o Areas/MedicalQuestionnaire/Pages -n Questions

Again, you might need to adjust the generated namespaces.

Within the Questions.cshtml file, add a link:

<h1>Medical Questionnaire</h1>

<a href="/">Finish<a>

Try running your host project with dotnet run and run through the entire UI!

Composite UIs With Blazor Components

When working with these kinds of loosely coupled modules a question arises:

What happens when we need to display information from multiple bounded contexts on the same screen?

Usually, we resort to building composite UIs.

These are UIs where various parts of the UI are rendered and controlled by components owned by a specific bounded context, service or module.

As you can see in the image above, each component might be owned by a different back-end module and would be isolated from and loosely coupled to the other modules.

Let's build a simple composite UI using some exciting new .NET Core 3.0 technologies!

Configure Blazor Components

We need to configure our host application to support the new blazor/razor components.

Add the following script to Pages/Shared/_Layout.cshtml:

<script src="_framework/blazor.server.js"></script>

In Startup.cs in ConfigureServices add:

services.AddServerSideBlazor();

In the Configure method:

app.UseEndpoints(endpoints =>
{
 endpoints.MapRazorPages();
 endpoints.MapBlazorHub(); // Add this one.
});

Create Insurance Application Component

Navigate to the InsuranceApplication project's root directory and execute the following:

dotnet new razorcomponent -o ./Components -n ApplicationDashboard

Replace the contents of your new razor component with the following:

@using Microsoft.AspNetCore.Components.Web

<h3>Application Dashboard</h3>

<p>
    Time: @currentTime
</p>

<button @onclick="Update">Update</button>

@code {
    private string currentTime = DateTime.Now.ToString();

    private void Update()
    {
        currentTime = DateTime.Now.ToString();       
    }
}

Create Medical Questions Component

Do the same steps within the MedicalQuestions module to create a dummy razor component (but change the title of the component).

Putting It Together

Now, back in your host application project, within the Pages/Index.cshtml page, add the following to the top of the file:

@page
@using InsuranceApplication.Components // Add this.
@using MedicalQuestions.Components // Add this.
@model IndexModel

Then, in the middle of your HTML page, add:

<div class="row">
 <div class="col-6">
 @(await Html.RenderComponentAsync<ApplicationDashboard>(RenderMode.ServerPrerendered))
 </div>
 <div class="col-6">
 @(await Html.RenderComponentAsync<QuestionsDashboard>(RenderMode.ServerPrerendered))
 </div>
</div>

Finally... start your host project using dotnet run!

Any time one of the components needs to be changed, the host application will (most likely) not need to change. Both bounded contexts are still isolated from each other but our architecture allows us to be smart about how we display that information.

Note: There are other ways to build these types of components: view components, tag helpers, tag helper components, javascript components, etc.

Other Considerations

Modules can be deployed as NuGet packages so that they ultimately can be managed by independent teams and be isolated as true modules.
Each bounded context can:
- Use its own isolated database
- Have its own self-contained business logic
- Communicate with other bounded contexts by using messaging and domain events
- etc.
In terms of service/module interaction, any calls between modules could use a shared abstractions library which would be configured in DI at run-time to use the concrete service from the appropriate bounded context:

It doesn't show on the diagram, but the Medical Questions context might need to get some information from the Insurance Application context in some scenarios where using messaging isn't appropriate.

In this case, the Medical Questions code would use the IInsuranceApplicationService interface (from DI) to make those calls.

Then, at run-time, the host process would configure the concrete implementation (InsuranceApplicationFacade) to be given to anyone who asks for the interface.

This technique keeps each module loosely coupled and enforces a strict contract in terms of what one bounded context can be explicitly be "asked for".

However, some might say the need to make direct calls to another bounded context is a sign that the boundaries are incorrect...a topic for another day.

Conclusion

I hope you found this article helpful and informative. I'm sure you can see that razor class libraries are a really exciting feature of .NET Core that hasn't been talked about too much.

This is one way to use them and for some cases it might work really well!

Here are some resources about the topics we covered:

My Book

Check out my book about keeping your code healthy!

Keep In Touch

Don't forget to connect with me on:

Refactoring TypeScript: Black Friday Discount 👌

James Hickey — Thu, 28 Nov 2019 02:28:44 +0000

My book Refactoring TypeScript: Keeping Your Code Healthy is on sale until Cyber Monday (Dec. 2, 2019) on Leanpub!

Click on the image to get the coupon/discount:

You can grab one book for $4.99 or 5 books for $13.99 (that would be great for a dev team, club, etc.)

For some samples of the book, you can check out these articles which are excerpts:

Forem: James Hickey

Databases For Web Developers: Indexing

Indexing

Why Indexes Work

Index Basics

Multi-Column Indexes

How And When To Use An Index

Looking up specific records by id

Selecting a subset of records

Conditions On Multiple Columns

Order Matters?

Index columns used in joins

Joins With Foreign Keys

Foreign Key Joins In Update And Delete Statements

Joins Using Non-Foreign Key Columns

Cascading Operations

Index Trade-Offs

Storage

Memory Pressure

Write Performance

More Index Types

My Final Tip

15 Common .NET & C# Blunders Based On My Experience

#1: Async/Await…

async void: doesn’t that just support asynchronous code and the caller can "fire-and-forget"?

Can’t I just put a synchronous call inside Task.Run to make it asynchronous?

Can’t I use Task.Run to make really performant parallelized code?

If I don’t await a Task doesn’t that make my method way more performant?

Isn’t Task.Factory.StartNew the same as Task.Run?

#2: Parallel Processing While Handling HTTP Requests

#3: Trying To Self-Manage Database Connections

#4: HttpClient

#5: Calling ToList() Unnecessarily

#6: Ignoring Database Fundamentals Because The ORM Will "Take Care Of It"

#7: Long-running Or Overnight Jobs That Keep Running Out Of Memory!

#8: Creating A Ton Of Different Separate Projects For One Business/Domain Context

#9: Allowing Unlimited Network Requests To Your APIs

#10: Using PostAsync Instead Of PostAsJsonAsync

#11: Building Proprietary "Frameworks" And Wrappers Around Built-in .NET Functionality

#12: Using Stored Procedures Everywhere

#13: Dogmatically Never Using Stored Procedures

#14: Microservices

#15: Building Your Own Background Job Scheduler

Conclusion

High-Performance .NET CRON Jobs

Code/Repository

Building A Basic CRON Job Process

Increasing Batch Sizes

Adding More CRON Jobs

Real-World Scenario: Running In Limited Docker Containers

Coravel Schedule Workers

More Power!

Distributed Processing

A Potential Issue

Decoupling Distributed CRON Jobs

Separate Scheduling Logic From Job Logic

An Alternate Way To Measure Performance

Basic Scheduler Results

Schedule Workers Results

Distributed Instances

Conclusion

What Are Aggregates In Domain-Driven Design?

What An Aggregate Is Not

Bubbles

Use Case: Teams

Database All The Things

Real Business Is Not So Simple

More Requirements That Complicate Things

Aggregates

Securing Your API With Long-Lived Authentication Keys (With A Deep-Dive!)

Minimum Viable API

The Big Decision

Deep-Dive: Long-lived API Token Authentication

The Anatomy Of An API Key

Public Identifier

Secret Token

Other Security Considerations

Bearer Token

Best Hashing Algorithm?

Storing API Tokens

`async void`: doesn’t that just support asynchronous code and the caller can "fire-and-forget"?

Can’t I just put a synchronous call inside `Task.Run` to make it asynchronous?

Can’t I use `Task.Run` to make really performant parallelized code?

If I don’t `await` a `Task` doesn’t that make my method way more performant?

Isn’t `Task.Factory.StartNew` the same as `Task.Run`?

#10: Using `PostAsync` Instead Of `PostAsJsonAsync`