Forem: Jeremy Likness ⚡️

Geospatial Data: A New Dimension for .NET

Jeremy Likness ⚡️ — Mon, 12 Oct 2020 15:49:00 +0000

Geospatial data refers to any data that represents a location on earth. A traditional relational database model might associate a café with a park using a relational table. What if instead you could store the coordinates of the café along with the coordinates and shape of the park and use geospatial logic to determine their relationship?

In this episode of the EF Core Community Standup, the team talks with Joe Amenta who demos the spatial (GIS) library NetTopologySuite and shows how to use EF Core with SQL to add a new dimension to your app.

You can find the links that were discussed in the show at:

https://www.theurlist.com/efcore-standup-2020-09-30.

Have feedback for the EF Core team? File an issue or share your feedback and suggestions.

Run EF Core Queries on SQL Server From Blazor WebAssembly

Jeremy Likness ⚡️ — Sun, 20 Sep 2020 07:04:59 +0000

It started with a simple idea.

💡 What if I could write a LINQ query on a client the same way I would on a server, and execute it remotely with minimal configuration, setup, ritual and ceremony?

It’s not a new idea. The idea sparks important questions such as:

How do I prevent users from writing dangerous queries that impact performance or return excessive rows?
How do I keep users from calling methods that have negative side effects?
How do I account for differences between database providers?

What if you could write fewer than five lines of configuration code and have something like this work seamlessly … from a Xamarin or Blazor WebAssembly client?

var list = await DbClientContext<ThingContext>
   .Query(context => context.Things)
      .Where(t => t.IsActive == ActiveFlag &&
         EF.Functions.Like(t.Name, $"%{nameFilter}%"))
      .OrderBy(
         t => EF.Property<DateTime>(t, nameof(Thing.Created)))
   .ExecuteRemote()
   .ToListAsync();

The experience of working with data from .NET applications is extremely important to me. OK, it’s technically my job. I built many production line of business applications in C# over almost two decades and have many unique experiences to learn from. A very common problem to solve for any application is how the various clients will connect to the source of truth to query, mutate, and synchronize data. Most operations are straightforward to implement using protocols like REST or gRPC. The surface area of an insert, update, or delete operation is relatively small, even when applying business rules and validation, compared to queries. The query scenario is more complex. There are options to make your life easier (like Swagger and the OpenAPI specification) and you can tap into existing solutions like OData or GraphQL, but those often require significant setup and impose limitations.

To solve problems like this, I like to work backwards from the solution. I mocked some code that represents what I think a quality experience looks like, then set out to make it a reality. I chose to use Entity Framework Core not only because it is a product I help manage, but because it has successfully encountered and solved many problems and edge cases already. Instead of figuring out the database pieces, I can just figure out how to hand off my query to EF Core and let it take care of the rest. I spun up a “pet project” with the intent to learn as much as possible about maintaining an open source project. Even if I weren’t able to solve the problem, I knew I would learn much.

The Solution

The solution involves just two lines of code to setup and enable running queries remotely. Caveat: this code is alpha. There are a few things I know need to be done and likely a lot of things to be completed that I don’t know about yet. I haven’t focused on performance or rough edges. This is what’s possible for a Blazor WebAssembly project with an ASP.NET Core API backend:

Install the middleware on the server (you might need -prerelease and -Version options depending on when you read this)

Install-Package ExpressionPowerTools.Serialization.EFCore.AspNetCore

Add the line of code to Startup.cs to configure the middleware

app.UseEndpoints(endpoints =>
{
   endpoints.MapPowerToolsEFCore<ThingContext>();
   endpoints.MapRazorPages();
});

Install the client on the Blazor WebAssembly project

Install-Package ExpressionPowerTools.Serialization.EFCore.Http

Add the line of code to Program.cs to configure the client

builder.Services.AddExpressionPowerToolsEFCore(
   new Uri(builder.HostEnvironment.BaseAddress));

With that, you’re ready. Assuming you have a DbContext named ThingContext on the server connected to SQLite, SQL Server, MySQL, PostgreSQL or Oracle, you can run queries as if you’re on the server. As I wrote earlier, you can…

var list = await DbClientContext<ThingContext>
   .Query(context => context.Things)
      .Where(t => t.IsActive == ActiveFlag &&
         EF.Functions.Like(t.Name, $"%{nameFilter}%"))
      .OrderBy(
         t => EF.Property<DateTime>(t, nameof(Thing.Created)))
   .ExecuteRemote()
   .ToListAsync();

Notice that I’m using EF Core extension methods and functions as part of the query! Although I haven’t tested it, this should also work perfectly fine with Azure Cosmos DB. It was a very exciting moment to see the Sample project run and return results the first time. I plan to build a more involved sample based on the reference contacts app and MVVM. The rest of this post is about the journey. I won’t cover everything, but it has been an amazing journey the past several months.

Here are just a few things I learned during this project:

What Language Integrated Queries LINQ really are and how they are composed with extension methods and expressions
Expression trees, inside and out
How the .NET type system works
How to take an incredibly complex type and reconstruct it on demand
How to reverse engineer the XML documentation comments bizarre key format and build an algorithm to map members to XML comments
How to uniquely identify members like fields, properties, methods and constructors
How to build ASP.NET Core middleware using endpoint routing
How to build a reusable HTTP-based client using IHttpClientFactory
How to apply a common set of style guidelines across projects using StyleCop and .editorConfig
How to use xUnit.net for unit tests
How to adopt a versioning strategy using tools like Nerdbank GitVersion
How to apply common project attributes
How to automate the generation of useful API documentation through XML comments
How to automate builds, tests, and NuGet packaging with GitHub actions

Explore the Expression Power Tools on GitHub

Let the journey begin!

Comparisons

The first thing I had to tackle is verification. If I write a query on the client, how do I verify it is correctly reassembled on the server? It turns out that comparing expressions is not straightforward. I came across a few libraries in my research but ultimately decided to build a comparison engine on my own. If nothing else, this forced me to dig deep into the internals of expressions.

⭐ Tip : I wish I knew this earlier, but ultimately I found two resources that helped me tremendously with understanding expressions. The first is a single class in the .NET Core source. Expressions provide a surprisingly detailed representation when you call ToString() and that is due to the ExpressionStringBuilder class. Because it must deal with every expression type, it is a single source to understand how to navigate and read parts of expressions. The second resource is the EF Core source code. It takes a while to understand but I consider the code in the EFCore/Query subfolder to be a graduate course curriculum in expressionology.

I chose to go with a recursive approach. I started with the simplest case: the ConstantExpression. This expression holds a value and the value’s type (in case it is null.) The logic to compare two constant expressions looks like this:

Do they share the same reference? If yes, they are equivalent. If not…
Is one or the other null? Then they are not equivalent
The types of the expressions must be equal to each other
If the type of the expression is another expression, apply the equivalency check to the values
Otherwise, try to do a value comparison

It turns out that answering the question, “are these two values equal” without context can be difficult. I settled on a “good enough” approach that looks like this (ValuesAreEquivalent in ExpressionEquivalency.cs):

Reference equals
If values are types, compare types
If values are expressions, compare expressions
If values are MemberInfo (properties, fields, methods, or constructors), compare members
If values are dictionaries, compare the dictionary keys and values
If values implement IEquatable then cast and invoke
If values implement IComparable then cast and invoke
If values are exceptions, compare the types and messages
If values are enumerable, enumerate and compare recursively (except for strings, which are enumerable as characters)
Last resort: Object.Equals

After doing this for several expressions, I realized it is possible to encapsulate the rules in expressions themselves. For example, this is true:

public static Expression<Func<T, T, bool>> True<T>()
   where T : Expression => (_, __) => true;

For combing fragments of rules, here’s a logical OR implementation:

public static Expression<Func<T, T, bool>> Or<T>(
   this Expression<Func<T, T, bool>> left,
   Expression<Func<T, T, bool>> right)
   where T : Expression
{
   var expr = Expression.Invoke(right, left.Parameters);
   return Expression.Lambda<Func<T, T, bool>>(
      Expression.OrElse(left.Body, expr), 
      left.Parameters);
}

The left expression already exists, so it has parameters that point to the expressions to compare. To pass the parameters to the right expression, we create an invocation. This can then be combined with the left expression using the Expression.OrElse method. The trick is that somewhere at the top of the tree there are parameters set with the expressions, so these are passed down as the tree is built. You can utilize existing method calls, like this example:

public static Expression<Func<T, T, bool>> ExpressionsMustBeEquivalent<T>(
   Func<T, Expression> member)
   where T : Expression => (s, t) =>
      eq.AreEquivalent(member(s), member(t));

And build combinations:

public static Expression<Func<T, T, bool>>
   AndExpressionsMustBeEquivalent<T>(
   this Expression<Func<T, T, bool>> rule,
   Func<T, Expression> member)
   where T : Expression => And(
      rule, 
      ExpressionsMustBeEquivalent(member));

Why bother? Because of readability. Here is the rule for lambda expressions:

public static Expression<Func<LambdaExpression, LambdaExpression, bool>>
   DefaultLambdaRules { get; } =
   rules.MembersMustMatch<LambdaExpression>(e => e.Name)
      .AndMembersMustMatch(e => e.TailCall)
      .AndExpressionsMustBeEquivalent(e => e.Body);

It should be fairly clear! The rules are all implemented in DefaultComparisonRules.cs. I’ll talk about “similarity” in a minute.

Compiled expressions vs. “raw code”? What about performance? I haven’t run any performance benchmarks yet, but compiled expressions perform exceptionally well. I will eventually write some benchmarks, but in case it becomes a problem down the line, I maintain a separate rules implementation that is direct code. Ultimately, I’d probably include logic to emit Intermediate Language (IL) from the expressions on frameworks that support it and fallback to code on those that don’t.

Again, this was a fun project to explore expressions and use them.

Sidebar: Lambdas in APIs

Expressions are extremely helpful for building APIs that infer information from strongly typed references. For example, I wrote Ensure.NotNull to validate parameters the old-fashioned, pre “nullable types” way. It’s not necessary because new keywords and attributes like CallerMemberName and nameof can resolve references at compile time, but it was a fun learning experience. Here’s the code to extract the value and name of a parameter using an expression like this: () => parameter:

// extract the name
public static string MemberName<T>(
   this Expression<Func<T>> expr) =>
   ((MemberExpression)expr.Body).Member.Name;

// compile and check the value
public static void NotNull<T>(Expression<Func<T>> value)
{
   var fn = value.Compile();
   if (fn() == null)
   {
      throw new ArgumentNullException(value.MemberName());
   }
}

A more interesting example of this came when I needed a way to identify methods, constructors, properties and fields for the permissions engine. Instead of forcing you to use reflection to find the member, I wanted to provide flexible options, like:

// type and name
var member = helper.OfType<MyClass>()
   .WithName(nameof(MyClass.Property));

// but what if there are multiple overloads, as with a method? How about a template:
var member = helper<MyClass>(
   mc => { mc.DoStuff(1, 2); }); // never called

More on that later. After tackling equivalency, I added a few extension methods and arrived at my first milestone: verifying that expressions match.

[Fact]
public void GivenIEquatableImplementedAndTrueThenAreEquivalentShouldReturnTrue()
{
   var source = new IdType().AsConstantExpression();
   var target = new IdType
   {
      Id = ((IdType)source.Value).Id,
      IdVal = ((IdType)source.Value).IdVal
   }.AsConstantExpression();
   Assert.True(eq.AreEquivalent(source, target));
}

There was a problem, though. For some reason, comparing a complex expression based on a query was failing. And I realized it’s not always practical to compare the full expression. Let’s assume I’m unit testing a view model and want to verify it applies the Skip property to the expression. How can I verify that Skip(Skip) is a part of the expression without having to compare the entire expression?

Unroll the Expression Tree

The first step was to have a consistent way to parse the tree. Expression trees can be very deep and complex, so I wrote ExpressionEnumerator and related extension methods to easily parse the tree. For example, I can extract all of the constants like this:

var constants = expr.AsEnumerable()
   .OfType<ConstantExpression>().ToList();

Even better, if I want to examine the integers in the tree:

var integers = expr.AsEnumerable()
   .ConstantsOfType<int>().ToList();

This let me implement IsPartOf so I can assert:

Assert.True(exprChild.IsPartOf(exprParent));

The child might be the “take” portion of the expression while the parent is the whole query. This would fail with equivalency because the parameters won’t match. Take(10) is a method call with two parameters: the source entities to take from (this is what the extension method starts with), and the number of entities to take. On the other hand, Where(t => t.IsActive).Take(10) has the same value parameter (10) but the other parameter is a method call (Where) that must be resolved to serve as input. Recognizing this, I built a similarity system to make relative comparisons. The IsPartOf implementation unrolls the parent tree and looks for any expressions that are similar:

return targetTree.Any(t => AreSimilar(source, t));

How is similarity implemented?

Similarity

When the Take method is encountered, the following rule is applied:

public static Expression<Func<MethodCallExpression, MethodCallExpression, bool>>
   DefaultMethodSimilarities { get; } =
   rules.TypesMustBeSimilar<MethodCallExpression>()
      .AndTypesMustBeSimilar(
         e => e.Method.DeclaringType)
      .AndMembersMustMatch(e => e.Method.Name)
      .AndMembersMustMatch(e => e.Arguments.Count)
      .AndIf(
         condition: (s, t) => s.Object != null,
         rules
            .SourceMustBePartofTarget<MethodCallExpression>(
               e => e.Object))
      .And((s, t) => 
          ExpressionSimilarity.ArgumentsAreSimilar(
             s.Arguments,
             t.Arguments));

The method must be the same, but arguments only need to be similar. Ultimately we end up with the rule for constants. The rule eventually uses equivalency to compare the values, but there is a concession in the code:

var type = source.GetType();
if (type.IsGenericType && 
   type.GetGenericTypeDefinition() == typeof(EnumerableQuery<>))
{
   var targetType = target.GetType();
   return targetType.IsGenericType &&
      targetType.GetGenericTypeDefinition() == 
         typeof(EnumerableQuery<>) &&
     !targetType.GenericTypeArguments
         .Except(type.GenericTypeArguments).Any();
}

This basically checks to make sure the query is of the same type. This allows the expression to match independent of the implementation. Because we only need the parameter to be “a part of”, the fragment with just Take is compared to the Where method, which has its own parameter that is a queryable, and those arguments will match.

[Fact]
public void 
GivenTargetWithSimilarQueryPartsWhenIsPartOfCalledThenShouldReturnTrue()
{
   var target = query.Skip(2).Take(3);
   var source = target.CreateQueryTemplate().Take(3);
   Assert.True(eq.IsPartOf(
       source.Expression, 
       target.Expression));
}

That’s a ✅ passing test.

Hosts and Providers

In building the core library I decided it was important to be able to take a snapshot of a query when executed (useful for validating tests) and mutate a query (useful for doing things like security checks). I imagined I would have a set of rules to apply against an expression to transform it into compliance. It turns out when I finally implemented the serialization, I was able to apply rules as I rebuilt the expression. That’s fine because the capability is still there. I wrote about this at length in Inspect and Mutate IQueryable Expression Trees. The library provides a generic QueryHost that allows you to track additional metadata about a query. I use it in my solution to track the original EF Core source so I can route the request appropriately. There are two built-in providers as well.

The QuerySnapshotProvider exposes a callback that is invoked right before a query runs. You can snapshot for logs or even inspect it and throw an error to prevent execution. The QueryInterceptingProvider goes one step further and allows you to transform the query (for example, strip out unwanted method calls or add business logic automatically).

A query is nothing more than a wrapper around an expression tree that provides a GetEnumerator() method for iterating the result. The provider is what makes it happen. The provider has an Execute(Expression) method for compiling and running the expression tree against the source. This is where you can tap into the pipeline. There is also a CreateQuery(Expression) method that turns an expression into a queryable. This happens during the course of processing existing expression trees that project to other types. If you write a query like this:

var query = things.Select(t => new SubThing(t.Id));

The query starts out as an IQueryable<Thing> but ends up as an IQueryable<SubThing>. When the projection is encountered, the provider will create a query with the new type for the subtree. Another place this is useful is in a serialization pipeline. I serialize the expression tree and use CreateQuery to turn it back in the query that EF Core runs.

Dependency Injection

It quickly became evident I needed a dependency injection system. Instead of taking a dependency on an external provider, I decided to implement my own. Again, this was mostly a learning experience, but I also wanted to minimize dependencies and make it more self-contained so that overriding defaults is the exception, not the null. The system allows for registering types that are implemented on the fly, and singletons. I may have made a bad decision when I chose to implement the solution as a static host, but it was the easiest way to make it globally available. The caveat is that I have to run tests in single-threaded to avoid conflicts with the services mutating as part of test runs. I am considering refactoring to allow partitions as part of Issue #20 but for now I haven’t encountered major issues outside of testing.

The ServiceHost contains a static reference to a Services instance. The services are a simple dictionary for mapping types and singletons. To avoid conflicts I chose to create a “registration window” for registering services that doesn’t allow mutations after. The goal is not to allow users to override services, but to easily define them internally and provide a way to mock or substitute for tests. This is why the “service domain” concept is appealing to me. By default, the services are loaded from the static constructor. You can call Initialize to register overrides. This passes back an interface with the registration methods that is can be chained to register in one pass:

ServiceHost.Initialize(register =>
{
   var evaluator = new ExpressionEvaluator();
   register.RegisterSingleton(Services)
      .RegisterSingleton(DefaultRules)
      .RegisterSingleton<IExpressionEvaluator>(evaluator)
      .Register<IExpressionEnumerator, ExpressionEnumerator>()
      .RegisterGeneric(
         typeof(IQuerySnapshotHost<>), 
         typeof(QuerySnapshotHost<>))
      .RegisterGeneric(
         typeof(IQueryHost<,>), 
         typeof(QueryHost<,>))
      .RegisterGeneric(
         typeof(IQueryInterceptingProvider<>), 
         typeof(QueryInterceptingProvider<>))
      .RegisterGeneric(
         typeof(IQuerySnapshotProvider<>), 
         typeof(QuerySnapshotProvider<>));
});

The service exposes a GetService and a recommended GetLazyService method. The latter allows code to reference the service but not actually implement it until it is used to potential avoid conflicts during registration.

Satellite Assemblies

When I started the second project for serialization, I immediately realized I needed a way for dependent assemblies to register their own services. I decided the easiest way is to simply implement an interface. When ServiceHost is initialized, it scans the app domain assemblies for types that implement IDependentServiceRegistration and calls RegisterDefaultServices. By convention, you’ll see a file in the root of my projects named Registration that implements the interface to register dependencies. I also added an AfterRegistered hook. This allows scenarios such as initializing configuration and default options after the services are registered. For example, the ASP.NET Core middleware uses this to allow EF Core extension methods:

public void AfterRegistered()
{
   ServiceHost
      .GetService<IRulesConfiguration>()
   .RuleForType(
      typeof(Microsoft.EntityFrameworkCore.EF))
   .RuleForType(
      typeof(Microsoft.EntityFrameworkCore.DbFunctionsExtensions));
}

But now I’m getting ahead of myself. Before we can talk about serialization rules, we should talk about serialization!

Serialization

As with everything else, I took a step-by-step test-driven approach and started with the easiest option: ConstantExpression. There are just two parts: a value and a type. What could possibly go wrong?

It turns out that types aren’t as easy as they may seem. You can’t simply save a type name and easily recreate it. Simple types are fine, but types that have generic parameters are another story. I built a system that seems to work, but I also am spiking a different solution that shows promise. I’ll cover what’s there now in a moment. The strategy I decided to go with was to create some complementary classes to expressions that are easy to serialize. The first class is the base class for all of these building blocks: SerializableExpression. It contains just one thing: the ExpressionType. This is exactly what’s needed to construct a new expression. Some types map one-to-one, such as Constant. Other types like Add and And both belong to BinaryExpression.

The unit of serialization is the Constant class. This holds a type for the constant, a value, and a type for the value. Why the difference? It turns out you can potentially have a constant with an interface type. The value obviously can’t be an interface but must be an implementation of the interface. The value type allows me to deserialize the value while the constant type enables recreating the expression. This is done recursively in the ConstantSerializer. It’s recursive because the value of the constant can be another expression.

You’ll notice that the serialization uses System.Text.Json. I may refactor it later to an encoded stream that can be transported via other methods, but for now this was the simplest approach. I wanted to minimize dependencies on third-party libraries and take advantage of the performance of the library. Let’s talk types.

Types

To capture types, I came up with a hierarchical set of primitives named SerializableType with a handful of properties (don’t look for it in the code, it’s gone because I’ve already refactored it):

FullTypeName was a human-readable, friendly look at the full hierarchy including generic type arguments
TypeName was the machine-readable name used to generate the type
GenericTypeArguments were used to find the right generic type or close the generic type as needed

During serialization, the SerializationState maintains a list of types. Unless the option is disabled, types are compressed. This means their index is swapped for their full name. Consider a list like this:

0 IQueryProvider<T>
0->0 - T
1 IQueryHost<T, IQueryProvider<T>>
1->0 - T
1->1 - IQueryProvider<T>
1->1->0 -- T
2 CustomProvider<T>
2->0 - T
3 QueryHost<T, CustomProvider<T>>
3->0 - T
3->1 - CustomProvider<T>
3->1->0 -- T
4 T
5 QueryHost<string, CustomProvider<string>>
5->0 - System.String
5->1 - CustomProvider<string>
5->1->0 -- System.String
6 CustomProvider<string>
6->0 - System.String
7 System.String

After compression, the list looks like this:

0 IQueryProvider<T>
0->0 - ^4
1 IQueryHost<T, IQueryProvider<T>>
1->0 - ^4
1->1 - ^0
2 CustomProvider<T>
2->0 - ^4
3 QueryHost<T, CustomProvider<T>>
3->0 - ^4
3->1 ^2
4 T
5 QueryHost<string, CustomProvider<string>>
5->0 - ^7
5->1 - ^6
5->1->0 -- ^7
6 CustomProvider<string>
6->0 ^7
7 System.String

This is easily compressed and decompressed.

The New Type System

It was frustrating me to have so much overhead for types and a different system to serialize types verse other members like constructors. I realized the XML doc comments already have a workable algorithm to uniquely identify types. So, I reverse engineered the algorithm to implement a new member management system. Now all members are represented as strings. Base types map directly to nodes in XML comments. For example:

Here is the representation of a generic type.

typeof(IThing<,,>)

"T:ExpressionPowerTools.Core.Tests.MemberAdapterTests+IThing`3"

The single back tick means it’s a type with three generic parameters. This is how the base system works. I extended it to allow me to reference closed types as well, like this:

typeof(IThing<ThingImplementation, IComparable<ThingImplementation>, char>)

"T:ExpressionPowerTools.Core.Tests.MemberAdapterTests+IThing{ExpressionPowerTools.Core.Tests.MemberAdapterTests+ThingImplementation,System.IComparable{ExpressionPowerTools.Core.Tests.MemberAdapterTests+ThingImplementation},System.Char}"

Notice that closed types may be nested. Methods with generic method parameters get two back ticks. Here’s a dummy method to demonstrate:

public static TResult Result<T, TResult>(
   T entity, string name) => default;

Representation:

"M:ExpressionPowerTools.Core.Tests.MemberAdapterTests.Result--2(--0,System.String)"

(Replace the dashes with back ticks, I couldn't figure out how to escape them). The two back ticks mean there are two closed types. In the parameters, the 0 refers to the first closure. So where is the second? If it’s not in the parameters, it’s the return type. Knowing that, I can “close” the method:

Expression<Func<int>> closedResultExpr = 
   () => Result<MemberAdapterTests, int>(
      new MemberAdapterTests(),
      nameof(indexer));

And serialize it like this:

"M:ExpressionPowerTools.Core.Tests.MemberAdapterTests.Result{System.Int32}(ExpressionPowerTools.Core.Tests.MemberAdapterTests,System.String)"

If you want to have fun seeing how I stretched the limits of my ability to figure out an algorithm, check out the member adapter. I plan to add extension methods to make it easier for others to use, for example when generating documentation. I’m confident I have some optimization opportunity there. I added some cache and improved performance 50% but my spike to change string manipulation to use Span<T> did not perform better so I pulled it to minimize dependencies.

Members

Members are tricky to serialize. Members include types, methods, constructors, properties, and fields. It’s fairly straightforward to ask for ToString() on the System.String type, but what if you have IMyType<T, TImpl> where TImpl : IOther<T> and then a method that is TImpl SetImplementation<Z>(Z input) where Z : IRoutable<T>? It can get very complex quickly. The initialize approach I settled on before I figured out the universal method system was to create unique building blocks per member, then expose a “key” that is essentially everything unique about that member, combined into a string. That way I can verify I’ve deserialized to the right member and use properties to build implementations of generic types. Consider a constructor:

[Serializable]
public class Ctor : MemberBase
{
   public Ctor()
   {
   }
   public Ctor(ConstructorInfo info)
   {
      DeclaringType = SerializeType(info.DeclaringType);
      MemberValueType = DeclaringType;
      ReflectedType = SerializeType(info.ReflectedType);
      IsStatic = info.IsStatic;
      Name = $"{DeclaringType.FullTypeName}()";
      Parameters = info.GetParameters().Select(
         p => new
         {
            p.Name,
            Type = SerializeType(p.ParameterType),
         })
     .ToDictionary(p => p.Name, p => p.Type);
   }
   public override int MemberType
   {
      get => (int)MemberTypes.Constructor;
      set { }
   }
   public bool IsStatic { get; set; }
   public string Name { get; set; }
   public Dictionary<string, SerializableType> 
      Parameters { get; set; }
      = new Dictionary<string, SerializableType>();
}

It contains the parent type, the static flag, and parameters. The “key” for this looks like;

public override string CalculateKey() =>
   string.Join(
   ",",
   new[] { "C:" }
   .Union(Parameters.Keys.ToArray())
   .Union(Parameters.Values.Select(p =>
   GetFullNameOfType(p)).ToArray())
   .Union(new[]
   {
      GetFullNameOfType(ReflectedType),
      IsStatic.ToString(),
      GetFullNameOfType(DeclaringType),
      Name,
   }
));

To deserialize, I found the parent type, applied any type arguments needed, then search constructors to find a match. I’m currently working on a new approach that may simplify things in the future.

Now I just use the new universal string system I explained earlier. So much easier!

Security

The biggest concern with an approach to serializing queries is security. A malicious user could issue a purposefully complex query designed to hinder performance. The attacker may try to invoke methods on the server that involve file I/O. To preventthis, I built an opt-in security model. The queries can access fields and properties but by default are not authorized to use methods or constructors. You can further ban entire types you don’t want to expose. The rule builder is fluent and hierarchical. For example, the default rules ensure you can query and enumerate, manipulate strings and dates, and use the math library. The only thing allowed on the base object class is ToString().

rules.RuleForType(typeof(Math))
   .RuleForType(typeof(Enumerable))
   .RuleForType(typeof(Queryable))
   .RuleForType<string>()
   .RuleForType<DateTime>()
   .RuleForMethod(
      selector =>
      selector.ByNameForType<MethodInfo, object>(
         nameof(object.ToString)));

The EF Core libraries also allow some EF Core extension methods:

public void AfterRegistered()
{
   ServiceHost.GetService<IRulesConfiguration>()
   .RuleForType(
      typeof(Microsoft.EntityFrameworkCore.EF))
   .RuleForType(
      typeof(
      Microsoft.EntityFrameworkCore.DbFunctionsExtensions));
}

For performance, rules are cached as they are evaluated. I suspect this is the area I’ll spend the most time on to take this to production. Security is incredibly important, especially hardening the application to prevent access to types that expose sensitive data. I am thinking about denying properties and fields by default as well, then allowing assembly-level config so you can opt-in your model project, etc.

The Selector

As I mentioned earlier, finding the actual definition of a member can be hard. For this reason, I build a selector to make it easier to define types. If you want all overloads of a method to be available, you can use this convention:

SelectMember<MethodInfo>(selectMethod =>    
   selectMethod
      .ByNameForType<MethodInfo, SelectorTests(
         nameof(SelectorMethod)));

If you want a specific member, you can use a resolver template. This mimics a call but is never actually executed and only used to find the member. Here’s an example that resolves a specific constructor overload:

SelectMember<ConstructorInfo>(selectCtor =>
    selectCtor
      .ByResolver<ConstructorInfo, Nested>(
         n => 
            new Nested(
               nameof(Nested).GetHashCode())));

That will find the constructor that takes an integer parameter. This is done, of course, using expressions. The lambda expression is “unrolled” using the enumerable extension, then the relevant expression that references the member is extracted.

private static void Resolve<T, TTarget>(
   MemberSelector<T> memberSelector,
   Expression expr)
   where T : MemberInfo
{
   if (typeof(T) == typeof(MethodInfo))
   {
      var methodCall = expr.AsEnumerable()
      .OfType<MethodCallExpression>().FirstOrDefault();
      memberSelector.Member = 
         new[] { methodCall.Method as T };
   }
   else if (typeof(T) == typeof(ConstructorInfo))
   {
      var ctor = expr.AsEnumerable()
      .OfType<NewExpression>().FirstOrDefault();
      memberSelector.Member = 
         new[] { ctor.Constructor as T };
   }
   else
   {
      var memberAccess = expr.AsEnumerable()
      .OfType<MemberExpression>().FirstOrDefault();
      memberSelector.Member = 
         new[] { memberAccess.Member as T };
   }
}

At this point I hit my goal and was able to successfully serialize, deserialize, and execute queries. However, when it came to testing across machines, something interesting came up. An expression like this would invariably fail:

var filterActive = true;
var activeFilter = false;
var skip = 5;
var query = things.Where(
   t =>
   (filterActive && 
   t.IsActive == activeFilter) || !filterActive)
   .Skip(skip)
   .Take(10);

Can you spot the challenge?

Let’s unroll the query. Here it is in a Jupyter notebook. I’ll follow up with more on notebooks in another post.

I installed the power tools and unrolled the query. Notice the MemberAccess nodes.

The variables filterActive and activeFilter are defined in my scope. If I serialize the query and send it to the server, there is no way to reference the variables! Instead, I have to transform the expression tree to pull out what I can.

Compression and Partial Compilation

To prepare the query for serialization, I first compile it to compress types. Local variables can be resolved to constants, but parameterized expressions must remain (these are ones that will be passed query entities or sent to the database for evaluation). Basically, when you see t => t.Id == myVariable the t.Id must be resolved later, whereas myVariable can be resolved now. The process takes multiple passes. First, a nominator recursively iterates expressions to find what candidates exist.

public override Expression Visit(Expression expression)
{
   if (expression != null)
   {
      bool saveCannotBeEvaluated = cannotBeEvaluated;
      cannotBeEvaluated = false;
      base.Visit(expression);
      if (!cannotBeEvaluated)
      {
         if (expression.NodeType != 
            ExpressionType.Parameter)
         {
            if (expression.Type.Namespace != null
               && expression.Type.Namespace
            .StartsWith(
               $"{nameof(System)}.{nameof(System.Linq)}"))
            {
               cannotBeEvaluated = true;
            }
            else
            {
               candidates.Add(expression);
            }
         }
         else
         {
            cannotBeEvaluated = true;
         }
      }
      cannotBeEvaluated |= saveCannotBeEvaluated;
   }
   return expression;
}

The code is looking at a branch on the tree. If the branch has a parameter, it is not a candidate. If it doesn’t have parameters, it can be collapsed. Consider this:

BinaryExpression Equals
Left: Member Access t => t.Id (parameter)
Right: Member Access myVariable

The BinaryExpression has parameters so it is not a candidate. The left uses those parameters, so it is not a candidate. The right doesn’t take parameters as it simply accesses the variable directly, so it is tagged as a candidate. I also disqualify LINQ methods because those may try to evaluate expressions too soon (like Sum or Count). The candidates are then resolved:

LambdaExpression lambda = Expression.Lambda(e);
Delegate fn = lambda.Compile();
return Expression.Constant(fn.DynamicInvoke(null), e.Type);

This turns the variable into a constant with the variable value. The final passes simplify nodes. I’m still expanding on this, but consider the following:

BinaryExpression AND
Left BinaryExpression Equals
Left myVariable
Right myOtherVariable
Right BinaryExpression Equals
Left Member Access t => t.IsActive
Right Member Access myVariable

Let’s assume myVariable is false and myOtherVariable is true. In the evaluation pass it will transform it to look like this:

BinaryExpression AND
Left BinaryExpression Equals
Left false
Right true
Right BinaryExpression Equals
Left Member Access t => t.IsActive
Right Constant false

We won’t crunch the right side due to parameters. BUT we can certainly collapse the “equals” node:

BinaryExpression AND
Left false
Right BinaryExpression Equals
Left Member Access t => t.IsActive
Right Member Access myVariable

According to the rules for logical AND operations, if one side is false, the entire expression is false. Therefore, I can turn this node into:

Constant false

Here’s the code that parses an AND expression:

private Expression CompressAnd(BinaryExpression node)
{
   if (TryCheckBoolean(node.Left, out bool left))
   {
      if (!left)
      {
         treeModified = true;
         return Expression.Constant(false);
      }
      if (
         TryCheckBoolean(
            node.Right, 
            out bool innerRight))
      {
         treeModified = true;
         return Expression.Constant(innerRight);
      }
      treeModified = true;
      return Visit(node.Right);
   }
   if (TryCheckBoolean(
         node.Right, 
         out bool right))
   {
      if (!right)
      {
         treeModified = true;
         return Expression.Constant(false);
      }
      treeModified = true;
      return Visit(node.Left);
   }
   return base.VisitBinary(node);
}

private bool TryCheckBoolean(
   Expression expression, 
   out bool value)
{
   if (expression is ConstantExpression ce
      && ce.Type == typeof(bool))
   {
      value = (bool)ce.Value;
      return true;
   }
   value = false;
   return false;
}

Because the expression is now a constant, it is possible its parent can be simplified, too. The algorithm iterates multiple times until no expressions are modified. That closes the loop. After implementing the compression I could successfully serialize a query and deserialize it. Time to move the code into the Blazor scenario!

ASP.NET Middleware

I chose to go with ASP.NET Middleware and allow you to set up a route. To support multiple contexts, the endpoint includes the type of the DbContext and the name of the collection. A set of handlers parse the route information and map to the appropriate type and set. Assume you have a context named ThingsContext with a DbSet<Thing> named Things. The default routing will POST the serialized query to /efcore/ThingsContext/Things. We can use reflection to get the property for the collection:

if (typeof(DbContext).IsAssignableFrom(context))
{
   var match = collection.ToLowerInvariant().Trim();
   dbSet = context.GetProperties().FirstOrDefault(
      p => p.Name.ToLowerInvariant() == match
      && p.PropertyType.IsGenericType
      && typeof(DbSet<>)
         .IsAssignableFrom(
         p.PropertyType.GetGenericTypeDefinition()));
   return dbSet != null;
}

Then we can build the AsQueryable extension to get an IQueryable. Here collection is the PropertyInfo of the collection we obtained in the previous code.

var dbSet = collection.GetValue(context);
var asQueryable =
dbSet.GetType()
   .GetMethod(nameof(Queryable.AsQueryable));
return asQueryable.Invoke(dbSet, null) as IQueryable;

Next, we can take the serialized query and build it on top of the IQueryable we just generated.

var request = 
   await JsonSerializer
      .DeserializeAsync<SerializationPayload>(json);
var query = Serializer.DeserializeQuery(
               template, 
               request.Json);

Finally, we execute the query by applying a method like ToArray and serialize the results:

private readonly MethodInfo toArray =
   typeof(Enumerable).GetMethods()
   .First(m => m.Name == nameof(Enumerable.ToArray)
      && m.IsGenericMethodDefinition
      && m.GetParameters().Length == 1);
var typeList = new[] { query.ElementType };
var parameters = new object[] { query };
var arrayMethod = toArray.MakeGenericMethod(typeList);
result = arrayMethod.Invoke(null, parameters);
var json = JsonSerializer.Serialize(result);
var bytes = Encoding.UTF8.GetBytes(json);
return response.WriteAsync(bytes, 0, bytes.Length);

Running the sample app, we can see the serialized query going out and the serialized result coming in.

A Client Approach

The last step is to tie things together on the client. I wanted to allow you to shape your queries based on the existing DbContext, so I built a DbClientContext<T> where T: DbContext. The entry into a query is a lambda expression that references the DbSet like this:

var myQuery = DbClientContext<ThingsContext>(
   t => t.Things)
   .Where(t => t.Id...)

The query uses a custom query host and provider that work like any other IQueryable but capture the context and collection. Instead of risking collision with existing extension methods, I decided to use ExecuteRemote to cast to the IRemoteQueryable<T> interface and cast the extensions from that. For example, here is the code for a query that returns a list:

public static Task<IList<T>> ToListAsync<T>(
   this IRemoteQueryable<T> query)
{
   Ensure.NotNull(() => query);
   return resolver.Value
      .ToListAsync(
         query as RemoteQuery<T, RemoteQueryProvider<T>>);
}

The resolver is an interface that is implemented using HttpClient but could be extended to use another protocol like gRPC-Web. The main work is done in this method:

if (!(query is IRemoteQuery remoteQuery))
{
   throw new ArgumentException(
      $"{query.GetType()} => {typeof(IRemoteQuery)}",
      nameof(query));
}
var compressedQuery =
   query.Provider.CreateQuery<T(
      compressor.EvalAndCompress(query.Expression));
var json = Serializer.Serialize(compressedQuery);
var payload = new SerializationPayload(type)
{
   Json = json,
};
var options = GetJsonSerializerOptions();
var transportPayload = JsonSerializer
                     .Serialize(payload, options);
var requestContent = new StringContent(transportPayload);
var path = PathTransformer(remoteQuery);
var client = GetHttpClient();
var contentJson = 
   await client
         .FetchRemoteQueryAsync(
            path, 
            requestContent);

The query is compiled, serialized, and sent over the wire and response is fetched from the middleware.

Summary

I have learned quite a lot on the journey to build a seamless client for EF Core. The journey still continues, and I’ll post my findings as the project evolves. A mentor once told me, “It takes a lot of technology to create the illusion of simplicity.” Although that’s true, it’s also true that sometimes a simple, iterative approach is best way to tackle complex problems. Start at the desire result and work backwards. I hope that sharing this journey gives you knowledge and insights and that you continue to learn more about and appreciate the power of expressions.

Explore the Expression Power Tools on GitHub

Regards,

Synchronize Databases with Only Five Lines of Code

Jeremy Likness ⚡️ — Fri, 04 Sep 2020 13:44:31 +0000

DotMim.sync is a framework for synchronizing relational databases. It is a .NET Standard 2.0 library, which means it will run on multiple platforms including Windows, Linux, iOS and Android (via Xamarin) and more. It is production-ready and has been evolving for nearly a decade. The following code sample shows how easy it is to get started. This will automatically sync a client, including creating the tables and populating the data if the database doesn't already exist.

// Sql Server provider, the "server" or "hub".
SqlSyncProvider serverProvider = new SqlSyncProvider(
    @"Data Source=.;Initial Catalog=AdventureWorks;" +
    "Integrated Security=true;");

// Sqlite Client provider acting as the "client"
SqliteSyncProvider clientProvider = new
    SqliteSyncProvider("advworks.db");

// Tables involved in the sync process:
var tables = new string[] {
    "ProductCategory", "ProductDescription", 
    "ProductModel", "Product", 
    "ProductModelProductDescription",
    "Address", "Customer", "CustomerAddress", 
    "SalesOrderHeader", "SalesOrderDetail" };

// Sync agent
SyncAgent agent = new SyncAgent(
    clientProvider, serverProvider, tables);

do
{
    var result = await agent.SynchronizeAsync();
    Console.WriteLine(result);

} while (Console.ReadKey().Key != ConsoleKey.Escape);

In this episode of the EF Core Community Standup, the team talks with creator Sébastien Pertus who shares the project's history and demos several scenarios that illustrate how fast and easy the framework is to use.

You can find the links that were discussed in the show at:

https://www.theurlist.com/efcore-standup-2020-09-02.

Have feedback for the EF Core team? File an issue or join one of our many online discussions.

Take Control of your Queries: Inspect and Mutate IQueryable Expression Trees

Jeremy Likness ⚡️ — Sun, 16 Aug 2020 07:11:25 +0000

In the last blog post of this series, I shared a look behind the IQueryable curtain. Using a special “query host,” I demonstrated how to iterate the expression tree, and even modify it, so the query executes differently. The problem is that the approach is limited because inspection/modification only works when you are in control of the end-to-end query. You lose control and insights when the query is handed off, extended, and/or run again. How do products like EF Core allow you to write whatever queries you like, then successfully intercept them to run SQL commands? The secret is in the provider.

To help explain these concepts, I built an example project that you can browse and download here:

JeremyLikness / QueryEvaluationInterceptor

An example of intercepting IQueryable executions to parse and transform the query.

QueryEvaluationInterceptor

An example of intercepting IQueryable executions to parse and transform the query.

View on GitHub

The “query host” is really just an implementation of IQueryable. The basic properties on the interface include:

The ElementType the query is for
The Expression that represents the definition/structure of the query
The Provider to implement the expression and materialize the results

Think of the query host as the container that queries are built against. This is the definition. The IQueryProvider is the implementation. This should be evident by the main methods on the interface:

CreateQuery that is responsible for producing the IQueryable the expression represents.
Execute that is responsible for running the query.

As long as you are defining filters, projections, sorts, and other parts of a query (for inspiration, you can take a look at the various Queryable methods) you are working with the host. The instant you start to iterate the results of a query, the provider comes into play. For default “LINQ to Objects” this involves sorting and filtering objects in memory. For EF Core, this involves generating the database statements that make sense for the provider being used, such as SQL.

The general flow looks like this:

Build the query: apply extension methods against the IQueryable host
Execute the query, i.e. begin to iterate or call ToList()
1. To provide the results, GetEnumerator() is called on the query host
2. This results in a call to CreateQuery() on the provider
3. Ultimately, the expression tree is executed to return the results
Any time a new type is encountered (i.e. when you have a subquery, or project results to an anonymous type), another call to CreateQuery is made using the new type.

The Provider

I’m not ready to build a full provider yet. Instead, I created a hybrid provider that intercepts the process to provide a hook to inspect and/or mutate the expression tree. It then passes the modified expression to the original provider. This allows it to work equally well with built-in providers like LINQ to Objects and external ones like EF Core. The interface for the basic provider looks like this:

public interface ICustomQueryProvider<T> : IQueryProvider
{
    IEnumerable<T> ExecuteEnumerable(Expression expression);
}

This is just a custom method for the host to call with the expression. This is where the interception will happen.

An abstract base class takes care of the default implementation:

public abstract class CustomQueryProvider<T> : ICustomQueryProvider<T>
{
    public CustomQueryProvider(IQueryable sourceQuery)
    {
        Source = sourceQuery;
    }

    protected IQueryable Source { get; }

    public abstract IQueryable CreateQuery(Expression expression);

    public abstract IQueryable<TElement> CreateQuery<TElement>(Expression expression);

    public virtual object Execute(Expression expression)
    {
        return Source.Provider.Execute(expression);
    }

    public virtual TResult Execute<TResult>(Expression expression)
    {
        object result = (this as IQueryProvider).Execute(expression);
        return (TResult)result;
    }

    public virtual IEnumerable<T> ExecuteEnumerable(Expression expression)
    {
        return Source.Provider.CreateQuery<T>(expression);
    }
}

There are two key features of the custom provider: it captures the original query (therefore the original provider), and when the “execute” methods are called, it passes the expression to the original provider.

The query host and provider go hand in hand. Here is the definition of a QueryHost that is ready to use the new provider (yet to be defined):

public class QueryHost<T> : IQueryHost<T, IQueryInterceptingProvider<T>>
{
    public QueryHost(
        IQueryable<T> source)
    {
        Expression = source.Expression;
        CustomProvider = new QueryInterceptingProvider<T>(source);
    }

    public QueryHost(
        Expression expression,
        QueryInterceptingProvider<T> provider)
    {
        Expression = expression;
        CustomProvider = provider;
    }

    public virtual Type ElementType => typeof(T);

    public virtual Expression Expression { get; }

    public IQueryProvider Provider => CustomProvider;

    public IQueryInterceptingProvider<T> CustomProvider { get; protected set; }

    public virtual IEnumerator<T> GetEnumerator() =>
        CustomProvider.ExecuteEnumerable(Expression).GetEnumerator();
}

The key features to note are the capture of the custom provider in the constructor calls, and the implementation of GetEnumerator that passes the request to the custom provider. What does the custom provider look like?

The Expression Transformer

To intercept and modify the query requires a function that takes in the original expression and returns the mutated one. Here is the definition of the transformation:

public delegate Expression ExpressionTransformer(Expression source);

The provider needs to be aware of the transformation. Here is an interface that defines how to register it:

public interface IQueryInterceptor
{
    void RegisterInterceptor(ExpressionTransformer transformation);
}

Now onto the intercepting provider…

The Query Interception Provider

This is the code for the intercepting provider:

public class QueryInterceptingProvider<T> :
    CustomQueryProvider<T>, IQueryInterceptingProvider<T>
{
    private ExpressionTransformer transformation = null;

    public QueryInterceptingProvider(IQueryable sourceQuery)
        : base(sourceQuery)
    {
    }

    public override IQueryable CreateQuery(Expression expression)
    {
        return new QueryHost<T>(expression, this);
    }

    public override IQueryable<TElement> CreateQuery<TElement>(Expression expression)
    {
        if (typeof(TElement) == typeof(T))
        {
            return CreateQuery(expression) as IQueryable<TElement>;
        }

        var childProvider = new QueryInterceptingProvider<TElement>(Source);

        return new QueryHost<TElement>(
            expression, childProvider);
    }

    public void RegisterInterceptor(ExpressionTransformer transformation)
    {
        if (this.transformation != null)
        {
            throw new InvalidOperationException();
        }

        this.transformation = transformation;
    }

    public override object Execute(Expression expression)
    {
        return Source.Provider.Execute(TransformExpression(expression));
    }

    public override IEnumerable<T> ExecuteEnumerable(Expression expression)
    {
        return base.ExecuteEnumerable(TransformExpression(expression));
    }

    private Expression TransformExpression(Expression source) =>
        transformation == null ? source :
        transformation(source);
}

So, what’s going on? In a nutshell, the custom provider does three things.

When CreateQuery is called, it ensures that the new query also uses the custom provider. If it is the same type, it uses the same instance of provider, otherwise it creates a new one.
It implements the interface to register a transformation an stores it as a property. It ensures the transformation is only registered once.
When the execute methods are called, it transforms the expression before passing them to the source provider.

That’s it! The query host uses the custom provider. The custom provider ensures any new query hosts that are created also use the custom provider. It also transforms the expression before passing it to the original provider.

A Query Snapshot

The basic building blocks are in place and we are ready to test the implementation. I’m using a Thing that generates random properties and half-truths to create a nice spread to query against.

public class Thing
{
    private static readonly Random Random = new Random();
    public string Id { get; private set; } = Guid.NewGuid().ToString();
    public int Value { get; private set; } = Random.Next(int.MinValue, int.MaxValue);
    public DateTime Created { get; private set; } = DateTime.Now;
    public DateTime Expires { get; private set; } = DateTime.Now.AddDays(Random.Next(1, 999));
    public bool IsTrue { get; private set; } = Random.NextDouble() < 0.5;
}

A static method generates 10,000 things. Next, I set up the interceptor. The first interception doesn’t change the expression. It just writes it to the console (the default ToString() override provides a very clear representation of the entire expression tree).

static Expression ExpressionTransformer(Expression e)
{
    Console.WriteLine(e);
    return e;
}

Next, the API sets up the query for consumption. It wraps it in the query host and registers the “transformation”. The query is now ready for consumption.

var query = new QueryHost<Thing>(ThingDbQuery);
query.CustomProvider.RegisterInterceptor(ExpressionTransformer);

The consumer is simply presented with an IQueryable<Thing> to build queries against (this is what is implemented by QueryHost.)

Here is a query that is immediately parsed to a list:

var list = query.Where(t => t.IsTrue &&
    t.Id.Contains("aa") &&
    t.Expires < DateTime.Now.AddDays(100))
    .OrderBy(t => t.Id).ToList();

Console.WriteLine($"Retrieved {list.Count()} items.");

Running this results in the following output:

If you step through in debug mode, you’ll see that the provider is called and immediately “transforms” the expression, resulting in the output to the console. That’s it for an interception. Next, how about a mutation?

Query “Guard Rails”

One challenge with exposing IQueryable is that fact that literally any type of expression chain might be applied. This can be problematic when the query is abused, such as returning large record sets or performing complex joins with a negative impact on performance. What if you could enforce a simple rule, such as “This query may never return more than ten items”?

Here’s a query with no limits - the same one that returned 53 results in the previous example.

var results = query.Where(t => t.IsTrue &&
    t.Id.Contains("aa") &&
    t.Expires < DateTime.Now.AddDays(100))
    .OrderBy(t => t.Id);

Here’s the registration:

static Expression ExpressionTransformer(Expression e)
{
    Console.WriteLine($"Before: {e}");

    var newExpression = new GuardRailsExpressionVisitor().Visit(e);

    Console.WriteLine($"After: {newExpression}");
    return newExpression;
}

// wrap and intercept
var query = new QueryHost<Thing>(ThingDbQuery);
query.CustomProvider.RegisterInterceptor(ExpressionTransformer);

Notice that the expression is passed to the GuardRailsExpressionVisitor. If you recall from previous posts, the ExpressionVisitor class provides an easy way to traverse and modify expression trees. The tree is recursively iterated by the base class, then methods are called that can be overridden to inspect the node on the tree. The overridden methods either return the original expression or a modified one, resulting in the transformation of the tree. To implement the guard rails, there are three possibilities:

The “take” exists but is less than ten (no change)
The “take” exists but is greater than ten (“take” is modified to 10)
The “take” doesn’t exist (“take of ten” is added)

This implementation is simple and assumes a constant is being used. Keep in mind a reliable implementation must handle edge cases, such as nested “take” statements in inner queries. Other edge cases include statements that use lambda expressions to reference properties and using the result of a method call instead of a constant.

When you specify “take” on a query, what really happens is an extension method of IQueryable is called that exists on the Queryable class. A query like this:

var results = query.Take(10);

Uses an extension method that looks like this:

public static IQueryable<TSource> Take<TSource>(this IQueryable<TSource> source, int count)
{
    if (source == null)
    {
        throw Error.ArgumentNull(nameof(source));
    }

    return source.Provider.CreateQuery<TSource>(
        Expression.Call(
            null,
            IQueryable<object>(Queryable.Take).GetMethodInfo()
                .GetGenericMethodDefinition().MakeGenericMethod(TSource),
            source.Expression,
            Expression.Constant(count)));
}

Here’s the code to intercept the method call and enforce the “take” rule:

protected override Expression VisitMethodCall(MethodCallExpression node)
{
    if (node.Method.Name == nameof(Queryable.Take))
    {
        TakeFound = true;

        if (node.Arguments[1] is ConstantExpression constant)
        {
            if (constant.Value is int valueInt)
            {
                if (valueInt > 10)
                {
                    var expression = node.Update(
                        node.Object,
                        new[] { node.Arguments[0] }
                        .Append(Expression.Constant(10)));
                    return expression;
                }
            }
        }
    }

    return base.VisitMethodCall(node);
}

The expression is only overridden when the value is out of range. Instead of rebuilding the original expression from scratch, the Update method (available on most expressions) makes it easy for you to create a copy of the expression and override the parameters you want to. The first argument points back to the parent query, so the second one is what we override to the new value. Running this results in the following:

Notice that the higher take of 50 is modified while the lower take of five stays the same. This works well for replacing an existing take, but what if no take is specified at all?

The solution is here:

public override Expression Visit(Expression node)
{
    if (first)
    {
        first = false;
        var expr = base.Visit(node);

        if (TakeFound)
        {
            return expr;
        }

        var existing = expr as MethodCallExpression;

        var newExpression = Expression.Call(
            typeof(Queryable),
            nameof(Queryable.Take),
            existing.Method.ReturnType.GetGenericArguments(),
            existing,
            Expression.Constant(10));
        return newExpression;
    }

    return base.Visit(node);
}

Remember that Visit is called recursively, so the first flag ensures the top-level logic is only run once. The expression tree is parsed and if the take expression was encountered, it simply returns the modified expression. Otherwise, it creates the take expression. The top-level expression is always a call that returns the result. To implement take, we simply create our own method that applies the take to the wrapped method and returns the results. The story is told through the Expression.Call parameters.

typeof(Queryable) is the static class that defines Take
nameof(Queryable.Take) is the name of the method to call (Take<T>())
existing.Method.ReturnType.GetGenericArguments() closes the generic type because Take requires a type parameter. The query is an IQueryable<T> so the existing method must implement T as the generic argument (in this case, T is Thing).
existing is the first parameter, or what the extension method applies the Take to. This is the original expression, so the result of that expression is what the Take will be applied to.
Expression.Constant(10) is the number of items to Take.

You can see the Take added here and confirm the result set is limited.

The expression has been intercepted and transformed! For the next (and last) example I tackled a real-world problem.

Intercept Binary Evaluations

A customer of ours asked if there is a way to understand why certain items are selected and others are filtered out when a query is executed. In other words, for each item, what is the part of the query that failed? The solution is tricky. For example, if the query is sent to the database then there is really no decision to intercept: the EF Core provider will create the SQL syntax and pass it along to the database. For in-memory selection it is possible to capture the decision. First, I created a BinaryInterceptorVisitor<T> to transform the expression tree for a given type.

When you build a query with filters, the “where” clause becomes a set of binary expressions. For example, the following query:

var list = query.Where(
    t => t.IsTrue &&
        t.Id.Contains("aa") &&
        t.Expires < DateTime.Now.AddDays(100))
    .OrderBy(t => t.Id).ToList();

Ends up looking like this for the filter:

AndAlso(t.IsTrue,
    AndAlso(
        t.Id.Contains("aa"),
        IsLessThan(
            t.Expires,
            DateTime.Now.AddDays(500))))

First, let’s capture the value of t.

Capture the Entity

I added a field named instance to capture the entity being evaluated. To keep it simple, I just declare it as an object:

private static object instance;

I created a method to set the instance:

private static void SetInstance(object instance)
{
    BinaryInterceptorVisitor<T>.instance = instance;
}

To make it easier to build an expression that calls the method, I added some helper methods that resolve the data structure for method information called MethodInfo. These are the flags needed to find a static method:

private static readonly BindingFlags GetStatic =
    BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static;

This is the method to get the MethodInfo data:

private static MethodInfo GetMethod(string methodName) =>
    typeof(BinaryInterceptorVisitor<T>)
        .GetMethod(methodName, GetStatic);

Here is the reference to the method:

private static readonly MethodInfo SetInstanceMethod = GetMethod(nameof(SetInstance));

Lambda Expressions

The “where” clause of the query is built as a LambdaExpression with this signature:

Func<T, bool> Where;

Each entity is passed into the filters and ultimately the filter either passes or not. Here is the overload to intercept a lambda expression:

protected override Expression VisitLambda<TValue>(Expression<TValue> node)

The expression to intercept has a specific signature. It will have exactly one parameter of type T and a return value of type bool:

if (node.Parameters.Count == 1 &&
    node.Parameters[0].Type == typeof(T) &&
    node.ReturnType == typeof(bool))

When the right lambda expression is intercepted, our code sneaks in to nab the entity:

var returnTarget = Expression.Label(typeof(bool));

var lambda = node.Update(
    Visit(node.Body),
    node.Parameters.Select(p => Visit(p)).Cast<ParameterExpression>());

var innerInvoke = Expression.Return(
    returnTarget, Expression.Invoke(lambda, lambda.Parameters));

var expr = Expression.Block(
    Expression.Call(SetInstanceMethod, node.Parameters),
    innerInvoke,
    Expression.Label(returnTarget, Expression.Constant(false)));

return Expression.Lambda<Func<T, bool>>(
    expr,
    node.Parameters);

The key to intercepting the type is to use a BlockExpression. This expression allows you to combine multiple expressions that run sequentially. Only the result of the final expression is used for the result (although you can capture the result of other expressions using variables). The steps in the code look like this:

Create a “return target.” Think of this as a variable to capture the return value of the original lambda expression. In expression parlance, this is a “label.”
Transform the lambda by using Update and visiting its parts (the body of the expression and the parameters). This is important because the body contains the binary expressions that also will be transformed, so the lambda expression itself must be replaced.
Create an invocation that calls the original lambda expression and captures its return value.
Create an expression block:
1. The first expression grabs the parameters from the lambda and uses them to call the SetInstance method. The parameter is the entity being evaluated, so this effectively captures the entity for inspection.
2. The second expression is the invocation of the original lambda expression.
3. The final parameter is the “binding” to the return value with a default value of false. This is always overwritten with the actual result of the original expression.
Finally, a lambda expression is returned with the same signature as the original.

The code takes this:

Func<T, bool> lambda = t => Where(t);

And transforms it into this:

Func<T, bool> lambda = t =>
{
    BinaryInterceptorVisitor<T>.SetInstance(t);
    return Where(t);
};

Now that the value is captured, the next step is to understand which binary expressions succeed or fail.

Wrap a Binary Expression

The key to understanding how to capture the result is logical. Very logical. As in, logic gates. Right now, we have a Func<T,bool> that could be true or false. The trick is to inject our own code that captures the rule before it is evaluated, then captures the result whether it is true or false. This must be done using only binary expressions. Here’s some pseudo code:

Grab the rule
If the rule succeeds, capture the success
Else if the rule fails, capture the failure

Logic provides a means to wrap the expression in a way that preserves the original value. Let’s review our basic logic statements:

Logical OR left OR right = OrElse(left, right)

If left
1. Then true
Else
1. If right
2. Then true
3. Else false

Notice that the right is only run if the left is false. Essentially, think of it as “when the left is false, pass through to the right.” This is how we intercept the expression before it’s called:

OrElse([our code => return false], [existing code]);

…and another logic statement:

Logical AND left AND right = AndAlso(left, right)

If left
1. Then
2. If right
  1. Then true
  2. Else false
3. Else false

Basically, you can think of it as a flipped OR because “when the left is true, pass through to the right.”

This makes it easy to capture success:

AndAlso([existing code], [our code => return true]);

But how do we capture failure? What we need is to call our code for failure when the previous logic returns false:

OrElse(AndAlso..., [our code => return false]);

Therefore, the full interception requires four parts:

binary - the original rule to intercept
orLeft - the snapshot of the rule before it is run
andRight - capture the success case
orRight - capture the failure case

This is the actual code to replace the existing expression:

return Expression.OrElse(
    orLeft,
    Expression.OrElse(
        Expression.AndAlso(binary, andRight),
        orRight));

So, what do the implementations of orLeft and our other interceptors look like? There are two fields defined that capture the level of nesting:

private static int evalLevel = 0;
private int binaryLevel = 0;

The binaryLevel keeps track of the levels as the expressions are parsed. This is used during the definition phase as the structure is being updated. The evalLevel keeps track of the nested level during the runtime or implementation phase.

The BeforeEval method checks to see if it is at the top level. If so, it writes the value of the current entity that was captured by the lambda expression override and resets the evaluation level. Otherwise, it increments the level. Then, it writes the rule. The text is indented based on the current level for readability.

public static void BeforeEval(int binaryLevel, string node)
{
    if (binaryLevel == 1)
    {
        Console.WriteLine($"with {instance} => {{");
        evalLevel = 0;
    }
    else
    {
        evalLevel++;
    }

    Console.WriteLine($"{Indent}[Eval {node}: ");
}

The AfterEval method writes out the result (success or failure) and decrements the level. It also writes a closing brace if the evaluation is after the top expression.

public static void AfterEval(int binaryLevel, bool success)
{
    var result = success ? "SUCCESS" : "FAILED";

    Console.WriteLine($"{Indent}{result}]");

    evalLevel--;

    if (binaryLevel == 1)
    {
        Console.WriteLine("}");
    }
}

In the BinaryExpression override method, the method calls are built with the appropriate parameters:

binaryLevel++;

var before = Expression.Call(
    BeforeEvalMethod,
    Expression.Constant(binaryLevel),
    Expression.Constant($"{node}"));

var afterSuccess = Expression.Call(
    AfterEvalMethod,
    Expression.Constant(binaryLevel),
    Expression.Constant(true));

var afterFailure = Expression.Call(
    AfterEvalMethod,
    Expression.Constant(binaryLevel),
    Expression.Constant(false));

Next, the binary expression inputs are created. These use expression blocks that call the desired method then return the proper bool.

var orLeft = Expression.Block(
    before,
    Expression.Constant(false));

var andRight = Expression.Block(
    afterSuccess,
    Expression.Constant(true));

var orRight = Expression.Block(
    afterFailure,
    Expression.Constant(false));

The existing binary expression is replaced with a call to visit the left and right nodes. This ensures downstream expressions are also intercepted.

var binary = node.Update(
    Visit(node.Left),
    node.Conversion,
    Visit(node.Right));

binaryLevel--;

Finally, the entire expression is replaced with the logic shown earlier.

Logically, the binary expression is replaced with this tree:

Here is the result of a run:

In the first example, IsTrue is true so the evaluation continued to the inner rule. The expiration date is December 10th, 2022, which is more than 500 days in the future, so it failed. This results in the entire expression failing.

In the second example, IsTrue is false so the expression failed immediately.

Finally, IsTrue is true and the expiration date of September 3rd, 2020 is less than 500 days in the future, so the entire expression succeeded.

💡 Tip : the approach can be enhanced by handling more than binary expressions. For example, the expression AndAlso requires a bool for each side, so you could override the member access (the expression that gets the value of IsTrue) and show that result like a nested rule. You could also intercept the evaluation of expressions like GreaterThan to show the actual values being compared.

Conclusion

The code covered in this blog post is available here:

JeremyLikness / QueryEvaluationInterceptor

An example of intercepting IQueryable executions to parse and transform the query.

QueryEvaluationInterceptor

An example of intercepting IQueryable executions to parse and transform the query.

View on GitHub

I hope these explorations of expressions help shed light on a powerful tool in the .NET toolbox. To make expressions more approachable, I’ve been working on an Expression Power Tools library to simplify working with expressions. For example, this code:

var query = new QueryHost<Thing>(ThingDbQuery);
query.CustomProvider.RegisterInterceptor(ExpressionTransformer);

Is simplified to this:

var query = ThingDbQuery.CreateInterceptedQueryable(ExpressionTranformer);

If you’re interested in expressions, take a look!

Regards,

High Performance Database Access with EF Core

Jeremy Likness ⚡️ — Fri, 07 Aug 2020 15:50:58 +0000

EF Core is an object relational mapper that allows developers to use a model-driven approach for database access. Recently, Microsoft MVP Phil Japikse and VS Toolbox host Robert Green created a video series named "EF Core In-Depth." We spoke with both of them about their motivation behind the series.

Phil and Robert shared demos that show the massive performance improvements in EF Core on .NET Core compared to Entity Framework 6 on the .NET Framework. In addition to performance, the team discussed the motivation behind EF Core architecture decisions, how to decide what version of EF Core to use (nightly builds? latest stable? most recent preview?), optimistic and pessimistic concurrency, and more.

You can find the links that were discussed in the show at:

https://www.theurlist.com/efcore-standup-2020-08-05.

Have feedback for the EF Core team? File an issue or join one of our many online discussions.

Keep your Code and Database in Sync

Jeremy Likness ⚡️ — Wed, 29 Jul 2020 15:13:45 +0000

Entity Framework Core is an object mapper that enables database interactions using familiar C# code. It supports multiple backend databases including SQLite, SQL Server, Oracle, PostgreSQL, MySQL and Cosmos DB. Using nothing more than LINQ queries, developers can write code that interacts with C# classes that is automatically translated into database calls.

In this episode of the EF Core Community Standup, software architect Tony Sneed shows how to extend a built-in feature of EF Core called scaffolding, or reverse engineering, by using templates (via Handlebars) to take control of generated code.

You can find the links that were discussed in the show at:

https://www.theurlist.com/efcore-standup-2020-07-22.

Have feedback for the EF Core team? File an issue or join one of our many online discussions.

Master your Database using C# with EF Core 5.0

Jeremy Likness ⚡️ — Wed, 22 Jul 2020 18:22:42 +0000

In this episode of the EF Core Community Standup, engineering lead Arthur Vickers walks through demos of some of the latest features that are available now in preview releases and will launch with .NET 5.0 later this year.

Have feedback for the EF Core team? File an issue or join one of our many online discussions.

Look Behind the IQueryable Curtain

Jeremy Likness ⚡️ — Sun, 19 Jul 2020 07:47:29 +0000

In a previous blog post, I explored the power of expressions and used them to dynamically build a rules engine based on a JSON payload. In this post I flip everything upside down and start with the expression. Given the variety of possible expression types and complexity of expression trees, what is the best way to decompose the tree? And, while we’re at it, can we mutate the expression to make it behave differently? I’ll explain why that’s important.

First, if you haven’t read the first article, take a few minutes to check it out:

Dynamically Build LINQ Expressions

If you want to master Language Integrated Query (LINQ), you first need to understand the expressions it is based on. This blog post demonstrates how to dynamically build a LINQ expression tree with multiple nested conditions, use the compiled query to filter an in-memory list, then reuse the expression query a relational database.

Ready to chop down the expression tree? Let’s get started.

(If you're eager to jump into code, the repo is right here):

JeremyLikness / ExpressionExplorer

A simple console application to explore and transform expressions using ExpressionVisitor.

ExpressionExplorer

A simple console application to explore and transform expressions using ExpressionVisitor.

Read the related blog post: IQueryable and Expression Trees.

View on GitHub

Setting the Stage

First, let’s start with a few assumptions. I’m working with a thing. It’s just a plain old Common Language Runtime (CLR) entity (you may have heard it referred to as POCO) named Thing. Here’s the definition:

public class Thing
{
    public Thing()
    {
        Id = Guid.NewGuid().ToString();
        Created = DateTimeOffset.Now;
        Name = Guid.NewGuid().ToString().Split("-")[0];
    }

    public string Id { get; set; }
    public string Name { get; set; }
    public DateTimeOffset Created { get; private set; }

    public string GetId() => Id;

    public override string ToString() =>
        $"({Id}: {Name}@{Created})";
}

To cheat, I added a static method that makes it easy to generate N number of things:

public static IList<Thing> Things(int count)
{
    var things = new List<Thing>();
    while (count-- > 0)
    {
        things.Add(new Thing());
    }
    return things;
}

Now I can generate a source and query it. Here’s a LINQ expression that generates 500 things and queries them:

var query = Thing.Things(500).AsQueryable()
    .Where(t => 
        t.Name.Contains("a", StringComparison.InvariantCultureIgnoreCase) &&
        t.Created > DateTimeOffset.Now.AddDays(-1))
    .Skip(2)
    .Take(50)
    .OrderBy(t => t.Created);

If you call ToString() on query you get this:

System.Collections.Generic.List`1[ExpressionExplorer.Thing]
    .Where(t => 
        (t.Name.Contains("a", InvariantCultureIgnoreCase) 
            AndAlso 
        (t.Created > DateTimeOffset.Now.AddDays(-1))))
    .Skip(2)
    .Take(50)
    .OrderBy(t => t.Created)

The formatting was added by me. One thing you may not have noticed is that query has a property named Expression.

Behind every good IQueryable is a proper Expression. — Me

The way the expression is built shouldn’t be too mysterious. Starting with the list, the Enumerable.Where method is called. The first parameter is an enumerable source (our list of things) and the second is a predicate that takes an item and returns true if it will be included. Inside the predicate, string.Contains is called, some comparisons happen, then two results are skipped. The Enumerable.Skip method takes an enumerable list and an integer representing the count. Although the syntax to build the query looks straightforward, you can visualize it as a series of progressive filters. The Skip call is an extension method of enumerable that is taking the result from the Where call and so forth.

Just for you, and exclusive to this post, I used my award-winning illustration skills to diagram the concept:

If you wanted to parse the expression tree, however, you may be in for a surprise. There are many different expression types, and each one is parsed a different way. For example, a BinaryExpression has a Left and a Right, but a MethodCallExpression has a list of Arguments that are expressions. That’s a lot of type checks and casting just to walk the tree!

Another Visitor

“Stay awhile. Stay … forever!” — Mission Impossible, Commodore 64 Edition

To make life easier, LINQ provides a special class named ExpressionVisitor. It contains all the logic necessary to recursively parse an expression tree. You simply pass an Expression into the Visit method and it will visit every node and return the expression back (more on that later). It contains methods specific to node types that can be overridden to intercept the process. Here’s a basic implementation that simply overrides certain methods to write information to the console and then passes the ball back to the base class.

public class BasicExpressionConsoleWriter : ExpressionVisitor
{
    protected override Expression VisitBinary(BinaryExpression node)
    {
        Console.Write($" binary:{node.NodeType} ");
        return base.VisitBinary(node);
    }

    protected override Expression VisitUnary(UnaryExpression node)
    {
        if (node.Method != null)
        {
            Console.Write($" unary:{node.Method.Name} ");
        }
        Console.Write($" unary:{node.Operand.NodeType} ");
        return base.VisitUnary(node);
    }

    protected override Expression VisitConstant(ConstantExpression node)
    {
        Console.Write($" constant:{node.Value} ");
        return base.VisitConstant(node);
    }

    protected override Expression VisitMember(MemberExpression node)
    {
        Console.Write($" member:{node.Member.Name} ");
        return base.VisitMember(node);
    }

    protected override Expression VisitMethodCall(MethodCallExpression node)
    {
        Console.Write($" call:{node.Method.Name} ");
        return base.VisitMethodCall(node);
    }

    protected override Expression VisitParameter(ParameterExpression node)
    {
        Console.Write($" p:{node.Name} ");
        return base.VisitParameter(node);
    }
}

To use it, simply create an instance and pass it an expression. Here, we’ll pass it the expression behind our query:

new BasicExpressionConsoleWriter().Visit(query.Expression);

Running it yields the following uninspiring output:

call:OrderBy  call:Take  call:Skip  call:Where  
constant:System.Collections.Generic.List`1[ExpressionExplorer.Thing]  unary:Lambda  
binary:AndAlso  call:Contains  member:Name  p:t  constant:a  
constant:InvariantCultureIgnoreCase  binary:GreaterThan  member:Created  p:t  
call:AddDays  member:Now  constant:-1  p:t  constant:2  constant:50  
unary:Lambda  member:Created  p:t  p:t

Notice the order things are visited. It may take a minute, but the logic makes sense:

OrderBy is the outermost call (last in, first out), and it takes a list and a field…
The first parameter to OrderBy is the list, which is provided by Take…
Take needs a list, which is provided by Skip…
Skip needs a list, which is provided by Where…
Where needs a list, which is provided by the Thing list…
The second parameter to Where is a predicate lambda expression…
…which is a binary AndAlso…
The left side of the binary is a Contains call…
(skipping a bunch of logic)
The second parameter to Take is 50…
The second parameter to Skip is 2…
The OrderBy property is Created…

You get the point? Of course, knowing how the tree is parsed is the key to making our visitor more readable. Here’s a more informed pass:

public class ExpressionConsoleWriter
    : ExpressionVisitor
{
    int indent;

    private string Indent => 
        $"\r\n{new string('\t', indent)}";

    public void Parse(Expression expression)
    {
        indent = 0;
        Visit(expression);
    }

    protected override Expression VisitConstant(ConstantExpression node)
    {
        if (node.Value is Expression value)
        {
            Visit(value);
        }
        else
        {
            Console.Write($"{node.Value}");
        }
        return node;
    }

    protected override Expression VisitParameter(ParameterExpression node)
    {
        Console.Write(node.Name);
        return node;
    }

    protected override Expression VisitMember(MemberExpression node)
    {
        if (node.Expression != null)
        {
            Visit(node.Expression);
        }
        Console.Write($".{node.Member?.Name}.");
        return node;
    }

    protected override Expression VisitMethodCall(MethodCallExpression node)
    {
        if (node.Object != null)
        {
            Visit(node.Object);
        }
        Console.Write($"{Indent}{node.Method.Name}( ");
        var first = true;
        indent++;
        foreach (var arg in node.Arguments)
        {
            if (first)
            {
                first = false;
            }
            else
            {
                indent--;
                Console.Write($"{Indent},");
                indent++;
            }
            Visit(arg);
        }
        indent--;
        Console.Write(") ");
        return node;
    }

    protected override Expression VisitBinary(BinaryExpression node)
    {
        Console.Write($"{Indent}<");
        indent++;
        Visit(node.Left);
        indent--;
        Console.Write($"{Indent}{node.NodeType}");
        indent++;
        Visit(node.Right);
        indent--;
        Console.Write(">");
        return node;
    }
}

The new entry method, Parse, is introduced to set the indent to 0 and kick off the visits. The Indent property emits a newline and the right number of tabs based on the current indent value. It’s called by various methods to format the output.

The VisitMethodCall and VisitBinary methods should shed some insight into how this works. In VisitMethodCall the method name is printed with an open parenthesis representing the parameters, or arguments. These are then visited in turn, and the logic will continue recursively for each argument until done. Then the closing parenthesis is printed. Because the method explicitly visited the child nodes, instead of calling the base class, the node is simply returned. This is because the base class would also recursively visit the arguments and result in duplicates. For the binary expression, an opening angle is printed, then the left node visited, followed by the type of binary operation, then the right node, and finally the closure. Again, the base method is not called because the nodes were already visited.

Running this new visitor:

new ExpressionConsoleWriter().Visit(query.Expression);

Results in something a little more readable (formatting via the tool, not me):

OrderBy(
    Take(
        Skip(
            Where( System.Collections.Generic.List`1[ExpressionExplorer.Thing]
            ,
                <t.Name.
                    Contains( a
                    ,InvariantCultureIgnoreCase)
                AndAlso
                    <t.Created.
                    GreaterThan.Now.
                        AddDays( -1) >>t)
        ,2)
    ,50)
,t.Created.t)

For a full implementation, look no further than LINQ itself. The ExpressionStringBuilder contains everything needed to print an expression tree in a friendly format. You can view the source code here: ExpressionStringBuilder.

The ability to parse expression trees is quite powerful. I’ll dig deeper into what’s possible in another blog post. Before I go, I want to address the elephant in the room: aside from helping parse the expression tree, what is the point of the main Visit method returning an expression? It turns out that ExpressionVisitor can do more than just inspect your query!

Invasion of the Query Snatchers

A magical quality of the ExpressionVisitor is the ability to shape a query on-the-fly. To understand why, consider this scenario: you’re tasked with standing up an order entry system that has powerful query capabilities and you must finish it fast. You read my articles and decide to use Blazor WebAssembly and write LINQ queries on the client. You use a custom visitor to cleverly serialize the query and pass it to the server, where you deserialize and run it. Everything works splendidly until the security audit. There, it’s determined that the query engine is too wide open. A malicious client could issue extremely complex queries that return massive result sets to bring down the system. What do you do?

One benefit of using the visitor approach is that you don’t have to reconstruct the entire expression tree just to modify a leaf node. Expression trees are immutable, but the visitor can return an entirely new expression tree! You write the logic that mutates the tree and receive the full tree complete with modifications at the end. To illustrate how this works, let’s make a special visitor named ExpressionTakeRestrainer.

public class ExpressionTakeRestrainer : ExpressionVisitor
{
    private int maxTake;
    public bool ExpressionHasTake { get; private set; }

    public Expression ParseAndConstrainTake(
        Expression expression, int maxTake)
    {
        this.maxTake = maxTake;
        ExpressionHasTake = false;
        return Visit(expression);
    }
}

The special ParseAndConstrainTake method will kick off a visit and return the expression. Notice that it sets ExpressionHasTake to false. Presumably, if an Enumerable.Take is encountered, this will be set to true. This allows you to append a Take to the end of queries that don’t have it to constrain results. Let’s assume we’re very paranoid and only ever want to return five results. Theoretically, you could just always add the take to the end of the query:

var myQuery = theirQuery.Take(5);
return myQuery.ToList();

But where’s the fun in that? Let’s modify an expression tree. We’ll only override one method, and that’s the VisitMethodCall:

protected override Expression VisitMethodCall(MethodCallExpression node)
{
    if (node.Method.Name == nameof(Enumerable.Take))
    {
        ExpressionHasTake = true;
        if (node.Arguments.Count == 2 && 
            node.Arguments[1] is ConstantExpression constant)
        {
            var takeCount = (int)constant.Value;
            if (takeCount > maxTake)
            {
                var arg1 = Visit(node.Arguments[0]);
                var arg2 = Expression.Constant(maxTake);
                var methodCall = Expression.Call(
                    node.Object, 
                    node.Method, 
                    new[] { arg1, arg2 } );
                return methodCall;
            }
        }
    }
    return base.VisitMethodCall(node);
}

The logic checks to see if the method call is to Enumerable.Take. If it is, it sets the ExpressionHasTake flag. The second argument is the number to take, so the value is inspected and compared against the max. If it exceeds the maximum allowed, a new node is built that constrains it to the maximum value. This new node is returned instead of the original one. If the method is not Enumerable.Take then the base class is called, and everything is parsed “as usual.”

We can test it by running this:

new ExpressionConsoleWriter().Parse(
   new ExpressionTakeRestrainer()
      .ParseAndConstrainTake(query.Expression, 5));

Check out the following result: the query has been modified to only take five!

OrderBy(
    Take(
        Skip(
            Where( System.Collections.Generic.List`1[ExpressionExplorer.Thing]
            ,
                <t.Name.
                    Contains( a
                    ,InvariantCultureIgnoreCase)
                AndAlso
                    <t.Created.
                    GreaterThan.Now.
                        AddDays( -1) >>t)
        ,2)
    ,5)
,t.Created.t)

But wait… has it?! Try running this:

var list = query.ToList();
Console.WriteLine($"\r\n---\r\nQuery results: {list.Count}");

And, unfortunately, what you’ll see is 50… the original “take” amount. The problem is that we generated a new expression, but we didn’t replace it on the query. In fact, we can’t… it’s a read-only property and expressions are immutable. So now what?

And Now, Your Host

We host it! We can make our own query host simply by implementing IOrderedQueryable<T>. The interface is a collection of other interfaces. Here’s a breakdown of what the interface requires:

ElementType - this is simply the type of element being queried.
Expression - the backing expression behind the query.
Provider - this is the query provider. It does the actual work of applying the query. We won’t write our own provider, but will use the one built-in, which in this case is LINQ-to-Objects.
GetEnumerator - this is called when it’s time to run the query. You can build, extend, and modify all you like, but once this is called the query is materialized.

Here’s an implementation of TranslatingHost because it translates the provided query.

public class TranslatingHost<T> : IOrderedQueryable<T>, IOrderedQueryable
{
    private readonly IQueryable<T> query;

    public Type ElementType => typeof(T);

    private Expression TranslatedExpression { get; set; }

    public TranslatingHost(IQueryable<T> query, int maxTake)
    {
        this.query = query;
        var translator = new ExpressionTakeRestrainer();
        TranslatedExpression = translator
            .ParseAndConstrainTake(query.Expression, maxTake);
    }

    public Expression Expression => TranslatedExpression;

    public IQueryProvider Provider => query.Provider;

    public IEnumerator<T> GetEnumerator()
        => Provider.CreateQuery<T>(TranslatedExpression)
        .GetEnumerator();

    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

It’s fairly simple. It takes in an existing query, then uses the ExpressionTakeRestrainer to generate a new expression. It uses the existing provider (so, for example, if this is a query from a DbSet<T> using EF Core on SQL Server, it will translate to a SQL statement). When the enumerator is requested, instead of passing the original expression, it passes the translated one.

Let’s use it!

var transformedQuery =
   new TranslatingHost<Thing>(query, 5);
var list2 = transformedQuery.ToList();
Console.WriteLine($"\r\n---\r\nModified query results: {list2.Count}");

This time the result is what we want… only five records are returned.

So far, I’ve covered inspecting an existing query and swapping it out. This is helpful when you execute the query. If your code is what executes query.ToList() then you can modify the query to your heart’s content. But what about when your code isn’t responsible for materializing the query? What if you expose a library, like a repository, that has this interface?

public IQueryable<Thing> QueryThings { get; }

Or, in the case of EF Core:

public DbSet<Thing> Things { get; set; }

How do you “intercept” the query when someone downstream calls ToList()? That requires a provider and is something I’ll cover in more detail in the next post for this series.

Until then, happy expression hunting!

Repo:

JeremyLikness / ExpressionExplorer

A simple console application to explore and transform expressions using ExpressionVisitor.

ExpressionExplorer

A simple console application to explore and transform expressions using ExpressionVisitor.

Read the related blog post: IQueryable and Expression Trees.

View on GitHub

Regards,

EF Core Community Standup: Blazor

Jeremy Likness ⚡️ — Tue, 14 Jul 2020 18:02:04 +0000

Blazor is a powerful platform for building Single Page Applications (SPA) using C# and .NET. It allows you to build reusable components and access existing .NET Standard libraries. Blazor Server is a hosting model that manages rendering and processes events on the server and communicates with your browser via a lightweight JavaScript shim and SignalR.

This episode walks through a reference line of business application. It features dynamic filtering, paging, and sorting, with add, edit, and delete capabilities. It implements optimistic concurrency for the case when two users edit the same information at the same time. It features a background audit that both timestamps entities as they are modified and generates a historic snapshot of properties.

All of the links for the show are available here:

https://www.theurlist.com/efcore-standup-2020-06-24

Filter Your Grids: How to Dynamically Build LINQ Expressions

Jeremy Likness ⚡️ — Wed, 03 Jun 2020 14:10:49 +0000

LINQ stands for Language Integrated Query and is one of my favorite .NET and C# technologies. Using LINQ, developers can write queries directly in strongly typed code. LINQ provides a standard language and syntax that is consistent across data sources.

The Basics

Consider this LINQ query (you can paste this into a console application and run it yourself):

using System;
using System.Linq;

public class Program
{
   public static void Main()
   {
      var someNumbers = new int[]{4, 8, 15, 16, 23, 42};
      var query =
         from num in someNumbers
            where num > 10
            orderby num descending
            select num.ToString();
      Console.WriteLine(string.Join('-', query.ToArray()));
      // 42-23-16-15
   }
}

Because someNumbers is an IEnumerable<int>, the query is parsed by LINQ to Objects. The same query syntax can be used with a tool like Entity Framework Core to generate T-SQL that is run against a relational database. LINQ can be written using one of two syntaxes: query syntax (shown above) or method syntax. The two syntaxes are semantically identical and the one you use depends on your preference. The same query above can be written using method syntax like this:

var secondQuery = someNumbers.Where(n => n > 10)
    .OrderByDescending(n => n)
    .Select(n => n.ToString());

Every LINQ query has three phases:

A data source is set up, known as a provider, for the query to act against. For example, the code shown so far uses the built-in LINQ to Objects provider. Your EF Core projects use the EF Core provider that maps to your database.
The query is defined and turned into an expression tree. I’ll cover expressions more in a moment.
The query is executed, and data is returned.

Step 3 is important because LINQ uses what is called deferred execution. In the example above, secondQuery defines an expression tree but doesn’t yet return any data. In fact, nothing actually happens until you start to iterate the data. This is important because it allows providers to manage resources by only delivering the data requested. For example, let’s assume you want to find a specific string using secondQuery, so you do something like this:

var found = false;
foreach(var item in secondQuery.AsEnumerable())
{
    if (item == "23")
    {
        found = true;
        break;
    }
}

A provider can handle the enumerator so that it feeds data one element at a time. If you find the value on the third iteration, it is possible that only three items were actually returned from the database. On the other hand, when you use the .ToList() extension method, all data is immediately fetched to populate the list.

The Challenge

For my role as PM for .NET Data, I frequently speak to customers to understand their needs. Recently, I had a discussion with a customer who wants to use a third-party control in their website to build business rules. More specifically, the business rules are “predicates” or a set of conditions that resolve to true or false. The tool can generate the rules in JSON or SQL format. SQL is tempting to pass along to the database, but their requirement is to apply the predicates to in-memory objects as a filter on the server as well. They are considering a tool that translates SQL to expressions (it’s called Dynamic LINQ if you’re interested). I suggested that the JSON format probably is fine, because it can be parsed into a LINQ expression that is run against objects in memory or easily applied to an Entity Framework Core collection to run against the database.

The spike I wrote only deals with the default JSON produced by the tool:

{
  "condition": "and",
  "rules": [
    {
      "label": "Category",
      "field": "Category",
      "operator": "in",
      "type": "string",
      "value": [
        "Clothing"
      ]
    },
    {
      "condition": "or",
      "rules": [
        {
          "label": "TransactionType",
          "field": "TransactionType",
          "operator": "equal",
          "type": "boolean",
          "value": "income"
        },
        {
          "label": "PaymentMode",
          "field": "PaymentMode",
          "operator": "equal",
          "type": "string",
          "value": "Cash"
        }
      ]
    },
    {
      "label": "Amount",
      "field": "Amount",
      "operator": "equal",
      "type": "number",
      "value": 10
    }
  ]
}

The structure is simple: there is an AND or OR condition that contains a set of rules that are either comparisons, or a nested condition. My goal was twofold: learn more about LINQ expressions to better inform my understanding of EF Core and related technologies, and provide a simple example to show how the JSON can be used without relying on a third party tool.

One of my earliest open source contributions was the NoSQL database engine I named Sterling because I wrote it as a local database for Silverlight. It later gained popularity when Windows Phone was released with Silverlight as the runtime and was used in several popular recipe and fitness apps. Sterling suffered from several limitations that could have easily been mitigated with a proper LINQ provider. My goal is to ultimately master enough of LINQ to be able to write my own EF Core provider if the need arises.

The Solution: Dynamic Expressions

I created a simple console app to test my hypothesis that materializing the LINQ from the JSON would be relatively straightforward.

JeremyLikness / ExpressionGenerator

Example of creating dynamic LINQ expressions

ExpressionGenerator

This example parses a JSON file to dynamically build a LINQ predicate that can be used to filter objects.

To get started, simply clone and run. Make sure one of the .json source files is in your current directory (this is done automatically from Visual Studio). Each project has its own copy.

The ExpressionGenerator project can be set as startup for LINQ to Objects.

The DatabaseTest will apply the expressions to a locally created SQLite database.

For a full explanation and walk through, read Dynamically Build LINQ Expressions.

View on GitHub

Set the startup project to ExpressionGenerator for the first part of this post. If you run it from the command line, be sure that rules.json is in your current directory.

I embedded the sample JSON as rules.json. Using System.Text.Json to parse the file is this simple:

var jsonStr = File.ReadAllText("rules.json");
var jsonDocument = JsonDocument.Parse(jsonStr);

I then created a JsonExpressionParser to parse the JSON and create an expression tree. Because the solution is a predicate, the expression tree is built from instances of BinaryExpression that evaluate a left expression and a right expression. That evaluation might be a logic gate (AND or OR), or a comparison (equal or greaterThan), or a method call. For the case of In i.e. we want property Category to be in one of several items in a list, I flip the script and use Contains. Conceptually, the referenced JSON looks like this:

                         /-----------AND-----------\
                         |                         |
                      /-AND-\                      |
Category IN ['Clothing']   Amount eq 10.0        /-OR-\
                        TransactionType EQ 'income'  PaymentMode EQ 'Cash'

Notice that each node is binary. Let’s start parsing!

Introducing Transaction

No, this isn’t System.Transaction. This is a custom class used in the example project. I didn’t spend much time on the vendor’s site, so I guessed at what the entity might look like based on the rules. I came up with this:

public class Transaction
{
   public int Id { get; set; }
   public string Category { get; set; }
   public string TransactionType { get; set; }
   public string PaymentMode { get; set; }
   public decimal Amount { get; set; }
}

I then added some additional methods to make it easy to generate random instances. You can see those in the code yourself.

Parameter Expressions

The main method returns a predicate function. Here’s the code to start:

public Func<T, bool> ParsePredicateOf<T>(JsonDocument doc)
{
   var itemExpression = Expression.Parameter(typeof(T));
   var conditions = ParseTree<T>(doc.RootElement, itemExpression);
}

The first step is to create the predicate parameter. A predicate can be passed to a Where clause, and if we wrote it ourselves it would look something like this:

var query = ListOfThings.Where(t => t.Id > 2);

The t => is the first parameter and represents the type of an item in the list. Therefore, we create a parameter for that type. Then we recursively traverse the JSON nodes to build the tree.

Logic Expressions

The start of the parser looks like this:

private Expression ParseTree<T>(
    JsonElement condition,
    ParameterExpression parm)
    {
        Expression left = null;
        var gate = condition.GetProperty(nameof(condition)).GetString();

        JsonElement rules = condition.GetProperty(nameof(rules));

        Binder binder = gate == And ? (Binder)Expression.And : Expression.Or;

        Expression bind(Expression left, Expression right) =>
            left == null ? right : binder(left, right);

There’s a bit to digest. The gate variable is the condition, i.e. “and” or “or.” The rules statement gets a node that is the list of related rules. We are keeping track of the left and right side of the expression. The Binder signature is shorthand for a binary expression, and is defined here:

private delegate Expression Binder(Expression left, Expression right);

The binder variable simply sets up the top-level expression: either Expression.And or Expression.Or. Both take left and right expressions to evaluate.

The bind function is a little more interesting. As we traverse the tree, we need to build up the various nodes. If we haven’t created an expression yet (left is null) we start with the first expression created. If we have an existing expression, we use that expression to merge the two sides.

Right now, left is null, and then we start enumerating the rules that are part of this condition:

foreach (var rule in rules.EnumerateArray())

Property Expressions

The first rule is an equality rule, so I’ll skip the condition part for now. Here’s what happens:

string @operator = rule.GetProperty(nameof(@operator)).GetString();
string type = rule.GetProperty(nameof(type)).GetString();
string field = rule.GetProperty(nameof(field)).GetString();
JsonElement value = rule.GetProperty(nameof(value));
var property = Expression.Property(parm, field);

First, we get the operator (“in”), the type (“string”), the field (“Category”) and the value (an array with “Clothing” as the only element). Notice the call to Expression.Property. The LINQ for this rule would look like this:

var filter = new List<string> { "Clothing" };
Transactions.Where(t => filter.Contains(t.Category));

The property is the t.Category part, so we create it based on the parent property (t) and the field name.

Constant and Call Expressions

Next, we need to build the call to Contains. To simplify things, I created a reference to the method here:

private readonly MethodInfo MethodContains = typeof(Enumerable).GetMethods(
  BindingFlags.Static | BindingFlags.Public)
  .Single(m => m.Name == nameof(Enumerable.Contains)
      && m.GetParameters().Length == 2);

This grabs the method on Enumerable that takes two parameters: the enumerable to use and the value to check for. The next bit of logic looks like this:

if (@operator == In)
{
    var contains = MethodContains.MakeGenericMethod(typeof(string));
    object val = value.EnumerateArray().Select(e => e.GetString())
        .ToList();
    var right = Expression.Call(
        contains,
        Expression.Constant(val),
        property);
    left = bind(left, right);
}

First, we use the Enumerable.Contains template to create an Enumerable<string> because that is the type we are looking for. Next, we grab the list of values and turn it into a List<string>. Finally, we build our call, passing it:

The method to call (contains)
The value that is the parameter to check for (the list with “Clothing”, or Expression.Constant(val))
The property to check it against (t.Category).

Our expression tree is already fairly deep, with parameters, properties, calls, and constants. Remember that left is still null, so the call to bind simply sets left to the call expression we just created. So far, we look like this:

Transactions.Where(t => (new List<string> { "Clothing" }).Contains(t.Category));

Iterating back around, the next rule is a nested condition. We hit this code:

if (rule.TryGetProperty(nameof(condition), out JsonElement check))
{
    var right = ParseTree<T>(rule, parm);
    left = bind(left, right);
    continue;
}

Currently, left is assigned to the “in” expression. right will be assigned as the result of parsing the new condition. I happen to know it’s an OR condition. Right now, our binder is set to Expression.And so when the function returns, the bind call leaves us with:

Transactions.Where(t => (new List<string> { "Clothing" }).Contains(t.Category) && <something>);

Let’s look at “something”.

Comparison Expressions

First, the recursive call determines a new condition exists, this time a logical OR. The binder is set to Expression.Or and the rules begin evaluation. The first rule is for TransactionType. It is set as boolean but from what I infer, it means the user in the interface can check to select a value or toggle to another value. Therefore, I implemented it as a simple string comparison. Here’s the code that builds the comparison:

object val = (type == StringStr || type == BooleanStr) ?
    (object)value.GetString() : value.GetDecimal();
var toCompare = Expression.Constant(val);
var right = Expression.Equal(property, toCompare);
left = bind(left, right);

The value is deconstructed as a string or decimal (a later rule will use the decimal format). The value is then turned into a constant, then the comparison is created. Notice it is passed the property. The variable right now looks like this:

Transactions.Where(t => t.TransactionType == "income");

In this nested loop, left is still empty. The parser evaluates the next rule, which is the payment mode. The bind function turns it into this “or” statement:

Transactions.Where(t => t.TransactionType == "income" || t.PaymentMode == "Cash");

The rest should be self-explanatory. A nice feature of expressions is that they overload ToString() to generate a representation. This is what our expression looks like (I took the liberty of formatting for easier viewing):

( (value(System.Collections.Generic.List`1[System.String]).Contains(Param_0.Category)
      And (
          (Param_0.TransactionType == "income")
          Or
          (Param_0.PaymentMode == "Cash"))
      )
  And
  (Param_0.Amount == 10)
)

It looks good… but we’re not done yet!

Lambda Expressions and Compilation

The expression tree represents an idea. It needs to be turned into something material. In case the expression can be simplified, I reduce it. Next, I create a lambda expression. This defines the shape of the parsed expression, which will be a predicate (Func<T,bool>). Finally, I return the compiled delegate.

var conditions = ParseTree<T>(doc.RootElement, itemExpression);
if (conditions.CanReduce)
{
    conditions = conditions.ReduceAndCheck();
}
var query = Expression.Lambda<Func<T, bool>>(conditions, itemExpression);
return query.Compile();

To “check my math” I generate 1000 transactions (weighted to include several that should match). I then apply the filter and iterate the results so I can manually test that the conditions were met.

var predicate = jsonExpressionParser
                .ParsePredicateOf<Transaction>(jsonDocument);
var transactionList = Transaction.GetList(1000);
var filteredTransactions = transactionList.Where(predicate).ToList();
filteredTransactions.ForEach(Console.WriteLine);

As you can see, the results all check out (I average about 70 “hits” each run.)

Beyond Memory to Database

The generated delegate isn’t just for objects. We can also use it for database access.

Set the startup project to DatabaseTest for the rest of this post. If you run it from the command line, be sure that databaseRules.json is in your current directory.

First, I refactored the code. Remember how expressions require a data source? In the previous example, we compile the expression and end up with a delegate that works on objects. To use a different data source, we need to pass the expression before it is compiled. That allows the data source to compile it. If we pass the compiled data source, the database provider will be forced to fetch all rows from the database, then parse the returned list. We want the database to do the work. I moved the bulk of code into a method named ParseExpressionOf<T> that returns the lambda. I refactored the original method to this:

public Func<T, bool> ParsePredicateOf<T>(JsonDocument doc)
{
   var query = ParseExpressionOf<T>(doc);
   return query.Compile();
}

The ExpressionGenerator program uses the compiled query. The DatabaseTest uses the raw lambda expression. It applies it to a local SQLite database to demonstrate how the expression is parsed by EF Core. After creating and inserting 1000 transactions into the database, the code retrieves a count:

var count = await context.DbTransactions.CountAsync();
Console.WriteLine($"Verified insert count: {count}.");

This results in the following SQL:

SELECT COUNT(*)
FROM "DbTransactions" AS "d"

If you’re wondering why there are two contexts, it is due to logging. The first context inserts 1000 records and if logging is turned on, it will run very slow as the inserts are written to the console. The second context switches on logging so you can see the evaluated statement.

The predicate is parsed (this time from a new set of rules in databaseRules.json) and passed to the Entity Framework Core provider.

var parser = new JsonExpressionParser();
var predicate = parser.ParseExpressionOf<Transaction>(
   JsonDocument.Parse(
      await File.ReadAllTextAsync("databaseRules.json")));
var query = context.DbTransactions.Where(predicate)
   .OrderBy(t => t.Id);
var results = await query.ToListAsync();

With Entity Framework Core logging turned on, we are able to retrieve the SQL and see the items are fetched in one pass and evaluated in the database engine. Notice the PaymentMode is checked for “Credit” instead of “Cash”.

SELECT "d"."Id", "d"."Amount", "d"."Category", "d"."PaymentMode", "d"."TransactionType"
FROM "DbTransactions" AS "d"
WHERE ("d"."Category" IN ('Clothing') &
        ((("d"."TransactionType" = 'income') AND "d"."TransactionType" IS NOT NULL) |
          (("d"."PaymentMode" = 'Credit') AND "d"."PaymentMode" IS NOT NULL))) &
      ("d"."Amount" = '10.0')
ORDER BY "d"."Id"

The example app also prints one of the selected entities to spot-check.

Conclusion

LINQ expressions are a very powerful tool to filter and transform data. I hope this example helped demystify how expression trees are built. Of course, parsing the expression tree feels a little like magic. How does Entity Framework Core walk the expression tree to produce meaningful SQL? I’m exploring that myself and with a little help from my friend, ExpressionVisitor. More posts to come on this!

Regards,

Azure AD Secured Serverless Cosmos DB from Blazor WebAssembly

Jeremy Likness ⚡️ — Mon, 25 May 2020 07:00:38 +0000

A Blazor 🔥 WebAssembly 🕸 app can be deployed as a set of static website assets and hosted on any server that publishes static files. As a browser application, it is capable of making calls to HTTP API endpoints. As a Single Page Application (SPA) it can render new pages dynamically from the data. Because it supports .NET Standard, it is possible to load NuGet packages that target .NET Standard, including the Cosmos DB 🌍 SDK. This makes it possible to connect to the Azure Cosmos DB SQL API backend directly. There is one major challenge: how can you connect securely without storing credentials in the client and hosting your own identity service?

The solution is to use Azure Active Directory for authentication and communicate securely with a serverless Azure Function. The function app uses securely stored master credentials to connect to Cosmos DB and generate an ephemeral token that grants limited access to a specific user for up to five hours. The client app then connects directly to Cosmos DB using the provided token.

The repository for this solution is at:

JeremyLikness / AzureBlazorCosmosWasm

A completely serverless solution to access Cosmos DB from Blazor WebAssembly using EF Core.

Azure Active Directory from a Blazor WebAssembly App to Access Cosmos DB

A completely serverless solution to access Cosmos DB from Blazor WebAssembly using EF Core.

To configure your application and learn more, read:

Azure AD Secured Serverless Cosmos DB from Blazor WebAssembly

"Quick" start

Optional: fork the repo
git clone https://github.com/jeremylikness/AzureBlazorCosmosWasm.git (or your fork)
Create an app registration for the Blazor app (there is an Azure CLI script for this linked in the comments of the blog post)
Update wwwroot/appsettings.json in the Blazor WebAssembly client to use the tenant (directory) and client id
Create a Cosmos DB database and seed some initial values
Deploy the Azure Functions app
Add the Cosmos DB connection string as "CosmosConnection" under connection strings for the Azure Functions app
Update authentication for the Azure Functions app to use Azure AD
Update wwwroot/appsettings.json in the Blazor WebAssembly project to point to your functions app (under…

View on GitHub

The app requires some initial configuration that is explained in this blog post. This post explores how I built the solution and how you can build and configure your own app from the repo.

Blazor WebAssembly and Authentication

The first step is to setup authentication in the Blazor WebAssembly app. I’m no Azure AD expert, so it was extremely helpful to find dedicated documentation on how to use Blazor WebAssembly with Azure AD.

Special thanks to Javier Calvarro Nelson for investing his time to provide guidance for me to properly configure the solution.

I’ll summarize the high-level steps:

Register your application in the Azure Active Directory App Registrations. This enables your application to authenticate the user against Azure AD and to make requests on behalf of the user using their identity and credentials.
Be sure to note two important components of the app registration: the tenant (or directory) the registration is a part of (this is like the zip code of the Azure AD authentication service) and the client id that uniquely identifies your app (like a postal address).
Generate the Blazor WebAssembly app using the built-in Azure AD template:

dotnet new blazorwasm -au SingleOrg --client-id "{CLIENT ID}" --tenant-id "{TENANT ID}"

This will scaffold an application that provides full integration with Azure AD. The document referenced earlier explains what code and components are generated by the authentication template and how they work. To use the sample project for this blog post, start by creating an application registration. After that is setup, update the appsettings.json file under the wwwroot folder in the AzureBlazorWasm project to use your tenant and client. It will look like this:

"AzureAd": {
   "Authority": "https://login.microsoftonline.com/{directory id}",
   "ClientId": "{clientid}",
   "ValidateAuthority": true
}

This information is stored as part of the client app and is “in the clear” meaning the user can easily access and read it. Fortunately, there are no secrets here. The configuration simply provides the endpoint to request authentication, using your organization’s tenant, and the unique identifier (client id) of your Blazor app. The login process redirects you to Azure AD where you are prompted to login using whatever process was configured for your tenant. This may include two-factor authentication. If authentication succeeds, a signed token is sent back to the Blazor WebAssembly client.

Now you have a secure Blazor client. You can use the [Authorize] attribute on any Razor component to prevent access by unauthorized users. I’ll show you that later in this post.

Secure Azure Functions with Azure AD

The next step is to create a secure connection to Azure Functions <⚡>. The functions app will communicate with Cosmos DB 🌍 to retrieve a token, so you don’t want just anybody accessing it. You can start with the existing CosmosAuthentication app and publish it to Azure. If you’re not familiar with the process, it is documented here: Develop Azure Functions using Visual Studio: Publish to Azure. The functions application is not yet secured. However, the code checks for an authenticated user and will fail by returning a 401 Unauthorized HTTP status code.

Follow the steps in this document to secure your functions app: Secure an Azure Functions App with Azure AD: Express Setup. I used the express option and created a new app registration. I chose the option to “Allow anonymous requests (no action)” because the code itself validates the identity of the connected user and the API is not intended to be accessed manually. The express setup will automatically create an application registration for the function app.

There is some additional configuration to perform. In the Azure Portal, navigate to “Azure Active Directory” then “App registrations” and finally open the registration for your functions app (not your Blazor WebAssembly client). Open “Authentication” and make sure that “Implicit grant” for “ID tokens” is set.

Next, we need to expose the API for consumption by other applications. Open the “Expose an API” tab. There are two steps. First, if it doesn’t already exist, you need to add a scope for user_impersonation which is in essence a permission that applies to the API. Tap “Add a Scope” and name it user_impersonation. If your access allows, choose “Admins and users” for the consent. Before the functions app can use any credentials, users must explicitly opt-in by providing consent. Some organizations require an administrator to provide consent. You can revoke consent at any time and effectively disable access to the functions app. Be sure the scope is enabled.

The scope is just a permission that the client app requests. In this case, the client app is the Blazor WebAssembly app. To allow it to work with the functions app, you need to authorize it. At the bottom of the same page is a section for “Authorized client applications.” Tap “Add a client application” and paste in the client id that was generated for your Blazor WebAssembly app registration. Under “Authorized scopes” ensure that user_impersonation is checked.

Note The full scope is the URL of the app registration (typically the same as your function app) and the scope name. You’ll need that later. The Blazor client assumes the functions app registration has the same name as the functions app.

My function app is named cosmosauthenticationfunc so my full scope is:

https://cosmosauthenticationfunc.azurewebsites.net/user_impersonation

Now is a good time to pause. What did we do?

We created an app registration to secure a Blazor WebAssembly app with Azure AD.
We generated the necessary code to authenticate with Azure AD from the Blazor WebAssembly app.
We created an app registration to secure an Azure Functions app with Azure AD.
We enabled support for tokens.
We ensured the user_impersonation scope (permission) is available.
We authorized our Blazor WebAssembly app to securely access the Azure Functions app using the user_impersonation scope.

We’re not quite done yet! Even though the functions app is configured to grant permissions to the Blazor WebAssembly app, the Blazor app must explicitly ask for permissions. This is the last step to close the loop. In Azure Active Directory app registrations, navigate to your Blazor WebAssembly registration. Tap on “API Permissions” then “Add a permission.” Choose “My APIs” and select the Azure Functions app registration. Tap “delegated permissions” and make sure that user_impersonation is checked. Click “Add permissions” to apply and save.

Now the functions app is locked down! The job is not done yet. If you try to access the Azure Function from the Blazor WebAssembly app, even after you are logged in, you will receive a 401 Unauthorized HTTP status code. That is because the request doesn’t contain any authentication information. The HttpClient must be configured to stash the Azure AD token in the request header.

Configure Blazor WebAssembly to use the Azure AD Token

In Blazor WebAssembly apps, the HttpClient is injected using dependency injection. The default client is configured with the same base URL as the client, so that you can make calls to the local server using a partial path, like this:

var entity = await _client.GetJsonAsync<MyEntity>("/api/entity/get");

This is configured in Program.cs:

builder.Services.AddTransient(sp => new HttpClient()
   { BaseAddress = new Uri(builder.HostEnvironment.BaseAddress) });

The Azure Functions endpoint is at a different base URL and requires the request header to contain the authentication information. To configure the client, I added the Microsoft.Extensions.Http NuGet package that provides helper methods for creating an HttpFactory. This allows configuration of specific instances of the client that can be requested by name. More on that in a minute, first we need to set up authorization.

The application has a built-in AuthorizationMessageHandler that automatically configures the request header to contain the authorization token for local requests. We need a custom handler that allows the token to be sent to the remote functions address. In the Data folder, I created this:

public class CosmosAuthorizationMessageHandler : AuthorizationMessageHandler
{
    public CosmosAuthorizationMessageHandler(
        IConfiguration config,
        IAccessTokenProvider provider,
        NavigationManager navigation) : base(provider, navigation)
    {
        var section = config.GetSection(nameof(TokenClient));
        var endpoint = section.GetValue<string>(nameof(TokenClient.Endpoint));
        ConfigureHandler(new[] { endpoint });
    }
}

The main purpose of the custom handler is to authorize storing the Azure AD token in the request headers when the client accesses the functions endpoint. We’ll access the function endpoint from a class named TokenClient. To configure the endpoint, I created a section in appsettings.json that is deployed under wwwroot. This makes it possible to update the endpoint value as part of your CI/CD process. The section in the configuration file looks like this:

"TokenClient": {
   "Endpoint": "https://{functionapp}.azurewebsites.net/"
}

Replace the setting in your file with your endpoint. The handler uses this configuration to grab the endpoint. Now that the message handler is done, we can set up the HttpClient factory in Program.cs. First, I created a local helper method to retrieve the functions endpoint from configuration. Although it is configured for the authorized endpoint by the message handler, we need it to configure the base address of the HttpClient instance and to request the scope.

static string functionEndpoint(WebAssemblyHostBuilder builder) =>
   builder.Configuration
      .GetSection(nameof(TokenClient))
      .GetValue<string>(nameof(CosmosAuthorizationMessageHandler.Endpoint));

Next, I updated the template-generated authentication configuration to explicitly request the user_impersonation scope and registered my custom message handler.

builder.Services.AddMsalAuthentication(options =>
{
    options.ProviderOptions
    .DefaultAccessTokenScopes.Add($"{functionEndpoint(builder)}user_impersonation");
    builder.Configuration.Bind("AzureAd", options.ProviderOptions.Authentication);
});
builder.Services.AddTransient<CosmosAuthorizationMessageHandler>();

Finally, I registered the custom HttpClient to use the custom message handler and default to the base address of the functions endpoint.

// configure the client to talk to the Azure Functions endpoint.
builder.Services.AddHttpClient(nameof(TokenClient),
    client =>
    {
        client.BaseAddress = new Uri(functionEndpoint(builder));
    }).AddHttpMessageHandler<CosmosAuthorizationMessageHandler>();

// register the client to retrieve Cosmos DB tokens.
builder.Services.AddTransient<TokenClient>();

The first step uses the extensions to register a “named” HttpClient with the name “TokenClient.” The second registers the client we’ll use to retrieve tokens. Now we can look at the TokenClient implementation to see how this is used.

public class TokenClient
{
    private readonly HttpClient _client;

    public TokenClient(IHttpClientFactory factory)
    {
        _client = factory.CreateClient(nameof(TokenClient));
    }

    public async Task<CosmosToken> GetTokenAsync()
    {
        return await _client.GetJsonAsync<CosmosToken>($"api/RequestToken");
    }
}

The client is very simple. It uses the injected factory to get the named HttpClient configured with the message handler that will apply the Azure AD token to the request. Notice that it doesn’t use a fully qualified domain name because that has already been configured as the base URL. The service exposes the method GetTokenAsync() to request a token. When I built the app, I started by returning an empty token. The token definition is shared across projects and contains the Cosmos DB endpoint and the ephemeral token (key) that will be retrieved from Cosmos DB.

public class CosmosToken
{
   public string Endpoint { get; set; }
   public string Key { get; set; }
}

After this step, I was able to successfully retrieve an empty token from the Azure Functions app. Next, the Azure Functions app needs to retrieve the authenticated user’s information and use that to generate the Cosmos DB token.

Serverless Cosmos DB Token Generation

The Azure AD token is a signed and encoded payload that includes a set of trusted claims. These claims contain information including the user’s unique identity. You can configure custom claims as well. For example, to access the user’s email you would add an email claim. When the user is prompted to provide consent to your app, it will include email as one of the items being consented to.

Here is my rough conceptualization of the token being passed from the Blazor WebAssembly app to the Azure Functions endpoint.

I used email as an illustration, but I don’t use it in this example.

On the Azure Functions side, we can now do this:

[FunctionName("RequestToken")]
public async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] 
        HttpRequest req,
    ILogger log,
    ClaimsPrincipal principal)
{
    if (principal == null)
    {
        log.LogWarning("No principal.");
        return new UnauthorizedResult();
    }

    if (principal.Identity == null)
    {
        log.LogWarning("No identity.");
        return new UnauthorizedResult();
    }

    if (!principal.Identity.IsAuthenticated)
    {
        log.LogWarning("Request was not authenticated.");
        return new UnauthorizedResult();
    }

    var id = principal.FindFirst(ClaimTypes.NameIdentifier).Value;
    log.LogInformation("Authenticated user {user}.", id);

    return new OkResult();
}

Notice that I include ClaimsPrincipal in the parameters and Azure Functions automatically passes in the principal associated with the request, if one exists. The log messages are how I confirmed the token authentication was working. It’s not useful yet because we don’t have a token, but we can at least get the user’s unique NameIdentifier claim. To grab the token, I created a CosmosClientWrapper that is injected into the function class. If you’re not familiar with how to setup dependency injection in Azure Functions, read: Use Dependency Injection in Azure Functions.

The client wrapper uses dependency injection to grab the configuration. It expects a configuration string named “CosmosConnection.” Be sure to set that up in the ConnectionStrings portion of your configuration, whether in your local JSON settings or in the actual application’s app settings. This connection string uses the master key so it can manipulate permissions.

A logger is also injected to illustrate what is happening.

private readonly string CosmosConnection = nameof(CosmosConnection);

public CosmosClientWrapper(
    IConfiguration config,
    ILogger<CosmosClientWrapper> logger)
{
    _logger = logger;
    _client = new CosmosClient(config.GetConnectionString(CosmosConnection));
}

Cosmos DB Users and Permissions

Cosmos DB only supports Azure AD for the “control plane” or administration of the Cosmos DB account in the portal or using the command line interface. The “data plane” (what we’re interested in) requires master keys or resource tokens. This is why we use an Azure Function to “broker” the transaction and securely retrieve a token for the user. You can read more about Cosmos DB permissions in the previously linked article, but here is a brief summary:

Create a named user that is scoped to the database.
Apply a named permission that specifies if the user can update or only read data, what container they have access to, and optionally a partition key for multi-tenant solutions.
Request a token to operate under the set of named permissions.

The token lasts for one hour by default. It can be configured to expire anywhere between ten seconds and five hours. Any time you request a token for a set of permissions, you receive a new token with a new expiration.

Conceptually, this is what we’ll do:

First, we need to check if the user exists. If not, we create the user. The user has the same name as the NameIdentifier claim so it is unique to each user. Although it is not part of this example, you could write code to delete the user when their Azure AD access is revoked.

private async Task<User> CreateOrReadUserAsync(
    Database database, string id)
{
    _logger.LogInformation("User request for {user}.", id);
    var user = database.GetUser(id);
    UserResponse userResult = null;
    try
    {
        userResult = await user.ReadAsync();
    }
    catch (CosmosException cex)
    {
        if (cex.StatusCode != System.Net.HttpStatusCode.NotFound)
        {
            throw;
        }
    }
    if (userResult?.Resource == null)
    {
        _logger.LogInformation("User {user} not found.", id);
        var newUser = await database.CreateUserAsync(id);
        user = newUser.User;
        _logger.LogInformation("User {user} created.", id);
    }
    else
    {
        _logger.LogInformation("User {user} exists.", id);
    }
    return user;
}

The following code creates a unique set of permissions for the user. We are granting read access to the container that holds blog posts. Instead of worrying about if the permissions already exist, we use the upsert functionality to either add or update them. We then request and return the token.

public async Task<CosmosToken> GetTokenForId(string id)
{
    var database = _client.GetDatabase(BlogContext.MyBlogs);
    var cosmosUser = await CreateOrReadUserAsync(database, id);
    var permissionId = $"Permission-{id}-blogs";
    var container = database.GetContainer(nameof(BlogContext));
    var permissions = new PermissionProperties(
        id: permissionId,
        permissionMode: PermissionMode.Read,
        container: container);
    await cosmosUser.UpsertPermissionAsync(permissions);
    _logger.LogInformation("Permissions upsert for {user} successful.", id);
    var token = await cosmosUser.GetPermission(permissionId).ReadAsync();
    return new CosmosToken
    {
        Endpoint = _client.Endpoint.ToString(),
        Key = token.Resource.Token
    };
}

The function app assumes that you already created a database named “myblogs”. The container, named “BlogContext” is created by Entity Framework Core. I explain how to set that up next.

Set up the Database

At this stage, you have everything you need to access Cosmos DB using a resource token. I use Entity Framework Core in this example to be consistent with my previous blog post that uses an ASP.NET Core hosted solution. That blog post explains the rationale for using EF Core and how the data model is setup.

To create the database for the first time and seed some data, update the SeedData class in the BlogData project with the blogs and posts you want to start with, and call it from a console application. You’ll need to modify the example to pass in a DbContextOptionsBuilder that configures the Cosmos DB connection string using the master keys. Install the EF Core Cosmos Provider then upgrade the referenced Cosmos DB SDK to version 3.9.1 or later by adding an explicit NuGet package reference.

Finishing the Client

Now all of the foundational pieces are in place. To retrieve the data from the Blazor WebAssembly app, I created a special client. It is designed to return a DbContext, but you can easily modify it to return a CosmosClient instance instead if you wish to use the SDK directly instead of via EF Core.

public class BlogClient
{
    private CosmosToken _credentials;

    private readonly TokenClient _tokenClient;

    public BlogClient(TokenClient tokenClient)
    {
        _tokenClient = tokenClient;
    }

    public async Task<BlogContext> GetDbContextAsync()
    {
        if (_credentials == null)
        {
            _credentials = await _tokenClient.GetTokenAsync();
        }

        BlogContext context = null;

        CosmosToken getCredentials() => _credentials;

        var options = new DbContextOptionsBuilder<BlogContext>()
            .UseCosmos(getCredentials().Endpoint,
                getCredentials().Key,
                Context.MyBlogs,
            opt =>
                opt.ConnectionMode(Microsoft.Azure.Cosmos.ConnectionMode.Gateway));

        try
        {
            context = new BlogContext(options.Options);
        }
        catch
        {
            _credentials = await _tokenClient.GetTokenAsync();
            context = new BlogContext(options.Options);
        }

        return context;
    }
}

The client creates an instance of the context to access the database and holds a reference to the resource token. If the constructor throws, it assumes it is due to the expired token and fetches a new one to try a second time. If it fails the second time, it is a different issue and the error will propagate.

The Index.razor page has a few interesting elements to point out. It uses:

@attribute [Authorize]

The attribute requires authorization for the component. The user will be forced to login before the component renders. The route is set to /cosmos so the default route can render a page that doesn’t require authentication. When it requests the context, it captures the AccessTokenNotAvailableException that is thrown by the message handler when a token is not available, revoked, or expired. The exception class provides a convenient Redirect() method that redirects the user to sign in again.

When I run the application against my Cosmos DB instance I see this render of blogs and posts:

This is what the resulting Azure Functions logs look like:

The browser’s network inspector shows this sequence of handshakes:

The requests are appear double because the first one is a CORS request followed by the actual API call. In order the calls:

Request the token
Authenticate with Cosmos DB and retrieve connection details
Fetch details about the container
Setup querying the container
Retrieve information about partition keys (not configured for this)
Retrieve the documents

The payload returned by the documents request looks like this:

It is automatically parsed and materialized by EF Core.

Conclusion

That’s the end of the story. I would love to point you to a working example, but because this is a solution based on Azure AD it requires a configured instance you can log into. My goal with sharing this solution is to demonstrate the flexibility of Blazor, the power of having direct access to Cosmos DB and the steps necessary to properly secure a client application without storing any secrets or credentials.

As always, I appreciate your thoughts, feedback, questions, and comments, so scroll down to join or start the conversation!

Regards,

EF Core and Cosmos DB with Blazor WebAssembly

Jeremy Likness ⚡️ — Thu, 14 May 2020 07:26:20 +0000

It all began as a simple idea. I’ve been speaking with customers about Entity Framework Core (EF Core) and was intrigued by the popularity of using EF Core to access Cosmos DB. I’m a huge fan of Cosmos DB and have worked with it since it evolved from Azure DocumentDB. It wasn’t clear to me why anyone would want to use EF Core when Cosmos DB has its own SDK. I started asking questions and Julie Lerman answered with this tweet for context:

// Detect dark theme var iframe = document.getElementById('tweet-1245732530160050176-161'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1245732530160050176&theme=dark" }

Many customers shared the same feedback:

EF Core presents a common interface and set of APIs for data access
Developers fluent in that API prefer to use it over other approaches
EF Core hides many of the nuances of setting up the Cosmos DB client to access documents
EF Core by convention creates a container named after the context and then stores all entity types in that table with a special type discriminator. This means less work for the developer
In some scenarios it is possible to reuse code across providers

The last point was interesting because I spoke to an internal team working on a process that distributes data to Azure SQL and Cosmos DB. They are using Azure Functions and chose EF Core so that most of the code remains the same between connectors. For the most part, they just plug in the appropriate provider to make it work.

Speaking with these teams sold me on the idea of using EF Core for Cosmos DB. What happened next was unexpected.

The Idea

Blazor WebAssembly is one of the most amazing new technologies I’ve worked with in my career. I know that’s a bold statement, but it’s not just based on personal experience. Enabling .NET apps to run in the browser efficiently and without plug-ins is revolutionary. Customers are migrating desktop and legacy web apps to the new platform. C# developers are rejoicing that they can build responsive Single Page Apps without getting sucked into choosing a JavaScript framework and learning an entirely new language and platform.

As powerful as the technology is, it is constrained by the fact it runs as a client in the browser. This means it can’t make direct TCP connections, for example, which limits the databases it can connect to directly. All the code runs in the browser, so credentials can’t be secured the same way they are on the webserver. Therefore most access is done over HTTP APIs and creates the challenge of deciding how to serialize complex queries over the wire.

Although there are new solutions like OData and GraphQL, I wondered if they are always necessary. Firebase is a popular solution and, from what I understand, developers can easily (but securely) connect directly from the browser. Would a similar experience be possible with Blazor?

A final push for the idea came from friend and coworker Brady Gaster. He was working on a special demo and shared how much he enjoyed using EF Core for Cosmos DB. We chatted about the potential to use it in Blazor apps and considered the implications of Blazor Server vs. Blazor WebAssembly. That’s when I had my “Ah-hah!" moment (thanks, Brady!).

Why Cosmos DB

The Cosmos DB client allows two connection modes: direct (TCP) and gateway (HTTPS). Gateway mode should be usable from the browser. The challenge is how to secure credentials. There is no safe way to ship them in the browser and if anyone were to intercept them, your entire database will be compromised. I decided to first tackle if it could work, then tackle whether it should be done based on security concerns. But why even consider Cosmos DB in the browser? There are a few reasons:

Cosmos DB can be configured with multiple regions and will automatically route to the closest (and fastest) node
Building it into the client app removes the need to build an additional API tier (with exceptions, more on that later)
Enterprise apps often have complex grids and search mechanisms. A client solution allows applying those queries directly to the database instead of figuring out some way to serialize them and re-materialize over the wire
The most popular web frameworks tend to be easy to use with minimal moving parts, and I’d love to see us offer something similar

Things began to click into place. I decided I had what I need to build a quick experiment. I immediately set to work, and just as fast, hit my first roadblock. More on that in a minute.

Why Entity Framework Core

My decision to use Entity Framework Core goes beyond the fact that I work closely with the team. I’ve learned in my conversations with developers that they love using EF Core as a data access API. It also simplifies the strategy of how to store documents and discriminate between varying types - this is all done automatically. I was confident that EF Core could run in Blazor because it targets .NET Standard 2.1. The question was only whether the provider would work. I had also just demoed some new EF Core features in the ASP.NET Community Standup.

I used Cosmos DB and built an example app that handles blogs and blog posts. To keep my experiment fast and easy, I simply copied the setup from that app into my Blazor WebAssembly project. I was up and running in just a few minutes. The moment of truth had arrived. For now, I brazenly embedded my secrets as hard-coded string literals that I used to configure the database context. I hit CTRL+F5 to “Start without Debugging” and eagerly waited for the blog posts to render.

Instead, what I got was an exception.

Mono Blocker

To reproduce the error and eliminate EF Core as the culprit, I used this very simple statement:

var client = new CosmosClient(
   "[CosmosEndpoint]",
   "[CosmosKey]",
   new CosmosClientOptions()
   {
      ConnectionMode = ConnectionMode.Gateway,
      LimitToEndpoint = true
   }
);

Before even attempting to issue a request, the client was failing. The exception had a very large call stack with this at the top:

System.Threading.SynchronizationLockException: Cannot wait on monitors on this runtime.

Blazor doesn’t allow blocking code, and something was blocking. I filed GitHub Issue #1429 and set aside my hopes and dreams for a while. OK, maybe it wasn’t that dramatic. The Cosmos DB SDK team responded quickly and figured out how to detect the mono runtime to change some settings with the HTTP client. I was excited to receive this email from Matias:

Subject: RE: Cosmos client on Blazor WebAssembly
Jeremy,
The 3.9.0-preview2 Nuget package is now out,
with the changes for WebAssembly,
you can now try it yourself 😊
Matias

I pulled down the preview package, recompiled and ran it. Success! (Thanks, Matias!) But there was still one problem left to solve.

Security Concerns

Blazor WebAssembly runs entirely in the browser. This creates a major security concern. In most applications, the secrets are hidden in an API tier. That tier communicates with the database on behalf of the user. It is capable of restricting access based on their role. If the user credentials are compromised, a hacker can impersonate that user but is constrained to the subset of actions and data their role allows. This is far more containable than the case of a hacker obtaining administrative credentials. They could then read sensitive information or even destroy the entire database.

Storing the credentials is therefore a problem. Even if the credentials are encrypted, there has to be a way to decrypt them in order to use them. If your code can decrypt them, assume the user who has your code running in their browser can too. Therefore, ⛔ embedded credentials are a no-go.

What about requesting the credentials from the server? HTTPS is a secure protocol. It is why we trust web applications in the first place. When I log into my bank, I know a complicated handshake occurred to encrypt my data for that session so that would-be attackers are unable to steal data from the wire. This is all fine, but that data must be decrypted in the browser to be used. I can easily look at my own traffic to see what is being sent and received. Therefore ⛔ sending me master credentials over a secured connection is meaningless because I can view them on my end and use them to take over the database.

So how does Firebase work? Although Firebase “feels” like a serverless solution, my understanding is that the service itself hosts information about users. When a user authenticates from the browser, they are giving their credentials (username and password) and receiving a connection specifically configured for them. That connection is completely controlled through role-based access (RBAC) so the scope of what they can do is limited. The question is, can we create a similar scenario with Cosmos DB? It turns out the answer is, “Yes.”

📱 It turns out this problem has already been solved in mobile applications. Although there are secure ways to store secrets on phones, no one wants to risk shipping the master keys. So how do they do it?

The official documentation for Cosmos DB includes an article to learn how to secure access to data. In that article you can read about “resource tokens.” These are ephemeral (short-lived) keys for access that are generated specifically for a user. The keys allow very refined access permissions and even allow you to scope access to a particular partition key for multi-tenant scenarios. The maximum lifespan of a token is five hours, but you can configure it to a shorter duration. The default is just one hour.

Handing the user their own token only allows them to do what we want them to do anyway. If the token is somehow intercepted, the hacker only has the same level of access they would via a Web API scenario: they can only act based on the permissions the user has. They also have a limited timeline to do so.

I would be remiss to not include a major security notice here. I believe there are viable scenarios that make sense if you follow the principle of least privilege and issue only the most restrictive tokens possible (the least amount of access acceptable for the app).

Here’s a poorly drawn diagram of the process:

Let’s review pros and cons:

Pros:

Simplified access from the client
Removes an additional round-trip to an API service for subsequent calls (after the token request) resulting in lower latency
Takes advantage of Cosmos DB global distribution to route to the nearest cluster
Token is ephemeral and restricted
Cross Original Resource Sharing (CORS) is configurable to constrain access (I was unable to access my endpoint until I set this up for my localhost port)

Cons:

The token can be intercepted or stolen
The endpoint of your Cosmos DB database is exposed, opening up a potential attack vector
You have to either disable your IP access control policy or somehow capture all potential IP endpoints for access

⚠ The last two concerns aren’t unique to Blazor WebAssembly apps. They also exist for mobile apps that access Cosmos DB directly. I’m curious about your thoughts: do you consider the endpoint to be a “secret” and how much risk do you feel is involved when it is revealed? Would love to hear your comments below.

💡 For reference, here’s the documentation for accessing Cosmos DB from Xamarin (mobile) apps. Does anything look familiar? They had professionals draw their diagram. It turns out we’re not in completely unchartered territory here.

The Final Solution

After I was successful, I posted a teaser to Twitter. I did not anticipate the magnitude of the response I received. The biggest (well-placed) concerns are around how security is implemented.

// Detect dark theme var iframe = document.getElementById('tweet-1260340804351975424-920'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1260340804351975424&theme=dark" }

I apologize to anyone looking for an out of the box solution. My project uses the latest preview bits and relies on ASP.NET Core Identity for authentication, so it has a lot of moving parts that will likely be out of date before I publish this. Instead, I’ll walk through the relevant steps and share the code I used along the way.

The first step is to create a Blazor WebAssembly project. I used the Release Candidate (3.2.0-rc1.20223.4). Choose “ASP.NET Core Hosted” so you can host authentication, then pick the option to include authentication in the app. This will set up a SQL database to store user profiles. It requires you to run an initial migration to set up the database (thanks to template magic, it will give you an option to do this directly from an error page the first time you run the app). This is just to facilitate an authentication mechanism for the example. If you have an external authentication provider, feel free to use that instead.

Your project structure will include:

BlazorApp.Client
BlazorApp.Server
BlazorApp.Shared

The preview packages target .NET 5, but the server solution defaults to ASP.NET Core 3.1. To fix that, I simply opened up the .csproj file for the server and changed the target framework to 5.0:

<PropertyGroup>
   <TargetFramework>net5.0</TargetFramework>
   <UserSecretsId>Caught you looking</UserSecretsId>
</PropertyGroup>

The client hosts your Blazor WebAssembly project. The server hosts authentication and any other APIs you choose to stand up. Shared is just what it implies. Now for some additions. From the NuGet package manager, install the Azure Cosmos DB SDK (Microsoft.Azure.Cosmos) version 3.9.0-preview3 or later for the server project. Install EF Core for Cosmos (Microsoft.EntityFrameworkCore.Cosmos) on the client project. I used 5.0.0-preview.3.20181.2 but anything later should work fine.

⚠ Important Step when I installed EF Core, it automatically pulled in an older version of the Azure Cosmos SDK for the client project. To fix this, I simply went into the NuGet package manager and upgraded it in the client project to the latest version.

In the server project, store your Cosmos DB connection string (this is a master key) somewhere secure. For my local test I put it in applicationSettings.Development.json under the key “CosmosConnection” (if you do this in a repo, make sure your .gitignore is setup so you don’t accidentally check it in). In production this would get put into environment variables, Azure Key Vault and/or application settings. This is the project that is not exposed to the client.

That’s the project setup, now for the custom code.

Set up the Cosmos DB User

The first step to generate a resource token is to add the user to Cosmos DB and configure permissions. You can choose how to identify your user (I will provide an example with email address, but you can use the id or name or anything else that is unique). You can also configure different sets of permissions and name them however you like. When and how you do this depends on your application design. If they should have access by default, you can add them to Cosmos DB the same time their profile is set up. Other options include the first time they try to access the resource, or as the result of some administrative action. Programmatically, the code looks like this to create the user. Because permissions are ephemeral, I’ll cover those later:

var client = new CosmosClient(CONNECTION_STRING);
var db = Client.GetDatabase("myblogs");
var cosmosUser = await db.CreateUserAsync(email);

Assuming the email test@test.com this will create a user on the myblogs database with the same name.

This step only needs to happen once. It creates the user, but it is the permissions that generate the tokens. In the Code Snippets section I share a gist that shows how to override Identity to automatically add the user to Cosmos DB.

Generate the Token

The two properties needed to connect to a Cosmos DB account are the endpoint and the key. I created a shared class to represent these properties:

public class CosmosCredentials
{
   public string EndPoint { get; set; }
   public string Key { get; set; }
}

Next, create an empty Web API controller in the server project. I named mine CosmosController.cs.

using System.Threading.Tasks;
using IConfiguration = Microsoft.Extensions.Configuration.IConfiguration;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Azure.Cosmos;
using Microsoft.Extensions.Configuration;
using BlazorCosmosWasm.Shared;
using IdentityServer4.Extensions;
using System.Linq;
using Microsoft.AspNetCore.Identity;
using BlazorCosmosWasm.Server.Models;
namespace BlazorCosmosWasm.Server.Controllers
{
   [Authorize]
   [Route("api/[controller]")]
   [ApiController]
   public class CosmosController : ControllerBase
   {
      private readonly UserManager<ApplicationUser> Manager;
      public CosmosController(IConfiguration config, UserManager<ApplicationUser> manager)
      {
         Config = config;
         Manager = manager;
         Client = new CosmosClient(Config.GetValue<string>("CosmosConnection"));
      }
   private CosmosClient Client { get; }
   public IConfiguration Config { get; set; }
   }
}

Notice this expects configuration to be injected for connection strings, but also takes an instance of the user manager. This is important for us to find the email of the logged in user. The [Authorize] attribute requires the user to be logged in to access any controller methods. When the authenticated user requests the controller endpoint, their security identity includes a set of claims. The username, email, and other information don’t exist. One of the claims contains a unique identifier for the user, which can be used to obtain the remaining information. Here’s a helper method to get the email address of the logged in user:

private async Task<string> GetEmailAsync()
{
   if (User.IsAuthenticated())
   {
      var id = User.Claims.SingleOrDefault(
         key => key.Type == System.Security.Claims.ClaimTypes.NameIdentifier)?.Value;
      var user = await Manager.FindByIdAsync(id);
      return user?.Email;
   }
   return null;
}

Now that we have the email, we can request and return the token. There are three basic steps:

Retrieve the user
Grant the user permissions
Retrieve the token

This example will grant the test@test.com user access to only the BlogContext container. The permission is named Permission-test@test.com-blogs and only grants “read” access. As mentioned earlier, you can further restrict access to a specific partition key. If you want to override the token expiration, pass the desired timeout in seconds after the PermissionProperties parameter.

[HttpGet]
public async Task<IActionResult> Get()
{
   var user = await GetEmailAsync();
   if (user == null)
   {
      return NotFound();
   }
   var db = Client.GetDatabase("myblogs");
   var container = db.GetContainer(nameof(BlogContext));
   var cosmosUser = db.GetUser(user);
   var permissionId = $"Permission-{user}-blogs";
   await cosmosUser.UpsertPermissionAsync(new PermissionProperties(
      id: permissionId,
      permissionMode: PermissionMode.Read,
      container: container));
   var token = await cosmosUser.GetPermission(permissionId).ReadAsync();
   return Ok(
      new CosmosCredentials
      {
         EndPoint = Client.Endpoint.ToString(),
         Key = token.Resource.Token
      });
}

The code proactively creates or updates the permissions, then grabs the token. Each time the permissions are updated, the token expiration is updated as well. Every time the permission is read, a new token is generated. You can proactively manage tokens by tracking expiration and requesting a new one, or reactively by capturing the authorization exception when it expires and request a new one.

The Data Model

OK, we’re done with the server. Let’s work on using the token. In the shared project, add the following code to define blogs, posts, and an EF Core context. I’m keeping this example simple (read only) so you may want to write some bootstrap code to insert a few sample documents. The EF Core docs explain how to seed Cosmos DB.

public class Blog
{
    public Blog()
    {
        Id = Guid.NewGuid();
    }
    public Guid Id { get; set; }
    public string Name { get; set; }
    public string Url { get; set; }
    public List<Post> Posts { get; set; } = new List<Post>();
}

[Owned]
public class Post
{
    public string Title { get; set; }
    public bool IsActive { get; set; }
}

public class BlogContext : DbContext
{
    public DbSet<Blog> Blogs { get; set; }

    public BlogContext(): base()
    {

    }

    public BlogContext(DbContextOptions<BlogContext> options): base(options)
    {

    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Blog>().Property<string>("ETag").IsEtagConcurrency();
        base.OnModelCreating(modelBuilder);
    }
}

Using DbContextOptions allows you to defer configuration and standup the context with the provider of your choice. The model building code simply adds a shadow property to track concurrency as I showed in the standup.Let’s move over to the client project.

The Client Service

The ordinary steps to register a data context won’t work here because the token needs to be retrieved before it is initialized. Therefore, I created a service to handle the additional steps. The interface is simple:

public interface IBlogClient
{
   Task<BlogContext> GetDbContextAsync();
}

The implementation looks like this:

public class BlogClient : IBlogClient
{
    private CosmosCredentials Credentials;
    private readonly HttpClient Http;

    public BlogClient(HttpClient client)
    {
        Http = client;
    }

    public async Task<BlogContext> GetDbContextAsync()
    {
        if (Credentials == null)
        {
            await GetCredentialsAsync();
        }

        var options = new DbContextOptionsBuilder<BlogContext>()
            .UseCosmos(Credentials.EndPoint, Credentials.Key, "myblogs",
            opt =>
                opt.ConnectionMode(Microsoft.Azure.Cosmos.ConnectionMode.Gateway));

        return new BlogContext(options.Options);
    }

    private async Task GetCredentialsAsync()
    {
        Credentials = await Http.GetFromJsonAsync<CosmosCredentials>("api/Cosmos");
    }
}

The CosmosCredentials class is from the shared project. The HttpClient is injected to use the special client properties that are configured by Blazor.

The service will cache the token. Right now, the service doesn’t deal with expiration; that would involve either setting an expiration and checking it, or capturing an authorization error and re-requesting the token. Either way, the endpoint will only return a token if the user is authenticated and already added to the Cosmos DB database. The options builder then specifies the Cosmos DB provider, passes the endpoint and token, and sets the connection mode to “gateway.” This is used to create a new context that is passed back.

⭐ Normally, creating a new context is a fast operation. The preferred pattern is to create a context per operation, and this scales well under most providers (except when you are using the change tracking features to add, update, and/or delete entities). The service here gives you the flexibility to generate and discard the context or hang onto it.

I registered it as “scoped” in the client project Program.cs:

builder.Services.AddScoped<IBlogClient, BlogClient>();

The final step is to use the service and consume some data.

The View Component

Next, create a Razor component named cosmos.razor. The heading should look like this:

@page "/cosmos"
@using Microsoft.AspNetCore.Authorization
@using Microsoft.AspNetCore.Components.WebAssembly.Authentication
@using BlazorCosmosWasm.Shared
@using BlazorCosmosWasm.Client.Data
@using Microsoft.EntityFrameworkCore
@attribute [Authorize]
@inject IBlogClient Client

A few things are going on here:

The route is set to “cosmos” for navigation
The [Authorize] attribute is used to require authorization for the page
The blog service is injected

Next, the template to render the blog content:

@if (Blogs != null)
{
    <h1>Blogs</h1>
    @foreach (var blog in Blogs)
    {
        <h2><a href="@blog.Url" target="_blank">@blog.Name</a></h2>
        <ul>
            @foreach (var post in blog.Posts)
            {
                @if (post.IsActive)
                {
                    <li><strong>@post.Title</strong></li>
                }
                else
                {
                    <li>(inactive) @post.Title</li>
                }
            }

        </ul>
    }
}

I also added a “loading” prompt and an error message:

@if (Blogs == null && string.IsNullOrWhiteSpace(error))
{
   <p class="alert alert-info">Loading...</p>
}
@if (Blogs == null && !string.IsNullOrWhiteSpace(error))
{
   <p class="alert alert-danger">Bad news: @error</p>
}

The code exposes an error property and a list of blogs:

public List<Blog> Blogs = null;
public string error;

Finally, the initialization code:

protected override async Task OnInitializedAsync()
{
    BlogContext blogContext = null;
    try
    {
        blogContext = await Client.GetDbContextAsync();
    }
    catch (AccessTokenNotAvailableException exception)
    {
        exception.Redirect();
    }
    catch (Exception ex)
    {
        error = ex.Message;
    }

    if (string.IsNullOrEmpty(error))
    {
        using (blogContext)
        {
            Blogs = await blogContext.Blogs.ToListAsync();
        }
    }
}

If the user is not authorized when the token endpoint is accessed, an exception is thrown that redirects them to a page to login or register. Any other errors are simply captured and displayed. The context is used to fetch the list and then disposed immediately. I’ve been asked how fast it takes to render. This is a real-time capture of the process.

That’s a wrap!

Code Snippets

For a more complete look at the code, I created several gists:

CosmosExtensions.cs - a few helper methods.
CosmosController.cs - the entire code of my token controller.
ApplicationCosmosDbContext.cs - a custom context that derives from the Identity context. It uses the context to find new users, then automatically adds them as Cosmos DB users (if you decide to use this, don’t forget to intercept the removal or deactivation of users as well). This gets registered in place of ApplicationDbContext in the startup methods.
BlogClient.cs - a version of the client that will retry on failure in case the exception is due to an expired token.

Conclusion

I believe that this experience can be a game changer for building applications, especially if the majority of effort is based on reading and querying data. There are definite security concerns to consider for this approach, but I’m not convinced they are much different than the experience of connecting from mobile. Although Cosmos DB was my first target for using EF Core in the client, there are other possibilities as well. Other teams have successfully implemented in-browser versions of SQLite. It will be interesting to explore how to set that up with the EF Core provider and consider scenarios like offline apps and disconnected sync.

I was surprised by how much attention my original tweet generated, so I’m very interested to hear your thoughts. What are your thoughts and concerns around Cosmos DB in Blazor WebAssembly? Do you have use cases this may work well for? Is there a security risk I didn’t consider that you would like to relate? How do you feel about using EF Core as a “database dialect” in your Blazor apps, regardless of whether it is working through a direct provider or simply transports the requests to the server and materializes results over the wire?

Please comment below!

Regards,