Forem: Luce Carter

Getting Started With Queryable Encryption With the MongoDB EF Core Provider

Luce Carter — Sun, 09 Nov 2025 18:18:32 +0000

This tutorial was written by Luce Carter.

With the release of MongoDB 7.0 in August 2023, MongoDB introduced a new feature called Queryable Encryption (QE), the first of its kind. With Queryable Encryption, your data is encrypted, even at rest, with the database server unable to read it either but still able to execute queries against it. You can configure what fields to encrypt so you can decide to encrypt as much or as little of your document’s fields as you need.

The great news is, not only is this available for all paid tiers and in our C# driver, but it’s now also available in the MongoDB Entity Framework Core (EF) provider.

In this tutorial, we are going to take an existing Blazor application that uses EF and the EF Core Provider, and configure it to use QE to securely encrypt certain fields. Let’s get started!

Prerequisites

In order to follow along with this tutorial, you will need a few things in place already:

A MongoDB M10 tier or above—this is because QE is an enterprise feature and only available on our paid tiers
Your MongoDB Atlas connection string to your cluster .NET 9 or later
The GitHub repo forked and cloned
- Add your connection string to the placeholder in appsettings.Development.json
- For example, "MongoDBConnectionString": "mongodb+srv://Luce:MyP4ssw0rd@cluster0.87zhs.mongodb.net/?retryWrites=true&w=majority&appName=enterprisehealthcaredotnet-efcorewithqe"

Note: If you want to see the full code, there is a branch on the GitHub repo called with-efcore-and-qe which is a working sample application. You will just need to follow the next section on adding the Automatic Encryption Library to run it locally and ensuring you add your connection string.

Adding the Automatic Encryption Shared Library

The first thing to do is add the Automatic Encryption Shared Library to the project. You can find this in our download center. Be sure to select crypt_shared from the package dropdown box.

Download the correct version for your platform and then unzip it to the root of the project. You should end up with a folder named something like mongo_crypt_shared_v1-macos-arm64-enterprise-8.2.1. The final folder name will change depending on your platform and version downloaded but the contents should be the same.

Ensure that the folder name matches the value in appsettings.Development.json.

Add NuGet package

Now we have the code set up and ready to go, it is time to add the NuGet package that contains the methods for using QE with the MongoDB EF Core Provider.

dotnet add package MongoDB.Driver.Encryption

Create helper class

The first thing we are going to do is create a helper class inside the Services folder called QueryableEncryptionHelpers.cs. This will provide code for creating and handling the encryption keys that are used to encrypt your data.

Create the new file and then paste the following code that we will discuss afterwards:

using System.Security.Cryptography;
using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Driver.Encryption;
namespace EnterpriseHealthcareDotNet.Services;

public class QueryableEncryptionHelpers
{
  private readonly IConfigurationRoot _appSettings;
  private readonly string _cryptSharedLibPath;
  public QueryableEncryptionHelpers(IConfigurationRoot appSettings)
  {
    _appSettings = appSettings;

    // Make sure your CryptSharedLibPath folder name is added to app settings, otherwise add here
    var relativeLibPath = _appSettings["CryptSharedLibPath"] ??                          "mongo_crypt_shared_v1-macos-arm64-enterprise-8.2.0/lib/mongo_crypt_v1.dylib";
    var projectRoot = Directory.GetParent(AppContext.BaseDirectory)!.Parent!.Parent!.Parent!.FullName;
    _cryptSharedLibPath = Path.GetFullPath(Path.Combine(projectRoot, relativeLibPath));
  }

  public Dictionary<string, IReadOnlyDictionary<string, object>> 
GetKmsProviderCredentials(string kmsProviderName,
        bool generateNewLocalKey)
  {
    if(kmsProviderName == "local")
    {
      if (generateNewLocalKey)
      {
        File.Delete("customer-master-key.txt");

        // start-generate-local-key
        using var randomNumberGenerator = RandomNumberGenerator.Create();
        try
        { 
          var bytes = new byte[96];
          randomNumberGenerator.GetBytes(bytes);
          var localCustomerMasterKeyBase64 = Convert.ToBase64String(bytes);
          File.WriteAllText("customer-master-key.txt", localCustomerMasterKeyBase64);
        }
        catch (Exception e)
        {
          throw new Exception("Unable to write Customer Master Key file due to the following error: " + e.Message);
        }
    // end-generate-local-key
    }

    // start-get-local-key
    // WARNING: Do not use a local key file in a production application
   var kmsProviderCredentials = new Dictionary<string, IReadOnlyDictionary<string, object>>();
   try
   {
     var localCustomerMasterKeyBase64 = File.ReadAllText("customer-master-key.txt");
     var localCustomerMasterKeyBytes = Convert.FromBase64String(localCustomerMasterKeyBase64);

     if (localCustomerMasterKeyBytes.Length != 96)
     {
       throw new Exception("Expected the customer master key file to be 96 bytes.");
     }

     var localOptions = new Dictionary<string, object>
     {
       { "key", localCustomerMasterKeyBytes }
     };

     kmsProviderCredentials.Add("local", localOptions);
     }
      // end-get-local-key
     catch (Exception e)
     {
       throw new Exception("Unable to read the Customer Master Key due to the following error: " + e.Message);
     }
 return kmsProviderCredentials;

 }

 throw new Exception("Unrecognized value for KMS provider name \"" + kmsProviderName + "\"  encountered while retrieving KMS credentials.");
 }

public AutoEncryptionOptions GetAutoEncryptionOptions(CollectionNamespace keyVaultNamespace,
        IReadOnlyDictionary<string, IReadOnlyDictionary<string, object>> kmsProviderCredentials)
{
  var extraOptions = new Dictionary<string, object>
  {
    { "cryptSharedLibRequired", true },
    { "cryptSharedLibPath",  _cryptSharedLibPath }
  };

var autoEncryptionOptions = new AutoEncryptionOptions(
keyVaultNamespace,
kmsProviderCredentials,
extraOptions: extraOptions);
// end-auto-encryption-options

  return autoEncryptionOptions;
}

}

Note: If you have not updated appsettings.Development.json with the folder name of the Automatic Encryption Library, be sure to do that or update the name in the relativeLibPath variable declaration in the constructor.

This code in the constructor builds up the path to the platform-specific file for the automatic encryption to seamlessly handle platform path differences.

Then, it handles creating an encryption key if it doesn’t already exist, or a new one has been requested, and saves it to a local .txt file. Of course, make sure you never share this file or commit it to GitHub, for security reasons! Once the key is available, it is returned to the code that called it.

There is a second method, GetAutoEncryptionOptions, which helps build up the auto encryption options that will be required later.

Note: Although outside the scope of this tutorial, this class is also where you could add code to handle other key provider options such as Azure, GCP, or AWS. You can see an example on the with-queryable-encryption branch that doesn’t use EF Core.

Update DbContext

This is building off of an existing Blazor application with EF Core configured so HealthcareDbContext already exists in the Services folder, but we need to update it to configure our encrypted fields.

First, we are going to add a record type at the bottom of the file, outside the class definition so we can reference it in the constructor of the class:

public record EncryptionKeys(Guid SsnKeyId, Guid DobKeyId);.

Now that this is defined, replace the class definition with the following, which uses a primary constructor:

public class HealthcareDbContext(DbContextOptions<HealthcareDbContext> options, EncryptionKeys keys) : DbContext(options)

This will cause some later code to error as it is missing the new parameter value, but we don’t use that now, so delete the Create method that is erroring.

After entity.ToCollection(“Patients”);, add a new line to let EF Core know that the Id field in our Patient model will be an ObjectID in MongoDB which is the default data type for the unique _iq field:

  entity.Property(p => p.Id)
                .HasBsonRepresentation(BsonType.ObjectId);

This is where the fun begins as it is time to configure the two fields we are going to encrypt: DateOfBirth and SSN.

First up is the DateOfBirth property which is a root field on the document:

entity.Property(p => p.DateOfBirth)
                .IsEncryptedForRange(new DateTime(1900, 1, 1),
                    new DateTime(2100, 1, 1),
                    keys.DobKeyId);

There are two types of queries currently supported, equality and range queries. We want to be able to query between a range of dates, so here, we use .IsEncryptedForRange. This takes three parameters: a possible starting range, a possible ending range (you can of course tweak this to your liking), and then the GUID that will be used as the data encryption key. This is why we added the new record for EncryptionKeys in the file and then passed it in the primary constructor.

Finally, we want to update the property for SSN to add that it is encrypted for equality and passed the SsnKeyId from the keys object:

pr.Property(r => r.SSN).IsEncryptedForEquality(keys.SsnKeyId).HasElementName("sSN");

Just like that, we have properties configured for encryption!

Update Program.cs

Now that we have the configuration handled, it is time to link it all up in Program.cs.

Before we do anything else, let’s add any additionally required using statements at the top of the class that aren’t already there:

using EnterpriseHealthcareDotNet.Models;
using MongoDB.Driver;
using MongoDB.Driver.Encryption;
using MongoDB.EntityFrameworkCore;

Now, delete the code between the call to AddRazorComponents and before the line to register PatientService as a scoped service in Dependency Injection (DI). This code was there to originally configure the DbContext and MongoDB when the application doesn’t use QE, but this will be done differently later. Add the variables that will be used to set up the Queryable Encryption Helpers class and MongoDB after PatientService is registered:

var configuration = builder.Configuration;
var qeHelpers = new QueryableEncryptionHelpers(configuration);

string uri = configuration["MongoDBConnectionString"]!;
string keyVaultDb = configuration["KeyVaultDatabase"] ?? "encryption";
string keyVaultColl = configuration["KeyVaultCollection"] ?? "__keyVault";
string kmsProviderName = configuration["KmsProvider"] ?? "local";
string encryptedDb = configuration["EncryptedDatabase"] ?? "MongoDBMedical";
string cryptSharedLibPath = configuration["CryptSharedLibPath"] ?? throw new ArgumentNullException("CryptSharedLibPath", "Path to the Automatic Encryption Shared Library must be provided in appsettings.Development.json");

After that, add the following code to set up the keyvault namespace, which is the collection in the database where the keys will be stored, and fetch the value using the method in QueryableEncryptionHelpers we set up earlier:

var keyVaultNamespace = CollectionNamespace.FromFullName($"{keyVaultDb}.{keyVaultColl}");


// Generate/reuse KMS provider credentials
var kmsProviders = qeHelpers.GetKmsProviderCredentials(
    kmsProviderName,
    generateNewLocalKey: !File.Exists("customer-master-key.txt"));

Next, add the following which will configure the MongoDB client specifically with settings for QE:

// Configure MongoDB client settings for QE
MongoClientSettings.Extensions.AddAutoEncryption();
var clientSettings = MongoClientSettings.FromConnectionString(uri);
clientSettings.AutoEncryptionOptions = qeHelpers.GetAutoEncryptionOptions(
    keyVaultNamespace,
    kmsProviders);

using var clientEncryption = new ClientEncryption(
    new ClientEncryptionOptions(new MongoClient(clientSettings), keyVaultNamespace, kmsProviders));


var equalityKey = clientEncryption.CreateDataKey("local", new DataKeyOptions());
var rangeKey = clientEncryption.CreateDataKey("local", new DataKeyOptions());

Finally, let’s add everything we have configured to the DI container:

builder.Services.AddSingleton(qeHelpers);
builder.Services.AddSingleton(new EncryptionKeys(equalityKey, rangeKey));
builder.Services.AddScoped<Patient>();

// Register DbContext with QE-enabled client
builder.Services.AddDbContext<HealthcareDbContext>(options =>
{
    options.UseMongoDB(new MongoOptionsExtension()
        .WithClientSettings(clientSettings)
        .WithDatabaseName(encryptedDb)
        .WithKeyVaultNamespace(keyVaultNamespace)
        .WithCryptProvider(CryptProvider.AutoEncryptSharedLibrary, cryptSharedLibPath)
        .WithKmsProviders(kmsProviders));
});

Test it out

It hasn’t even taken that much change addition or code changes, yet here we are, ready to test it out! This is testament to how easy it is to configure QE in EF Core.

Go ahead and run the application. Make sure your connection string and correct CryptSharedLib folder name are set in appsettings.json or appsettings.Development.json (recommend running in debug and using the development version).

Play around adding a new patient and then use a tool like MongoDB Compass to view the collection, and you should see your new patient with the date of birth and sSN encrypted.

Summary

Wow, how amazing! We have added Queryable Encryption to an existing EF Core application using the latest features of the EF Core Provider for MongoDB to safely encrypt private information—in this case, the dates of birth and social security numbers of patients—not just in transit but also in the database itself, while still being able to query it.

Go ahead and give it a go! If you want to see the full code, it can be found on GitHub.

Plus, if you want to level up your MongoDB skills and get a Credly-backed badge you can show off on LinkedIn, why not try the MongoDB Overview Skills Bage

MongoDB Vector Search With EF Core

Luce Carter — Mon, 27 Oct 2025 13:07:49 +0000

This post was written by Arthur Vickers, Staff Engineer at MongoDB

The 8.3.3 and 9.0.3 releases of the EF Core database provider for MongoDB introduce support for Atlas Vector Search in LINQ queries. This enables vector similarity search integrated with MongoDB's powerful document mapping and distributed architecture. Vector searches are used in AI, but can also be used for powerful similarity matching in non-AI applications.

In this post, we will create a simple console application that will demonstrate the capabilities of MongoDB vector similarity search with EF Core. The application uses a Movie entity type and contains embeddings for the movie's plot such that searches for similar movies can be made.

The application

We will start with a simple console application targeting .NET 9, and install the latest version of the EF Core provider for MongoDB.

Tip: The completed application can be found on GitHub. If anything isn’t clear from the code snippets, then please refer to the code on GitHub.

EF Core requires a DbContext class and entity types to be defined. For this post, we will use a simple model with a single entity type representing a movie:

[BsonIgnoreExtraElements]
public class EmbeddedMovie
{
    public ObjectId Id { get; }
    public required string Title { get; set; }
    public required int Year { get; set; }
    public string? Plot { get; set; }
    public float[]? PlotEmbedding { get; set; }
}

The MoviesDbDbContext class includes a DbSet<Movie> property as the root for LINQ queries, and configuration of the mapping from C# names (e.g., "Plot") to lower-case names (e.g., "plot") in the documents. In addition, embeddings are configured to use an efficient binary format with HasBinaryVectorDataType(BinaryVectorDataType.Float32).

public class MoviesDbContext : DbContext
{
    protected override void OnConfiguring(
        DbContextOptionsBuilder optionsBuilder)
        => optionsBuilder
            .UseMongoDB(
                "mongodb://localhost:50934/?directConnection=true&appname=ef_core_vector_search",
                "sample_mflix")
            .LogTo(Console.WriteLine, LogLevel.Information);

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder
            .Entity<EmbeddedMovie>(b =>
            {
                b.ToCollection("embedded_movies");

                b.Property(e => e.Id);
                b.Property(e => e.Title).HasElementName("title");
                b.Property(e => e.Plot).HasElementName("plot");
                b.Property(e => e.Year).HasElementName("year");

                b.Property(e => e.PlotEmbedding)
                    .HasElementName("plot_embedding_voyage_3_large")
                    .HasBinaryVectorDataType(BinaryVectorDataType.Float32);
            });
    }

    public DbSet<EmbeddedMovie> Movies => Set<EmbeddedMovie>();
}

Optionally, the binary format can be configured directly on the property using BinaryVectorAttribute. For example:

[BinaryVector(BinaryVectorDataType.Float32)]
public float[]? PlotEmbedding { get; set; }

Tip: See the EF Core documentation for more information about DbContext and EF Core in general, and the MongoDB provider documentation for more information about using EF Core with MongoDB.

Using MongoDB Atlas local

We will test our code against MongoDB Atlas running locally in Docker, which is fast, affordable, and convenient. Follow the instructions for setting up Atlas local; it's very easy, even if you don't know Docker. Alternatively, you can sign up for a free Atlas deployment, or use an existing MongoDB Atlas deployment.

Tip: If you already have an Atlas deployment with the "mflix" sample data loaded, then you may need to refresh this data, since the dataset is updated periodically. Consider using MongoDB Compass or mongosh to examine and/or delete existing data.

Get the connection string for MongoDB Atlas local using the command atlas deployments connect, and copy it into the call to UseMongoDB. For example:

arthur.vickers@M-MG6X9WX2PC mongo-efcore-provider % atlas deployments connect                                                              
? Select a deployment local3537 (Local)
? How would you like to connect to local3537? connectionString
mongodb://localhost:50934/?directConnection=true
arthur.vickers@M-MG6X9WX2PC mongo-efcore-provider %

Import sample data

Realistic vector data is hard to create manually. Instead, we will import the "MFlix" sample database, which contains embeddings for the plot of each movie. When running MongoDB Atlas locally, the sample data can be downloaded using curl:

curl  https://atlas-education.s3.amazonaws.com/sampledata.archive -o sampledata.archive

Then, copy it into the database using mongorestore:

mongorestore --archive=sampledata.archive --port=<port-number>

Make sure to use the port number from your connection string.

Why use a vector query?

The sample data contains "embeddings" for movie plots. Each of these embeddings encodes, across many dimensions, the various aspects of the plot. For example, a James Bond movie will have an embedding indicating it is a spy action movie, with a character called James Bond, among many other aspects of the plot.

If we now create an embedding for a plot, or part of a plot, we can use a vector search to find the other movies with a similar plot, searching across this large number of possible dimensions. To demonstrate this, we will first use a normal MongoDB query to fetch the plot for the James Bond film Octopussy:

using var db = new MoviesDbContext();

var octopussy = await db.Movies.SingleAsync(e => e.Title == "Octopussy");

Console.WriteLine($"Found '{octopussy.Title}' with plot '{octopussy.Plot}'");

Now, we can use the embedding for Octopussy's plot to find movies that are similar. For example:

var similarMovies = await db.Movies.VectorSearch(
        e => e.PlotEmbedding,
        octopussy.PlotEmbedding,
        limit: 10)
    .ToListAsync();

But wait! If you run this query now in our sample project without any addition setup, you will see the following error:

Unhandled exception. System.InvalidOperationException: A vector query for 'EmbeddedMovie.PlotEmbedding' could not be executed because the vector index for this query could not be found. Use 'HasIndex' on the EF model builder to specify the index,or specify the index name in the call to 'VectorQuery' if indexes are being managed outside of EF Core.
   at MongoDB.EntityFrameworkCore.Query.Visitors...

This illustrates a critical point: Vector queries require a vector index before they will return results.

Creating a vector index

EF Core can help us create this index, but first, we need to define it in the EF Core model. For example, add the following code to the configuration for Movie in the OnModelCreating method of MoviesDbContext:

b.HasIndex(e => e.PlotEmbedding).IsVectorIndex(
    VectorSimilarity.DotProduct,
    dimensions: 2048,
    options => options
        .HasQuantization(VectorQuantization.Scalar)
        .AllowsFiltersOn(e => e.Title)
        .AllowsFiltersOn(e => e.Year));

This defines an index with 2048 dimensions and adds some filter fields, which are discussed below. However, running the query after adding this code to the model still generates a warning:

warn: 14/10/2025 11:23:16.057 MongoEventId.VectorSearchReturnedZeroResults[35014] (Microsoft.EntityFrameworkCore.Query) 
      The vector query against 'EmbeddedMovie.PlotEmbedding' using index 'PlotEmbeddingVectorIndex' returned zero results. This could be because either there is no vector index defined in the database for the query property, or because vector data (embeddings) have recently been inserted and the index is still building. Consider disabling index creation in 'DbContext.Database.EnsureCreated' and performing initial ingestion of embeddings, before calling 'DbContext.Database.CreateMissingVectorIndexes' and 'DbContext.Database.WaitForVectorIndexes'.

This is because we have defined the vector index in the EF Core model, but it has not yet been created in the MongoDB database.

Creation of vector indexes generally takes time, and the index must be fully created before vector searches can be performed. This is normally done outside of the normal application flow at the time the embeddings are created. EF Core can help with this by creating vector indexes and waiting for them to be ready.

Calling DbContext.Database.EnsureCreatedAsync will create any missing vector indexes by default. Alternatively, this can be done explicitly at any time using DbContext.Database.CreateMissingVectorIndexes and DbContext.Database.WaitForVectorIndexes. In our sample, we make these calls before running any queries to ensure that the vector index is created and ready before the query runs. For example:

using var db = new MoviesDbContext();

await db.Database.CreateMissingVectorIndexesAsync();
await db.Database.WaitForVectorIndexesAsync(timeout: TimeSpan.FromMinutes(2));

Tip: The default timeout for vector index creation is one minute. Creation of very large indexes in the cloud may take longer than this. We account for this in the code above by passing in a longer timeout.

Running vector queries

Now, let's re-run the vector query for movies with plots like Octopussy, and print out the results:

var similarMovies = await db.Movies.VectorSearch(
        e => e.PlotEmbedding,
        octopussy.PlotEmbedding,
        limit: 10)
    .ToListAsync();

int i;
PrintMovieResults(similarMovies, "the plot of 'Octopussy'");

void PrintMovieResults(
    List<EmbeddedMovie> movies,
    string? embeddingDescription)
{
    Console.WriteLine(
        $"Found {movies.Count} movies with plots " +
        $"similar to {embeddingDescription}:");

    i = 1;
    foreach (var movie in movies)
    {
        Console.WriteLine(
            $"  {i++}: '{movie.Title}' " +
            $"with plot '{movie.Plot?.Substring(0, 20)}...'");
    }
}

Running this code results in the following output:

Found 10 movies with plots similar to the plot of 'Octopussy':
  1: 'Octopussy' with plot 'A fake Fabergè egg a...'
  2: 'Never Say Never Again' with plot 'A SPECTRE agent has ...'
  3: 'The Spy Who Loved Me' with plot 'James Bond investiga...'
  4: 'From Russia with Love' with plot 'James Bond willingly...'
  5: 'Goldfinger' with plot 'Investigating a gold...'
  6: 'GoldenEye' with plot 'James Bond teams up ...'
  7: 'Diamonds Are Forever' with plot 'A diamond smuggling ...'
  8: 'Thunderball' with plot 'James Bond heads to ...'
  9: 'The World Is Not Enough' with plot 'James Bond uncovers ...'
  10: 'For Your Eyes Only' with plot 'Agent 007 is assigne...'

As we might expect, all the top 10 most similar movies also feature James Bond!

Tip: Only 10 results were returned because we set that limit in the query. Limiting the query in this way is a best practice to help reduce the amount of work done by the vector search.

The plot embedding used in the query does not have to come from an existing movie in the database. Instead, an embedding can be created for any user-supplied fragment of a plot. This is achieved by calling an endpoint provided by the embedding model, passing in the text or other data to be embedded. The endpoint returns a list of floating point numbers that represent the embedding.

This process is described in How to Create Vector Embeddings for the "voyage-3-large" model we are using here. Some code adapted from the linked instructions is also included in the GitHub repo for this post. Since the sample data contains embeddings with 2048 dimensions, we need to instruct Voyage AI to do the same by specifying "the output_dimension" when calling the endpoint. For example:

var requestBody = new
{
    input = texts,
    model = EmbeddingModelName,
    truncation = true,
    output_dimension = 2048
};

Assume we have obtained an embedding for "time travel".

Tip: The "time travel" embedding is included in the code on GitHub, along with three other sample embeddings for you to use without creating your own.

We can now use this embedding to find movies about time travel:

var timeTravelEmbedding = new[] {-0.034731735, ... , 040714234};

var timeTravelMovies = await db.Movies.VectorSearch(
        e => e.PlotEmbedding,
        timeTravelEmbedding,
        limit: 10)
    .ToListAsync();

PrintMovieResults(timeTravelMovies, "'time travel'");

The returned movies are all related to "time travel":

Found 10 movies with plots similar to 'time travel':
  1: 'About Time' with plot 'At the age of 21, Ti...'
  2: 'Retroactive' with plot 'A psychiatrist makes...'
  3: 'Timecop' with plot 'An officer for a sec...'
  4: 'A.P.E.X.' with plot 'A time-travel experi...'
  5: 'Back to the Future Part II' with plot 'After visiting 2015,...'
  6: 'Thrill Seekers' with plot 'A reporter, learning...'
  7: 'Timerider: The Adventure of Lyle Swann' with plot 'Lyle, a motorcycle c...'
  8: 'The Time Machine' with plot 'Hoping to alter the ...'
  9: 'The Time Traveler's Wife' with plot 'A romantic drama abo...'
  10: 'Stargate: Continuum' with plot 'Ba'al travels back i...'

Pre-filtering vector searches

Often, there are other constraints on the results returned from a vector search. The search is more efficient if unwanted results are filtered out before looking for similarities. For example, imagine we only want to find time travel movies from the 1980s. We can add this filter to our vector query:

var eightiesTimeTravelMovies = await db.Movies.VectorSearch(
        e => e.PlotEmbedding,
        preFilter: e => e.Year >= 1980 && e.Year < 1990,
        timeTravelEmbedding,
        limit: 10)
    .ToListAsync();

PrintMovieResults(eightiesTimeTravelMovies, "'80s 'time travel'");

Tip: For a field to be used in a vector search filter, it must be included in the vector index as a "filter" field. Refer back to the index definition above to see how "Year" and "Title" are included for filtering.

Looking at the results from the 1980s only, it is clear that some of the movies at the bottom of the list are not very time travel-like.

Found 10 movies with plots similar to 'time travel' from the 1980s:
  1: 'Back to the Future Part II' with plot 'After visiting 2015,...'
  2: 'Timerider: The Adventure of Lyle Swann' with plot 'Lyle, a motorcycle c...'
  3: 'The Final Countdown' with plot 'A modern aircraft ca...'
  4: 'Time Bandits' with plot 'A young boy accident...'
  5: 'Peggy Sue Got Married' with plot 'Peggy Sue faints at ...'
  6: 'Tommy Tricker and the Stamp Traveller' with plot 'When the joker Tommy...'
  7: 'Mirror for a Hero' with plot 'Two not quite simila...'
  8: 'The Navigator: A Medieval Odyssey' with plot 'Men seeking relief f...'
  9: 'Warlock' with plot 'A warlock flees from...'
  10: 'The Karate Kid, Part II' with plot 'Daniel accompanies h...'

If we want to find out, relatively, how close each of the matches is, then we can project out the "score" for each result. Creating a projection also means we can avoid fetching the large embedding models in each document, which are not normally needed on the client. For example, let's project out the ID, title, plot, and score for each match:

var projectedMoviesWithScore = await db.Movies.VectorSearch(
        e => e.PlotEmbedding,
        preFilter: e => e.Year >= 1980 && e.Year < 1990,
        timeTravelEmbedding,
        limit: 10)
    .Select(e => new
    {
        e.Id,
        e.Title,
        e.Plot,
        Score = EF.Property<float>(e, "__score")
    }).ToListAsync();

Console.WriteLine(
    $"Projecting details and score from " +
    $"{projectedMoviesWithScore.Count} movies:");

i = 1;
foreach (var movie in projectedMoviesWithScore)
{
    Console.WriteLine(
        $"  {i++}: (Score: {movie.Score}) '{movie.Title}' " +
        $"with plot '{movie.Plot?.Substring(0, 20)}...'");
}

Tip: Tip: This projection uses the EF Property method to request the query metadata value "__score" even though there is no Score property in the entity itself.

Let's look at the scores for "time travel" movies limited to the 1980s:

Projecting details and score from 10 movies:
  1: (Score: 0.7521951) 'Back to the Future Part II' with plot 'After visiting 2015,...'
  2: (Score: 0.7504921) 'Timerider: The Adventure of Lyle Swann' with plot 'Lyle, a motorcycle c...'
  3: (Score: 0.74751997) 'The Final Countdown' with plot 'A modern aircraft ca...'
  4: (Score: 0.7193389) 'Time Bandits' with plot 'A young boy accident...'
  5: (Score: 0.7188964) 'Peggy Sue Got Married' with plot 'Peggy Sue faints at ...'
  6: (Score: 0.71291494) 'Tommy Tricker and the Stamp Traveller' with plot 'When the joker Tommy...'
  7: (Score: 0.7125094) 'Mirror for a Hero' with plot 'Two not quite simila...'
  8: (Score: 0.70869136) 'The Navigator: A Medieval Odyssey' with plot 'Men seeking relief f...'
  9: (Score: 0.70750093) 'Warlock' with plot 'A warlock flees from...'
  10: (Score: 0.7039094) 'The Karate Kid, Part II' with plot 'Daniel accompanies h...'

And compare to the scores for all "time travel" movies:

Projecting details and score from 10 movies:
  1: (Score: 0.7706156) 'About Time' with plot 'At the age of 21, Ti...'
  2: (Score: 0.7603297) 'Retroactive' with plot 'A psychiatrist makes...'
  3: (Score: 0.7581115) 'Timecop' with plot 'An officer for a sec...'
  4: (Score: 0.75784063) 'A.P.E.X.' with plot 'A time-travel experi...'
  5: (Score: 0.7521951) 'Back to the Future Part II' with plot 'After visiting 2015,...'
  6: (Score: 0.751307) 'Thrill Seekers' with plot 'A reporter, learning...'
  7: (Score: 0.7504921) 'Timerider: The Adventure of Lyle Swann' with plot 'Lyle, a motorcycle c...'
  8: (Score: 0.7499349) 'The Time Machine' with plot 'Hoping to alter the ...'
  9: (Score: 0.74950624) 'The Time Traveler's Wife' with plot 'A romantic drama abo...'
  10: (Score: 0.7475202) 'Stargate: Continuum' with plot 'Ba'al travels back i...'

Notice how, relatively speaking, the scores for the filtered list drop off much faster than the scores for the full list. This aligns with the observation that the movies at the end of the filtered list don't have that much to do with time travel as expected.

Summary

Vector queries add powerful similarity searches for both AI and non-AI applications to the MongoDB repertoire. EF Core provides modelling for vector indexes, along with utilities for creating the vector indexes in MongoDB and waiting for them to be ready. EF Core exposes vector queries in LINQ, and vector queries can be pre-filtered using a LINQ predicate. Projections from a vector query can include the score for each result.

Want to show off your new found knowledge of Vector Search? Want to learn more about it? Take our Vector Search Skills Badge, a quick 10 question quiz that when you pass, can be shared on LinkedIn to show off your new found knowledge as part of the Licences & Certifications section!

Using OIDC with .NET to connect to MongoDB Atlas

Luce Carter — Mon, 22 Sep 2025 08:39:37 +0000

This tutorial was written by Luce Carter

In any kind of application, especially enterprise applications, developers will want to keep their data and the access to that data secure. It is common for businesses to use identity and access management (IAM) systems such as Azure EntraID (formerly Azure Active Directory) to manage access to an application, including its data.

In this tutorial, we are going to look at adding OpenID Connect (OIDC) using EntraID to a .NET and MongoDB Atlas-backed application, ensuring only those with access can log in with their Microsoft credentials and use the application.

Note: We are using Azure EntraID in this tutorial but this will work with IAM solutions offered by AWS and Google Cloud as well, as long as you have the required configuration values for MongoDB Atlas that we will cover later in the tutorial.

If you want to see the finished result, you can visit the GitHub repo for the application on the with-oidc branch.

Prerequisites

You will need a few things in place in order to follow this tutorial:

Azure EntraID configuration details
- Because this is an enterprise feature, you will need to ensure you are not using a free or trial subscription.
- In order to configure Workforce Federation in Atlas, you will need your application (client) ID and your OIDC metadata doc.
- You can obtain this from the Azure Portal under EntraID.
- You will also need your domain value. You can find this by clicking the settings cog icon at the top of any page in the Azure Portal and it will show in the page that opens.
The GitHub repo forked to your own account and cloned
- Make sure you are on the start-oidc branch.
A MongoDB Atlas M10 or above cluster deployed
- OIDC support via Workforce Federation in MongoDB Atlas is an enterprise feature and thus not available on the free tier.
MongoDB Atlas Workforce Federation configured including the step for adding a Database User Group linked to EntraID to allow full access to the database
The connection string for your cluster added to appsettings.json and appsettings.Development.json in place of the placeholder text
- Note: You will not need the username and password but instead just the clustername and address—for example, mongodb+srv://cluster0.lz13sat.mongodb.net.

Validating MongoDB Atlas Workforce Federation

Before we get into writing any code and updating the existing EnterpriseHealthcareDotNet application to add support for OIDC, we first need to ensure that MongoDB Atlas Workforce Federation is set up with the configuration values obtained from EntraID (or your cloud provider of choice) as mentioned in the prerequisites.

The first step is to go to the landing page of your organization where you will see Federation listed on the left under Identity & Access.

Then click the button to Open Federation Management App which will open in a new tab. Under the left-side menu, select Identity Providers. This will show you the configured identity provider, with the values populated.

You will see a “Complete” label next to the Identity Provider in Workforce Federation that shows it has been configured.

Also under the Project in MongoDB Atlas in Database Access, ensure the user has been added successfully. The ‘User’ value should match your Tenant ID value from EntraID.

Adding NuGet packages

Now that MongoDB Atlas Workforce Federation is configured for OIDC, in this case with Azure EntraID, it is time to move on to working on the application.

You will need to add the following NuGet packages to the application, either via the NuGet Package Manager in your IDE/Text Editor of choice, or via CLI:

Microsoft.Identity.Web
Microsoft.Identity.Web.UI
Microsoft.AspNetCore.Authentication.OpenIdConnect
Microsoft.Identity.Client

Adding these four packages will make available other packages implicitly, so getting setup for OIDC in your .NET 9 Blazor application is as simple as adding those packages—neat!

Updating appsettings and Program.cs

We have the required packages in place, so now it is time to start to set it up. We will be adding configuration secrets to the application, so these will be added to appsettings.

Add the following to both appsettings.json and appsettings.Development.json, below the existing MongoDB entry:

"AzureEntraID": {
    "Authority": "<your OIDC Metadata doc URL until the 2.0 suffix>",
    "Domain": "<your domain>.onmicrosoft.com",
    "ClientId": "<your EntraID client id>",
    "ClientSecret": "<your EntraID client secret value>",
    "CallbackPath": "/signin-oidc",
    "RedirectUri": "http://localhost:<your default .NET app port>/signin-oidc"
  }

Be sure to update the placeholder entries with your own values. Of course, the domain value also will be slightly different here if not using Azure EntraID.

Now that we have the configuration values in place, we can update Program.cs to set up our authentication, and be able to reference those values.

Add the following using statements to the top of the class below the existing ones:

using Microsoft.AspNetCore.Authentication.OpenIdConnect;
using Microsoft.Identity.Web;
using Microsoft.IdentityModel.Logging;

Then add the following code below the line var builder = WebApplication.CreateBuilder(args);:

IdentityModelEventSource.ShowPII = true;

var entraDetails = builder.Configuration.GetSection("AzureEntraID");

builder.Services.AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApp(entraDetails)
    .EnableTokenAcquisitionToCallDownstreamApi(new[] { "User.Read" })
    .AddDistributedTokenCaches();

builder.Services.AddDistributedMemoryCache();

builder.Services.AddAuthorization(options =>
{
    options.FallbackPolicy = options.DefaultPolicy;
});

This is all standard as part of the Microsoft Identity Web authentication library which is available in ASP.NET Core, rather than being unique to MongoDB.

Replace the existing first call to builder.Services with the following:

builder.Services.AddRazorComponents()
    .AddInteractiveServerComponents()
    .AddMicrosoftIdentityConsentHandler();

builder.Services.AddControllers();
builder.Services.AddHttpContextAccessor();

You will also need to change the call to add MongoDBService from AddSingleton to AddScoped:

builder.Services.AddScoped<MongoDBService>();

This is because authentication uses scoped services under the hood, and if you leave it as AddSingleton, you will get errors at runtime about it.

Before the call to app.UseHttpsRedirection(); further down the class, add the following:

app.UseAuthentication();
app.UseAuthorization();
app.MapControllers();

None of this is doing anything specific to MongoDB, but it is setting everything up for authentication and authorization. This is such a common feature in .NET applications, especially ASP.NET applications such as those built with Blazor, like this demo repo, that Microsoft makes it as easy as possible with simple calls to use certain features.

Updating the MongoDB service

Things are slowly coming together nicely. Now that the application is configured to use authentication, we can update the existing MongoDBService.cs class to set up authentication on the MongoDB Atlas side, using the MongoDB C# Driver.

Add the following extra using statements to the top of the file:

using Microsoft.AspNetCore.Authentication;
using Microsoft.Identity.Web;
using MongoDB.Driver.Authentication.Oidc;

Then add these new local readonly variables below private IMongoClient? _client;:

private readonly IHttpContextAccessor _httpContextAccessor;
private readonly ITokenAcquisition _tokenAcquisition;

Next, we need to add some additional parameters to the constructor, so replace the constructor declaration with the following:

public MongoDBService(IConfiguration appSettings, IHttpContextAccessor httpContextAccessor, ITokenAcquisition tokenAcquisition)

Then initialize the local variables with the new parameters:

_httpContextAccessor = httpContextAccessor;
 _tokenAcquisition = tokenAcquisition;

Delete the following calls to initialize the client and collection as we will be doing that later now instead:

_client = new MongoClient(appSettings["MongoDBConnectionString"] ??
                                  throw new ArgumentNullException("MongoDbUri cannot be null"));

        _patientsCollection = _client.GetDatabase("MongoDBMedical").GetCollection<Patient>("Patients");

The crux of the changes to the service class comes next with a method and an inner class to handle authentication.

First, let’s define the method and then talk through what is happening:

public async Task InitializeAsync()
    {
        var httpContext = _httpContextAccessor.HttpContext;
        if (httpContext == null)
        {
            // No HTTP context, do not initialize (let UI handle login prompt)
            return;
        }


        // Use Microsoft.Identity.Web to get the access token for the user
        string[] scopes = new[] { "<your client id>/.default" }; // Replace with your API scope
        var accessToken = await _tokenAcquisition.GetAccessTokenForUserAsync(scopes);

        if (string.IsNullOrEmpty(accessToken))
        {
            // User not authenticated, do not initialize (let UI handle login prompt)
            return;
        }

        var authenticatedConnString = _appSettings["MongoDBConnectionString"] + "?authMechanism=MONGODB-OIDC&authSource=$external";

        var mongoDBClientSettings = MongoClientSettings.FromConnectionString(authenticatedConnString);
        // Pass the acquired access token directly to the OIDC callback
        mongoDBClientSettings.Credential = MongoCredential.CreateOidcCredential(new AccessTokenOidcCallback(accessToken));

       _client = new MongoClient(mongoDBClientSettings);
        try
        {
            var result = await _client.GetDatabase("admin").RunCommandAsync<BsonDocument>(new BsonDocument("ping", 1));

            if (result.GetValue("ok") == 1.0)
            {
                _patientsCollection = _client.GetDatabase("medicalRecords").GetCollection<Patient>("patients");
            }
        } catch (Exception ex)
        {
            Console.WriteLine($"Error connecting to MongoDB: {ex.Message}");
            throw;
        }

    }

Note: Be sure to update the placeholder value in the string[] scopes assignment with your client id.

The code then builds up the connection string by appending on additional values that you can add as parameters to the base connection string that MongoDB uses to know that your application is using OIDC and an external authentication source—in this case, EntraID.

It then uses the new connection string and the access token received from authentication, to initialize the MongoClient object we added earlier and carry out some best practice steps to check the connection. This step will automatically check that the authentication has been configured correctly both in the application and in MongoDB Atlas.

This code will currently give you an error/red squiggly line because we haven’t defined the class yet that was mentioned earlier, so let’s do that now.

private class AccessTokenOidcCallback : IOidcCallback
{
    private readonly string _accessToken;
    public AccessTokenOidcCallback(string accessToken)
    {
        _accessToken = accessToken;
    }
    public OidcAccessToken GetOidcAccessToken(OidcCallbackParameters parameters, CancellationToken cancellationToken)
    {
        return new OidcAccessToken(_accessToken, expiresIn: null);
    }
    public Task<OidcAccessToken> GetOidcAccessTokenAsync(OidcCallbackParameters parameters, CancellationToken cancellationToken)
    {
        return Task.FromResult(GetOidcAccessToken(parameters, cancellationToken));
    }
}

This class implements an interface available in the MongoDB C# Driver as part of its support for OIDC, which provides helpful token management methods to make it easier to work with and add OIDC features.

Adding a login page

The backend side of things is set up now and ready for login, but we need to add a login page to trigger the authentication flow in the frontend, so let’s do that now.

Add a new Blazor Component page called Login.razor in the Components/Pages folder and paste the following code to replace any existing code that is automatically created:

@page "/login"
@using EnterpriseHealthcareDotNet.Services
@using Microsoft.AspNetCore.Components.Authorization

@inject NavigationManager NavigationManager
@inject MongoDBService MongoDBService
@inject AuthenticationStateProvider AuthenticationStateProvider

<h3>Login</h3>

@if (IsAuthenticated)
{
    <p>You are signed in.</p>
}
else
{
    <a href="/auth/login" class="btn btn-primary">Authorize with Entra ID</a>
}

@code {
    private bool IsAuthenticated = false;

    private bool ShouldRedirect = false;

protected override async Task OnInitializedAsync()
{
    var authState = await AuthenticationStateProvider.GetAuthenticationStateAsync();
    var user = authState.User;

    if (!user.Identity?.IsAuthenticated ?? true)
    {
        ShouldRedirect = true;
    }
    else
    {
        await MongoDBService.InitializeAsync();
        NavigationManager.NavigateTo("/", forceLoad: true);
    }
}

protected override async Task OnAfterRenderAsync(bool firstRender)
{
if (firstRender && ShouldRedirect)
    {
        NavigationManager.NavigateTo("/login", forceLoad: true);
    }
}
}

This is a pretty straightforward page that gives the ability to log in if not already logged in and calls the InitalizeAsync method we added to the service class in the previous section.

Adding the username to the home page

Almost there! The last thing we want to do is add the username of the logging in user to the top right of the home page, just so that if they are logged in, the user can see that they are because their username is displayed.

We will need to add some new code to Home.razor to fetch the user’s information for displaying. So add the following inject statement to the top of the class:

@inject Microsoft.AspNetCore.Components.Authorization.AuthenticationStateProvider AuthenticationStateProvider

Then add the following code block at the bottom of the file, to make the user information available to be referenced in our razor code:

@code {
    private string? userName;
    private bool isAuthenticated;

    protected override async Task OnInitializedAsync()
    {
        var authState = await AuthenticationStateProvider.GetAuthenticationStateAsync();
        var user = authState.User;
        isAuthenticated = user.Identity != null && user.Identity.IsAuthenticated;
        if (isAuthenticated && user.Identity != null)
        {
            userName = user.Identity.Name ?? string.Empty;
            await MongoDbService.InitializeAsync();
        }
    }
}

Finally, after the first div container definition in the file, add the following div which will display a welcome message to the user in the top-right corner:

<div class="d-flex justify-content-end align-items-center mb-2">
        @if (isAuthenticated && !string.IsNullOrEmpty(userName))
        {
            <span class="text-muted">Welcome, <strong>@userName</strong></span>
        }
    </div>

Testing it all out

Now, it is time to run the application for the first time and test logging in!

It is important to know that this code is not production-ready and does not have any cache for storing sessions. For this reason, you will need to empty your browser cache and cookies between runs of the application or you will get an MSAL error related to authentication not being passed.

But the first time you run the app, you will be asked to log in to a Microsoft account. This is because we are using EntraID. Make sure you use an account in the tenant you used when setting up EntraID. But once you sign in, you will get redirected to the homepage and will see your username in the top right corner.

Summary

There we have it, OIDC integrated into your .NET application, linked up to MongoDB Atlas for a secure experience!

This tutorial used Azure EntraID but MongoDB Atlas’ OIDC support is not exclusive to Azure. You can use other identity providers from Google and AWS, as well. You just need the client ID, client secret, and OIDC URL from them to configure it in MongoDB Atlas’ Workforce Federation as an external provider.

You could even combine this with Queryable Encryption (QE), MongoDB’s encryption capability that not only encrypts the data client-side for transport over the network but even at rest, for an extra level of security, while still allowing the encrypted data in the database to be queried against! In fact, there is an earlier tutorial I wrote which shows you how to add QE support to the same application. Why not try them together for even more security and let us know in the comments how you got on?

And if you want to learn more about security with MongoDB, you can take a skills badge on security that you can share on LinkedIn to show off your new knowledge!

Pre-computing Statistics for a Time Frame With MongoDB

Luce Carter — Tue, 12 Aug 2025 15:24:32 +0000

This article was written by MongoDB Champion Markus Wildgruber

Especially in IoT contexts, you often have the requirements that your users want to view aggregated data over a specific time frame. Imagine we have a system where a number of sensors reports a temperature value in a variable interval. The users of the system want to know the average temperature and the number of readings over the last seven days. For systems with a small amount of data, you might be able to calculate those statistics on the fly for each request, especially if you make use of time series collections or the Bucket Pattern to reduce the number of documents involved in the query.

As your system grows, calculating the statistics over and over again is not sustainable. Furthermore, users often want to view those data in a table and be able to filter and sort by those statistics to identify the sensors that reported the highest average temperature or the lowest number of readings in the time frame. In this case, you need to pre-compute the statistics and store them in a document per sensor, e.g.:

{
  "_id": /* Sensor Id */,
  "name": "Sensor name",
  "avgTemp": 23.45,
  "readingCount": 18,
  // other properties
}

This schema, while simplified, covers our requirements. You can create indexes to support filtering and sorting the documents for optimal performance. The indexes you need depend on your specific scenario, but as we store the average temperature and reading count in dedicated properties on the document, those are good candidates for an index if you often filter or sort by them. For instance, if you display the 10 sensors with the highest average temperature in the time frame on your dashboard, you could create an index with the following fields to cover the query:
avgTemp: -1
name: 1,
_id: 1

From a read perspective, our work is done at this point. In terms of writing, we need to consider two reasons to update the document:

A sensor reports new data.
The day changed and we need to ensure that the data of the document is up-to-date.

A simple approach for updating and its downsides

Updating statistics when new data are reported often is feasible without too much consideration as:

a) The amount of data that is involved is restricted to a single sensor.

b) We also know that the update for this sensor is necessary.

But what about the daily update for the statistics?

A straightforward way to update the documents is to have a job update the documents once per day. If you have been working with aggregation pipelines, you can easily imagine an aggregation pipeline that reads the values in the time frame from a time series collection, groups them by sensor, calculates the statistics, and merges the outcome into the sensors’ collection.

Especially in systems with a high number of sensors or in multi-tenant scenarios, this approach has serious downsides, because it puts a lot of load on the cluster during the update process. If you have sensors that might not report a reading each day, this update also might include sensors that are not outdated. Also, the approach includes sensors that have been updated when new data has been reported even if there is no need to update them.

Evolving to a more sophisticated approach

Our main goal is to reduce the load on the system when carrying the values forward for a new day. A more sophisticated approach achieves this by updating statistics for a sensor when new data is reported. This moves the load from the daily run to the ingestion of new values and thus spreads the load.
Also, the approach excludes documents from the daily update that do not need to be updated.

Fortunately, MongoDB's flexible data schema provides us with the toolset to achieve this by extending the sensor document with:

An array with statistics aggregated by day (more on that later).
The date property outdatedFrom that contains the date when the data needs to be updated in the daily run:

{
  "_id": /* Sensor Id */,
  "name": "Sensor name",
  "avgTemp": 23.45,
  "readingCount": 18,
  "statsByDate": [
    { "date": ISODate("2025-07-19T00:00:00Z"), readings: 7, totalTemps: 164.15 },
    { "date": ISODate("2025-07-21T00:00:00Z"), readings: 3, totalTemps: 70.35 },
    { "date": ISODate("2025-07-23T00:00:00Z"), readings: 5, totalTemps: 117.25 },
    { "date": ISODate("2025-07-24T00:00:00Z"), readings: 3, totalTemps: 70.35 },
  ],
  "outdatedFrom": ISODate("2025-07-26T00:00:00Z")
  // other properties
}

Let's consider our current date is July 25th, 2025. If the sensor reports new data for the day, a new sub-document is added to the statsByDate array, and the statistics are updated, but the outdatedFrom property keeps its value. The document will not be included in the daily run on July 25th, 2025.

On the next day, July 26th, 2025, data for July 19th is not part of the seven-day window for the statistics anymore. In this case, the daily job will remove the sub-document for this day from the array and update outdatedFrom to July 28th, 2025. Also, if new data for the sensor is reported before the daily job runs, the document will be updated and is not included in the daily job anymore.

As you can see, this approach keeps statistics up to date and also limits the documents in the daily run to the ones that really need to be updated.

Let's have a closer look at the two main reasons to update the data.

Ingesting new data for the sensor

As described above, we want to update all statistics for the sensor when new data is received. This builds upon the information that is already present in the document so that we either need to read the document from the database and perform the changes in code, or—better and much more efficient—use an update with aggregation pipeline. As updates on a single document in MongoDB are atomic, the latter also means that MongoDB will handle concurrency issues for us.

The following Javascript code (from a MongoDB playground) performs the update:

const sensorId = /* Sensor Id */;
const today = new Date("2025-07-25T00:00:00Z");
const daysInTimeframe = 7;
const cutOffDate = new Date(today.getTime() - daysInTimeframe * 24 * 60 * 60 * 1000);

db.getCollection("sensors").updateOne({ _id: sensorId }, [
  // Initialize array if not present (omit this if you are sure that the array is always present)
  {
    $set: {
      statsByDate: { $ifNull: ["$statsByDate", []] },
    },
  },
  // Cut off redundant data from array
  {
    $set: {
      statsByDate: {
        $filter: {
          input: "$statsByDate",
          cond: {
            $gt: ["$$this.date", cutOffDate],
          },
        },
      },
    },
  },
  // Add/update data in array
  {
    $set: {
      statsByDate: {
        $cond: {
          if: {
            $in: [today, "$statsByDate.date"],
          },
          then: {
            $map: {
              input: "$statsByDate",
              in: {
                $cond: {
                  if: { $eq: ["$$this.date", today] },
                  then: {
                    date: today,
                    totalTemps: { $add: ["$$this.totalTemps", temp] },
                    readings: { $add: ["$$this.readings", 1] },
                  },
                  else: "$$this",
                },
              },
            },
          },
          else: {
            $concatArrays: [
              "$statsByDate",
              [
                {
                  date: today,
                  totalTemps: temp,
                  readings: 1,
                },
              ],
            ],
          },
        },
      },
    },
  },
  // Update reading count
  {
    $set: {
      readingCount: { $sum: "$statsByDate.readings" },
    },
  },
  // Update avg temp (separate to reuse new value of readingCount)
  {
    $set: {
      avgTemp: {
        $cond: {
          if: { $gt: ["$readingCount", 0] },
          then: {
            $divide: [
              { $sum: "$statsByDate.totalTemps" },
              { $sum: "$statsByDate.readings" },
            ],
          },
          else: null,
        },
      },
    },
  },
  // Update outdatedFrom
  {
    $set: {
      outdatedFrom: {
        $cond: {
          if: { $gt: ["$readingCount", 0] },
          then: {
            $dateAdd: {
              startDate: { $min: "$statsByDate.date" },
              unit: "day",
              amount: daysInTimeframe,
            },
          },
          else: null,
        },
      },
    },
  },
]);

The aggregation pipeline above guards against cases where the array is not present at all and also where there are no values in the time frame so that readingCount is zero and a division by its value would fail. In this case, both avgTemp and outdatedFrom are set to null.

The most complex part is to add the new reading to the statsByDate array if there are no values for the date yet or update the existing sub-document in the array. The outer $cond checks if there is a sub-document for the date. If there is, the existing array is transformed using a $map statement that updates the sub-document for the date and returns the existing sub-document for other dates.

You could simplify this significantly if you forgo the aggregation by day and add each new reading to the array when it arrives. Keep in mind that sub-arrays in MongoDB documents should have a limited amount of items and should not be unbounded. If you get a lot of readings per day or if your time frame is large, it is preferable to aggregate values by date.

Also, if your scenario is based on a different unit for time frames—e.g., hours—adjust the aggregation accordingly so that the number of sub-documents in the array is limited while still allowing for cutting off redundant data that are not part of the time frame anymore.

Updating outdated documents

After being able to ingest new readings, we now take a look at the daily job that carries the values of outdated documents forward. Luckily, the last section already contains many building blocks that we can reuse in the aggregation pipeline for the job:

The initialization of the array if not present as a safety measure
Cutting of sub-documents in the array if they are not in the time frame anymore
The update of readingCount, avgTemp, and outdatedFrom

This leads to the following sample code in JavaScript:

const today = new Date("2025-07-25T00:00:00Z");
const daysInTimeframe = 7;
const cutOffDate = new Date(
  today.getTime() - daysInTimeframe * 24 * 60 * 60 * 1000
);

db.getCollection("sensors").updateMany(
  {
    outdatedFrom: {
      $lte: today,
    },
  },
  [
    // Initialize array if not present
    {
      $set: {
        statsByDate: { $ifNull: ["$statsByDate", []] },
      },
    },
    // Cut off redundant data from array
    {
      $set: {
        statsByDate: {
          $filter: {
            input: "$statsByDate",
            cond: {
              $gt: ["$$this.date", cutOffDate],
            },
          },
        },
      },
    },
    // Update reading count
    {
      $set: {
        readingCount: { $sum: "$statsByDate.readings" },
      },
    },
    // Update avg temp
    {
      $set: {
        avgTemp: {
          $cond: {
            if: { $gt: ["$readingCount", 0] },
            then: {
              $divide: [
                { $sum: "$statsByDate.totalTemps" },
                { $sum: "$statsByDate.readings" },
              ],
            },
            else: null,
          },
        },
      },
    },
    // Update outdatedFrom
    {
      $set: {
        outdatedFrom: {
          $cond: {
            if: { $gt: ["$readingCount", 0] },
            then: {
              $dateAdd: {
                startDate: { $min: "$statsByDate.date" },
                unit: "day",
                amount: daysInTimeframe,
              },
            },
            else: null,
          },
        },
      },
    },
  ]
);

The update identifies the documents that need to be updated by filtering on the outdatedFrom property. It then reuses the stages from the ingestion and applies the relevant changes to the documents.

Additional considerations

This sample provides an outline on how to pre-compute time frame statistics in MongoDB documents. It can be a starting point for tackling various other requirements regarding time frames—e.g., comparing statistics with the previous time frame. In this case, you need to double the number of days that are preserved in the statsByDate array and filter the items when calculating the statistics for the current and previous time frame. Also, outdatedFrom needs to consider both when an item leaves the time frame and when it moves from the current to the previous time frame.

If you are providing statistics for a time frame, chances are that users want to view and compare developments between different time frames. In this case, a key decision is whether to have a single document that keeps data for all time frames for a sensor or if you have a document per sensor and time frame. The main factor here is the size of your documents and how often data for different time frames is accessed. If your documents are not too big and users often switch between time frames, having all time frames in the same document is advisable from a performance point of view. This also allows you to show sensor data in a tabular view that contains columns for different time frames. On the other hand, having all time frames in the same document adds further complexity to the update statements.

This sample demonstrates how MongoDB's flexible data schema has a significant impact on system performance. The documents for the sensors are self-sufficient so that you can update the data in the sensors' collection without the need to access other collections. If you do not need to store the single readings of the sensors in the database, you can discard them (or save them in cold storage) so that your database is not bloated by these fine-grained data just for updating statistics over time.

How do you handle statistics for time frames? Do you have tips that you can share with fellow developers? Let us know in the comments!

How to Implement MongoDB Change Streams in C#

Luce Carter — Fri, 23 May 2025 13:25:38 +0000

In database systems, you need a way to know what has happened, as this allows you to recover from any issues. MongoDB is no different and has an operational log (oplog) for recording all operations that modify the database.

But if you want to be aware of changes to the database from your application, such as inserting or updating new documents, you don’t want to have to mess around with the oplog. So for this reason, back in MongoDB 6.0, we introduced Change Streams. This allows you to easily access the changes in real-time from your applications. It is available in multiple programming languages and that includes C#.

So in this tutorial, we are going to implement change stream functionality to an existing healthcare application. We will take advantage of SignalR for real-time communication from a Blazor application and hook it up to Change Streams using the C# driver.

By the end, there will be a page to simulate a pharmacy that will be alerted every time a new health condition is added for a patient. In the real world, this might be a great way to notify a pharmacy of a new prescription as it is prescribed to a patient in real-time.

If you just want to view the code, there is a branch of the repo available called with-change-streams where you can view the full functioning version.

Prerequisites

This tutorial is kept as simple as possible to follow along, but you will still need a few things in place:

A MongoDB cluster on MongoDB 6.0 or later and your connection string copied to the clipboard
A forked and cloned copy of the repo on the start-change-streams branch with the copied connection string added to app settings
.NET 9
An IDE/text editor that supports .NET/C#, such as Rider or VS Code

Implementing HealthConditionChangeStreamService

There is already lots of skeleton code available to you in the repo to minimise the amount of code required to be added outside the scope of the change stream feature itself. One of these semi-implemented classes available is HeathConditionChangeStreamService.cs inside the Services folder.

You will see there are already local variables available for the MongoDB Service class, SignalR hub, conditions cache, and an IMongoCollection<Patient> variable that will allow us to access the patients collection in our database. Let’s go ahead and add the missing code to the ExecuteAsync method that is already there.

We will do it in small sections and discuss what is happening. But all code will go inside this method. Start by pasting the following code:

        var allPatients = await _patientsCollection.Find(_ => true).ToListAsync(stoppingToken);
        foreach (var patient in allPatients)
        {
            var patientId = patient.Id.ToString();
            var conditionNames = patient.PatientRecord.HealthConditions.Select(hc => hc.Name).ToList();
            _conditionCache[patientId] = conditionNames;
        }

Change streams will let you know when any part of a document has been modified in some way. But in the case of array fields, such as the HealthConditions field, it can provide the new array but not what has changed. So we build up a cache first, containing the names of the health conditions as they currently are stored.

Now, we will start to configure our change stream to return the full document, create an empty pipeline as we don’t want to apply any filtering to the stream, and then open the stream. Paste the following code into the method below the code you just added:

var options = new ChangeStreamOptions
        {
            FullDocument = ChangeStreamFullDocumentOption.UpdateLookup
        };


        var pipeline =
            PipelineDefinition<ChangeStreamDocument<Patient>, ChangeStreamDocument<Patient>>.Create(new BsonDocument[]
                { });

        using var cursor = await _patientsCollection.WatchAsync(pipeline, options, stoppingToken);

Finally, for this section, we will add a larger chunk of code to loop through the update when it arrives:

await cursor.ForEachAsync(change =>
    {
        if (change.OperationType != ChangeStreamOperationType.Replace && change.OperationType != ChangeStreamOperationType.Create && change.FullDocument is null)
            return;

        var patient = change.FullDocument;
        var patientId = patient.Id.ToString();
        var newConditionNames = patient.PatientRecord.HealthConditions.Select(hc => hc.Name).ToList();

        // Retrieve the old list from the cache
        if (_conditionCache.TryGetValue(patientId, out var oldConditionNames))
        {
            // Determine added and removed conditions
            var addedConditions = newConditionNames.Except(oldConditionNames).ToList();
            var removedConditions = oldConditionNames.Except(newConditionNames).ToList();
     var updatedConditions = newConditionNames
.Where(newCondition => oldConditionNames.Any(oldCondition => oldCondition != newCondition))
                    .ToList();


            // Handle added conditions
            if (addedConditions.Any())
            {
                _hubContext.Clients.All.SendAsync("NewHealthConditionAdded", new
                {
                    patient = patient.PatientName,
                    condition = addedConditions
                }, cancellationToken: stoppingToken);
            }

            // Handle removed conditions
            if (removedConditions.Any())
            {
                _hubContext.Clients.All.SendAsync("HealthConditionRemoved", new
                {
                    patient = patient.PatientName,
                    condition = removedConditions
                }, cancellationToken: stoppingToken);
            }

// Handle updated conditions
                if (updatedConditions.Any())
                {
                    _hubContext.Clients.All.SendAsync("HealthConditionUpdated", new
                    {
                        patient = patient.PatientName,
                        condition = updatedConditions
                    }, cancellationToken: stoppingToken);
                }


            // Update the cache with the new list
            _conditionCache[patientId] = newConditionNames;
        }
        else
        {
            _conditionCache[patientId] = newConditionNames;

            // Send alert if there are any conditions
            if (newConditionNames.Any())
            {
                _hubContext.Clients.All.SendAsync("NewHealthConditionAdded", new
                {
                    patient = patient.PatientName,
                    condition = newConditionNames
                });
            }
        }
    }, stoppingToken);

This is quite a lot of code but what it is doing is fairly straightforward. It is looking at only replaced and new documents because when editing a patient in a different page, the update method in MongoDBService actually replaces the whole document. So if we selected to watch for updates, it would never trigger.

Then, if it is the correct type of change stream update, it processes the changes, including checking the difference between the cached health conditions array and the new one in the document and sending out an alert to all registered clients of our SignalR hub.

It passes stoppingToken which is a cancellation token to the driver as part of the loop, so if the cancellation is ever triggered, it stops iterating through the change stream cursor.

Updating Pharmacy page to listen for updates

There is already an existing Pharmacy page in the Components/Pages folder and it has been set up to have a table for viewing updates, and a hub created to listen to the existing pharmacy hub available in the repo.

All we need to do is paste the following code below the hub declaration in OnInitializedAsync() and before the call to StartAsync:

 hub.On<object>("NewHealthConditionAdded", (data) =>
        {
            var patient = (string)((JsonElement)data).GetProperty("patient").GetString();
            var conditions = ((JsonElement)data).GetProperty("condition").EnumerateArray().Select(x => x.GetString()).ToList();
            InvokeAsync(() =>
            {
                foreach (var condition in conditions)
                {
                    alerts.Insert(0, $"{patient} has a new condition: {condition}");
                }
                StateHasChanged();
            });
        });

hub.On<object>("HealthConditionUpdated", (data) =>
        {
            var patient = (string)((JsonElement)data).GetProperty("patient").GetString();
            var conditions = ((JsonElement)data).GetProperty("condition").EnumerateArray().Select(x => x.GetString()).ToList();
            InvokeAsync(() =>
            {
                foreach (var condition in conditions)
                {
                    alerts.Insert(0, $"{patient} condition {condition} has been updated");
                }
                StateHasChanged();
            });
        });

        hub.On<object>("HealthConditionRemoved", (data) =>
        {
            var patient = (string)((JsonElement)data).GetProperty("patient").GetString();
            var conditions = ((JsonElement)data).GetProperty("condition").EnumerateArray().Select(x => x.GetString()).ToList();
            InvokeAsync(() =>
            {
                foreach (var condition in conditions)
                {
                    alerts.Insert(0, $"{patient} condition {condition} has been removed");
                }
                StateHasChanged();
            });
        });

This is listening for any new, updated, or deleted conditions, getting the value and then adding it to the list of alerts. The UI is then informed that the state has changed so that it will display the alerts. And that’s it!

Testing it out

We now have a working pharmacy alert system that we can test out. Run the application and open two tabs, one for the main application and one set to /pharmacy. If you add new patients with health conditions or edit an existing patient to add a new health condition, you will see it update in the table of pharmacy alerts!

Summary

And there we go! In a matter of minutes, we have a working application that adds MongoDB Change Stream functionality to a .NET/C# Blazor application, using SignalR.

The full code can be found in the GitHub repo on the with-change-streams branch.

Why not start applying it to your applications today? If you have any questions or want to share how you got on, you can visit our Community Forums.

Advanced MongoDB Aggregation Pipeline Stages With C# Samples

Luce Carter — Mon, 14 Apr 2025 09:24:11 +0000

This is a guest author post from community contributor Markus Wildgruber.

My last two articles on aggregation pipelines with MongoDB and C# focussed on the basics of running aggregation pipelines, and presented ways to handle complex aggregation pipelines. This time, we will take a look at several aggregation pipeline stages and how you can use them to implement common scenarios in C#.

Instead of using LINQ, we will use the fluent interface so you know exactly how the MongoDB C# driver will serialize the pipeline when it is sent to the database.

One of the main goals is to make good use of the static types that C# provides so that we define the stages in a type-safe manner and can also use the results seamlessly after the aggregation.

The samples that are used throughout this article are based on the movies collection in the samples_mflix database. We will use the following data transfer object (DTO) class as a starting point:

[BsonIgnoreExtraElements]
public class Movie
{
    [BsonRepresentation(MongoDB.Bson.BsonType.ObjectId)]
    public string Id { get; set; } = ObjectId.GenerateNewId().ToString();

    [BsonElement("title")]
    public string Title { get; set; } = string.Empty;

    [BsonElement("cast")]
    public List<string> Cast { get; set; } = new();
}

Calculating properties

One of the great benefits of an aggregation pipeline is that you can calculate values based on the document contents. For instance, you might want to return not all elements of a sub-array, but only three of them. Suppose you want to display the movies in a list that provides limited space. Instead of showing all actors in the cast, a maximum of three can be displayed. In order to reduce the amount of data that is transported from the database to the API, you want to limit the number of array elements directly in the aggregation pipeline:

var result = await movies.Aggregate()
    // Match or other stages
    .Set(x => new Movie()
    {
        Cast = x.Cast.Take(3).ToList(),
    })
    .ToListAsync();

The above sample uses a $set stage and takes only the first three actors in the cast array. In the result, all the other properties of the movie document (that are mapped in the class definition) are still contained; the result is a list of Movie objects.
This image shows the first results—the second movie, “The Great Train Robbery,” has four actors in the database, but only three in our result:

What if you want to add a new property in a type-safe way? Say you want to include a flag that shows that there are more actors available. You could create a type that adds this specific flag by just deriving a class from Movie:

[BsonIgnoreExtraElements]
[BsonNoId]
public class MovieWithActorsFlag : Movie
{
    public bool HasMoreActors { get; set; }
}

The following pipeline will provide the results:

var result = await movies.Aggregate()
    .Set(x => new Movie()
    {
        Cast = x.Cast ?? new List<string>()
    })
    .Set(x => new MovieWithActorsFlag()
    {
        HasMoreActors = x.Cast.Count > 3,
        Cast = x.Cast!.Take(3).ToList(),
    })
    .As<MovieWithActorsFlag>()
    .ToListAsync();

The first Set stage is used to initialize the Cast array if it is null. Otherwise, the call to Count will fail as the $size operator needs an array as input and does not handle null well. Please also note that Set itself does not do a type conversion—hence the call to As as the last stage of the pipeline. This method is very handy in case you need to ensure that the data in the pipeline is mapped to the corresponding C# POCOs.

Projecting to a different form

In the last section, we had a look at how to add calculated properties to a result. But what if you want to reduce the schema to return just the properties that your use case requires? Let's stick with the example of showing a list of movies with the top three actors and a hint that there are more actors. To restrict the amount of data that is returned, we would use a $project stage or the following C# code:

[BsonIgnoreExtraElements]
public class MovieListItem
{
    [BsonRepresentation(MongoDB.Bson.BsonType.ObjectId)]
    public string Id { get; set; } = ObjectId.GenerateNewId().ToString();

    [BsonElement("title")]
    public string Title { get; set; } = string.Empty;

    [BsonElement("cast")]
    public List<string> Cast { get; set; } = new();

    public bool HasMoreActors { get; set; }
}

var result = await movies.Aggregate()
    .Set(x => new Movie()
    {
        Cast = x.Cast ?? new List<string>()
    })
    .Project(x => new MovieListItem()
    {
        Id = x.Id, 
        Title = x.Title, 
        Cast = x.Cast!.Take(3).ToList(),
        HasMoreActors = x.Cast.Count > 3,
    })
    .ToListAsync();

In contrast to the previous code, we need to set all properties that we want to be included in the result. However, as our goal is to reduce the amount of data, there should not be too many.

Adding information from other collections

Let's assume that we want to display the details of a movie along with the comments for that movie. As before, we create POCOs to hold the information:

[BsonIgnoreExtraElements]
[BsonNoId]
public class Comment
{
    [BsonElement("movie_id")]
    [BsonRepresentation(BsonType.ObjectId)]
    public required string MovieId { get; set; }

    [BsonElement("name")]
    public string Name { get; set; } = string.Empty;

    [BsonElement("text")]
    public string Text { get; set; } = string.Empty;
}

[BsonIgnoreExtraElements]
public class MovieWithComments : Movie
{
    public List<Comment> Comments { get; set; } = new();
}

Now, we can use Lookup to retrieve the comments with the movie:

var comments = db.GetCollection<Comment>("comments");
var details = await movies.Aggregate()
    .Match(x => x.Title == "The Wizard of Oz")
    .Lookup<Movie, Comment, MovieWithComments>(comments, x => x.Id, x => x.MovieId, x => x.Comments)
    .ToListAsync();

Please keep in mind that $lookup is an expensive operation in terms of performance so you should not use it on large amounts of data. As you have already heard for sure, MongoDB offers a flexible data schema that makes it easy to prepare data in the form that supports your application best. The decision of how to handle relationships in your data schema is a very important one and there are several patterns available that you can apply to depict relationships between your data entities.

Paging data with a single request

In order to use bandwidth responsibly, applications often display only a subset of the data to the user and use paging to retrieve data one page after the other. The user can "turn the pages" and load the next set of documents. In addition, the total count of documents that fulfill the current filter criteria is displayed. This often means that two queries are run against the database: one to get the count of documents and another to get the data of the current page. Of course, it would be more efficient to get both results in a single request to the database. This is where the $facet stage comes in; it allows splitting the aggregation pipeline and finishing the pipeline with several different sub-pipelines. The results of the sub-pipelines are returned as a single result at the end—exactly what we need for an efficient implementation of the paging request.

The following piece of code first creates two pipelines for the facets: one for the count and another one that retrieves the data for page six:

var pipelineCount = new EmptyPipelineDefinition<Movie>()
    .Count();
var facetCount = AggregateFacet.Create("count", pipelineCount);

var pipelinePage = new EmptyPipelineDefinition<Movie>()
    .Sort(Builders<Movie>.Sort.Ascending(x => x.Title))
    .Skip(50)
    .Limit(10);
var facetPage = AggregateFacet.Create("page", pipelinePage);

Afterward, these facets are appended to a pipeline that filters the documents:

var pageResult = await movies.Aggregate()
    .Match(x => x.Title.StartsWith("The "))
    .Facet(facetCount, facetPage)
    .SingleAsync();
var count = pageResult.Facets[0].Output<AggregateCountResult>().Single();
var page = pageResult.Facets[1].Output<Movie>().ToList();

Of course, there are lots of other use cases for $facet besides paging that benefit from reducing the number of requests to the database.

Summary

These samples show how to integrate MongoDB aggregations with C# in a type-safe way so that both MongoDB and C# can play to their strengths. The MongoDB C# driver offers lots of other aggregation methods that you can use to set up aggregation pipelines. If this is not enough, check out my article on handling complex aggregation pipelines with C# that shows how to include aggregation pipeline stages in JSON notation.

Which aggregation pipeline stages do you use most often? Are there cases that you wish should be supported in the driver? Let us know in the comments!

Getting Started With Queryable Encryption in the MongoDB C# Driver

Luce Carter — Tue, 08 Apr 2025 08:30:00 +0000

With the release of MongoDB 7.0 in August 2023, we introduced a feature called Queryable Encryption, the first of its kind. With queryable encryption, your data is encrypted, even at rest, with the server unable to read it either but still able to execute queries against it. You can specify what fields to encrypt so you can encrypt as much or as little of your document as you need.

The great news is, not only is this available for all tiers, but it is supported in our C# driver too!

In this tutorial, we are going to add queryable encryption to a healthcare application, ensuring that private information, such as social security number (SSN) and date of birth, is encrypted.

Plus, we will show how you can carry out both equality and range searches, even against encrypted fields!

Prerequisites

To follow along with this tutorial, you will need a few things in place:

.NET 9.
A forked and cloned copy of the GitHub repo. Ensure you are on the start-qe branch as we will build off this starting point.
Any MongoDB cluster on version 7.0 or later. I will be using a free-forever M0 tier cluster on Atlas.

Note: If you want to see the final code, there is a branch on the GitHub repo called with-queryable-encryption which is a working sample. You will just need to follow the next section on adding the automatic encryption library to run it locally.

Adding the automatic encryption shared library

The first thing to do is add the automatic encryption shared library to the project. You can find this in our download center. Be sure to select crypt_shared from the package dropdown box.

Download the correct version for your platform and then unzip it to the root of the project. You should end up with a folder named something like mongo_crypt_shared_v1-macos-arm64-enterprise-8.0.4. The final folder name will change depending on your platform and version downloaded but the contents should be the same. Next, we will get the path to the mongo_crypt file that we will add to appsettings.

On Windows:
Copy the path to the .dll file found in the bin folder.

On MacOS:
Copy the path to the .dylib file in the lib folder.

Updating appsettings

Now you have the path to the mongo_crypt file on your clipboard, it is time to update appsettings.json and appsettings.Development.json.

First, update the placeholder for the CryptSharedLibPath with the file path you copied to the clipboard.

After that, update the MongoDBConnectionString value with the connection string for your MongoDB cluster.

Adding QueryableEncryptionHelpers.cs

Now, we are going to add a new class in the Services folder. This class will act as a helper with methods for handling key management for the encryption.

Inside the Services folder, add a new class called QueryableEncryptionHelpers.cs and paste the following code:

using System.Security.Cryptography;
using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Driver.Encryption;
namespace EnterpriseHealthcareDotNet.Services;

public class QueryableEncryptionHelpers
{
  private readonly IConfigurationRoot _appSettings;
  public QueryableEncryptionHelpers(IConfigurationRoot appSettings)
  {
    _appSettings = appSettings;
  }

  public Dictionary<string, IReadOnlyDictionary<string, object>> 
GetKmsProviderCredentials(string kmsProviderName,
        bool generateNewLocalKey)
  {
    if(kmsProviderName == "local")
    {
      if (generateNewLocalKey)
      {
        File.Delete("customer-master-key.txt");

        // start-generate-local-key
        using var randomNumberGenerator = RandomNumberGenerator.Create();
        try
        { 
          var bytes = new byte[96];
          randomNumberGenerator.GetBytes(bytes);
          var localCustomerMasterKeyBase64 = Convert.ToBase64String(bytes);
          File.WriteAllText("customer-master-key.txt", localCustomerMasterKeyBase64);
        }
        catch (Exception e)
        {
          throw new Exception("Unable to write Customer Master Key file due to the following error: " + e.Message);
        }
    // end-generate-local-key
    }

    // start-get-local-key
    // WARNING: Do not use a local key file in a production application
   var kmsProviderCredentials = new Dictionary<string, IReadOnlyDictionary<string, object>>();
   try
   {
     var localCustomerMasterKeyBase64 = File.ReadAllText("customer-master-key.txt");
     var localCustomerMasterKeyBytes = Convert.FromBase64String(localCustomerMasterKeyBase64);

     if (localCustomerMasterKeyBytes.Length != 96)
     {
       throw new Exception("Expected the customer master key file to be 96 bytes.");
     }

     var localOptions = new Dictionary<string, object>
     {
       { "key", localCustomerMasterKeyBytes }
     };

     kmsProviderCredentials.Add("local", localOptions);
     }
      // end-get-local-key
     catch (Exception e)
     {
       throw new Exception("Unable to read the Customer Master Key due to the following error: " + e.Message);
     }
 return kmsProviderCredentials;

 }

 throw new Exception("Unrecognized value for KMS provider name \"" + kmsProviderName + "\"  encountered while retrieving KMS credentials.");
 }

public BsonDocument GetCustomerMasterKeyCredentials(string kmsProvider)
{
  if (kmsProvider == "local")
  {
    // start-kmip-local-cmk-credentials
    var customerMasterKeyCredentials = new BsonDocument();
    // end-kmip-local-cmk-credentials
    return customerMasterKeyCredentials;
  }
  else
    {
      throw new Exception("Unrecognized value for KMS provider name \"" + kmsProvider + "\"  encountered while retrieving Customer Master Key credentials.");
    }
  }

public AutoEncryptionOptions GetAutoEncryptionOptions(CollectionNamespace keyVaultNamespace,
        IReadOnlyDictionary<string, IReadOnlyDictionary<string, object>> kmsProviderCredentials)
{
  var kmsProvider = kmsProviderCredentials.Keys.First();

  // start-auto-encryption-options
  var extraOptions = new Dictionary<string, object>
  {
    { "cryptSharedLibPath", _appSettings["CryptSharedLibPath"] } 
     // Path to your Automatic Encryption Shared Library
  };

var autoEncryptionOptions = new AutoEncryptionOptions(
keyVaultNamespace,
kmsProviderCredentials,
extraOptions: extraOptions);
// end-auto-encryption-options

  return autoEncryptionOptions;
}

public ClientEncryption GetClientEncryption(IMongoClient keyVaultClient,
        CollectionNamespace keyVaultNamespace, Dictionary<string, IReadOnlyDictionary<string, object>> kmsProviderCredentials)
{
  var kmsProvider = kmsProviderCredentials.Keys.First();

  // start-client-encryption
  var clientEncryptionOptions = new ClientEncryptionOptions(
                keyVaultClient: keyVaultClient,
                keyVaultNamespace: keyVaultNamespace,
                kmsProviders: kmsProviderCredentials
  );
  var clientEncryption = new ClientEncryption(clientEncryptionOptions);
            // end-client-encryption
            return clientEncryption;
  }
}

This code uses features from both the MongoDB Driver and the additional MongoDB.Driver.Encryption package inside methods for client and master key management.

This only provides code for local keys as this is what we are using in this tutorial, but you can find examples for also handling Azure, GCP, AWS, and KMIP in the same file on the GitHub repo on the with-queryable-encryption branch.

Note: Local keys are great for development but not recommended for production! So if you are looking to deploy to production, be sure to take a look at using one of the cloud providers!

Updating MongoDBService.cs

Now we have the helpers class available to use, we can start working on updating MongoDBService.cs to call the newly available methods.

At the top of the class, add a new local variable:

private readonly QueryableEncryptionHelpers _qeHelpers;

Then, inside the constructor before the call to InitAsync, initialize the variable:

_qeHelpers = new QueryableEncryptionHelpers((IConfigurationRoot)_appSettings); 
_kmsProviderCredentials = _qeHelpers.GetKmsProviderCredentials(_kmsProviderName, generateNewLocalKey: true);

Next, add the following code to InitAsync:

var camelCaseConvention = new ConventionPack { new CamelCaseElementNameConvention() };
ConventionRegistry.Register("CamelCase", camelCaseConvention, type => true);

MongoClientSettings.Extensions.AddAutoEncryption(); // .NET/C# Driver v3.0 or later only
var clientSettings = MongoClientSettings.FromConnectionString(_uri);
        clientSettings.AutoEncryptionOptions = _qeHelpers.GetAutoEncryptionOptions(
            _keyVaultNamespace,
            _kmsProviderCredentials);
var encryptedClient = new MongoClient(clientSettings);

var keyDatabase = encryptedClient.GetDatabase(_keyVaultDatabaseName);

var encryptedFields = new BsonDocument
{
  {
    "fields", new BsonArray
    {
      new BsonDocument
        {
          { "keyId", BsonNull.Value },
          { "path", "patientRecord.ssn" },
          { "bsonType", "string" },
          { "queries", new BsonDocument("queryType", "equality") 
         }
        },
       new BsonDocument
         {
           { "keyId", BsonNull.Value },
           { "path", "dateOfBirth" },
           { "bsonType", "date" },
           { "queries", new BsonDocument("queryType", "range") }
           }
          }
        }
      };

var patientDatabase = encryptedClient.GetDatabase(_encryptedDatabaseName);

var clientEncryption = _qeHelpers.GetClientEncryption(encryptedClient,
_keyVaultNamespace,
_kmsProviderCredentials);

var customerMasterKeyCredentials = _qeHelpers.GetCustomerMasterKeyCredentials(_kmsProviderName);

if(!encryptedClient.GetDatabase(_encryptedDatabaseName).ListCollectionNames().ToList().Contains(_encryptedCollectionName))
{
  try
  {
    // start-create-encrypted-collection
    var createCollectionOptions = new CreateCollectionOptions<Patient>
    {
    EncryptedFields = encryptedFields
    };

    await clientEncryption.CreateEncryptedCollectionAsync(patientDatabase,
   _encryptedCollectionName, createCollectionOptions, _kmsProviderName, customerMasterKeyCredentials);
   // end-create-encrypted-collection
   }
   catch (Exception e)
     {
       throw new Exception("Unable to create encrypted collection due to the following error: " + e.Message);
     }
   }
   _patientsCollection = encryptedClient.GetDatabase(_encryptedDatabaseName).
GetCollection<Patient>(_encryptedCollectionName);

Let’s talk through what is happening in this code.

First, it sets up a case convention that automatically handles the difference in casing between the fields in the document when stored, and the casing used for the properties in Patient model, found in the Models folder.

Then, it sets up an encrypted client, configuring the connection string and key database and turning on auto encryption. Afterward, the encrypted fields are defined. We don’t need to encrypt every field in the document—just the ones that are private, such as the social security number (SSN) and date of birth. If this was being used in production, a healthcare provider might want to find all patients born between certain dates, to target them for specific vaccines, so the date of birth field is specifically configured to allow range queries.

This defines the encrypted fields for the collection. It is used when creating the collection for the first time and is not needed on every startup.

Next, it sets up the database, fetches the credentials, and then creates the collection if it doesn’t already exist, configuring it to encrypt the earlier defined fields.

Get all patients

Everything is in place now for configuring and setting up queryable encryption in our application, so it is time to put it into place.

Still inside MongoDBService.cs, add a new method after InitAsync:

public async Task<List<Patient>> GetPatientsAsync()
{
  if (_patientsCollection == null)
    throw new InvalidOperationException("Patients collection is not initialized");

  return await _patientsCollection.Find(_ => true).ToListAsync();
}

The repo already has pages created for interacting with patients, but we need to add the code to talk to our service class and call our newly added method.

Inside the @code block in Components/Pages/Patients.razor, add the following inside the OnInitializedAsync method: patients = await MongoDbService.GetPatientsAsync();.

This will populate the List<Patients> Patients variable with all the patients from the database, and display them on the page.

You will notice that there is an existing method available to navigate to a specific patient but that functionality is not defined in code yet, so let’s do that now.

Get patient by Id

In MongoDBService.cs, add the following method, below the method for getting all patients:

public async Task<Patient> GetPatientAsync(string id)
{
  if (_patientsCollection == null)
    throw new InvalidOperationException("Patients collection is not initialized");

  return await _patientsCollection.Find(p => p.Id == ObjectId.Parse(id)).FirstOrDefaultAsync();
}

Next, we want to call another override method in the code block of PatientDetails.razor, this time, for when the parameters update:

protected override async Task OnParametersSetAsync()
{
  if (!string.IsNullOrEmpty(Id))
  {
    patient = await MongoDbService.GetPatientAsync(Id);
  }
}

This already has the Id set up as a query parameter so when the page loads, the method will be called and the patient’s details will be fetched from the service class using that id.

Of course, there are currently no patients so if we ran the application now, we would see a message on the patients page saying there are no patients. We wouldn’t be able to access this page with a valid id. So let’s change that.

Add patients

Back in MongoDBService.cs, add the following to save new patients to the database:

public async Task AddPatientAsync(Patient patient)
{
  if (_patientsCollection == null)
    throw new InvalidOperationException("Patients collection is not initialized");

  await _patientsCollection.InsertOneAsync(patient);
}

The form for adding new patients is already defined in AddPatient.razor. we just need to call our new method from the submit method.

Inside HandleSubmit, before the call to navigate back to /patients, add await MongoDbService.AddPatientAsync(newPatient);

Updating a patient

Now we have the create and read methods from the CRUD (create, read, update, delete) operations defined, it is time to handle updating.

We are going to take advantage of the ability with the driver to replace whole documents. In MongoDBService.cs, add the following method after the previous:

public async Task UpdatePatientAsync(string id, Patient patient)
{
  if(_patientsCollection == null)
    throw new InvalidOperationException("Patients collection is not initialized");

  await _patientsCollection.ReplaceOneAsync(p => p.Id == 
  ObjectId.Parse(id), patient);
}

Now, we are actually going to change PatientDetails.razor with more changes than just calling the new method, as we want to add the ability to edit an existing patient and then call the new method to update it in the database.

Replace the whole file with the following code:

@page "/patient/{id}"
@using EnterpriseHealthcareDotNet.Models
@using EnterpriseHealthcareDotNet.Services
@using MongoDB.Bson
@inject MongoDBService MongoDbService
@inject NavigationManager NavigationManager

<h3>Patient Details</h3>

@if (patient == null)
{
    <p>Loading...</p>
}
else
{
    <div>
        <p><strong>Name:</strong> <input type="text" @bind="patient.PatientName" readonly="@(!isEditMode)" /></p>
        <p><strong>Date of Birth:</strong> <input type="date" @bind="patient.DateOfBirth" readonly="@(!isEditMode)" /></p>
        <p><strong>SSN:</strong> <input type="text" @bind="patient.PatientRecord.SSN" readonly="@(!isEditMode)" /></p>
        <p><strong>Health Conditions: </strong></p>
        <table class="table table-hover table-responsive-sm">
            <thead>
            <tr>
                <th>Name</th>
                <th>Date</th>
                <th>Status</th>
            </tr>
            </thead>
            <tbody>
            @if(patient.PatientRecord.HealthConditions == null || !patient.PatientRecord.HealthConditions.Any())
            {
                <tr>
                    <td colspan="3">No health conditions found.</td>
                </tr>
            }
            else
            {
                @foreach (var condition in patient.PatientRecord.HealthConditions)
                {
                    <tr>
                        <td><input type="text" @bind="condition.Name" readonly="@(!isEditMode)" /></td>
                        <td><input type="date" @bind="condition.Date" readonly="@(!isEditMode)" /></td>
                        <td>
                            <select @bind="condition.Status" disabled="@(!isEditMode)">
                                @foreach (var status in Enum.GetValues(typeof(Status)))
                                {
                                    <option value="@status">@status</option>
                                }
                            </select>
                        </td>
                    </tr>
                }
            }
            </tbody>
        </table>

        <button class="btn btn-primary btn-sm" @onclick="ToggleEditMode">@editButtonText</button>
        @if (isEditMode)
        {
            <button class="btn btn-success btn-sm" @onclick="SaveChanges">Save</button>
        }
        </div>
}

@code {
  [Parameter]
  public string Id { get; set; }

  private Patient? patient;
  private bool isEditMode = false;
  private string editButtonText = "Edit";

  protected override async Task OnParametersSetAsync()
  {
    if (!string.IsNullOrEmpty(Id))
    {
      patient = await MongoDbService.GetPatientAsync(Id);
    }
  }

  private void ToggleEditMode()
  {
    isEditMode = !isEditMode;
    editButtonText = isEditMode ? "Cancel" : "Edit";
  }

 private async Task SaveChanges()
 {
   if (patient != null)
   {
     await MongoDbService.UpdatePatientAsync(patient.Id.ToString(), patient);
    isEditMode = false;
    editButtonText = "Edit";
   }
 }
}

A lot of this code is still the same, but it now has a boolean for isEditable which toggles edit mode on and off, as well as buttons to edit, save, or cancel the changes and calls our UpdatePatientAsync method in the service class.

Delete a patient

Finally, it is time to add the ability to delete a patient. First, we will add the method inside MongoDBService.cs:

public async Task DeletePatientAsync(string id)
{
  if(_patientsCollection == null)
    throw new InvalidOperationException("Patients collection is not initialized");

  await _patientsCollection.DeleteOneAsync(p => p.Id == ObjectId.Parse(id));
}

Then, we are going to add a delete button in two places: in the list of patients in Patients.razor and on the individual patient in PatientDetails.razor.

Let’s add a delete button to our Patients.razor page first. Within the table block, update the foreach loop to add the following delete button after the existing view details button:

<button class="btn btn-danger btn-sm" @onclick="() => DeletePatient(patient.Id)">Delete</button>

This will error for now as we next need to implement the DeletePatient method in the code block:

private async Task DeletePatient(ObjectId patientId)
{
  await MongoDbService.DeletePatientAsync(patientId.ToString());
        patients = await MongoDbService.GetPatientsAsync();
}

Next, we will add the button to the PatientDetails.razor page. Add the following code to the page, after the if check for isEditMode:

<button class="btn btn-danger btn-sm" @onclick="() => DeletePatient(patient.Id)">Delete</button>

Then, add the following method to the code block on the page, to call the service class and navigate back to the list of patients on deletion:

private async Task DeletePatient(ObjectId patientId)
{
  await MongoDbService.DeletePatientAsync(patientId.ToString());
        NavigationManager.NavigateTo("/patients");
}

Searching encrypted fields

One of the things that makes queryable encryption such an incredible feature is that not only are your fields encrypted at rest, but you can still carry out queries against them!

The SSN field is configured to allow equality searches, and as mentioned earlier the date of birth field is configured to allow range queries. So let's add the ability to carry out those searches!

First, we will implement the functionality in MongoDBService.cs so it is available to be called from the page later.

Copy and paste the following code into the class after existing methods:

 public async Task<List<Patient>> SearchPatientsBySSNAsync(string searchSsn)
    {
        if (_patientsCollection == null)
            throw new InvalidOperationException("Patients collection is not initialized");
        var filter = Builders<Patient>.Filter.Eq("patientRecord.sSN", searchSsn);

        var patients = await _patientsCollection.Find(filter).ToListAsync();
        return patients;
    }

    public async Task<List<Patient>> SearchPatientsByDOBAsync(DateTime startDate, DateTime endDate)
    {
        if (_patientsCollection == null)
            throw new InvalidOperationException("Patients collection is not initialized");

        var filter = Builders<Patient>.Filter.And(
            Builders<Patient>.Filter.Gte("dateOfBirth", startDate),
            Builders<Patient>.Filter.Lte("dateOfBirth", endDate)
        );

        var patients = await _patientsCollection.Find(filter).ToListAsync();
        return patients;
    }

This code is the same as you would use for queries against non-encrypted data as it is all part of the MongoDB driver so you don't have to learn anything new or different. Yay!

Now we can implement the front end. The search page already exists in the repo with the forms for searching, but it lacks the calls to our new methods in the relevant handle methods.

Replace the existing handle methods in the code block with the following:

private async Task HandleSearchBySSN()
    {
        SearchResults = await MongoDbService.SearchPatientsBySSNAsync(searchModel.SSN);
        searchModel.SSN = "";
    }

    private async Task HandleSearchByDOB()
    {
        if (searchModel.StartDate.HasValue && searchModel.EndDate.HasValue)
        {
            SearchResults = await MongoDbService.SearchPatientsByDOBAsync(searchModel.StartDate.Value, searchModel.EndDate.Value);
        }
    }

Testing

Now we have everything in place, it is time to test it out. Run the application and give it a try.
The home page has links to view patients, add patients and search patients that you can visit.

If you add a patient and then view your cluster, there will be a database called medicalRecords and a collection in that database called patients where you will find your newly inserted document, complete with encrypted fields! This is queryable encryption at work. The data is encrypted in the database and in transit, but because the application has access to the keys used to encrypt the data, it isn’t encrypted when displayed!

You can even edit a patient document from the PatientDetails page now, or delete any existing patients, and you will see all the changes reflected in your collection.

Once you have patients added, as well as editing them you can now search them for as well via the search page!

Summary

Just like that, we have a working application showing queryable encryption in action. The QueryableEncryptionHelpers.cs class can even be reused across applications so you have a head start to make you super productive!

Then, all you need to do is configure the encryption settings and encrypted fields and continue to use it as you would with the C# driver without any encryption. So the barrier to entry for queryable encryption is really low.

The full code can be found in the GitHub repo on the with-queryable-encryption branch.

Why not start applying it to your enterprise applications today? If you have any questions or want to share how you got on, you can visit our Community Forums.

Migrating From PostgreSQL to MongoDB in a .NET EF Core Application

Luce Carter — Thu, 20 Feb 2025 16:00:00 +0000

For many reasons—including scalability, flexibility, and modernization—more and more development teams are looking to migrate away from PostgreSQL to MongoDB. In the past, these teams might have been hesitant if they used Entity Framework Core (EF Core) as their object relational mapper (ORM).

With EF Core being one of the most popular ORMs for .NET development, we received a lot of requests from the community for a provider, and so did Microsoft! So it made sense for MongoDB to develop a provider for EF Core. This went generally available (GA) in May 2024 and has been greatly received by the community.

Plus, developers can now use MongoDB's Relational Migrator, a free tool that uses smart algorithms and GenAI to streamline migrating from legacy relational databases to MongoDB, accelerating application modernization initiatives. So now is a great time to take a look at how you too can reap the benefits of all this by migrating your .NET application that uses PostgreSQL with EF Core, to use the MongoDB EF Core Provider.

In this tutorial, we will start off with a simple application that uses PostgreSQL as the database under the hood, and update it to still take advantage of EF Core but use MongoDB instead. So let's get started!

Preparing the data

For this tutorial, we won’t be going through the data migration process as we already have content on that with our getting started documentation and even a short but great demo video.

If you want to have the PostgreSQL data however, it has been uploaded to Azure Storage

However, Relational Migrator was used to migrate the content, even making sure to take advantage of the benefits of the MongoDB data model, such as embedding documents.

In the original PostgreSQL database, there are seven tables:

movies
actors
theaters
users
comments
movie_cast: This table contains two foreign keys, movie_id and actor_id, to show the actors in a film.
movie_genres: This table contains one foreign key, movie_id, and a column for genre. This is to show what genres belong to which film in the movies table.

Here at MongoDB, we have a mantra to help you understand how to think about your data from an application usage perspective: “Data that is accessed together should be stored together.”
So when connecting Relational Migrator to my PostgreSQL database, I selected the option to follow recommendations on how to model the data in collections for MongoDB.

This means we have gone from seven tables to four collections, as some fields have been embedded.

The four collections it suggested are:

Movies: This contains all the movie information and embeds the comments as an array of comment documents inside that document, the same for movie_casts and movie_genre.
Actors.
Theaters.
Users.

Note: It is highly recommended that you refine any schemas within Relational Migrator to optimize your data model to suit your needs before migrating. The recommendations you can select within Relational Migrator are a fantastic start but not perfect.

However, you will need to have the data available in your own MongoDB cluster that will be used in the tutorial.

You can find the data stored as a ZIP file on Azure that you will need to download, unzip, and then restore to your own cluster (M0 is sufficient) with mongorestore.

Once you have mongorestore on your machine, you can run the following command from the terminal from within the folder where you extracted the ZIP file.

mongorestore --uri <your connection string>/migrated_mflix 
 migrated_mflix/

Note: If you are running on Windows, make sure that the /bin folder from the installation is added to your path, and update the above command to call mongorestore.exe.

The migrated_mflix folder will contain the database and collections within that database that contain all the documents we will make use of in this tutorial.

Prerequisites

In order to follow along with this tutorial, you will need a few things in place:

An M0 or above cluster with the data loaded, as mentioned in the previous section
The GitHub repo forked and cloned to your machine with the with-postgresql branch checked out - The connection string updated with your own value in appsettings.json and appsettings.Development.json

"ConnectionStrings": {
      "MongoDBAtlasConnectionString": "mongodb+srv://<username>:<password>@<your cluster url>/migrated_mflix"
    }

.NET 9 SDK installed IDE/editor of choice

If you choose to use VS Code, ensure you have the REST Client extension installed as we will use this later to test the endpoints.

Note: If you want to see the finished result, there is a branch on the repo called with-mongodb

Updating the NuGet package

The first thing we want to do in our code is update the NuGet packages from using Npgsql.EntityFrameworkCore.PostgreSQL to using MongoDB.EntityFrameworkCore.
You can do this either from the NuGet package manager inside your IDE of choice, or from the terminal with the .NET CLI:

dotnet remove package Npgsql.EntityFrameworkCore.PostgreSQL

dotnet add package MongoDB.EntityFrameworkCore

This will of course lead to errors in the code because we removed the package, but we are going to update the code in the next section so it's fine to see errors for now.

Updating the models

The fields in our migrated documents don’t map directly to the columns in the tables in PostgreSQL. Plus, we don’t have as many collections compared to the number of tables in the relational database. So for this reason, the first step we are going to take is deleting model classes that are no longer required, and updating the existing ones to use MongoDB instead.

Delete old models

As mentioned, we don’t need some of the classes anymore as they are not collections in our new database, but instead embedded in the collections where they are accessed. So go ahead and delete the following files:

Comment.cs
MovieCast.cs
MovieGenre.cs

Updating existing models

Now, we can go ahead and update the remaining models to use the properties we care about now and add the MongoDB related attributes, available from the provider, that tell it how to work with our data.

Actor.cs

using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.EntityFrameworkCore;

namespace MigratedMoviesEFCore.Models;

[Collection("actors")]
public class Actor
{
    [BsonId]
    [BsonElement("_id")]
    public ObjectId Id { get; set; }

    [BsonElement("name")] 
    public required string Name { get; set; }
    [BsonElement("dateOfBirth")] 
    public required DateTime DateOfBirth { get; set; }
    [BsonElement("placeOfBirth")] 
    public required string PlaceOfBirth { get; set; }
}

Movie.cs

using System.ComponentModel.DataAnnotations;
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.EntityFrameworkCore;

namespace MigratedMoviesEFCore.Models;

[Collection("movies")]
public class Movie
{
  [BsonId] [BsonElement("_id")] public ObjectId Id { get; set; } = ObjectId.GenerateNewId();

  [BsonElement("title")]
  public required string Title { get; set; }

  [BsonElement("year")]
  public required int Year { get; set; }

  [BsonElement("runtime")]
  public required int Runtime { get; set; }

  [BsonElement("plot")]
  public required string Plot { get; set; }

  [BsonElement("fullplot")]
  public string FullPlot { get; set; }

  [BsonElement("released")]
  public required DateTime Released { get; set; }

  [BsonElement("rated")]
  public required string Rated { get; set; }

  [BsonElement("poster")]
  public required string Poster { get; set; }

  [BsonElement("comments")] 
  public List<Comment>? Comments { get; set; } = new List<Comment>();

}
public class Comment
{
  [BsonElement("text")]
  public required string Text { get; set; }

  [BsonElement("commentDate")]
  public required DateTime CommentDate { get; set; }
}

Theater.cs

using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.EntityFrameworkCore;

namespace MigratedMoviesEFCore.Models;

[Collection("theaters")]
public class Theater
{

    [BsonId]
    [BsonElement("_id")]
    public ObjectId Id { get; set; }

    [BsonElement("theaterName")]
    public required string TheaterName { get; set; }

    [BsonElement("street1")]
    public string? Street1 { get; set; }

    [BsonElement("street2")]
    public string? Street2 { get; set; }

    [BsonElement("city")]
    public string? City { get; set; }

    [BsonElement("state")]
    public string? State { get; set; }

    [BsonElement("zipcode")]
    public string? Zipcode { get; set; }

    [BsonElement("location_lat")]
    public decimal? LocationLat { get; set; }

    [BsonElement("location_lon")]
    public decimal? LocationLon { get; set; }
}

User.cs

using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.EntityFrameworkCore;

namespace MigratedMoviesEFCore.Models;

[Collection("users")]
public class User
{
    [BsonId]
    [BsonElement("_id")]
    public ObjectId Id { get; set; }

    [BsonElement("name")]
    public string? Name { get; set; }

    [BsonElement("email")]
    public string? Email { get; set; }

    [BsonElement("password")]
    public string? Password { get; set; }

    [BsonElement("created_at")]
    public DateTime? CreatedAt { get; set; }
}

Updating DbContext

Next, we need to update our MoviesDbContext.cs class inside the Services folder so it no longer holds DbSet properties for our deleted classes, as well as update the overridden method for our new data model. As you will see, not much is going to change as it is still going to use the EF Core paradigms and interaction patterns that you are used to.

Delete the DbSet property entries for the following:

Comments
MovieCasts
MovieGenres

The next thing we need to do is update the code for OnModelCreating.
This is to handle instances where the comments field is missing on a document. Going forward, if a document is created without passing a comments array, it will create an empty one. But this may not be the case in migrated data. You may also experience this in the future with error handling arrays in your own migrated data so it is a best practice worth learning.

protected override void OnModelCreating(ModelBuilder modelBuilder)
{    
  modelBuilder.Entity<Movie>().HasMany(m => m.Comments).WithOne().IsRequired(false);
}

Updating MovieService

Next, we need to remove the redundant methods from our IMovieService.cs interface and the corresponding Movie.cs class and make some adjustments to the data type for the id from int to string and add some return statements.

Update your IMovieService.cs class to contain the following:

public interface IMovieService
{
    IEnumerable<Movie> GetAllMovies();

    Movie GetMovieById(string id);

    void UpdateMovie(string movieId, Movie movie);

    void DeleteMovie(Movie movieToDelete);

    string AddMovie(Movie movie);
    IEnumerable<Actor> GetAllActors(); 

    Actor GetActorById(string id);

    void UpdateActor(string id, Actor actor);

    void DeleteActor(Actor actorToDelete);

    string AddActor(Actor actor);

    IEnumerable<Theater> GetAllTheaters();

    IEnumerable<User> GetAllUsers();
}

We then need to make some changes to existing methods within MovieServices.cs.

GetAllMovies - Change the m.MovieId in the OrderBy call to be m.Title instead.
GetMovieById and GetActorById - Change the passed in parameter from int to string. We also need to parse from string to ObjectId (a special data type in MongoDB for the unique _id field in all documents.

    public Movie GetMovieById(string id)
    {
        return _moviesDbContext.Movies.Find(ObjectId.Parse(id));
    }

Make sure you have

using MongoDB.Bson;

added at the top if your editor doesn’t automatically add it.


public Actor GetActorById(string id)
    {
        return _moviesDbContext.Actors.Find(ObjectId.Parse(id));
    }

AddMovie and AddActor: We want to change the return type from void to string and return the id of the movie and actors we just added, respectively.

public string AddMovie(Movie movie)
    {
        _moviesDbContext.Movies.Add(movie);
        // Outputting for debugging purposes
        _moviesDbContext.ChangeTracker.DetectChanges();
        Console.WriteLine(_moviesDbContext.ChangeTracker.DebugView.LongView);

        _moviesDbContext.SaveChanges();

        return movie.Id.ToString();
    }

public string AddActor(Actor actor)
    {
        _moviesDbContext.Actors.Add(actor);
        // Outputting for debugging purposes
        _moviesDbContext.ChangeTracker.DetectChanges();
        Console.WriteLine(_moviesDbContext.ChangeTracker.DebugView.LongView);

        _moviesDbContext.SaveChanges();

        return actor.Id.ToString();
    }

UpdateMovie and UpdateActor now take an additional parameter for the id and the contents of each method have been changed slightly:

public void UpdateMovie(string movieId, Movie movie)
    {
        var movieToUpdate = _moviesDbContext.Movies.FirstOrDefault((m => m.Id == ObjectId.Parse(movieId)));
        _moviesDbContext.Movies.Update(movieToUpdate);

        _moviesDbContext.ChangeTracker.DetectChanges();
        Console.WriteLine(_moviesDbContext.ChangeTracker.DebugView.LongView);

        _moviesDbContext.SaveChanges();
    }

 public void UpdateActor(string actorId, Actor actor)
    {
        var actorToUpdate = _moviesDbContext.Actors.FirstOrDefault(a => a.Id == ObjectId.Parse(actorId));
        _moviesDbContext.Actors.Update(actorToUpdate);
        // Outputting for debugging purposes
        _moviesDbContext.ChangeTracker.DetectChanges();
        Console.WriteLine(_moviesDbContext.ChangeTracker.DebugView.LongView);

        _moviesDbContext.SaveChanges();
    }

Finally, if you haven’t already, make sure to delete the methods we removed from the interface for comments, moviecasts, and moviegenres.

Updating Program.cs

Phew! We are almost there. The last couple of steps involve updates in the Program.cs file. First, we need to change the code that configures EF Core to use MongoDB instead of PostgreSQL.

builder.Services.AddDbContext<MoviesDbContext>(options =>
options.UseMongoDB(builder.Configuration.GetConnectionString("MongoDBAtlasConnectionString"), "migrated_mflix").EnableSensitiveDataLogging());

This project uses the Minimal API’s feature, so all our endpoints are defined within Program.cs. The final step we are going to take is to update our endpoints to remove the ones we no longer need and update some existing ones that currently have errors.

For simplicity, I will paste the final code here and then explain what has changed.

using Microsoft.EntityFrameworkCore;
using MigratedMoviesEFCore.Models;
using MigratedMoviesEFCore.Services;

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddOpenApi();

builder.Services.AddDbContext<MoviesDbContext>(options =>
    options.UseMongoDB(builder.Configuration.GetConnectionString("MongoDBAtlasConnectionString"), "migrated_mflix")
           .EnableSensitiveDataLogging());

builder.Services.AddScoped<IMovieService, MovieService>();

builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.MapOpenApi();
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();

#region movies
app.MapGet("/movies", (IMovieService movieService) =>
{
    return movieService.GetAllMovies();
})
.WithName("GetMovies");

app.MapGet("/movies/{id}", (IMovieService movieService, string id) =>
{
    return movieService.GetMovieById(id);
}).WithName("GetMovieById");

app.MapPost("/movies", (IMovieService movieService, Movie movie) =>
{
    string newId = movieService.AddMovie(movie);
    return newId;
}).WithName("AddMovie");

app.MapPut("/movies/{id}", (IMovieService movieService, string id, Movie movieToUpdate) =>
{
    movieService.UpdateMovie(id, movieToUpdate);
}).WithName("UpdateMovie");

app.MapDelete("/movies/{id}", (IMovieService movieService, string id) =>
{
    var movieToDelete = movieService.GetMovieById(id);
    movieService.DeleteMovie(movieToDelete);
}).WithName("DeleteMovie");
#endregion

#region actors
app.MapGet("/actors", (IMovieService movieService) =>
{
    return movieService.GetAllActors();
}).WithName("GetActors");

app.MapGet("/actors/{id}", (IMovieService movieService, string id) =>
{
    return movieService.GetActorById(id);
}).WithName("GetActorById");

app.MapPost("/actors", (IMovieService movieService, Actor actor) =>
{
    string newId = movieService.AddActor(actor);
    return newId;
}).WithName("AddActor");

app.MapPut("/actors/{id}", (IMovieService movieService, string id, Actor actorToUpdate) =>
{
    movieService.UpdateActor(id, actorToUpdate);
}).WithName("UpdateActor");

app.MapDelete("/actors/{id}", (IMovieService movieService, string id) =>
{
    var actorToDelete = movieService.GetActorById(id);
    movieService.DeleteActor(actorToDelete);
}).WithName("DeleteActor");
#endregion

app.MapGet("/theaters", (IMovieService movieService) =>
{
    return movieService.GetAllTheaters();
}).WithName("GetTheaters");

app.MapGet("/users", (IMovieService movieService) =>
{
    return movieService.GetAllUsers();
}).WithName("GetUsers");

app.Run();

Updated the query parameter type from int to string in the endpoint for /movies/id and /actors/id
Removed the parsing to Int32 of the string id in the DELETE endpoints
Removed the endpoints that are no longer applicable

Just like that, we are ready to test it!

Testing with .http file

A feature that has been around for a few years in .NET, but that I always forget about, is .http files (shout-out to our amazing Champion Kev Smith who taught me they existed). These files act as a way to test your own APIs, or even endpoints on someone else’s API, without even needing to leave your IDE or VS Code (with an extension).

Note: If you would rather use Swagger, you can run the application in debug mode and access /swagger.

However, for this tutorial, we are going to test our endpoints by writing calls in the MigratedMoviesEFCore.http file in the root of the project.

Make sure to update appsettings.Development.json and appsettings.json with your connection string.

 "ConnectionStrings": {
      "MongoDBAtlasConnectionString": "<Your Atlas URL>/migrated_mflix"
    }

From the terminal, run your application by calling dotnet run as the application needs to be running and available to test the endpoints.

Testing /movies

Inside the .http file, replace the existing GET with the following code to test the endpoint to get all movies:

GET {{MigratedMoviesEFCore_HostAddress}}/movies
Accept: application/json

###

The existence of this declaration, plus the three # symbols, will tell it that this is a complete test and you will see a play button appear, or perhaps you will see ‘Send Request’ appear above it, depending on your IDE/editor of choice.

Send the request and you should get back a list of all the available movies.

Next, we are going to create a new movie. This will look very similar to GET but instead uses POST and passes a JSON object to use in the request.

POST {{MigratedMoviesEFCore_HostAddress}}/movies
Content-Type: application/json

{
  "title": "The Shawshank Redemption",
  "year": 1994,
  "released": "1994-09-23T00:00:00Z",
  "rated": "R",
  "runtime": 142,
 "plot": "Two imprisoned men plot an escape",
  "fullplot": "Andy Dufresne is sentenced to life in prison...",
  "poster": "https://www.imdb.com/title/tt0111161/mediaviewer/rm10105600.jpg"
}
###

If you run that, you will see in the response that there is an id value returned. Copy that to clipboard as we will use that to test getting a movie by its Id.

GET {{MigratedMoviesEFCore_HostAddress}}/movies/<your copied movie id>

###

Make sure to update the id with the value that you copied. This will return a document containing the data of the movie you added in the previous step.

We can create and read the document, so now, let’s test updating it:

PUT {{MigratedMoviesEFCore_HostAddress}}/movies/<your copied id>
Content-Type: application/json

{
  "title": "The Shawshank Redemption",
  "year": 1994,
  "released": "1994-09-23T00:00:00Z",
  "rated": "18",
  "runtime": 142,
 "plot": "Two imprisoned men plot an escape",
  "fullplot": "Andy Dufresne is sentenced to life in prison...",
  "poster": "https://www.imdb.com/title/tt0111161/mediaviewer/rm10105600.jpg"
}

###

Again, make sure to add your id value. This doesn't return anything but you should see a 200 OK value to show it updated.

You could always run the test for getting that specific movie again if you would like to see that your changes have been applied.

Finally, we have DELETE:

DELETE {{MigratedMoviesEFCore_HostAddress}}/movies/<your copied id>

###

Testing /actors

Now that we have tested the /movies endpoints, it’s time to move on to /actors. Like the /movies endpoint, we are going to write two GET tests, a POST, a PUT, and a DELETE.

GET {{MigratedMoviesEFCore_HostAddress}}/actors

###

POST {{MigratedMoviesEFCore_HostAddress}}/actors
Content-Type: application/json

{
  "name": "Morgan Freeman",
  "dateOfBirth": "1937-06-02",
  "placeOfBirth": "Memphis, Tennessee, USA"
}

###

Copy the returned id of the new actor on insertion for use in the next tests.

GET {{MigratedMoviesEFCore_HostAddress}}/actors/<your copied actor document id>

###

PUT {{MigratedMoviesEFCore_HostAddress}}/actors/<your copied actor document id>
Content-Type: application/json

{
  "name": "Leonardo DiCaprio",
  "dateOfBirth": "1974-11-11T00:00:00.000Z",
  "placeOfBirth": "Los Angeles, California, USA"
}
###

Lastly, we can finish off this endpoint by deleting the recently added document.

DELETE {{MigratedMoviesEFCore_HostAddress}}/actors/<your copied actor document id>

###

Testing /theaters and /users

We are going to combine the last two endpoints as these have been kept simple and only have a single GET endpoint for both to fetch all documents.

GET {{MigratedMoviesEFCore_HostAddress}}/theaters

###

GET {{MigratedMoviesEFCore_HostAddress}}/users

###

Summary

Amazing! We have been able to update from using PostgreSQL as the database under the hood, to the flexible, scalable, and modern solution of MongoDB, while still enjoying the paradigms from EF Core that we are used to.

You can find the final full code to this on GitHub.

Give it a try today and see how easy it is to migrate over, with the great partnership of the Relational Migrator and our EF Core Provider. If you have any questions or want to share how you got on, you can visit our Community Forums.