Forem: Santanu Paul

Improving the local development setup for running MongoDB Replica Set with Docker

Santanu Paul — Fri, 19 Mar 2021 19:36:21 +0000

All associated source code used in this blog can be found at the following Github repository:

sntnupl / devcontainers-mongodb-replica-set-with-docker

Docker based development container template to run a MongoDB replica set cluster in local machine.

In my previous post, we created a docker based development setup for running MongoDB Replica Set.

A flaw in that setup, was that although clients could connect to stand-alone nodes of the replica set, they could not connect to the replica set as a whole, as reported by this issue on the github repository.

In this blog post, we will see what can be done, to overcome this problem.

Root Cause

As per the MongoDB specifications:

the hostnames used in a replica set config are reachable from the client

In our previous blog, we initialized the following replica set configuration:

MONGODB1=mongo1
MONGODB2=mongo2
MONGODB3=mongo3
:
var cfg = {
    "_id": "rs0",
    ...
    "members": [
        {
            ...
            "host": "${MONGODB1}:27017",
            ...
        },
        {
            ...
            "host": "${MONGODB2}:27017",
            ...
        },
        {
            ...
            "host": "${MONGODB3}:27017",
            ...
        }
    ]
};

In essence, this would require the clients to able to reach the following hosts: mongo1:27017, mongo2:27017, mongo3:27017.

We can update /etc/hosts file to point 127.0.0.1 to mongo1, mongo2, mongo3 hostnames. But the portnumber 27017 would still cause the problem - we can't have 3 nodes (services) running on the same port.

Therefore, solution to the problem would be the following:

Update hosts file in client machines, to map hostnames mongo1, mongo2, mongo3 to 127.0.0.1
Run mongod service on the 3 nodes of the replica set. in different port numbers

Solution

Implementing the 1st step is trivial. One would need to update /etc/hosts file in Linux, or C:\Windows\System32\drivers\etc\hosts in Windows:

127.0.0.1 localhost mongo1 mongo2 mongo3

Let's go through the steps to implement the second step.

Changes to the docker compose file

...
mongo1:
  hostname: 'mongo1'
  container_name: 'mongo1'
  ...
  command: ["-f", "/etc/mongod.conf", "--port", "30001", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
  expose: 
    - 30001
  ports: 
    - 30001:30001 
  ...

mongo2:
  hostname: 'mongo2'
  container_name: 'mongo2'
  ...
  command: ["-f", "/etc/mongod.conf", "--port", "30002", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
  expose: 
    - 30002
  ports: 
    - 30002:30002 
  ...

mongo3:
  hostname: 'mongo3'
  container_name: 'mongo3'
  ...
  command: ["-f", "/etc/mongod.conf", "--port", "30003", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
  expose: 
    - 30003
  ports: 
    - 30003:30003 
  ...

This will bring up mongod services in the 3 nodes in 3 different ports.

Next, we modify the replica set configuration, by changing mongosetup.sh file:

mongo --host ${MONGODB1}:30001 -u ${MONGO_INITDB_ROOT_USERNAME} -p ${MONGO_INITDB_ROOT_PASSWORD} <<EOF
var cfg = {
    "_id": "rs0",
    ...
    "members": [
        {
            ...
            "host": "${MONGODB1}:30001",
            ...
        },
        {
            ...
            "host": "${MONGODB2}:30002",
            ...
        },
        {
            ...
            "host": "${MONGODB3}:30003",
            ...
        }
    ]
};

These are essentially all the necessary changes needed to be done, to solve the problem.

Final Result

Once we make these changes, we could connect to the replica set using mongo client:

$ mongo "mongodb://localhost:30001,localhost:30002,localhost:30003/<MONGO_INITDB_DATABASE>?replicaSet=rs0" -u <MONGO_INITDB_USERNAME> --authenticationDatabase admin

$ mongo "mongodb://mongo1:30001,mongo2:30002,mongo3:30003/<MONGO_INITDB_DATABASE>?replicaSet=rs0" -u <MONGO_INITDB_USERNAME> --authenticationDatabase admin

We can also view result of db.isMaster() command to debug details like hosts of the replicaset a client must connect to, if the node is primary, etc.

Test Driven Development of Azure Functions with C# Part 3: Integration Tests

Santanu Paul — Mon, 08 Mar 2021 07:13:38 +0000

In the previous post of this series, we saw how one can implement Unit Tests for an Azure Function.
While Unit Tests forms the base of our Testing Pyramid, they are not enough from TDD completion point of view.
The next phase would be to implement Integration Tests for our Azure Function.

Other Posts in this series

Part 1 - Introduction to our sample Azure Functions application
Part 2 - Unit Tests
Part 3 - This Article

I have created this repository on Github, which contains all the associated source code used in this blog series:

sntnupl / azure-functions-sample-unittest-integrationtest

Sample application to showcase how one can implement Unit and Integration testing for an Azure Functions application

The relevant Code for this post is under InvoiceProcessor.Tests.Integration project in this repository.

image source

Few of the characteristics that would separate these Integration tests from Unit Tests:

We would not mock the Event Triggers from Service Bus. We will make our Azure Function to get triggered by real messages coming from Service Bus Topic.
We would not be mocking the IBinder that our Azure function uses to read from Blob storage - we will use actual blob storage
We would not be mocking the IAsyncCollector to write to Azure Table Storage. We will be using real Table Storage that our Azure Function will write to, and we will be verifying those writes by reading from those real Table Storage.

As you can imagine, to achieve these feats, we would need to do some background work first.
Before going into the details of those "background work", let us understand some concepts first.

Azure WebJobs SDK and how it is related to Azure Functions

When we execute an Azure Functions app, it will start a console app.
However, the actual "function" can't operate on its own and magically listen for event triggers - there is no magic. It needs a runtime container, a Host - which has the capability of listening for event triggers and executing this Azure Function.
This host will be responsible for starting the Azure Functions app, and managing its lifetime. It will also set up Dependency Injection, Logging and Configuration for the Azure Function app.
This host is implemented via the JobHost class in Azure WebJobs SDK (source code here). Azure Functions is built on top of the Azure WebJobs SDK package, which allows it to use this JobHost class.

Of course, when we implement an Azure Function app, or even execute it, all of the details related to the JobHost will be abstracted away from us.
Inside an Azure Functions project, the only knobs you get to the Host is via the host.json file. A complete reference to all the possible host settings that can be tweaked via host.json is available at this link.

So once we understand this relationship, we can see that in order to setup our Integration Testing environment, we would need to

bring up this host
Tell it where to look for the Azure Functions to load

With this knowledge under our belt, lets go through our Integration Test project.

Note: All code discussed in the following sections is in the InvoiceProcessor.Tests.Integration Project.

Designing the Integration Tests

The SUT for our Integration Test is the Azure Function inside InvoiceWorkitemEventHandler class. This function essentially does 3 things:

Gets triggered as and when a new message arrives in a specific Azure Service Bus Topic, and parses this message to extract the location of an uploaded invoice
Reads the blob from the above extracted location, and parses the content of the uploaded invoice file
Finally it persists the content in a specific format inside Azure Table Storage

As such, our Integration Tests will need to

Upload a dummy invoice file into an Azure Blob Storage Container
Create message with the location of the above created blob and send it to the Service Bus Topic our Azure Function is listening to
Finally, wait for the corresponding entries to be created in the Azure Table Storage

Before doing these tasks, however, our Tests must ensure that any artifacts from previously run tests are cleaned off. In other words, it needs to:

Delete any Azure Blob Storage containers that previous tests would have created
Create a new Azure Blob Storage container that will be used by the tests in the current run
Delete any entries inside Azure Table Storage that was created by previously running tests.

The entity that takes care of these tasks is the EndToEndTestFixture class.
We leverage the Class Fixture pattern with this class, which allows us to create test context that will get shared across all tests in the ValidInvoiceTests test suite.

This class inherits from the IAsyncLifetime interface of xUnit as well, which allows us to perform asynchronous setup tasks during its initialization.

Writing our Integration Tests

Arrange Part

[Fact]
public async Task ValidInvoiceUploaded_GetsParsedAndSavedToTableStorage()
{
    string blobPath = await _testFixture.UploadSampleValidInvoice();

    // ...
}

Before doing anything else, we are uploading our sample invoice file to Azure Blob Storage.

[Fact]
public async Task ValidInvoiceUploaded_GetsParsedAndSavedToTableStorage()
{
    // ...
    IHost host = new HostBuilder()
        .ConfigureWebJobs()
        .ConfigureDefaultTestHost<InvoiceWorkitemEventHandler>(webjobsBuilder => {
            webjobsBuilder.AddAzureStorage();
            webjobsBuilder.AddServiceBus();
        })
        .ConfigureServices(services => {
            services.AddSingleton<INameResolver>(resolver);
        })
        .Build();

    using (host) {
        await host.StartAsync();
        // ..
    }

    // ...
}

This is the part where we set up a Host that will contain our Azure Functions.
Specifically we have used the ConfigureDefaultTestHost<InvoiceWorkitemEventHandler>() helper method to tell the host to look for Azure Functions inside InvoiceWorkitemEventHandler class.
We have also initialized the Azure Storage (AddAzureStorage()) as well as the ServiceBus (AddServiceBus()) extensions, as both of these are used by our SUT.
once the host is created, we invoke StartAsync() to start it.

Act Part

public async Task ValidInvoiceUploaded_GetsParsedAndSavedToTableStorage()
{
    // ...
    using (host) {
        // ...
        await _testFixture.SendInvoiceWorkitemEvent(blobPath);
        // ...
    }
}

Here we create a message with the location of the uploaded invoice file, and send it to the Azure Service Bus Topic.
As soon as we push this message, our SUT should get triggered, and if everything goes well, Azure Table Storage would get updated with corresponding entries.

Assert Part

public async Task ValidInvoiceUploaded_GetsParsedAndSavedToTableStorage()
{
    // ...
    using (host) {
        // ...

        List<AcmeOrderEntry> parsedOrders = await _testFixture.WaitForTableUpdateAndGetParsedOrders();
        _output.WriteLine($"Found {parsedOrders.Count} invoices.");
        Assert.True(parsedOrders.Count == 2);
    }
}

Here we are waiting for the entries to be uploaded in the Table Storage, and asserting that 2 entries were created (our sample invoice contained two entries).

Similar to running Unit Tests, we can use the Test Explorer in Visual Studio to execute this Integration Test as well.

Summary

In this series of post, we saw how we can effectively perform Test Driven Development of Azure Functions.
I have often seen developers worry about this aspect, when it comes to Azure Functions.
The fact that these functions involves triggeres external entities like Azure Blob Storage, Azure Service Bus, etc, makes developers believe that unit/integration testing the Azure Functions will be a complex endeavour.
As we have seen in this series of posts, that notion is not necessarily true.

I hope that this series helped you to embrace TDD in your Azure functions app, and hopefully helped you become a better developer by shipping higher quality code.

Thanks for reading! 🙂

Test Driven Development of Azure Functions with C# Part 2: Unit Tests

Santanu Paul — Mon, 08 Mar 2021 07:08:05 +0000

In Part 1 of this series we started to explore how to implement Test Driven Development of Serverless Azure Functions application.
We had an overview of some of the Event Bindings that allow Azure Functions to integrate with Azure Services like Azure Blob Storage, Azure Service Bus, etc.

In this post, we will see how we can add Unit Tests to our Azure Functions application.

Other Posts in this series

Part 1 - Introduction to our sample Azure Functions application
Part 2 - This Article
Part 3 - Integration Tests

I have created this repository on Github, which contains all the associated source code used in this blog series:

sntnupl / azure-functions-sample-unittest-integrationtest

Sample application to showcase how one can implement Unit and Integration testing for an Azure Functions application

The relevant Code for this post is under InvoiceProcessor.Tests.Unit project in this repository.

Adding Unit Test Project

We will be using xUnit Test Framework to implement Unit Tests for our Azure Function.

You can visit this official document, on how to get started with creating a new xUnit Test Project via Visual Studio.

Refer to InvoiceProcessor.Tests.Unit project within our Github repository, for all the relevant code discussed in this post.
File InvoiceWorkitemEventHandlerShould.cs hosts all the tests that we will use to Unit Test our Azure Function.

Ground work

Before writing the test cases, we would need to do create some resources that we are going to need for our testing. Let's take a look at them.

Mocked Logger
We need to pass an instance of ILogger to our Azure Function.
Going forward, let's address this as SUT.

To aid in our testing, we have created our own implementation of ILogger.
This implementation is available in ListLogger.cs class within our project.

Some implementations of Unit Testing uses NullLoggerFactory to create an instance of ILogger and pass the same in the Azure Function.
That works for sure, but in our testing strategy, we intend to capture the logs generated by our SUT to validate sucess/failure of our test cases.
As such we have created ListLogger, which essentially stores all the published logs from SUT into a List<string>. Within our test cases we will be checking the contents of this list to Assert certain behaviors.

Mocked AsyncCollector

Our SUT takes IAsyncCollector<AcmeOrderEntry> as its paramter. So we need to create a mock for this as well.
To aid in our testing, we have created our own implementation of IAsyncCollector, called AsyncCollector - you can refer to AsyncCollector.cs for the code.
This class essentially wraps over a List<T> as well. Any invocation to AddAsync() will be adding items to this internal List.
This again, allows us to verify that SUT invoked AddAsync() properly on the IAsyncCollector<AcmeOrderEntry> that got passed to it.

Other Mocks

Apart from the above, we have mocked the following entities in our Test Project

IBinder: this is passed as the method parameter to SUT
IAcmeInvoiceParser
- This is used within the SUT code, to parse the blob text.
- To keep things simple, we have kept this as a publicly accessible static property within InvoiceWorkitemEventHandler class, instead of using Dependency Injection.

All these mocks is created within the constructor of our Test Suite, the InvoiceWorkitemEventHandlerShould class.
XUnit has few powerful strategies to allow developers share these resources as a test context. I would suggest going through the official xUnit documentation on Shared Context between Tests, for more details.
To put things briefly, we wanted to recreate these mocks afresh for every test, hence we created them within the constructor of our Test Suite.

Writing our first Unit Test

While writing these test cases, we must not loose sight of the fact that at the end of the day, an Azure Function is just a simple method which can be invoked as any normal function.

public static async Task Run(
    [ServiceBusTrigger("workiteminvoice", "InvoiceProcessor", Connection = "ServiceBusReceiverConnection")] string msg,
    IBinder blobBinder,
    [Table("AcmeOrders", Connection = "StorageConnection")] IAsyncCollector<AcmeOrderEntry> tableOutput,
    ILogger logger)

So although we see the 1st parameter string msg decorated with the ServiceBusTrigger attribute, we dont need to do any extra work with respect to this attribute.
To simulate a message coming this Azure Function via the Service Bus, we just pass a string in the 1st parameter, thats it!

With that being said, lets go through one of the test cases which validates that an invalid message arriving via Service Bus would get rejected by our SUT.
This test is housed with RejectInvalidMessagesFromServiceBus() method of InvoiceWorkitemEventHandlerShould class.

Arrange Part

public async Task RejectInvalidMessagesFromServiceBus()
{
    _mockParser
        .Setup(x => x.TryParse(
            It.IsAny<Stream>(),
            _testLogger,
            out _mockParsedOrders))
        .Callback(new TryParseCallback((Stream s, ILogger l, out List<AcmeOrder> o) => {
            o = new List<AcmeOrder>();
        }))
        .Returns(true);

    //...
}

Here we are setting up our mocked IAcmeInvoiceParser. We have used the wonderful moq package to create these mocks.

This code essentially states that whenever SUT invokes TryParse() method on our mock, return a value of true.

We dont really care about the Stream parameter that the SUT passes to TryParse() - It.IsAny<Stream>()
But we do expect the SUT to pass our very own ILogger implementation (ListLogger) to be passed in TryParse() - _testLogger
The Callback() part is to essentially mock-populate the out parameter that SUT will be passing to TryParse() method.

public async Task RejectInvalidMessagesFromServiceBus()
{
    // ...

    _mockBlobBinder
        .Setup(x => x.BindAsync<Stream>(
            It.IsAny<BlobAttribute>(),
            default))
        .Returns(Task.FromResult<Stream>(null));

    var sut = new InvoiceWorkitemEventHandler();
    InvoiceWorkitemEventHandler.InvoiceParser = _mockParser.Object;

    // ...
}

Here we are setting up our mocked IBinder instance.
SUT can pass any BlobAttribute, and default value of CancellationToken (which is null), and our mock will return a Task<stream> with value null.

We have also created an instance of InvoiceWorkitemEventHandler class and injected our mocked IAcmeInvoiceParser to its public static property.

Act Part

public async Task RejectInvalidMessagesFromServiceBus()
{
    // ...

   await InvoiceWorkitemEventHandler.Run(
        "Invalid work item",
        _mockBlobBinder.Object,
        _mockCollector,
        _testLogger);

    // ...
}

We invoke the SUT, with an invalid message "Invalid work item".

Assert Part

public async Task RejectInvalidMessagesFromServiceBus()
{
    //...

    var logs = _testLogger.Logs;

    logs.Should().NotBeNull();
    logs.Should().NotBeEmpty();
    logs.Should().NotContain(l => l.Contains("Empty Invoice Workitem."));
    logs.Should().Contain(l => l.Contains("Invalid Invoice Workitem."));
    _mockBlobBinder.Verify(b => b.BindAsync<Stream>(It.IsAny<BlobAttribute>(), default), Times.Never);
}

As we mentioned earlier, we will be leveraging our ListLogger class to verify SUT behavior.

Inside the SUT test code, we first check if the message from Service Bus is empty. If it is, the SUT logs message Empty Invoice Workitem. and exit.
Then it checks if the message from Service Bus is invalid. If it is it will log message Invalid Invoice Workitem. and exit.

What we have asserted here is that the first message is not logged by SUT, as message was not empty: logs.Should().NotContain(l => l.Contains("Empty Invoice Workitem."));
However the second message should get logged, becuase the message was an invalid InvoiceWorkitemMessage: logs.Should().Contain(l => l.Contains("Invalid Invoice Workitem."));

Lastly, we also assert that SUT must not invoke BindAsync on the IBinder passed to it, because the method should have exited.

Here, we have leveraged the FluentAssertions package to write the descriptive Assertions messages.

To execute test cases like these, use the Test Explorer in Visual Studio.

In the last post of this series, we will see how we can perform Integration Testing on our Azure Function.

Test Driven Development of Azure Functions with C# Part 1: Introduction to the application

Santanu Paul — Mon, 08 Mar 2021 07:04:14 +0000

Over the last few years, Serverless computing has become one of the most appealing choice for developing backend application.

With PaaS offerings like Azure App Service, we already had the benefit of not having to manage and maintain our Server infrastructure.

The cloud service provider will do all the heavy lifting required, to manage the infrastructure, perform auto scaling and auto provisioning of the application.

Now with FaaS offerings like Azure Functions, AWS Lambda, backend developers have been given the additional benefit of not having to maintain all the application server code.

This is a very powerful abstraction, which makes perfect sense for event-driven applications. Once we write the piece of code that you want to execute for certain event(s), Azure Functions will do the hard work of

hosting this code (in some abstracted server infrastructure)
provide the glue between the actual event and your application (ensure that your code actually gets triggered as and when the actual event gets triggered)
auto-scale the application based on the volume of events received

For event driven applications that runs on the cloud, this is a very powerful model indeed.

These days, we are seeing more and more organizations curating their backend systems running on the Cloud as a Microservice based architecture.

In Azure for example, these microservices will leverage cloud services like Blob Storage, Message Queues, Service Bus, Event Hubs, Table Storage etc to communicate with each other.

Azure Functions allows us to register our code to get notified for activities happening on these services, by registering to get notified for specific "events".

For example, an Azure Function can get notified for a "Blob Event", that indicates that a blob was created at a certain path. Similarly, a message arriving on a Service Bus Topic can notify an Azure Function who has registered for it.

The icing on the cake is that registration for a certain event is simply via adding event bindings of these services in the methos parameters - as simple as it gets!

In this post, I would like to share a practical guide towards Test Driven Development (TDD) of Serverless Applications using Azure Functions.

When I started developing serverless applications using Azure Functions, writing Unit Tests and Integration tests were two aspects in which I could not find good, detailed guides.

Over the last few months, I do see ample resources guiding developers on how to Unit Test their Azure Functions app, but there is not much guidance on how to implement Integration Tests in these apps.

In this series of posts, I would try to share my learnings, which hopefully would help future developers in their journey.

This post will be the first of a three-part blog series, in which I shall introduce a sample Azure Functions application written in C#.

Other Posts in this series

Also, I have created this repository on Github, which contains all the associated source code used in this blog series:

sntnupl / azure-functions-sample-unittest-integrationtest

Sample application to showcase how one can implement Unit and Integration testing for an Azure Functions application

The Sample Application

Our sample application is an Azure Function, which will get triggered by an event triggered by Azure Service Bus, when a message is sent to Service Bus Topic.
This message essentially contains location to a blob in Azure Blob Storage.
Once our app is able to read the text contents of this blob, it will try to parse its contents as (fictious) Invoice entries of Acme Corp.
These parsed Acme Corp invoice entries will then be persisted in Azure Table Storage.

If you have worked with Azure functions a question you might be wondering, is that instead of using Service Bus trigger, why don't we use the trigger generated by the Blob storage itself.
That can be a valid solution, in certain use cases. I chose to architect our application such that we read the location from a service bus message because of the following reasons:

I can send additional metadata about the upload blob in a cleaner fashion. For e.g., I can pass metadata like transaction-id, user-id in the Service Bus message
I can choose to be flexible on the actual location of the uploaded file (what if some client wants to use AWS S3?)
Another application who wants these uploaded invoices to be processed, can choose to delay the processing, or may even want to batch them up. Both of these can be easily achieved if we choose Service Bus Trigger.

Project InvoiceProcessor hosts our Azure Function. inside InvoiceWorkitemEventHandler.cs.

[FunctionName("InvoiceWorkitemEventHandler")]
public static async Task Run(
    [ServiceBusTrigger("workiteminvoice", "InvoiceProcessor", Connection = "ServiceBusReceiverConnection")] string msg,
    IBinder blobBinder,
    [Table("AcmeOrders", Connection = "StorageConnection")] IAsyncCollector<AcmeOrderEntry> tableOutput,
    ILogger logger)
{
...
}

The line [ServiceBusTrigger("workiteminvoice", "InvoiceProcessor", Connection = "ServiceBusReceiverConnection")] string msg, shows just how easy it is, to hook to event triggers generated from Azure Service Bus.

All we need is to use the ServiceBusTrigger attribute to decorate the 1st method parameter.

It takes 3 arguments itself:

Topic Name,
Subscription Name, and
the name of the Configuration property, that stores the ServiceBus Connection string.

If you want more details on this attribute, I would suggest you to go through this article - which explains the Azure Service Trigger in more details.

The next method parameter to our Azure Function is an IBinder.

We will be using this to perform input-binding (i.e., bind to perform read operations) to a particular blob in Azure Blob Storage.

ServiceBusTrigger allowed us to imperatively perform input-binding to Service Bus messages. This works because we know that we will always be looking for messages in workiteminvoice topic, using InvoiceProcessor subscription.

We can't use the corresponding Blob Trigger - because we will know about the blob location at runtime, ONLY after we parse the message received from Service Bus.

For scenarios like these, when we need run time binding we can use the IBinder input parameter.

We will soon see, that once we know the blob location, how we can use BindAsync method on the IBinder instance to perform input binding to the blob.

The last parameter, allows us to perform output binding to Azure Table Storage: [Table("AcmeOrders", Connection = "StorageConnection")] IAsyncCollector<AcmeOrderEntry> tableOutput.

Imperative binding works here, because we know we will always be writing to the table named AcmeOrders, withing the Azure Storage account whose connection string is specified in the configuration property StorageConnection.

Also, since we might be writing (output-ing) multiple entries to this table, we need to use IAsyncCollector<T> as the concrete type. Think of these as out parameters to an async Azure Function.

We will be invoking AddAsync() method on this to write multiple entries to the Azure Table Storage.

The Code

Let us now take a look into the first part of this function:

InvoiceWorkitemMessage invoiceWorkItem;
logger.LogInformation($"C# ServiceBus topic trigger function processed message: {msg}");

if (string.IsNullOrEmpty(msg)) {
    logger.LogError("Empty Invoice Workitem.");
    return;
}

try {
    invoiceWorkItem = JsonSerializer.Deserialize<InvoiceWorkitemMessage>(msg, jsonSerializationOptions);
}
catch (JsonException ex) {
    logger.LogError(ex, "Invalid Invoice Workitem.");
    return;
}
catch (Exception ex) {
    logger.LogError(ex, "Unexpected Error.");
    return;
}

if (invoiceWorkItem.DataLocationType != DataLocationType.AzureBlob) {
    logger.LogError($"Unsupported data location type {invoiceWorkItem.DataLocationType}.");
    return;
}

if (string.IsNullOrEmpty(invoiceWorkItem.DataLocation)) {
    logger.LogError("Empty data location.");
    return;
}

Nothing too fancy here. We are essentially trying to parse the message received from Service Bus as an InvoiceWorkitemMessage.

If the message is valid, and we are able to parse it, we check that the blob location is of type Azure Blob Storage (currently we support ONLY Azure Blob Storage).

Once we know the blob path, we can use the IBinder instance to bind to the specific blob path. As discussed earlier, we will be using the BindAsync method to get a Stream of the blob contents.

var blobAttr = new BlobAttribute(invoiceWorkItem.DataLocation, FileAccess.Read) {
    Connection = "StorageConnection"
};

using (var blobStream = await blobBinder.BindAsync<Stream>(blobAttr)) {
   //...
}

We will then parse the contents of this blob, using InvoiceParser class instance.

if (!InvoiceParser.TryParse(blobStream, logger, out List<AcmeOrder> orders)) {
    logger.LogError("Failed to parse invoice.");
    return;
}

I will skip the inner behavior of the TryParse method, as it doesn't really add to our scope of discussion here.
You can look into the InvoiceParser.cs file for more details.

If everything goes fine, and we are able to parse the contents of the blob as one or more AcmeOrders, we can persist them in Azure Table Storage.
As discussed earlier, we use AddAsync() method on the IAsyncCollector<AcmeOrderEntry> to achieve this.

foreach (var order in orders) {
    await tableOutput.AddAsync(new AcmeOrderEntry(invoiceWorkItem.UserEmail, order));
    logger.LogInformation($"Added table record for BigBasket Order number: {order.OrderNumber}");
}

To perform TDD with this sample application, we need to write Unit Tests and Integration Tests against this code.

In the next post, we will discuss how we can implement Unit Tests for this Azure Function. See you there!

How to setup a MongoDB Replica set for development using Docker

Santanu Paul — Mon, 15 Feb 2021 15:41:19 +0000

In this post I will be sharing how I configure and run a MongoDB replica set on my local machine, using Docker.

While there are ample examples on the internet about running MongoDB containers, not many of them focus on how to bring up a MongoDB replica set.

Doing development against single node MongoDB is fine for starting, but as one starts using features like transactions, it becomes imperative that the MongoDB deployment has replica set enabled (or sharded cluster, a topic for future discussion).

Creating and developing against a MongoDB replica set in local development setup helps you learn the topic, and be more confident that your application code is already adapted to run against replica sets in production.

To follow along, you will need to have Docker, and Docker Compose installed in your machine.

To install Docker, I suggest following the official documentation at this link.

Once Docker is installed, install Docker Compose by following the links from this link.

I have created this Github repository, which contains all the required artifacts to follow along with this tutorial.

The Docker Compose file

Once Docker and Docker Compose are both installed on your machine, we can start writing the Docker Compose file, that contains directives to bring up the MongoDB replica set. I will show the full docker-compose.yml file first, and then explain the individual details.

version: '3.9'  # Docker Engine release 19.03.0+ [https://docs.docker.com/compose/compose-file/]

services: 

    # setup MongoDB cluster for production
    mongo-replica-setup:
        container_name: mongo-setup
        image: 'mongo:4.2'
        restart: on-failure
        networks:
            - netApplication
        volumes:
        - ./.docker/mongodb/scripts/mongosetup.sh:/scripts/mongosetup.sh
        entrypoint: ["bash", "/scripts/mongosetup.sh" ]
        env_file:
            - .env
        environment:
            MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
            MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
        depends_on:
        - mongo1
        - mongo2
        - mongo3

    mongo1:
        hostname: 'mongo1'
        container_name: 'mongo1'
        image: 'mongo:4.2'
        restart: 'on-failure'
        command: ["-f", "/etc/mongod.conf", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
        expose: 
            - 27017
        ports: 
            - 30001:27017 
        networks: 
            - netApplication
        volumes:
            - dataMongo1:/data/db
            - logMongo1:/var/log/mongodb
            - ./.docker/mongodb/initdb.d/:/docker-entrypoint-initdb.d/
            - ./.docker/mongodb/mongod.conf:/etc/mongod.conf
            - ./.docker/mongodb/file.key:/auth/file.key
        healthcheck:
            test: test $$(echo "rs.status().ok" | mongo -u $${MONGO_INITDB_ROOT_USERNAME} -p $${MONGO_INITDB_ROOT_PASSWORD} --quiet) -eq 1
            interval: 30s
            start_period: 60s
        env_file:
            - .env
        environment:
            MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
            MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
            MONGO_INITDB_DATABASE: ${MONGO_INITDB_DATABASE}

    mongo2:
        hostname: 'mongo2'
        container_name: 'mongo2'
        image: 'mongo:4.2'
        command: ["-f", "/etc/mongod.conf", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
        restart: 'on-failure'
        expose: 
            - 27017
        ports: 
            - 30002:27017  
        networks: 
            - netApplication
        volumes:
            - dataMongo2:/data/db
            - logMongo2:/var/log/mongodb
            - ./.docker/mongodb/mongod.conf:/etc/mongod.conf
            - ./.docker/mongodb/file.key:/auth/file.key
        env_file:
            - .env
        environment:
            MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
            MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
            MONGO_INITDB_DATABASE: ${MONGO_INITDB_DATABASE}
        depends_on: 
            - mongo1

    mongo3:
        hostname: 'mongo3'
        container_name: 'mongo3'
        image: 'mongo:4.2'
        command: ["-f", "/etc/mongod.conf", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
        restart: 'on-failure'
        expose: 
            - 27017
        ports: 
            - 30003:27017  
        networks: 
            - netApplication
        volumes:
            - dataMongo3:/data/db
            - logMongo3:/var/log/mongodb
            - ./.docker/mongodb/mongod.conf:/etc/mongod.conf
            - ./.docker/mongodb/file.key:/auth/file.key
        env_file:
            - .env
        environment:
            MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
            MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
            MONGO_INITDB_DATABASE: ${MONGO_INITDB_DATABASE}
        depends_on: 
            - mongo1

volumes: 
    dataMongo1:
    dataMongo2:
    dataMongo3:
    logMongo1:
    logMongo2:
    logMongo3:

networks: 
    netApplication:

The Setup

Here we are trying to setup a MongoDB replica set with 3 nodes.

One of these nodes will be the Primary node - all writes to the database will happen via this node.

The remaining two, are Secondary nodes, providing two-levels of data replication in the setup.

A solid explanation of how data replication works in MongoDB can be found at this link.

Exploring the docker-compose file

Coming back to the yaml file, let's go through the service declaration of mongo1 and see what we are trying to achieve.

The basic stuff

mongo1:
    hostname: 'mongo1'
    container_name: 'mongo1'
    image: 'mongo:4.2'
    restart: 'on-failure'
    :
    expose: 
        - 27017
    ports: 
        - 30001:27017 
    networks: 
        - netApplication

Nothing fancy here, we are targetting MongoDB server version 4.2 to run these containers.

We have declared hostname and container name, and set the container restart policy to on-failure.

On the ports front, we have exposed 27017 to other docker containers running on the same docker network. Also, this port is mapped to host port 30001.

Lastly, we will be running our replica set on a named network.

Starting the mongod instance

command: ["-f", "/etc/mongod.conf", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]

This will start the mongod service with the following:

-f /etc/mongod.conf
- specifies that the runtime configuration options should be picked up from the provided mongod.conf file.
- we will see the contents of this file later
-keyFile /auth/file.key
- In our replica set, the nodes will use the contents of a shared keyfile to authenticate to each other.
- Here we are specifying the path to the keyfile
--replSet ${MONGO_REPLICA_SET_NAME}
- This configures the node to run in a replica set.
- We are also specifying the name of the replica set, which is to be picked up from the environment variable ${MONGO_REPLICA_SET_NAME}
--bind_ip_all
- This specifies the mongod instance to bind to all IPv4 addresses (0.0.0.0).

Container volumes

Before proceeding it is worth mentioning again, that I have created this Github repository to house all the artifacts mentioned here.

volumes:
    - mongoData1:/data/db
    - mongoLog1:/var/log/mongodb
    - ./.docker/mongodb/initdb.d/:/docker-entrypoint-initdb.d/
    - ./.docker/mongodb/mongod.conf:/etc/mongod.conf
    - ./.docker/mongodb/file.key:/auth/file.key

Here we have setup a few Docker managed volumes to store the container's data in host's file system.

mongoData1: This will act as the peristent store for MongoDB's data.
mongoLog1: All of MongoDB's logs for this container will be stored here.
./.docker/mongodb/initdb.d/
- When mongod container starts for the very first time, it will look for a directory named /docker-entrypoint-initdb.d, and execute all files with extensions .sh and .js.
- We map this directory to ./.docker/mongodb/initdb.d/ in host.
- As we will see later, we will use this directory to store a bash script that allows us to create a user.
./.docker/mongodb/mongod.conf
- We are essentially providing the path (in host) from where MongoDB will pick up its config file.
./.docker/mongodb/file.key
- Like above volume, here we are specifying the keyfile that this MongoDB node will be using to authenticate itself to other nodes in the replica set.
- Steps to create this keyfile is provided in the Github repository that I have created.

env_file:
    - .env
environment:
    MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
    MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
    MONGO_INITDB_DATABASE: ${MONGO_INITDB_DATABASE}

Here we have specified that the docker compose will be using the .env file to read environment variables and inject them into the container.

This allows us to specify that the database's root username/password and initial database name to be picked up from specific environment variables.

In my github repository, I have provided a sample env file that you can use to create your own .env file.

Contents of the sample env file looks like this:

# MongoDB
MONGO_URL=mongodb://mongodb:27017
MONGO_INITDB_ROOT_USERNAME=<root_username>
MONGO_INITDB_ROOT_PASSWORD=<root_password>
MONGO_INITDB_DATABASE=<app_db-name>
MONGO_INITDB_USERNAME=<app_username>
MONGO_INITDB_PASSWORD=<app_password>
MONGO_REPLICA_SET_NAME=rs0

One must make sure not to commit this file in github, as these contains secrests like root username/password.

The above image provides a tree view of the files and directories discussed here.

Health Check

healthcheck:
    test: test $$(echo "rs.status().ok" | mongo -u $${MONGO_INITDB_ROOT_USERNAME} -p $${MONGO_INITDB_ROOT_PASSWORD} --quiet) -eq 1
    interval: 30s
    start_period: 60s

Here we are specifying the healthcheck command that docker will use to check if the container is up and running.

To summarize, the mongo1 service's specification looks like this:

mongo1:
    hostname: 'mongo1'
    container_name: 'mongo1'
    image: 'mongo:4.2'
    restart: 'on-failure'
    command: ["-f", "/etc/mongod.conf", "--keyFile", "/auth/file.key", "--replSet", "${MONGO_REPLICA_SET_NAME}", "--bind_ip_all"]
    expose: 
        - 27017
    ports: 
        - 30001:27017 
    networks: 
        - netApplication
    volumes:
        - mongoData1:/data/db
        - mongoLog1:/var/log/mongodb
        - ./.docker/mongodb/initdb.d/:/docker-entrypoint-initdb.d/
        - ./.docker/mongodb/mongod.conf:/etc/mongod.conf
        - ./.docker/mongodb/file.key:/auth/file.key
    healthcheck:
        test: test $$(echo "rs.status().ok" | mongo -u $${MONGO_INITDB_ROOT_USERNAME} -p $${MONGO_INITDB_ROOT_PASSWORD} --quiet) -eq 1
        interval: 30s
        start_period: 60s
    env_file:
        - .env
    environment:
        MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
        MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
        MONGO_INITDB_DATABASE: ${MONGO_INITDB_DATABASE}

The other two nodes - mongo1 and mongo2 is a subset of this specification. The only things missing are the healthcheck section, and a volume that maps to docker-entrypoint-initdb.d. This is because we have designed this setup such that mongo1 will become the primary and the rest secondary.

Being secondary nodes, they dont need the initdb part to work.

Create the Mongo Cluster

Once all the nodes in the replica set is up, we will need to initiate Replica set configuration on these nodes.

We will be using a 4th container - mongo-replica-setup to do this.

mongo-replica-setup:
    container_name: mongo-setup
    image: 'mongo:4.2'
    restart: on-failure
    networks:
        - netApplication
    volumes:
    - ./.docker/mongodb/scripts/mongosetup.sh:/scripts/mongosetup.sh
    entrypoint: ["bash", "/scripts/mongosetup.sh" ]
    env_file:
        - .env
    environment:
        MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
        MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
    depends_on:
        - mongo1
        - mongo2
        - mongo3

Once this container starts, it will connect to mongo1, and execute a script that will initiate a replica set over mongo1, mongo2 and mongo3.

Once the script is executed this container exits.

Its important to note that we have set the depends_on field of this container to mongo1, mongo2 and mongo3, which ensures that this container will start only after all those 3 containers are up. This is important, as you wont want to run the script without all the nodes of the replica set being up.

The following is the script that this container runs, to initiate the replica set:

#!/bin/bash

MONGODB1=mongo1
MONGODB2=mongo2
MONGODB3=mongo3

echo "**********************************************" ${MONGODB1}
echo "Waiting for startup.."
sleep 30
echo "done"

echo SETUP.sh time now: `date +"%T" `
mongo --host ${MONGODB1}:27017 -u ${MONGO_INITDB_ROOT_USERNAME} -p ${MONGO_INITDB_ROOT_PASSWORD} <<EOF
var cfg = {
    "_id": "rs0",
    "protocolVersion": 1,
    "version": 1,
    "members": [
        {
            "_id": 0,
            "host": "${MONGODB1}:27017",
            "priority": 2
        },
        {
            "_id": 1,
            "host": "${MONGODB2}:27017",
            "priority": 0
        },
        {
            "_id": 2,
            "host": "${MONGODB3}:27017",
            "priority": 0,
        }
    ]
};
rs.initiate(cfg, { force: true });
rs.secondaryOk();
db.getMongo().setReadPref('primary');
rs.status();
EOF

If you look closely, we have set the priority for mongo1 higher than the other two nodes, which ensures that initially, mongo1 will be acting as the Primary node.

Final Result

To start the replica set, run docker-compose up -d from command line.

Once everything comes up, you can run docker-compose status, and see something like this:

To connect to the replica set, you can use mongo client like so:

$ mongo "mongodb://localhost:30001,localhost:30002,localhost:30003/<MONGO_INITDB_DATABASE>" -u <MONGO_INITDB_USERNAME>