Forem: Andrea Grillo

Sending an alert in Microsoft Teams from Azure DevOps Pipelines

Andrea Grillo — Wed, 01 Feb 2023 22:54:52 +0000

Sometimes there is the need to warn your team about an on going deployment to the cloud. And if you are using Microsoft Teams as primary communication platform, you are likely to have a private channel with your team. This is my case, where anyone is going to run a release pipeline warns the rest of the team about it.
Since manually sending messages is annoying, we automated this operation by using Teams webhooks - without the help of Graph APIs 🐱‍👤.

Setup a channel webhook

In order to start, you need to set up a webhook for the channel of where you would like to send messages. So select your channel and then click the 3 dots in the upper right corner; open the Connectors menu and search for Incoming Webhook. Once found, click on Configure:

Here, set the name, optionally upload an image (it will be the avatar next to the channel posts) and finally click on Create.

An URL should be generated: copy and note it, you will need it later on.

Setup the pipeline

Select your pipeline and a task to run a bash script. Here write down the following code:

set -x
umask 0002

cat > ./post.json <<endmsg
      {
        "summary": "$System.DefinitionName is being deployed in $Release.EnvironmentName",
        "text": "[$System.DefinitionName]($Release.ReleaseWebUrl) ($BRANCH_NAME) is being deployed in $Build.SourceBranchName by $Release.RequestedFor"
      }
endmsg

curl -X POST -H "Content-Type: application/json" -d @post.json <url-you-copied-from-teams> -v

This script is simply creating a json file called post.json with some content supported by Teams. Then, a POST request is sent to the endpoint of the webhook we previously configured. You should now see the message on Teams:

In the example above, the variables System.DefinitionName, Release.EnvironmentName, Release.ReleaseWebUrl, Build.SourceBranchName and Release.RequestedFor are automatically provided by Azure DevOps in a release pipeline and they stand for:

the pipeline name
the pipeline stage name
a permanent link to the current pipeline run
the artifact branch name
the user who triggered the pipeline

Conclusions

As shown, sending a message to a Teams channel is really easy and you don't need to integrate Graph APIs unless you care about some advanced features. However, implementing Graph APIs in a pipeline is not a 5 minute job like this one, so find your tradeoff between utility and effort.

This tutorial sends a simple message as an example, but messages are customizable and also interactive as described in Microsoft documentation.

Even if this procedure is quite straightforward, for me it took few hours since at the time, it wasn't documented at all.

For this reason, I hope this has been helpful.

Happy coding and happy releases! 🐱‍👤

Implementing health checks PT.2 - Azure Application Insights configuration

Andrea Grillo — Sun, 03 Jul 2022 20:27:28 +0000

This is the second post in a series dedicated to health checks.

Implementing health checks PT.1 - Asp.Net Core 6 configuration
Implementing health checks PT.2 - Azure Application Insights configuration (this post)

Once the application is deployied, we may want to use some tool that periodically ping these endpoints and send us a notification when something is not properly working.

The tool I use most is Azure Application Insights.

Application Insights

Azure Application Insights is a wonderful service provided by Azure to monitor live apps. It provides tons of features, such as logs search and health checks ping as shown later, and it supports lot of languages - not just dotnet.

From official documentation:

Application Insights is a feature of Azure Monitor that provides extensible application performance management (APM) and monitoring for live web apps. Developers and DevOps professionals can use Application Insights to:

Automatically detect performance anomalies.

Help diagnose issues by using powerful analytics tools.

See what users actually do with apps.

Help continuously improve app performance and usability.

Application Insights:

Supports a wide variety of platforms, including .NET, Node.js, Java, and Python.

Works for apps hosted on-premises, hybrid, or on any public cloud.

Integrates with DevOps processes.

Has connection points to many development tools.

Can monitor and analyze telemetry from mobile apps by integrating with Visual Studio App Center.

Configuring availability checks in Application Insights (AI)

To get started, create a new AI instance in your Azure subscription and navigate to the overview page:

In particular, there are three charts:

Failed requests: shows the amount of requests that the application failed to process (4xx/5xx status codes)
Server response time: average server response time
Server requests: amount of requests per minute
Availability: the server availability over time

Of course, to have some data from the first three charts you need to link this AI resource to an another Azure resource such as Azure WebApp, Azure Functions or Azure Kubernetes Service.

Instead, the availability data is obtained from the ping requests that AI does to an endpoint of your choice.

To setup them up, navigate to Availability section and click "Add Standard (preview) test".

NB: the Standard test is still in preview but since it is only a ping service, I think it is safe enough to be used in production environments. It is not that different than the classic test but provides some additional options

Setup a name for the test, such as MyWebAppPingTest and the url to ping (in the previous post I did set health checks on routes like /alive and dbcontext, so I put <hostname>/alive or <hostname>/dbcontext).

Then there are several options that can be checked to enhance the accuracy of the test. Each of them is properly documentated by going hover the i icon with the mouse, so I don't go deep for each of them. However, I want to focus on some area:

you can set the HTTP Rest verb and eventually custom headers: it is useful when your resource is behind a gateway such as Azure API Management or you want to add some protection to your APIs.
it is possible to set some custom successfull criteria by checking response status code, response body and other details that give you enough flexibility to play with different scenarios of your app
the last section is about alerts, that can be configured only after the ping test creation. Enable it, we will use it in the next chapters

Click create and wait for the deploy.

Looking at availability results

After a while, data will start to be populated and a line in the chart is drawn:

Clicking on a category result on the right hand side (such as Successful or Failed), a new pane opens showing the list of the ping requests in the selected category. By clicking on one of them, the trace of that request is shown:

Going back to the Availability section, you can set alerting by selecting your ping test and clicking on the three dots menu Open Rules (Alerts) page:

Here you are basically in Azure Monitor and you see the list of alert rules in your subscriptions. Selecting the previously created alert, a new page where you can customize alerting rules opens:

Conclusions

These two posts explain how to write health checks in your dotnet application using the dedicated SDK provided by Microsoft. Then, the second post shows the Azure Application Insights setup to ping and consume these endpoints in a real-like environment, with alerting support.

Photo by nikko macaspac on Unsplash

Implementing health checks PT.1 - Asp.Net Core 6 configuration

Andrea Grillo — Mon, 30 May 2022 21:58:51 +0000

Implementing health checks PT.1 - Asp.Net Core 6 configuration (this post)
Implementing health checks PT.2 - Azure Application Insights configuration

When a production running application is not available to the customers due to technical issues, we are not only losing money but also losing trust and faith in our customers. In fact, they may spread a bad word about our product on social media, friends, store reviews or - even worst - move to competitors.

Basically, when an application is not available to the stakeholders, it is as if it didn't exist.

Of course it may happen. It might because of a human mistake, an erroneous configuration or a bug. Or maybe because our cloud provider has some connectivity issues.

Monitoring the application availability it also allows you to elaborate some metrics to define SLA of your service.

For these reasons, we must know immediately when application goes offline.

One of the most used and simplest techniques are health checks endpoint.
An health check endpoint, is an API exposed by an application that has no logic except for replying with a positive status code; if the caller doesn't receive any response or the status code is not the one expected, it means there is an error and an alert must be sent using the technology of your choice (email, push notification.. SMS and so on).

Of course, Asp.Net Core provides a built-in mechanism to implement an health check endpoint.

Implementing a simple health check endpoint

Create an Asp.Net Core project. For this demo (you can find the source code at the end of this post) I am going to use dotnet 6 with the new minimal host startup template.

Install the NuGet package Microsoft.Extensions.Diagnostics.HealthChecks and configurate it:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHealthChecks();

var app = builder.Build();

app.MapHealthChecks("/alive");

app.Run();

Run the application and navigate to localhost:<port>/alive and you should see something like this:

You if you try to call the same endpoint with Postman or a similar tool, you see the status code is 200 OK.

And that's it - the health check endpoint is ready to be regularly consumed by a watch dog application 🎉

Advanced scenario

It might be a scenario where the application is online but it can't communicate with the database for some reason. Potentially it is a big disservice to our customers and it is likely that users cannot perform almost all of the actions.

Entity Framework Health Checks

Also Entity Framework provides a built-in mechanism to monitor the connectivity between an application and the SQL instance database.

NB: to go ahead, of course you must have Entity Framework configured and pointing to a running SQL database

Install the package Microsoft.Extensions.Diagnostics.HealthChecks.EntityFrameworkCore from NuGet and configure it:

services.
AddHealthChecks()
.AddDbContextCheck<MyDbContext>("/dbcontext");

Running the application and pointing again to localhost:<port>/alive you should have the same result:

By the default, under the hood it checks for:

The DbContextHealthCheck calls EF Core's CanConnectAsync method. You can customize what operation is run when checking health using AddDbContextCheck method overloads.
The name of the health check is the name of the TContext type.

It now works but there is still a problem: no matter what health check endpoint we hit (/alive or /dbcontext), both checks are performed so exposing two different APIs is useless.

How is it possible to set different endpoints for different checks? The answer is, through health checks names.

Change the endpoints registration as follow:

const string ALIVE = "alive";
const string ALIVE_DBCONTEXT = "dbcontext";

...
builder.Services.AddHealthChecks().AddDbContextCheck<MyDbContext>(name: ALIVE_DBCONTEXT);

app.MapHealthChecks(
    "/alive",
    new HealthCheckOptions { Predicate = (c) => c.Name == ALIVE });

app.MapHealthChecks(
    "/dbcontext",
    new HealthCheckOptions { Predicate = (c) => c.Name == ALIVE_DBCONTEXT });

In this case, we are giving a name to a specific health check endpoint. When the name is matched, only the specific health check is executed.

To have a confirmation, we can do a test:

run the application and the database in a local Docker container
navigate to localhost:<port>/alive: you should get Healthy as result
stop the database container and refresh the page: you should still get Healthy as result
navigate to localhost:<port>/dbcontext: you should get Unhealthy as result
restart the database container and refresh the page: you should get Healthy as result

Custom checks: Azure Cosmos DB

Asp.Net allows to define some custom code while performing an health check. This is the case where we can test our connection to an Azure Cosmos DB instance, because the .NET SDK doesn't provide any mechanism to check the connection between the application and the database service.

The following example tests the connection using container.ReadContainerAsync(), a method exposed by the Cosmos SDK that needs database access to be executed.

// Cosmos initialization

builder.Services
    .AddHealthChecks()
    .AddDbContextCheck<MyDbContext>(name: ALIVE_DBCONTEXT)
    .AddCheck<CosmosHealthChecker>(name: ALIVE_COSMOS);

...

app.MapHealthChecks(
    "/cosmos",
    new HealthCheckOptions { Predicate = (c) => c.Name == ALIVE_COSMOS });

public class CosmosHealthChecker : IHealthCheck
{
    private readonly IServiceScopeFactory _serviceScopeFactory;
    private readonly IConfiguration _configuration;
    private readonly ILogger<CosmosHealthChecker> _logger;

    public CosmosHealthChecker(
        IServiceScopeFactory scopeFactory,
        IConfiguration configuration,
        ILogger<CosmosHealthChecker> logger)
    {
        _serviceScopeFactory = scopeFactory;
        _configuration = configuration;
        _logger = logger;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default)
    {
        try
        {
            string collectionName = _configuration["Cosmos:Collection"];

            using IServiceScope scope = _serviceScopeFactory.CreateScope();
            Database db = scope.ServiceProvider.GetRequiredService<Database>();
            Container container = db.GetContainer(collectionName);

            await container.ReadContainerAsync(cancellationToken: cancellationToken);
            return new HealthCheckResult(HealthStatus.Healthy);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Error pinging Cosmos db");

            return new HealthCheckResult(
                context.Registration.FailureStatus, "An unhealthy result.");
        }
    }
}

Conclusions

This post explains how we can provide some health check endpoints in an Asp.Net project with just few lines of code 😎

As usual the source code is available in my GitHub profile.

Photo by Bruno Nascimento on Unsplash

Public Service Announcement: I've joined the SRE team

Andrea Grillo — Sun, 13 Feb 2022 11:53:12 +0000

Cover image by Brianna Santellan on Unsplash

Since I joined my current company 9 months ago, I have always looked at the SRE team with interest. They look to be experts in everything: technology stack, application domain knowledge, software architecture, the ability to handle hot and risky tasks, DevOps techniques, cloud services and infrastructure. They literally looked so brilliant.

Since then, I simply wanted to become one of them. So I thought that I had to work hard to fill the knowledge gap between me and them and then ask to join the team one day. I never explicity asked for a team change but few - and apparently right - people knew I would like it.

If you don't known what an SRE is, it stands for Site Reliability Engineer. It is a software engineering approach invented by Google in the 00s but became public knowledge only in the last years. Lot of companies like Amazon and Netflix adopted this technique. It is like a DevOps subset approach, but teams members have a software development background and they can fill the gap between devs and operation teams. SREs have to make sure the system runs smoothly. Resources provisioning, infrastructure, bugs, check and fix of production data, automation of ripetitive tasks in order to remove the toil is what they do everyday.

In an afternoon about a month ago, my boss called me on Teams and I thought the reason was a task I was working on. Instead, he asked me if it was true that I was interested in joining the SRE team, because they probably needed more people. The question was unofficial because everything needed to be sorted out yet but obviously I gave the green light on my candidature. At that moment I didn't realize immediatly what was going on and I felt completely normal. After the call I started to understand what happened and I was so excited about the news! A couple of week later I received another call from my boss and he confirmed the move. Deal was done!

For the first time in my work career, I can finally say that I am not a software developer anymore, but an SRE! 🎉

Enforcing a consistent code style and quality in your team-wide .NET projects

Andrea Grillo — Thu, 06 Jan 2022 14:18:54 +0000

Each developer has his/her own coding style, usually inherited from personal preferences, habits and years of experience. Just think of naming conventions, indentation, spaces, curly braces, blank lines, etc.: there are many elements that can be typed in various ways, and none of them are wrong but just different.

Especially in big teams, mixing different styles has some consequences:

codebase is not coherent due to lot of discrepancies in code style
hundreds of compiler warnings/suggestions may hide the important ones
new developers don't know what style they shall use to write code in early days
PR reviews may become a fight between people with different point of views
reading the code could be difficult - just think at the variable naming convention that should help the reader to understand the variables scope

Then, the goal is creating a team style rather than a personal style. Moreover, keeping the solution free of warnings is definitely a good move.

I understand that enforcing code style rules in a team is important. But how can we achieve that?

Just few years ago, Microsoft added the .editorconfig file support to Visual Studio. This kind of file defines a set rules that override the local Visual Studio settings. So you can work on multiple projects owned by different teams with their own rules without having to change every time your local Visual Studio settings.

And above all, the whole team has the same settings.

Last but not least, it is possible to define rules about code quality as well. To give you an idea, it is possible to throw a compiler error if an IDisposable object is not properly disposed, or when an async method is not awaited.

NOTE: Visual Studio Code needs this extension to work with .editorconfig files.

Starting with .editorconfig files is quite simple: create a file named .editorconfig. and put it in the solution folder; then add it to the Visual Studio solution by right clicking on the solution file -> Add -> Existing Item:

Visual Studio displays an UI when you try to open the .editorconfig file, but personally I don't like it. I think it is way quicker to manually edit it using VS Code.

As covered later in this post, .editorconfig files support inheritance, so the first thing to do is to set the nearly created file as the top most using the root property:

root = true

Then, you can set any rule you would like to set. Personally, I start with the indentation rules for any file in the solution, and then I go deeper with more specific rules for each file extension and namespace. For example:

[*] # Any file uses space rather than tabs
indent_style = space

# Code files
[*.{cs,csx,vb,vbx}] # these files use 4 spaces to indent the code
indent_size = 4
insert_final_newline = true
charset = utf-8-bom

# XML project files
[*.{csproj,vbproj,vcxproj,vcxproj.filters,proj,projitems,shproj}]
indent_size = 2

# XML config files
[*.{props,targets,ruleset,config,nuspec,resx,vsixmanifest,vsct}]
indent_size = 2

# JSON files
[*.json]
indent_size = 2

# Powershell files
[*.ps1]
indent_size = 2

# Shell script files
[*.sh]
end_of_line = lf
indent_size = 2

# Dotnet code style settings:
[*.{cs,vb}]

You can set hundreds of rules, of any type. On Microsoft Docs there is an (almost) complete list of all the available rules that you can try locally and find the balance that suits best for your needs.
Each rule can have a specific severity: you can treat warnings as errors or demote them to suggestions, or even disable them. Be careful, the most bottom rule has the precedence over the rest.

By now, Visual Studio is reading the .editorconfig file and changing the local settings but as you may noticed, if you set some warning-default rule to error and try to build the project, you realized that the compiler works even if Visual Studio is showing the rule broke. This happens because you need to setup MSBuild by simply adding these two properties to each of your .csproj (or the Directory.Build.props parent file):

<PropertyGroup>
    <TargetFramework>...</TargetFramework>
    **<EnableNETAnalyzers>true</EnableNETAnalyzers>
    <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>**
</PropertyGroup>

_NOTE: By default, Visual Studio analyses open documents only. If you wish to scan the entire solution go in Tools -> Options -> Text Editor -> C# (or your favorite language) -> Advanced -> Set Background Analysis Scope to Entire Solution.

Namespace rules

Sometimes you want to define some rule (or exception) for a specific project or namespace. For example, I want to enforce the suffix Async to any asynchronous method in the solution but for unit tests and controllers.
Luckily .editorconfig files allow you to achieve a solution quite straightforward. Basically, you have two ways:

defining exceptions in the root .editorconfig file
use inheritance writing others files

Define exceptions in the root file

At the bottom of the .editorconfig, define the namespace of where you would like to apply the exception. For example:

[src/**{.UnitTests,/Controllers}/**.cs]
# Async methods should have "Async" suffix
dotnet_naming_rule.async_methods_end_in_async.severity = none

Using inheritance

Inheritance is a good alternative. Do you remember the root property we set at the beginning? That means that the given .editorconfig is applied to the whole solution. But however it is possible to create others .editorconfig files in subfolders that will inherit all the rules from the root file. Rules defined here takes the precedence. Getting back the previous example, you can navigate to the Controllers/UnitTests folders and create a new file. Do NOT define the root=true property and add the rule:

# Async methods should have "Async" suffix
dotnet_naming_rule.async_methods_end_in_async.severity = none

Shall I manually change any code formatting error?

This is a tricky question with several solutions.

The first solution is to use the built-in Visual Studio feature - Code clean up. Personally I don't like it for the following reasons:

it has a specific shortcut (CTRL+K, CTRL+E) that developers may forget to use every time they edit a file (we work under pressure sometimes!)
it's annoying
it's a local setting that must be configured and there is no way to share it with your team.

For these reasons, I had been using Productivity Power Tools, an (historical) extensions developed by Microsoft. It provides a feature to format the entire file when it is saved (through UI or the shortcut CTRL+S). It makes simply impossible to forget to format your file before committing it in source control.

But, unfortunately Microsoft decided to remove this feature from the PPT 2022 version - apparently without any reason, there are plenty of negative feedbacks for this.

Luckily, a developer named Elders has published an extension called Format document on Save and it does simply what its name says: it formats the document when the user save it!

I want to sort and remove unnecessary using statement in my project, but I have to do it manually even if I set the rule. Any way to automate it?

Yes, you can do it!
Going in Tools -> Options -> Format Document on Save you can set the commands to be executed when user saves the file.
In order to format the file and then sort and removing using statements, set it as follow:

Good, but any team member shall remember to set this feature on?

Nope! This extension is able to read a file named .formatconfig that you can check in your source control and share with your team. It doesn't need to be added to the Visual Studio solution, but just to stay in the same level of the solution file. Local user preferences are ignored.

Here's an example of definition:

root = true

[*.*]
enable = true
enable_in_debug = true
command = Edit.FormatDocument Edit.RemoveAndSort
allowed_extensions = .*
denied_extensions =

Anything familiar? 🐱‍💻

Working example

Last thing to do is leaving you a link to a repository where you can find a fully working solution and especially an .editorconfig file with lot of rules defined. You can get the remaining ones on Microsoft Docs.

GitHub

Enjoy! 🎉

UPDATE: Looking at the Visual Studio 2022 development roadmap, the next preview (17.2) includes a synchronization between the .editorconfig file and the code clean up profile. This is really exciting!

Hide sensitive data in Azure Application Insights logs of our ASP.NET (Core) APIs

Andrea Grillo — Fri, 24 Dec 2021 08:36:11 +0000

Azure Application Insights is a wonderful tool to log data and errors in your (not only) .NET applications.

But the Microsoft SDKs don't solve an issue I faced in one of the project I worked on: would it be possible to hide sensitive data from logs? Nobody should be able to read user sensitive data, especially from logs.

How we used to log APIs payloads

We have various log sources in our distributed system and one of them is the APIM (Azure Api Management): with a simple checkbox you can log APIs requests and responses bodies. But since it is a simple checkbox, you can't do anything more than that.
So, we moved our logging system out of it.

Where we log APIs payloads now

The easiest way to do log in front of all the APIs in an ASP.NET Core project, is to add a FilterAttribute class to the ASP.NET action invocation pipeline:


 csharp
// Startup.cs
protected override void ConfigureMvc(MvcOptions options)
{    
    options.Filters.Add<ApplicationInsightsActionFilterAttribute>();    
}

// ApplicationInsightsActionFilterAttribute.cs
public class ApplicationInsightsActionFilterAttribute : ActionFilterAttribute
{
    internal const string REQUEST_TELEMETRY_KEY = "Request-Body";
    internal const string RESPONSE_TELEMETRY_KEY = "Response-Body";

    private readonly ILogger<ApplicationInsightsActionFilterAttribute> _logger;

    public ApplicationInsightsActionFilterAttribute(
        ILogger<ApplicationInsightsActionFilterAttribute> logger)
    {
        _logger = logger;
    }

    public override async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
    {
        if (context.HttpContext is null)
        {
            await next();
            return;
        }

        try
        {
            string fromBodyParameter = context.ActionDescriptor.Parameters
                .Where(item => item.BindingInfo.BindingSource == BindingSource.Body)
                .FirstOrDefault()
                ?.Name.ToUpperInvariant();

            if (string.IsNullOrWhiteSpace(fromBodyParameter))
                return;

            KeyValuePair<string, object> arguments = context.ActionArguments
                .First(x => x.Key.ToUpperInvariant() == fromBodyParameter);

            string argument = SerializeDataUsingObfuscator(arguments.Value);

            RequestTelemetry request = context.HttpContext.Features.Get<RequestTelemetry>();
            request.Properties.Add(REQUEST_TELEMETRY_KEY, argument);
        }
        catch (Exception ex)
        {
            _logger.LogCritical(ex, "Error executing the response logger task");
        }
        finally
        {
            await next();
        }
    }

    public override async Task OnResultExecutionAsync(ResultExecutingContext context, ResultExecutionDelegate next)
    {
        if (context?.HttpContext is null)
        {
            await next();
            return;
        }

        try
        {
            if (context.Result is ObjectResult result &&
                result.Value != null)
            {
                string serializedObj = SerializeDataUsingObfuscator(result.Value);
                RequestTelemetry request = context.HttpContext.Features.Get<RequestTelemetry>();
                request.Properties.Add(RESPONSE_TELEMETRY_KEY, serializedObj);
            };
        }
        catch (Exception ex)
        {
            _logger.LogCritical(ex, "Error executing the response logger task");
        }
        finally
        {
            await next();
        }
    }

    private static string SerializeDataUsingObfuscator(object value) =>
        JsonConvert.SerializeObject(value,
            new JsonSerializerSettings
            {
                ContractResolver = new ObfuscatorContractResolver()
            });
}

The code above is defining two methods: OnActionExecutionAsync and OnResultExecutionAsync. The first one is fired before the incoming request triggers the controller action; the latter is fired when the controller action returns. Let's focus on OnActionExecutionAsync for a moment: firstly it gets the controller action parameters that have a body, if any - you can change that code to get the query strings, paths, or whatever you'd like - and then calls the method SerializeDataUsingObfuscator passing the CLR object that represents the request body. To better explain this concept, consider the following API:


 csharp
public async Task<ActionResult> MyWonderfulApi([FromBody] RequestDto dto)
{
    // code
    return Ok();
}

The object consumed by the SerializeDataUsingObfuscator method is of type RequestDto and it contains the incoming data.

As you can see, we are using a custom JSON.NET ContractResolver implementation, called ObfuscatorContractResolver. This class serializes the request body following our obfuscation rules - that we are going to define in a minute.

The Sensitive attribute

In order to define which properties must be obfuscated in ApplicationInsights logs, we defined a custom attribute named Sensitive. Moreover, this attribute allows some customization to obfuscate properties using different patterns - ie. truncate the string, show the first/last N chars, etc. Here's the definition and an usage example:



// WonderfulDto.cs
public class WonderfulDto
{
    [Sensitive(ObfuscationType.MaskChar)]
    [JsonProperty("pin")]
    public string Pin { get; set; }
}

// SensitiveAttribute.cs using the Factory pattern to separate responsibilities
[AttributeUsage(
    AttributeTargets.Property
    , AllowMultiple = false)]
public class SensitiveAttribute : Attribute
{
    private SensitiveBase Sensitive { get; }
    public int? TruncateAfter { get; }

    public SensitiveAttribute(
        ObfuscationType obfuscationType,
        int? truncateAfter)
    {
        TruncateAfter = truncateAfter;
        Sensitive = GetSensitive(obfuscationType);
    }

    public SensitiveAttribute(
        ObfuscationType obfuscationType)
    {
        TruncateAfter = null;
        Sensitive = GetSensitive(obfuscationType);
    }

    private SensitiveBase GetSensitive(ObfuscationType obfuscationType)
    {
        return obfuscationType switch
        {
            ObfuscationType.HeadVisible => new SensitiveHeadVisible(TruncateAfter),
            ObfuscationType.TailVisible => new SensitiveTailVisible(),
            ObfuscationType.MaskChar => new SensitiveMaskChar(TruncateAfter),
            _ => throw new NotImplementedException($"None type {nameof(obfuscationType)}: {obfuscationType}")
        };
    }

    internal string Obfuscate(string strValue)
    {
        return Sensitive.Obfuscate(strValue);
    }

    internal string Obfuscate(Guid guid) => Sensitive.Obfuscate(guid);

    internal string Obfuscate(DateTime dateTime) => Sensitive.Obfuscate(dateTime);

    internal string Obfuscate(Enum @enum) => Sensitive.Obfuscate(@enum);

    internal virtual string ObfuscateDefault(object value) => Sensitive.ObfuscateDefault(value);
}

Now that obfuscation rules are defined, I can show you how to use them in ObfuscatorContractResolver:



// ObfuscatorContractResolver.cs

public class ObfuscatorContractResolver : DefaultContractResolver
{
    protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
    {
        IList<JsonProperty> baseProperties = base.CreateProperties(type, memberSerialization);

        var sensitiveData = new Dictionary<string, SensitiveAttribute>();

        foreach (PropertyInfo p in type.GetProperties())
        {
            var customAttributes = p.GetCustomAttributes(false);

            var jsonPropertyAttribute = customAttributes
                .OfType<JsonPropertyAttribute>()
                .FirstOrDefault();

            if (jsonPropertyAttribute is null)
                continue;

            var sensitiveAttribute = customAttributes
                .OfType<SensitiveAttribute>()
                .FirstOrDefault();

            if (sensitiveAttribute is null)
                continue;

            var propertyName = jsonPropertyAttribute.PropertyName.ToUpperInvariant();

            sensitiveData.Add(propertyName, sensitiveAttribute);
        }

        if (!sensitiveData.Any())
            return baseProperties;

        var processedProperties = new List<JsonProperty>();

        foreach (JsonProperty baseProperty in baseProperties)
        {
            if (sensitiveData.TryGetValue(baseProperty.PropertyName.ToUpperInvariant(), out SensitiveAttribute sensitiveAttribute))
            {
                baseProperty.PropertyType = typeof(string);
                baseProperty.ValueProvider = new ObfuscatorValueProvider(baseProperty.ValueProvider, sensitiveAttribute);
            }

            processedProperties.Add(baseProperty);
        }

        return processedProperties;
    }
}

// ObfuscatorValueProvider.cs
    internal class ObfuscatorValueProvider : IValueProvider
{
    private readonly IValueProvider _valueProvider;
    private readonly SensitiveAttribute _sensitiveAttribute;

    public ObfuscatorValueProvider(
        IValueProvider valueProvider,
        SensitiveAttribute sensitiveAttribute)
    {
        _valueProvider = valueProvider;
        _sensitiveAttribute = sensitiveAttribute;
    }

    public object GetValue(object target)
    {
        var originalValue = _valueProvider.GetValue(target);

        var result = originalValue switch
        {
            null => null,
            string strValue => _sensitiveAttribute.Obfuscate(strValue),
            Guid guid => _sensitiveAttribute.Obfuscate(guid),
            Enum @enum => _sensitiveAttribute.Obfuscate(@enum),
            DateTime dateTime => _sensitiveAttribute.Obfuscate(dateTime),
            _ => _sensitiveAttribute.ObfuscateDefault(originalValue),
        };

        return result;
    }

    public void SetValue(object target, object value)
    {
        // we don't care
    }
}

And that's it: based on the Sensitive attribute definition over DTOs properties, JSON.NET understands if and how a given property should be serialized. The result will be sent to ApplicationInsights.

And what about my API response body?

The solution is pretty straightforward. The code above shows the OnResultExecutionAsync implementation: if code runs smooth and no exception is thrown, the response body serialized using the obfuscation rules is attached to the RequestTelemetry object.

Exception handling

If an exception occurs, you will realize that the OnResultExecutionAsync will not be fired. This happens because the flow runs out the normal ASP.NET action invocation pipeline.
But adding the exception handling middleware is enough to handle this:



public void Configure(IApplicationBuilder app)
{
    app.UseExceptionHandler(async (context) =>
    {
        RequestTelemetry request = context.Features.Get<RequestTelemetry>();

        string serializedObj = JsonConvert.SerializeObject(error,
            new JsonSerializerSettings
            {
                ContractResolver = new ObfuscatorContractResolver(),
            });

        request.Properties.Add(ApplicationInsightsActionFilterAttribute.RESPONSE_TELEMETRY_KEY, serializedObj);
    }));
}

Here you can see references to SensitiveHeadVisible, SensitiveMaskChar and SensitiveTailVisible classes that implement some code to apply different obfuscation rules - such as hiding the value with a specific char, showing the string tail, and so on. You are free to create your preferred ones.

Performance

After the implementation described in this post, we spent some time doing some benchmark to understand the performance impact over our APIs. So we searched the APIs with the biggest incoming and outgoing DTOs and then we stressed these APIs running 1000 requests each through Postman. We suddenly realized that we were going to lose 60-90ms for each request on each APIs! This performance drop was caused by the string serialization performed by JSON.NET, because the rest of the code needed a couple of milliseconds to run. Our code is fine then, how can we cut down the serialization time then?
After some research, we found a simple way to minimize the performance impact, keeping our APIs safe. Basically, we understood that we didn't need to wait that the serialization process to be finished before invoking the invocation pipeline!

Coming back at ApplicationInsightsActionFilterAttribute class, we did the following changes:



// OnActionExecutionAsync
var loggerTask = Task.Run(() =>
{
    string fromBodyParameter = context.ActionDescriptor.Parameters

    // same code as before
});

await next();

try
{
    if (loggerTask != null)
    {
        await loggerTask;
    }
}
catch (Exception ex)
{
    _logger.LogCritical(ex, "Error executing the response logger task");
}

// OnResultExecutionAsync
var loggerTask = Task.Run(() =>
{
    if (context.Result is ObjectResult result &&

    // same code as before
});

await next();

try
{
    if (loggerTask != null)
    {
        await loggerTask;
    }
}
catch (Exception ex)
{
    _logger.LogCritical(ex, "Error executing the response logger task");
}

This implementation doesn't wait for the serialization to be completed, but instead it runs a task that will be awaited AFTER the invocation pipeline is completed. Most of the times, the loggerTask is already completed when awaited, so the impact is almost nil.

Conclusion

We saw how to hide sensitive information from Application Insights logs, keeping our data (and users) safe.
Since all the logic is wrapped in a custom ContractResolver, you can obfuscate properties everywhere in your code! For example, we used the same approach for the log generated by our HttpClient instances.

I hope this tutorial was helpful! 🎉

.NET 6.0 console app - Configuration, tricks and tips

Andrea Grillo — Wed, 17 Mar 2021 09:24:09 +0000

One of the biggest .NET benefit is the flexibility and the portability of its features.
As ASP.NET Core relies on the IHostBuilder, we can do the same with a classic console application.
This approach will help us to have a basic infrastructure in our application, including the support for logging, dependency injection, app settings and so on.

To start with this, add the Microsoft.Extensions.Hosting NuGet package to your project and replace the "Hello World template" with this code:



class Program
{
    static async Task<int> Main(string[] args)
    {
        var host = CreateHostBuilder();
        await host.RunConsoleAsync();
        return Environment.ExitCode;
    }

    private static IHostBuilder CreateHostBuilder()
    {
        return Host.CreateDefaultBuilder();
    }
}

Doing this, we are asking for an HostBuilder instance with the default infrastructure implementation.
The RunConsoleAsync starts the host builder with the "classic console" capabilities, such as the listener for the CTRL+C shortcut. Finally, it returns the application exit code (default is 0).
This is a starting point; now we can extend our capabilities.

IHostedService

The Main method is the application entry point, but we now need to continue the execution of our app. To achieve this, add a class that implement the IHostedService interface and register it to the IoC container using the extension method AddHostedService:



private static IHostBuilder CreateHostBuilder()
{
    return Host.CreateDefaultBuilder()
        .ConfigureServices(services =>
        {
            services.AddHostedService<Worker>();
        });
}

public class Worker : IHostedService
{
    public Task StartAsync(CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }
}

Now at the startup, the application runs the StartAsync method;
Instead StopAsync is executed when the is going to be terminated.

Use the Dependency Injection

We can now use the dependency injection system provided by .NET.
In the ConfigureServices method, add the services you need to run your application:



public class MyService : IMyService
{
    public async Task PerformLongTaskAsync()
    {
        await Task.Delay(5000);
    }
}

public interface IMyService
{
    Task PerformLongTaskAsync();
}

public class Worker : IHostedService
{
    private readonly IMyService _myService;

    public Worker(IMyService service)
    {
        _myService = service ?? throw new ArgumentNullException(nameof(service));
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        await _myService.PerformLongTaskAsync();
    }
}

private static IHostBuilder CreateHostBuilder()
{
    return Host.CreateDefaultBuilder()
        .ConfigureServices(services =>
        {
            services.AddHostedService<Worker>();

            services.AddTransient<IMyService, MyService>();
        });
}

Logging and configuration

Using the HostBuilder's ConfigureLogging extension method we have a full access to the logging configuration. In this case, we want to replace the default .NET implementation with one of the most used logging library, Serilog.
First of all, install Serilog NuGet packages:

Serilog.Extensions.Hosting
Serilog.Settings.Configuration In this example, we are going to use the Console and the File sinks:
Serilog.Sinks.Console
Serilog.Sinks.File

Our goal is to log events in a log file when running in a Production environment; we stick instead to the console when debugging the app.
To start with this, edit the CreateHostBuilder method as follow:



return Host.CreateDefaultBuilder()
    .ConfigureLogging(logging =>
    {
        logging.ClearProviders();
    })
    .UseSerilog((hostContext, loggerConfiguration) =>
    {
        loggerConfiguration.ReadFrom.Configuration(hostContext.Configuration);
    })
    .ConfigureServices(services =>
    {
        services.AddHostedService<Worker>();

        services.AddTransient<IMyService, MyService>();
    });

The logging.ClearProviders() is used to clear all the default HostBuilder logging providers. These are:

Console
Debug
Event Source
EventLog (Windows only)

Then we read from the appsettings.json (added in the next paragraph) the Serilog configuration using loggerConfiguration.ReadFrom.Configuration(hostContext.Configuration);.

Now it's time to give a configuration to our app. Add the appsettings.json file to your project (ensure to set the BuildAction to Content and the Copy to Output Directory to Copy if newer). Then, paste the Serilog's configuration:



"Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "File",
        "Args": {
          "path": "log.txt",
          "rollingInterval": "Day",
          "outputTemplate": "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level:u3}] {Message:lj}{NewLine}{Exception}",
          "shared": true
        }
      }
    ],
    "Enrich": [ "FromLogContext" ]
  }

The snippet above is setting the Serilog default level to Information (with some override for the Microsoft and System namespaces) and configure the File sink. In particular, it rolls out a different log file each day.
But as stated above, we want to redirect all the application output to the console when debugging the app. Then we need to setup the appsetting.Development.json file to the project and override the Serilog configuration, changing the default minimum level and setting the console as output provider:



"Serilog": {
    "MinimumLevel": {
      "Default": "Debug",
      "Override": {
        "Microsoft": "Information",
        "System": "Information"
      }
    },
    "WriteTo": [
      {
        "Name": "Console",
        "Args": {
          "theme": "Serilog.Sinks.SystemConsole.Themes.AnsiConsoleTheme::Code, Serilog.Sinks.Console",
          "outputTemplate": "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} <s:{SourceContext}>{NewLine}{Exception}"
        }
      }
    ],
    "Enrich": [ "FromLogContext" ]
  }

We don't need to do anything else to support different environemnts than defining them in the launchSettings file. Add a folder called Properties and a file named launchSettings.json:



{
    "profiles": {
        "Development": {
            "commandName": "Project",
            "environmentVariables": {
                "DOTNET_ENVIRONMENT": "Development"
            }
        },
        "Production": {
            "commandName": "Project",
            "environmentVariables": {
                "DOTNET_ENVIRONMENT": "Production"
            }
        }
    }
}

The DOTNET_ENVIRONMENT variable is automatically read from the HostBuilder. It is possible to customize them, more info are available in the official doc.

We are now ready to use the logger and eventually the values coming from the app settings files. Inject the ILogger and the IConfiguration instances in your classes:



public class Worker : IHostedService
{
    private readonly IMyService _myService;
    private readonly string _configKey;
    private readonly ILogger<Worker> _logger;

    public Worker(IMyService service, IConfiguration configuration, ILogger<Worker> logger)
    {
        _myService = service ?? throw new ArgumentNullException(nameof(service));
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
        _configKey = configuration["ConfigKey"];
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Read {key} from settings", _configKey);

        await _myService.PerformLongTaskAsync();
    }

...
}

NOTE: In order to make this code working, you need to add a ConfigKey property in your app settings. If you do the same in the appsettings.Development.json but with a differenva lue, you could notice that value changes depending on the current environment

We could also add the user secrets and environment variables support using:



.ConfigureLogging(logging =>
{
    ...
})
.UseSerilog((hostContext, loggerConfiguration) =>
{
    ...
})
.ConfigureAppConfiguration((hostContext, builder) =>
{
    builder.AddEnvironmentVariables();

    if (hostContext.HostingEnvironment.IsDevelopment())
    {
        builder.AddUserSecrets<Program>();
    }
})

Terminates the app properly using Exit Code

Injecting the IHostApplicationLifetime instance in the Worker class, we can terminate properly the console process when the job is done. Calling StopApplication is enough and it will redirect the running path to the StopAsync method. Moreover, we can use Exit Code to tell the user why the app stopped:



private readonly IHostApplicationLifetime _hostLifetime;
private int? _exitCode;
...

public Worker(IMyService service, IConfiguration configuration, IHostApplicationLifetime hostLifetime, ILogger<Worker> logger)
{
    ...
    _hostLifetime = hostLifetime ?? throw new ArgumentNullException(nameof(hostLifetime));
}

public async Task StartAsync(CancellationToken cancellationToken)
{
    _logger.LogInformation("Read {key} from settings", _configKey);

    try
    {
        await _myService.PerformLongTaskAsync();

        _exitCode = 0;
    }
    catch (OperationCanceledException)
    {
        _logger?.LogInformation("The job has been killed with CTRL+C");
        _exitCode = -1;
    }
    catch (Exception ex)
    {
        _logger?.LogError(ex, "An error occurred");
        _exitCode = 1;
    }
    finally
    {
        _hostLifetime.StopApplication();
    }
}

public Task StopAsync(CancellationToken cancellationToken)
{
    Environment.ExitCode = _exitCode.GetValueOrDefault(-1);
    _logger?.LogInformation("Shutting down the service with code {exitCode}", Environment.ExitCode);
    return Task.CompletedTask;
}

Use command line arguments efficiently

One of the better way to handle the command line arguments in your app, is through the McMaster.Extensions.CommandLineUtils NuGet package.
It is a fork of the deprecated Microsoft library and allow you to easily manage the input commands. I show you a brief example of use. You need to modify the Program.cs as follow:



[Command(
    Name = "MyApp"
    , Description = "My app is very cool 😎")]
[HelpOption(
    "-h"
    , LongName = "help"
    , Description = "Get info")]
[VersionOptionFromMember(
    "-v"
    , MemberName = nameof(GetVersion))]
class Program
{
    [Option(
        "-o"
        , "Some option"
        , CommandOptionType.SingleValue
        , LongName = "option")]
    [Range(1, 5)]
    public int Option { get; } = 1;

    public static Task<int> Main(string[] args) =>
        CommandLineApplication.ExecuteAsync<Program>(args);

    public async Task<int> OnExecuteAsync(CancellationToken cancellationToken)
    {
        var host = CreateHostBuilder();
        await host.RunConsoleAsync(cancellationToken);
        return Environment.ExitCode;
    }

private static IHostBuilder CreateHostBuilder() {...}

private static string GetVersion()
    => typeof(Program).Assembly.GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion;

Here's a description:

Command: gives a name and a description to you app
HelpOption: enables the -h (or --help) option
VersionOption: returns the app version number
Option: represents the value passed as input argument to your app

Then, the Main method needs some refactoring. Instead, the OnExecuteAsync is called automatically by the framework (it becomes the old Main method basically). Not that OnExecuteAsync is not static, so it can access to the class properties such has Option - that you can register in the IoC container using the IOptions pattern:



public class MyOptions
{
    public int Value { get; set; }
}

class Program
{

    [Option(
        "-o"
        , "Some option"
        , CommandOptionType.SingleValue
        , LongName = "option")]
    [Range(1, 5)]
    public int Option { get; } = 1;

...

.ConfigureServices(services =>
    {
        services.AddHostedService<Worker>();

        services.Configure<MyOptions>(options =>
        {
            options.Value = Option;
        });

        services.AddTransient<IMyService, MyService>();
    });

...

public class Worker : IHostedService

    private readonly MyOptions _options;

    public Worker(IMyService service, IConfiguration configuration, IHostApplicationLifetime hostLifetime, ILogger<Worker> logger, IOptions<MyOptions> options)
    {
        ...
        _options = options?.Value ?? throw new ArgumentNullException(nameof(options));
        ...
    }

Here the CommandLineUtils in action:

Unfortunately my console doesn't support emoji yet 😒

Conclusions

I hope this tutorial gave you the right hint to start your next project based on a .NET console application! These are just an introduction of all the features you can add, basically with no limits! Yeah, it is true: you can use everything you are already using in your ASP.NET Core projects through the IHostBuilder interface. Sounds good, isn't it? 🐱‍👤

As usual the source code is available in my GitHub profile.

Xamarin Android and iOS .resx localization with MvvmCross

Andrea Grillo — Sat, 23 Jan 2021 00:10:21 +0000

Almost any mobile app published to the PlayStore or AppStore should be localizated in different languages in order to get more clients as possible from anywhere in the world.

MvvmCross is a 'MVVM framework' written exclusively to work with Xamarin and provides an easy way to share the localized resources among the different platforms using the same .resx files. In this post I will show you how to share localized resources between an Android and an iOS apps.

Let's localize it!

Our starting point is the standard MvvmCross solution structure, with a separate project for each platform and the Core project with the common stuff. I created a single view called HomeView with its view model HomeViewModel on both platforms:

Solution structure:

The HomeViews contain only a label in the middle of the screen with the text "Hello". Below the screenshots and the UI definition:

Android:

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
  xmlns:android="http://schemas.android.com/apk/res/android"
  xmlns:app="http://schemas.android.com/apk/res-auto"
  android:orientation="vertical"
  android:layout_width="match_parent"
  android:layout_height="match_parent">

  <androidx.appcompat.widget.AppCompatTextView
      android:layout_width="wrap_content"
      android:layout_height="wrap_content"
      android:text="Hello!"
      app:layout_constraintStart_toStartOf="parent"
      app:layout_constraintEnd_toEndOf="parent"
      app:layout_constraintBottom_toBottomOf="parent"
      app:layout_constraintTop_toTopOf="parent"/>

</androidx.constraintlayout.widget.ConstraintLayout>

iOS:

public override void ViewDidLoad()
{
    View.BackgroundColor = UIColor.White;

    base.ViewDidLoad();

    var label = new UILabel { Text = "Hello!" };

    Add(label);

    View.SubviewsDoNotTranslateAutoresizingMaskIntoConstraints();
    View.AddConstraints(
        label.WithSameCenterX(View),
        label.WithSameCenterY(View));

    var set = CreateBindingSet();
    set.Bind(label).ToLocalizationId("Hello");
    set.Apply();
 }

NOTE: For more information on how to setup MvvmCross, refer to the official documentation.

At this point, we must install the NuGet package MvvmCross.Plugin.ResxLocalization to the Core project.

Then creates a folder Localization and add a resource file, let's call it AppResources.resx. This file will contain the main language of our app.

Open it and be sure to set the Access Modifier to Public:

Let's add our first localized resource, as per in the screenshot below:

NOTE: the MvvmCross convention uses the string {ViewModelName}.{ResourceName} as key. In our case it is HomeViewModel.Hello.

Now we can proceed adding another resource file containing the translations for another language of our choices - Italian in my case. To do this, add a file called AppResources.it.resx still under Localization folder:

Please, note that now the Access Modifer has been set to No code generation. In fact, it's only the main resx file that is going to generate the necessary code.

Close the resx files and go in HomeViewModel and:

Implement the interface IMvxLocalizedTextSourceOwner
Define the following property:

public IMvxLanguageBinder LocalizedTextSource =>
            new MvxLanguageBinder("", GetType().Name);

NOTE: For demostration purposes I am doing this in the (only) page's view model, but in a real environment it would be better to add this property to a single base view model since almost every page in our application would need to localize resources.

Mvx.IoCProvider.RegisterSingleton<IMvxTextProvider>(new MvxResxTextProvider(AppResources.ResourceManager));

Now it's time to apply the translation via the binding set exposed by MvvmCross. Even you can do this by UI too, I prefer to set all the resources by code: in case of refactoring or changes, it's definitely easier to keep track of all the changes. Let's see how to do it.

Android

From the definition above, replace the android:text definition with android:id. You can see mine below:

<androidx.appcompat.widget.AppCompatTextView
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:id="@+id/txtHello"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintBottom_toBottomOf="parent"
    app:layout_constraintTop_toTopOf="parent"/>

We need the control's id (txtHello) in the code behind in the next step.

Go in HomeView.cs. Write code to get the reference of the UI control (txtHello) and to bind it to the MvxLanguageBinder instance we defined into the view model through a built-in converter:

protected override void OnCreate(Bundle bundle)
{
    base.OnCreate(bundle);
    SetContentView(Resource.Layout.home_view);

    var textView = FindViewById<AppCompatTextView>(Resource.Id.txtHello);

    var set = CreateBindingSet();

    set.Bind(textView)**.ToLocalizationId("Hello");**

    set.Apply();
}

Now your app is ready to rock! This is what you see if you press F5:

Then go to Android's settings and change the language to Italian. Then open the app again:

Urrah! 🎉🎉

iOS

In iOS everything works just as what we did on the Android side.

Creates the Mvx's binding set and apply the binding to the localized resource:

var set = CreateBindingSet();
set.Bind(label).ToLocalizationId("Hello");
set.Apply();

Change the iPhone language to Italian.. and here we go again 🎉

The source code is available on my GitHub