Forem: Miguel Bernard

First and Last line content | Solving the diamond kata with property-based testing

Miguel Bernard — Mon, 20 Jul 2020 17:19:03 +0000

Solving the diamond kata with property-based testing series

All code samples are available on github

Intro

We continue our adventure trying to solve the Diamond Kata while using Property-Based testing. Last time, we added our first test, Non-empty and discovered how to use input generators. Now let's figure out the next test.

First and last line content

In the diamond Kata, the first and last line of every diamond always contains A. Such regularity is perfect for a property. Even the particular case with input A, where the first line is also the last one, respects that property.

e.g.

input: A

A

input: E

----A----
---B-B---
--C---C--
-D-----D-
E-------E
-D-----D-
--C---C--
---B-B---
----A----

C# Tests

[Property(Arbitrary = new[] { typeof(LetterGenerator) })]
public Property FirstLineContainsA(char c)
{
    return Diamond.Generate(c).First().Contains('A').ToProperty();
}

[Property(Arbitrary = new[] { typeof(LetterGenerator) })]
public Property LastLineContainsA(char c)
{
    return Diamond.Generate(c).Last().Contains('A').ToProperty();
}

Here we used some built-in methods of the .NET library, which makes these tests simple to read and short to write. It almost reads like a sentence.

Diamond.Generate(c) Generates the diamond
First() / Last() Takes the first/last line of the generated diamond
Contains('A') Checks if the line contains the letter A and returns a bool
ToProperty() Transforms a boolean expression to a property

If you are wondering what's [Property(Arbitrary = new[] { typeof(LetterGenerator) })] it's probably because you missed my previous post

Wrapping up

We are making some good progress towards a fully functioning test suite. However, there are still some uncovered areas that we'll address with more tests next time.

Solving the diamond kata with property-based testing series

All code samples are available on github

Input generators in property-based tests with FsCheck

Miguel Bernard — Fri, 03 Jul 2020 12:04:40 +0000

Solving the diamond kata with property-based testing series

All code samples are available on github

Intro

Now that we have all the tools installed, we are finally ready to write our first Property-Based test.

FsCheck

Creating a property-based test is as simple as tagging a method with the [Property] attribute, similarly as you would do in xUnit with the [Fact] or [Theory] attributes. If you dig a little, you will realize that there's a lot of configurable options on the [Property] attribute. You can control how many random tests will be run each time, timeouts, etc. One of them is essential; the Arbitrary property. That property will define which method to use to generate random inputs. In our specific case, we are not interested in all the possible values of char. We only care about the 26 letters of the alphabet. To scope the input, we'll need to configure a generator that will do just that.

Generators

Generators are relatively simple.

Create a static class
Add a static method that returns Arbitrary<TypeYouWantToGenerateValueFor>
Voilà

You can write your randomization logic, but it's usually better to start from the built-in methods as they are already well balanced.

public static class LetterGenerator
{
    public static Arbitrary<char> Generate() =>
            Arb.Default.Char().Filter(c => c >= 'A' && c <= 'Z');
}

Not Empty Test

It may seem basic, but it's quite essential. We want to make sure we always have a non-empty result.

[Property(Arbitrary = new[] { typeof(LetterGenerator) })]
public Property NotEmpty(char c)
{
    return Diamond.Generate(c).All(s => s != string.Empty).ToProperty();
}

One thing I forgot to mention earlier, the test method also needs to return a property object. Fortunately for us, the library contains an extension method that can transform any boolean expression into a property object by calling ToProperty().

So our test reads like this:

Call the Diamond.Generate() method with the provided input char c.
Make sure all the lines of the returned diamond are not empty.
Use the ToProperty() method to wrap it into a property object.

Stay tuned

Stay tuned for my next posts on this series about solving the diamond kata with property-based testing. The following post will try to define properties for the first and last lines of our diamond.

Solving the diamond kata with property-based testing series

All code samples are available on github

How to get started with Property-based Testing in C#

Miguel Bernard — Mon, 29 Jun 2020 18:14:39 +0000

Solving the diamond kata with property-based testing series

All code samples are available on github

Intro

So far, we've talked about the benefits of property-based tests in my two previous blog posts here and here. I'm sure for most of you; this concept is still pretty abstract. We've seen the technique, but how do we implement it in C#? In this series, we'll go through all the steps to solve the Diamond Kata using TDD and property-based tests.

The Diamond Kata

Given an input letter, print a diamond starting with A and having the supplied letter at the widest point.

e.g., For input: D

---A---
--B-B--
-C---C-
D-----D
-C---C-
--B-B--
---A---

Why did I choose this specific Kata? Well, if you try to solve it with example-based tests, you effectively need 26 tests to make sure it works properly. It may not seem like a lot of tests, but if you try it, you'll soon discover that the first few tests are trivial, and as you approach Z, the tests are becoming harder to read and harder to write. It's usually a code smell that indicates that some property-based tests would be a better approach.

TDD

In this series, we'll also use TDD, and you'll see that it's much easier to use with Property-based tests than example-based tests. So in excellent TDD practitioners that we are, we are going to start with the most straightforward implementation of the algorithm that we can think of.

public class Diamond
{
    private static IEnumerable<string> Generate(char c)
    {
        yield return "A";
    }
}

Easy, right? It only works for the A case, but it's a good starting point.

I already hear the purists among you.

This is not the right way to do TDD! Red -> Green -> Refactor! You should start with a failing test, not with the implementation.

You are not wrong, but let's be realistic here. It's just a starting point, and I promise we'll do better TDD going forward :D.

The tools

Property-based tests is a simple enough concept, but generating proper pseudo-random inputs can be harder than it looks. Fortunately for us, other people solved that issue and built great tools. In our case, we'll use FsCheck, a property-based testing framework made for F#.

I'm confused, are we doing this in C# or F#?

That's the beauty of .NET, even though the library is made for F#, all the .NET languages compile to Intermediate Language (IL or MSIL) and thus are interoperable. It means we can call that library from C#.

Install the NuGet packages, and you are ready to go.

dotnet add package FsCheck --version 2.14.2
dotnet add package FsCheck.Xunit --version 2.14.2

Closing word

I think we have all the prerequisites to get started. Stay tuned for the next post where we'll implement our first test.

Solving the diamond kata with property-based testing series

All code samples are available on github

My full MS Build 2020 recap

Miguel Bernard — Sun, 31 May 2020 14:01:17 +0000

Intro

This year was a special edition of MS Build; for obvious reasons, it was 100% virtual. I must say that I had my doubts at first, but I'm forced to admit that Microsoft nailed it. The experience was fantastic. I liked the personal schedule builder and the rebroadcast in 3 different timezones, to only name a few things. Microsoft even took the time to ship the conference's gear to all attendees, DIRECTLY to their doorstep, what a nice touch.

There were over 600 sessions, running 24h/day continuously. Unfortunately, this recap only covers what I was able to watch. If you see something missing, don't hesitate to engage in the comments down below.

Recap

Empowering every developer, with Satya Nadella

Once again, Satya had a very compelling opening keynote. He remembered everyone that in a world constantly changing, the developer is more important than ever. Microsoft is building the best toolchain to empower those developers.

No big announcements from his part, but he highlighted some critical aspects of the conference to keep on eye on:

GitHub
VSCode
PowerPlatform
Azure
O365

Every developer is welcome, with Scott Hanselman and guests

Scott was demoing some new vital tools in the developer environment, starting at the workstation's tooling.

Winget

Winget is a new tool to manage and script the installation of software packages. It will save developers a ton of time when they need to rebuild or set up their machines for a new project.

PowerToys

He showed some of the features of PowerToys, namely fancy zones, which I must admit is pretty awesome to arrange your working space in a more efficient way.

WSL 2.0

WSL 2.0 now support graphical applications, and there's even a new Linux file explorer GUI available.

Windows terminal

Windows terminal is finally out of beta and just hit version 1.0.

CodeSpaces

Developing applications has always been made working locally. This approach comes with some downsides. For instance, you need to have all the tools and other dependencies of the project installed locally. It tends to create snowflakes environment. How many times have you heard "Strange it works on my machine".

Microsoft is coming with a new developer experience that runs entirely remotely. CodeSpaces (formerly VS online) is a new tool that enables you to spin a complete dev environment in the cloud, on a remote machine. Then you can develop directly in your browser or connect to it with your favorite tools like VS Code or Visual Studio.

It's also now possible to spin up a fully functioning environment from a GitHub repository. It will come pretty handly to review a pull request.

Collaboration is at the heart of this new experience. With LiveShare, you can collaborate with multiple people on that remote environment like you would in an Office document. This functionality will enable a whole new way of doing pair programming, mainly since most of us are still confined at home.

Azure: Invent with Purpose

That session started with some impressive statistics about the Microsoft Intelligent Cloud (a.k.a. Azure + O365).

Used by 95% of Fortune 500 companies
61 azure regions, more than any other cloud provider
MS Teams hosts 4.1 billion meeting minutes per day

These are some pretty impressive numbers, and it seems they keep growing quarter after quarter. All of these remarkable services are built on top of the same building blocks that are publically available to you in Azure; ServiceFabric, Kuberbernates, Media Services, Redis, FrontDoor, etc.

Microsoft wants to enable all the developers to focus on their application and not on the infrastructure supporting it. Speaking of which, they announced some new capabilities:

AKS Windows container support for lift and shift scenarios Spot pricing for low priority workloads
ComosDB A free tier that gives you 400 RU Serverless pricing where you only pay per consumption Auto-scale
AI New cognitive services
Synapse Synapse Link for real-time data analytics with no performance impacts on operational DB No ETL required Deep integration with PowerBI

The Journey to One .NET

First of all, there's a new kid on the block, Blazor WASM, which brings a new alternative for devs to run .NET in the browser to build rich client applications using the technology they know. It provides complete debugging capabilities in the browser. It's also possible to debug a full loop of Front-end and Back-end altogether in Visual Studio.

It's the beginning of the unification of the whole .NET stack.
Dotnet Core and Framework will be merged in the new version .NET 5.
MAUI will be the evolution of Xamarin Forms and will bring a single UI that can target any platform like Android, iPhone, Windows, etc.

Complete talk here

C# Today & Tomorrow

I think that session was my favorite. I must admit I'm a big fan of Functional Programming (FP) and especially F#. Microsoft showed some preview of what's coming in C# 9.0. Most of the new features are all about FP concepts and are implemented very similarly to F#, which will help bridge the gap between the two languages.

Immutability in C# classes
Simplicity, you can now skip some boilerplate code like Program.cs and the Main function
Structural equality
Better pattern matching

With all of this, Microsoft will reach an even broader audience and onboard them in the rich ecosystem of C#. It will also provide all the necessary tools for C# developers to start playing with FP concepts and write better code.

Making every developer and team more productive from dev box setup to coding to deployment and debugging

Microsoft has always invested a lot in great tooling. Call me a fanboy, but Visual Studio and VS Code are, in my opinion, the two best IDE out there. They are now bringing an improved experience within collaborative development with chat and voice integration directly in LiveShare. Git support also got a significant revamp, improving how VS handles git repositories. IntelliCode got smarter; it's now able to suggest coding standards that your teammates are following. It will save a lot of comments in PR when onboarding new developers in a project.

Infrastructure as code - build Azure applications with ARM templates

Infrastructure as Code (IaC) is the new trend in DevOps practices, and Microsoft knows it. They are doubling down in their IaC technology, the ARM Templates. They added a useful new parameter that you can use in PowerShell or Azure CLI when deploying an ARM template. What-if will output all the changes that will be executed if you deploy that ARM Template. A great feature to validate impacts before deploying. There are also some new features to ARM template themselves:

Auto-generation of parameter files
Intellisense for ResourceId
Improved schema

Microsoft added TemplateSpec, a new way to package, version, and store ARM templates. It will help simplify the deployment of big infrastructures.

The future of tech, with Kevin Scott and guests

Microsoft has built one of the biggest supercomputers on earth (Top 5), putting it at the service of AI for everyone's benefit. They are improving their models and offering it to you as a service. They are also running a lot of research projects on it.

In that session, they showed an AI that can automatically read a user requirement and write Python code that fulfills that requirement. What's impressive is that the generated code integrates flawlessly with existing code. Not only the code is accurate, but it's well written and easy to read. It's utterly indistinguishable from code written by a real person. I don't think this kind of capability will ever replace a good old developer, but it can indeed be an excellent tool to have to speed up the development of an application.

Microsoft is one of the few providers that brings the AI at the edge. It enables the use of AI in the most challenging environments that don't always have a stable internet connection. On one of the demos, the equip a robot from Boston Robotics to make it fully autonomous.

Accelerating enterprise app development with Power Platform

The Power platform is having a lot of traction lately. It allows super rapid application development, which requires little to no code at all. It's currently used by more than 97% of the Fortune 500. This rich environment of tools for power users is also adding new services. Power Virtual Agent is a new automated chatbot build on top of the Microsoft bot framework.

Other notable improvements

ApplicationInsights & Azure Monitor

Support for the OpenTelemetry standard
Transaction correlation support for Azure Service Bus
New visualization
Merge of application and infrastructure logs for better correlation and diagnostics capabilities

Azure SQL

Serverless option, only pay for what you use
Scale instantly
Can event completely pause/resume your DB automatically (e.g., at night when there's no traffic)

Azure Databricks and Sparks

AAD authentication
DataBricks pools
DataBricks container services
ZSkipping, a new algorithm that can bring up to 100x performance improvement
DeltaLake

Closing

It does not entirely cover the event, but it should give you a good overview of the significant announcements. Thank you again, Microsoft, for organizing such a great online and free event.

You can also consult the official book of news for the complete list of announcements.

You need these tools to be more efficient at Testing with AutoFixture and FakeItEasy

Miguel Bernard — Tue, 05 May 2020 15:22:04 +0000

There are many well-known benefits of testing your code, but for many developers, testing is still the part of their job they like the least. They find it hard and painful, but it doesn't have to be that way.
Nowadays, there's excellent tooling to make it fun and productive, and that's what this blog post is all about.

xUnit, AutoFixture, and FakeItEasy are among my favorites frameworks/tools. They all achieve something particular, and they all play well together. I won't go over the details for each one, because there's a ton of documentation with great examples out there, but here's a quick look at what they do.

xUnit

xUnit.net is a free, open-source, community-focused unit testing tool for the .NET Framework. Written by the original inventor of NUnit v2, xUnit.net is the latest technology for unit testing C#, F#, VB.NET, and other .NET languages. xUnit.net works with ReSharper, CodeRush, TestDriven.NET, and Xamarin.

AutoFixture

AutoFixture is an open-source library for .NET designed to minimize the 'Arrange' phase of your unit tests to maximize maintainability. Its primary goal is to allow developers to focus on what is being tested rather than how to set up the test scenarios by making it easier to create object graphs containing test data.

FakeItEasy

FakeItEasy is a .Net dynamic fake framework for creating all types of fake objects, mocks, stubs, etc.

In this post, what I'm more interested in is to show you how to integrate them. Let's do it with a hands-on example that we're going to complete together.

First of all, install the following NuGet packages in your test project.

xunit
AutoFixture
FakeItEasy
AutoFixture.Xunit2
AutoFixture.FakeItEasy

Class to test

Second of all, let's define a simple example of a class to test a.k.a. System Under Test (sut)

public class StringService
{
    private INumberService _service;

    public StringService(INumberService service)
    {
        this._service = service;
    }

    public int ConvertToNumber(string value)
    {
        // Service that returns a number for each char in a string
        // corresponding to that letter position in the alphabet
        var numbers = this._service.Execute(value);
        return numbers.Sum();
    }
}

Test - V1

[Theory]
[InlineData(6, [1,2,3], "abc")]
[InlineData(15, [4,5,6], "def")]
public void GivenAString_WhenConvertToNumber_ThenSumOfLetterPositionIsReturned(int expected, 
    int[] charValues, string value)
{
    // Given
    var service = A.Fake<INumberService>();
    A.CallTo(() => service.Execute(A<string>._).Returns(charValues);
    var sut = new StringService(service);

    // When
    var result = sut.ConvertToNumber(value);

    // Then
    Assert.Equal(expected, result);
}

Pretty simple, it's neither too big nor dirty. We are leveraging [Theory] and [InlineData] to get a helpful, clean test, but it seems like we might miss some test cases.

Test - V2

Let's change the test to use [AutoData] in order to generate random test data and cover more edge cases

[Theory]
[AutoData]
[InlineAutoData([1,2,3], "abc")]
[InlineAutoData([4,5,6], "def")]
public void GivenAString_WhenConvertToNumber_ThenSumOfLetterPositionIsReturned(int[] charValues,
    string value)
{
    // Given
    var fixture = new Fixture();
    var service = A.Fake<INumberService>();
    A.CallTo(() => service.Execute(A<string>._).Returns(charValues);
    fixture.Register<INumberService>(()=> service);
    var sut = fixture.Create<StringService>();

    // When
    var result = sut.ConvertToNumber(value);

    // Then
    var expected = charValues.Sum();
    Assert.Equal(expected, result);
}

It's nice to have some auto-generated test data, but the Given part of the test is getting quite big and harder to understand. We can probably do something with this.

Test - Final Version

Let's use everything in the toolbox, but to do so, we'll need to create our integration between the frameworks.

Here's the actual implementation of the [AutoDataFakeItEasy] and [InlineAutoDataFakeItEasy] attributes that will make all of the heavy lifting for you.

Magic Attributes

[AttributeUsage(AttributeTargets.Method)]
public class AutoFakeItEasyDataAttribute : AutoDataAttribute
{
    public AutoFakeItEasyDataAttribute()
    : base(FixtureFactory)
    {
    }

    private static IFixture FixtureFactory()
    {
        return new Fixture().Customize(new AutoFakeItEasyCustomization());
    }
}

[AttributeUsage(AttributeTargets.Method, AllowMultiple = true)]
public class InlineAutoFakeItEasyDataAttribute : InlineAutoDataAttribute
{
    private readonly object[] _values;

    public InlineAutoFakeItEasyDataAttribute(params object[] values)
        : base(new AutoFakeItEasyDataAttribute(), values)
    {
        this._values = values;
    }

    public override IEnumerable<object[]> GetData(MethodInfo testMethod)
    {
        var data = base.GetData(testMethod).ToList();

        for (var i = 0; i < this._values.Length; i++)
        {
            data[0][i] = this._values[i];
        }

        return data;
    }
}

The final test

[Theory]
[AutoDataFakeItEasy]
[InlineAutoDataFakeItEasy([1,2,3])]
[InlineAutoDataFakeItEasy([4,5,6])]
public void GivenAString_WhenConvertToNumber_ThenSumOfLetterPositionIsReturned(int[] charValues, 
    string value, [Frozen]INumberService service, StringService sut)
{
    // Given
    A.CallTo(() => service.Execute(A<string>._).Returns(charValues);

    // When
    var result = sut.ConvertToNumber(value);

    // Then
    var expected = charValues.Sum();
    Assert.Equal(expected, result);
}

We still get all the benefits of auto-generated test data and the flexibility to add our specific test cases. The best part, we got rid of all the boilerplate code that was initializing our sut and its dependencies. The attributes will make sure to initialize our test values with inputs from the attribute or random data otherwise. They'll then make sure to instantiate mocks and dummy classes, also taking care of all their dependencies. It gives us a very robust and maintainable test that will survive many refactors without needing any changes.

The only thing we need to define now is the behavior of the mock, calling the method under test and asserting the result. Wow, this is what every test should look like. Like we say: less is more.

Conclusion

It is a very simple sut and test. Imagine when you have a complex class to test, how much code and complexity you can save with these attributes.
As you can see, knowing the right tools can make you super efficient at writing good unit tests. Moreover, your tests will be simple to understand and maintain over time.

Why null in C# is so bad

Miguel Bernard — Sun, 26 Apr 2020 13:49:00 +0000

In my career, more than half of the bugs I encountered were related in a way or another to a null value being propagated where it shouldn't have been—therefore resulting in unexpected behaviors or a nasty NullReferenceException.

The problem

Why is null such a problem in C#? Well... the null problem (a.k.a the billion-dollar mistake) breaks many assumptions you can make about the programming language.

In more theoretical terms, it breaks the Liskov substitution principle. (One of the famous SOLID principles)

Let Φ(x) be a property provable about objects x of type T. Then Φ(y)should be true for objects y of type S where S is a subtype of T.

Did I lose you? Great! Now let's talk plain English.

It means that you should be able to use any subtype in place of its parent type without affecting the behavior of the application.

What does that have to do with null?

In the C# compiler, there's a rule to enforce and prevent multiple inheritances... except... for null. Null is a special instruction in the language that can substitute any type. Put in another way; it means that null inherits from all reference types.

Because of that, you cannot think of your code logically anymore. You always have to check if the value is of subtype null or not. You have to check this because null behave in a very different way than any other type. It needs special processing and/or logic branching.



var customer = this._customerRepository.Get();
if (customer != null)
{
    // rest of the logic here.
}
else
{
    // logic for the null case here.
}

This practice, often referenced as defensive programming, makes your code verbose and error-prone. It works fine-ishhh. To make it work, you need to do it everywhere in your system. It's really easy to forget one location and guess what? It's precisely where you're going to get a NullReferenceException popping up in production.

Solutions

Yes, of course, you can use the more beautiful Elvis operator ?., but it doesn't solve anything. It just makes the code more readable.

The only real solution is to get rid of all the null. That being said, unless you have very strict coding rules within your team, I doubt you'll be able to do it. It's always tempting as a developer to take a shortcut and return a null value on a function call.

C# 8.0 to the rescue

Fortunately, C# 8.0 adds a brand new feature to help us with the null witch hunt. That new option instructs the compiler to check for null values in all code paths. Simply add this line to your .csproj and voilà!



<Nullable>enable</Nullable>

Then you can start getting rid of those null values and make sure nobody introduces new ones behind your back.

Conclusion

Using that will most likely break everything on a brownfield project, but for greenfield projects, I highly recommend you enable it. It might seem more work at first, but writing code without passing null around will save you many long, painful debugging hours in production. Ask me how I know. For me, that upfront development cost is a no-brainer.

Good hunt!

How to backup your Azure Storage Accounts

Miguel Bernard — Sun, 19 Apr 2020 15:03:33 +0000

Azure Storage Account is one of the foundation services of Azure. Pretty much all other services use it under the cover in a way or another. This service is simple, works great, and has crazy SLA and redundancy capabilities. However, it doesn't provide a point in time restore. Meaning that if you corrupt or delete some data, there's no way to recover it. There are some solutions that you can find over the internet, but unfortunately, none of them is a silver bullet. Let's explore and compare some of them.

Requirements

Let's first define some goals and requirements before we jump into solutions

Backups must run on a schedule (e.g., every hour)
Backups must be fast
Minimize network traffic $$$
The solution must be easy to maintain and operate
Get notification or report on failures
Must support multi-environment configuration (dev, QA, prod, etc...)
Has support for both Blob and Table storage format

AzCopy

AzCopy is a great tool to move data around in Azure, and it's also what most people will suggest using for backups. Target a source and a destination backup storage account and voilà! The AzCopy command-line performs the processing of that operation inside Azure datacenter between the two target storage accounts instead of passing by the machine that runs AzCopy. This feature alone makes AzCopy an obvious choice as it covers 2 of the most important objectives:
No. 2: Backup must be fast
No. 3: Minimize network traffic $$$
That's why most backup solutions use this tool.

Solution 1: Running from your PC

AzCopy is a command-line tool, so it's pretty easy to run from your machine or an on-prem server to backup your storage accounts. You can create a windows schedule to run that command, and you are good to go. It works fine when you have a few storage accounts, but grows complex real fast when adding more storage.

Pros

Easy to set up and automate

Cons

No easy way to get notified on a failure
Not easy to set it up for multiple environments
Scalability

Solution 2: Azure Automation

Azure Automation allows you to run PowerShell script directly in Azure. With that service, you can easily create a script that launches AzCopy with the specified arguments to target your storage accounts.

Pros

Easy to set up a schedule

Cons

Not that simple to install a .exe and run it on their hosted agent (You need to run it with a Hybrid Runbook Worker)
Not easy to set it up for multiple environments
No notification on failures

Solution 3: Let's try thinking outside the box

What about using Azure DevOps? It seems crazy right, but let's look at it more closely. One of the common problems with previous solutions is the management of backups for multiple environments and notifications on failure. Another problem is to install AzCopy on the agent or machine that's going to run the backup. Good news though, AzCopy is already installed by default on all Azure DevOps agents.
So here's a general idea: Create a pipeline that runs on a schedule and performs backup for each environment.

Create a PowerShell module with utility methods to make the use of AzCopy easier and upload it to your git repository

EDIT: You can now download all these scripts from Github. Also, special thanks to my good friend Yohan who cleaned up the whole thing and added the restore functionality.

Backup-Storage-Accounts.psm1

function Build-AzCopyCmd 
{
  param(
    [Parameter(Mandatory)] 
    [string]$DestinationPath,
    [Parameter(Mandatory)] 
    [PSCustomObject]$SrcCtx,
    [Parameter(Mandatory)] 
    [PSCustomObject]$DestCtx,
    [Parameter(Mandatory)] 
    [string] $AzCopyParam,
    [Parameter(Mandatory)] 
    [string] $SourcePath
  )

  $srcStorageAccountKey = $SrcCtx.StorageAccount.Credentials.ExportBase64EncodedKey()
  $destStorageAccountKey = $DestCtx.StorageAccount.Credentials.ExportBase64EncodedKey()
  $destContainer = $DestCtx.StorageAccount.CreateCloudBlobClient().GetContainerReference($DestinationPath) 
  return [string]::Format("""{0}"" /source:{1} /dest:{2} /sourcekey:""{3}"" 
      /destkey:""{4}"" $AzCopyParam", "C:\Program Files (x86)\Microsoft 
      SDKs\Azure\AzCopy\AzCopy.exe", $SourcePath, $destContainer.Uri.AbsoluteUri, 
      $srcStorageAccountKey, $destStorageAccountKey)
}

function Invoke-AzCopyCmd 
{
  param(
    [Parameter(Mandatory)] 
    [string]$AzCopyCmd
  )

  $result = cmd /c $AzCopyCmd
  foreach($s in $result)
  {
    Write-Host $s 
  }

  if ($LASTEXITCODE -ne 0){
    Write-Error "Copy failed!";
    break;
  }
  else
  {
    Write-Host "Copy succeed!"
  }

  Write-Host "-----------------"
}

function Backup-Blobs 
{
  param(
    [Parameter(Mandatory)] 
    [string]$DestinationPath,
    [Parameter(Mandatory)] 
    [PSCustomObject]$SrcCtx, 
    [Parameter(Mandatory)] 
    [PSCustomObject]$DestCtx,
    [Parameter(Mandatory)]
    [array] $SrcStorageContainers
  )

  Process {
    foreach ($srcStorageContainer in $SrcStorageContainers)
    {
      if($srcStorageContainer.Name -like '*$*')
      {
        Write-Host "-----------------"
        Write-Host "Skipping copy: $($srcStorageContainer.Name)"
        Write-Host "-----------------"
        Continue;
      }

      Write-Host "-----------------"
      Write-Host "Start copying: $($srcStorageContainer.Name)"
      Write-Host "-----------------"

      $blobDestinationPath = $DestinationPath + "/blobs/" + $srcStorageContainer.Name
      $azCopyParam = "/snapshot /y /s /synccopy"
      $sourcePath = $srcStorageContainer.CloudBlobContainer.Uri.AbsoluteUri
      $azCopyCmd = Build-AzCopyCmd -DestinationPath $blobDestinationPath -SrcCtx
          $SrcCtx -DestCtx $DestCtx -AzCopyParam $azCopyParam -SourcePath 
          $sourcePath
      Invoke-AzCopyCmd -AzCopyCmd $AzCopyCmd
    }
  }
}

function Backup-Tables 
{
  param(
    [Parameter(Mandatory)] 
    [string]$DestinationPath,
    [Parameter(Mandatory)] 
    [PSCustomObject]$SrcCtx,
    [Parameter(Mandatory)] 
    [PSCustomObject]$DestCtx,
    [Parameter(Mandatory)]
    [array] $SrcStorageTables
  )

  Process {
    foreach ($srcStorageTable in $SrcStorageTables)
    {
      Write-Host "-----------------"
      Write-Host "Start copying: $($srcStorageTable.Name)"
      Write-Host "-----------------"

      $tableDestinationPath = $DestinationPath + "/tables/" + $srcStorageTable.Name
      $azCopyParam = "/y"
      $sourcePath = $srcStorageTable.CloudTable.Uri.AbsoluteUri        
      $azCopyCmd = Build-AzCopyCmd -DestinationPath $tableDestinationPath -SrcCtx
          $SrcCtx -DestCtx $DestCtx -AzCopyParam $azCopyParam -SourcePath 
          $sourcePath
      Invoke-AzCopyCmd -AzCopyCmd $AzCopyCmd
    }
  }
}

function Backup-StorageAccount
{
  param(
    [Parameter(Mandatory)] 
    [PSCustomObject]$SrcCtx,
    [Parameter(Mandatory)] 
    [PSCustomObject]$DestCtx
  )

  $currentDate = (Get-Date).ToUniversalTime().tostring('yyyy\/MM\/dd\/HH:mm')
  $SrcStorageAccountName = $srcCtx.StorageAccount.Credentials.AccountName
  $destinationPath = $SrcStorageAccountName + "/" + $currentDate    

  $srcTables = Get-AzureStorageTable -Context $srcCtx

  if($srcTables)
  {
    Backup-Tables -DestinationPath $destinationPath -SrcCtx $SrcCtx 
        -DestCtx $destCtx -SrcStorageTables $srcTables
  }

  $maxReturn = 250
  $token = $null

  do{      
    $srcContainers = Get-AzureStorageContainer -MaxCount $maxReturn 
        -ContinuationToken $token -Context $srcCtx

    if($srcContainers)
    {
      $token = $srcContainers[$srcContainers.Count -1].ContinuationToken;
      Backup-Blobs -DestinationPath $destinationPath -SrcCtx $SrcCtx -DestCtx
          $destCtx -SrcStorageContainers $srcContainers
    }
  }
  While ($token -ne $null)
}

function Get-StorageAccountContext 
{
  param(
    [Parameter(Mandatory)] 
    [string]$StorageAccountName,
    [Parameter(Mandatory)] 
    [string]$StorageAccountResourceGroup
  )

  $storageAccountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName
      $StorageAccountResourceGroup -AccountName $StorageAccountName).Value[0]
  return New-AzureStorageContext -StorageAccountName $StorageAccountName 
      -StorageAccountKey $storageAccountKey
}

Export-ModuleMember -Function Get-StorageAccountContext
Export-ModuleMember -Function Backup-StorageAccount

Create a script to launch the backup Backup-Storage-Account.ps1

$destCtx = Get-StorageAccountContext -StorageAccountName $destStorageAccountName 
    -StorageAccountResourceGroup $destResourceGroup

  foreach($srcStorageAccount in $srcStorageAccounts)
  {    
    $srcStorageAccountName = $srcStorageAccount.storageAccountName
    $srcResourceGroup = $srcStorageAccount.resourceGroup
    $srcCtx = Get-StorageAccountContext -StorageAccountName $srcStorageAccountName
        -StorageAccountResourceGroup $srcResourceGroup
    Backup-StorageAccount -SrcCtx $srcCtx -DestCtx $destCtx
  }

Create a task group in Azure DevOps
Add the Azure PowerShell task, so the script will run in an Azure context and will be authenticated to the Storage Accounts you want to backup (make sure that Azure DevOps has access to your Azure Subscription)
Here's the YAML of that task. > Note that I used variables to make it more reusable in different environments
$(subscription) -> So you can target different Azure subscriptions
$(RepositoryPath) -> Git repository path that contains the scripts
$(EnvironmentCode) -> Variable that can be used in your PowerShell script to make custom logic depending on the environment

steps:
- task: AzurePowerShell@3
  displayName: 'Run Backup Azure PowerShell Script'
  inputs:
    azureSubscription: '$(subscription)'
    ScriptPath: '$(RepositoryPath)\src\Backup\Scripts\Backup-Storage-Accounts.ps1'
    ScriptArguments: '-EnvironmentCode $(EnvironmentCode)'
    FailOnStandardError: true
    azurePowerShellVersion: LatestVersion

Create a new Release build pipeline with all your environments as stages
Add an artifact and select Azure Repos

Set a schedule to kick off the backups on all environments

Configure each environment to use your Task Group

Configure your variables for each environment

You can now look at all the backup runs and get notifications for free when it fails (default behavior of Azure DevOps when a build fails)

Pros

Easy to set up for multiple environments with the release pipeline
Audit trail of all the changes in the release pipeline
Support for various environments
Overview of all the backup runs
Built-in notifications on failure

Cons

You need an Azure DevOps account (but you can subscribe to a free plan here)

Conclusion

Sometimes the solution is more straightforward than it looks. You need to leverage existing tools that you know. Try to think outside the box, and you'll find out that your absurd ideas might not be that crazy after all. An essential skill for a developer is the ability to solve business problems in new creative ways using technology.

How many unit tests are good enough | Property-Based testing

Miguel Bernard — Mon, 13 Apr 2020 14:35:52 +0000

As I was mentioning in my previous post, the way we are doing unit testing today has many issues. One of the biggest problems is to know for sure when we have enough tests to be confident to release in production. Why is that? Well, we mostly base our tests on examples, so it's tough to know for sure if we covered all the edge cases and that we have all the required samples.

Tale of a developer

Let's use a story to demonstrate what I mean.

Notes:

Here, the gray circle will be defining what the client wants, the Xs will visually represent the tests, and the red lines will represent the actual output of the algorithm.

Once upon a time

Client: Could you create an algorithm that draws a circle given a radius? Something like that.

Developer: Of course! Easy peasy.

After some time

Developer: Alright I'm done, it's working.
Client: How do you know if it's working properly? Did you test it?
Developer: Well... I made it run on my machine. If you want, I can add a unit test to prove it.
Client: That would be great.
Developer: Here you go. I added a unit test that makes sure that the point (1,0) is part of the circle.

Client: How do you know that you are not only drawing a horizontal line that passes through that point?
Developer: Ok, here's another test that makes sure that the point (0,1) is part of the circle.

Client: What about coordinates that are not int?
Developer: Sure. I added another test for this.

Client: I don't want just a quarter of a circle. Can you make sure other quadrants are also supported?
Developer: Arrrrg... Sure. I added another test for this. I'm pretty sure we covered everything. We should ship it.

Client: Alright, after all, you are the expert. Let's ship it.

Once in production

Client: What the heck. It looks nothing like the circle I wanted.
Developer: I don't understand. All my tests are passing, which means this should work just fine according to your requirements.

Developer: Maybe if I add another test, I'll find the issue.

The truth is

In fact, with this kind of problem, you'll never be done adding examples. It's because a circle is composed of an infinity of points, which makes it almost impossible to validate with example-based tests.

What do we do then

At this point, I'm pretty sure we all agree that we need a better strategy to test this. That's precisely the kind of problem where Property-Based testing shines. Instead of validating points along the circle, let's try to understand and model the relationship between the input (the radius) and the output (the circle). It turns out that this relationship is well known and defined in the math world.

A circle can be defined as the locus of all points that satisfy the equation
x^2 + y^2 = r^2

It's an excellent property for a test. With only one test, we can validate any point of a circle given a radius. Voilà!

Conclusion

Stay tuned for my next blog post, where we'll solve a real problem with C# and property-based testing.

What's wrong with the current state of unit testing

Miguel Bernard — Sun, 05 Apr 2020 18:03:59 +0000

Unit testing nowadays is pretty much always done using a technique called example-based testing. The method is simple; we run a series of examples against an algorithm, and we validate that we get the expected result.

e.g.

[Fact]
public void GivenThree_WhenCalculate_ThenFiveIsReturned()
{
    // Given
    var input = 3;

    // When
    var result = Calculator.Calculate(input);

    // Then
    Assert.Equal(5, result);
}

This technique is relatively easy to understand and implement. However, it's far from being all rainbows and unicorns. In this article, we'll go over property-based testing, an alternative approach that shines where example-based testing tends to fall short.

The main problems with those tests are that:

They don't scale
They don't help you find edge cases
They are brittle
They don't explain the requirement
They tend to be too specific to the implementation

We all do this

If you are like most of us, mortals, you are probably terrible at using TDD, which means that in the best-case scenario, you add your tests after coding the actual method. Moreover, I bet that you write an empty test shell calling your method under test, take the output, and then write your assertion with that value. Let's be honest. The temptation to reach a better code coverage is just too strong that we have all done it at some point. Then, if you have a bug in your code, you effectively only coded a buggy test.

Property-based testing

Property-based testing is an alternative approach that can deal with all the shortcomings of example-based tests. However, it's by no means a full replacement, as example-based tests are still excellent in many cases. The best approach is to have the right mix of the two in your test suite.

What's a property

When we are talking about properties here, we don't mean properties of C# classes. We use the term property here in a more abstract way. Think of it as a characteristic, trait, attribute, feature, or quality of an algorithm.

Simple definition:

A property is a condition that will always be true for a set of valid inputs.

Testing method

The first thing to do when you want to create a property-based test is to understand and represent the relationship between inputs and outputs. Then, use randomly generated inputs to express deterministic expected results.

Said like that, it seems complex, but you'll see, it's more straightforward than it sounds.

Example

Let's try the technique with a simple example; the Add method. The signature looks like this.

int Add(int number1, int number2);

If you try to test this method with examples, you'll never be done for sure, as there's an infinite combination of numbers that you can add together. So how can we be sure that our Add method is behaving correctly in every situation?

First, try to think of what makes the addition different from the other mathematical operations like subtraction and multiplication.

Properties

The first property is called commutativity. For those of you that don't have a math major, here is a simple definition:

When I add two numbers, the result should not depend on the order of the parameters.

2+3 = 3+2  // True
2-3 != 3-2 // False
2*3 = 3*2  // True

This property is true for addition and multiplication, but not for subtraction. Translated to a test in C#:

// When
var res1 = Add(input1, input2);
var res2 = Add(input2, input1);

// Then
Assert.Equal(res1, res2);

The second property is called associativity.

When I add two numbers, the order of the operations doesn't matter.

(2+3)+4 = 2+(3+4)  // True
(2-3)-4 != 2-(3-4) // False
(2*3)*4 = 2*(3*4)  // True

Again, this property is true for addition and multiplication, but not for subtraction. Translated to a test in C#:

// When
var res1 = Add(Add(input1, input2), input3);
var res2 = Add(input1, Add(input2, input3));

// Then
Assert.Equal(res1, res2);

The third and last property of the addition that makes it unique is called identity.

Adding zero is the same as doing nothing.

2+0 = 2  // True
2-0 = 2  // True
2*0 != 2 // False

At last! We found something to differentiate the addition and the multiplication. Translated to a test in C#:

Assert.Equal(Add(input, 0), input);

Conclusion

With three simple tests, we've been able to cover an infinite number of examples. With these, it's impossible to write an implementation of the Add method that doesn't behave properly. That's precisely the way mathematicians define concepts. They describe the properties. If you look up the definition of the addition, you'll find something interesting. It's defined as commutativity, associativity, and identity.

What are default interface methods in C# 8.0?

Miguel Bernard — Sun, 29 Mar 2020 18:45:11 +0000

Also in this series

All code samples are available on github

Introduction

Last September, Microsoft released a bunch of new features with C# 8.0. One of the major features that they introduced is Default interface methods. In this post, we'll explain what they are, show a few examples and provide guidance to avoid common pitfalls.

What are default interface methods in C# 8.0

Interfaces in C# used to be definitions or contracts without any behavior. This assumption is no longer true. Now, with default interface methods, you can define a default body for any interface methods. When you do so, they mostly behave like virtual methods in an abstract class.

Why are default interface methods important

If they behave like virtual methods in an abstract class, why not simply use an abstract class? There is one big difference that justifies their existence:

In C#, you can only inherit from one base class, but you can implement as many interfaces as you want.

That subtle point makes all the difference when it comes to code reuse, a.k.a. the Don't Repeat Yourself (DRY) principle. That new feature will enable many scenarios that were previously difficult or impossible to achieve.

e.g.

Extend interfaces safely by adding methods with implementations
Create parameterized implementations to provide greater flexibility
Enable implementers to provide a more specific implementation in the form of an override

Real examples of default interface methods

Without default interface methods

internal interface ILogger
{
    void WriteCore(LogLevel level, string message);
    void WriteInformation(string message);
    void WriteWarning(string message);
    void WriteError(string message);
}

internal class ConsoleLogger : ILogger
{
    public void WriteCore(LogLevel level, string message)
    {
        Console.WriteLine($"{level}: {message}");
    }

    public void WriteInformation(string message)
    {
        this.WriteCore(LogLevel.Information, message);
    }

    public void WriteWarning(string message)
    {
        this.WriteCore(LogLevel.Warning, message);
    }

    public void WriteError(string message)
    {
        this.WriteCore(LogLevel.Error, message);
    }
}

internal class TraceLogger : ILogger
{
    public void WriteCore(LogLevel level, string message)
    {
        switch (level)
        {
            case LogLevel.Information:
                Trace.TraceInformation(message);
                break;

            case LogLevel.Warning:
                Trace.TraceWarning(message);
                break;

            case LogLevel.Error:
                Trace.TraceError(message);
                break;
        }
    }

    public void WriteInformation(string message)
    {
        this.WriteCore(LogLevel.Information, message);
    }

    public void WriteWarning(string message)
    {
        this.WriteCore(LogLevel.Warning, message);
    }

    public void WriteError(string message)
    {
        this.WriteCore(LogLevel.Error, message);
    }
}

For every implementation of ILogger we have to repeat this same code block:

public void WriteInformation(string message)
{
    this.WriteCore(LogLevel.Information, message);
}

public void WriteWarning(string message)
{
    this.WriteCore(LogLevel.Warning, message);
}

public void WriteError(string message)
{
    this.WriteCore(LogLevel.Error, message);
}

With default interface methods

internal interface ILogger
{
    void WriteCore(LogLevel level, string message);

    void WriteInformation(string message)
    {
        this.WriteCore(LogLevel.Information, message);
    }

    void WriteWarning(string message)
    {
        this.WriteCore(LogLevel.Warning, message);
    }

    void WriteError(string message)
    {
        this.WriteCore(LogLevel.Error, message);
    }
}

internal class ConsoleLogger : ILogger
{
    public void WriteCore(LogLevel level, string message)
    {
        Console.WriteLine($"{level}: {message}");
    }
}

internal class TraceLogger : ILogger
{
    public void WriteCore(LogLevel level, string message)
    {
        switch (level)
        {
            case LogLevel.Information:
                Trace.TraceInformation(message);
                break;

            case LogLevel.Warning:
                Trace.TraceWarning(message);
                break;

            case LogLevel.Error:
                Trace.TraceError(message);
                break;
        }
    }
}

Haaaaa... this is much better. Now the implementations only contain the relevant code.

Pitfall to avoid

As much as multiple inheritance is awesome, you also need to be very mindful of the dreaded diamond problem.

It's an ambiguity that arises when two classes B and C inherit from A, and class D inherits from both B and C. If there is a method in A that B and C have overridden, and D does not override it, then which version of the method does D inherit: that of B, or that of C?

Closing

There, you have another great tool to put in your toolbox to keep your code clean and maintainable. This approach is also very non-intrusive and won't conflict with existing code, which means it will be easy to slowly adopt it. If you want to hear more on this subject, let me know in the comments.

References

Also in this series

All code samples are available on github

C# 8.0 Indices and ranges

Miguel Bernard — Sun, 22 Mar 2020 01:02:58 +0000

Also in this series

All code samples are available on github

Compared to other languages, C# was way behind in capabilities to handle data efficiently. Well, those days are over now. Microsoft just improved the C# syntax, making it easier for developers to manage data in arrays.

Manipulating Arrays

Have you ever had to manipulate a large set of data in multiple arrays? Of course, you have! And I guess your experience wasn't that good. Compared to other languages, C# was way behind in capabilities to handle data efficiently. Well, those days are over now. Microsoft just improved the C# syntax, making it easier for developers to manage data in arrays.

The new guys

Two new operators have been introduced in C# 8.0 to give you all the power you need:

The 'index from the end' operator: ^, which specifies that an index is relative to the end of the sequence; and
The 'range' operator: .., which specifies the start and end of a range.

Important notes

The ^0 index is the same as sequence[sequence.Length].
- Be careful, sequence[^0] does throw an IndexOutOfRangeException, just as sequence[sequence.Length] does.
For any number n, the index ^n is the same as sequence.Length - n.
For ranges, the start of the range is inclusive, but the end of the range is exclusive.
The range [0..0^] represents the entire sequence, just as [0..sequence.Length] or [..].
A range doesn't need to be completely defined, e.g.
- [..3] -> give me everything from the start of the array to index 3.
- [2..] -> give me everything from index 2 until the end of the array.
- [..] -> give me everything

Examples

Confused? I promise it will all make sense after this. Let's look at a few examples.

private string[] words = new string[]
{
                // index from start    index from end
    "The",      // 0                   ^9
    "quick",    // 1                   ^8
    "brown",    // 2                   ^7
    "fox",      // 3                   ^6
    "jumped",   // 4                   ^5
    "over",     // 5                   ^4
    "the",      // 6                   ^3
    "lazy",     // 7                   ^2
    "dog"       // 8                   ^1
};

As you can see here words[^0] is equal to words[9], which is out of range

Give me some more

Alright, alright. Here are some more ways to use it.

var allWords = words[..]; // contains "The" through "dog".
var firstPhrase = words[..4]; // contains "The" through "fox".
var lastPhrase = words[6..]; // contains "the, "lazy" and "dog".
var lazyDog = words[^2..^0]; // contains "lazy" and "dog".

Index and Range are also .NET types, which means you can create variables of those types, name them for code clarity, and reuse them over and over.

Index the = ^3;
words[the];

Range phrase = 1..4;
words[phrase];

Conclusion

This is super powerful, I can't wait to use it in my projects. This will help so much with reducing noise when using index calculations as well as making the code more maintainable. Thank you Microsoft for this great addition to the language ❤️.

References

https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8

Also in this series

All code samples are available on github

Asynchronous streams in C# 8.0

Miguel Bernard — Sun, 15 Mar 2020 21:56:43 +0000

Also in this series

All code samples are available on github

Asynchronous streams: what are they?

Asynchronous streams are a new way to iterate over a stream of data. It was previously possible to do this synchronously in C#. Now you can do it even if the stream producer is asynchronous! Woot, woot!

What you need to get started

To enable asynchronous streams you need 3 things:

Your method must be async
The return type of your method must be IAsyncEnumerable<T>
The method body must contain at least one yield return

How is this useful?

To show the value of this feature, nothing is better than a real use case.

Use case

As a user, I want to count all the users' names that contain the letter 'n'.

Context

All the users' names are in an SQL database. That database contains millions of users.

The problem

The problem is that this dataset is really big and to calculate the final value we'll need to load it all. If you try to load that many records in memory, you'll probably run out of memory and get an OutOfMemoryException.

Yes, yes... I know, you can compute this value directly with a SQL query. Remember, this is just a simple example to show how the feature works.

Let's look at how it's done with async streams

Producer

public static async IAsyncEnumerable<string> GetAllNames()
{
    var pageIndex = 0;
    const int pageSize = 100;
    var hasMore = false;
    do
    {
        // **** IMPORTANT ****
        // Never EVER do this in a real application. The following code block is
        // not secure and could be subject to an attack known as SQL Injection.
        await using var conn = new SqlConnection("ConnectionString here");
        await using var cmd = new SqlCommand(
            @$"
            SELECT Name
            FROM Users
            ORDER BY Name
            OFFSET {pageIndex * pageSize} ROWS
            FETCH NEXT {pageSize} ROWS ONLY",
            conn);
        await using var reader = await cmd.ExecuteReaderAsync();
        while (reader.Read())
        {
            yield return reader.GetString(0); // This is the "Name" column
        }

        pageIndex++;
        hasMore = reader.HasRows;
    } while (hasMore);
}

Consumer

public async Task<int> CountNamesWithN()
{
    var namesContainingN = 0;
    await foreach (var name in GetAllNames())
    {
        if (name.Contains("n"))
        {
            namesContainingN++;
        }
    }

    return namesContainingN;
}

This is fairly straightforward. From the producer's side, simply use
yield return to stream some data on demand. On the consumer's side, notice the use of await right before the foreach keyword, this is where the magic happens. The rest behaves just like a plain old foreach over a synchronous
IEnumerable, but now it's asynchronous.

At first, this may look like magic, but under the hood, while iterating over the stream of data on the consumer's side, new database queries are issued seamlessly when it needs to load more data. This allows us to nicely stream our data without blowing up the memory, and all the complexity is managed by
IAsyncEnumerable.

Conclusion

With async and await that are now part of our daily development tools, this new feature will simplify a lot of code that you had to do manually. Let me know in the comments how you think this could help you in your day to day work.

Also in this series

All code samples are available on github