Forem: Petr Švihlík

Lessons Learned: Migrating from AppVeyor to GitHub Actions

Petr Švihlík — Fri, 05 Mar 2021 21:26:01 +0000

The third article about GitHub Actions in a row? One must be thinking I am up to something. And they'd be right. Over the past week, I migrated more than 10 repositories from AppVeyor to GitHub Actions.

"Why?" you are asking? My team did extensive research on which CI provider to use next as we started hitting some limits with Travis and we weren't happy with how scattered our build procedures were over multiple CIs. We've been using a different CI provider for nearly every stack. Considering the number of SDKs that we provide (JavaScript, .NET, PHP, Java, Swift, Android, Ruby), it should come as no surprise that we wanted to reduce the maintenance required. And we decided that GitHub Actions are simply the best to fit our needs, closely followed by CircleCI.

Over the course of the migration, I ran into multiple issues. Most of them are pretty easy to solve (at least if you pay attention to the documentation) so the intention of this article is not to give you solutions to simple problems but rather to give you an overview of what types of problems you might face along the way and how much time (based on the knowledge of your codebase) is required to convert a repo to GH Actions.

Linux vs. Windows

By default, all actions run on Linux by default. And if possible, it's a good idea to keep it that way as it's the least resource-demanding option (which you'll also learn if you check out the minute multiplier on the pricing page).
Another reason to stick to Linux is that, if you use Windows for local development, you have verification of cross-platform compatibility. Obviously, you can also use the strategy matrix to run multiple platforms at once.

PRO TIP: For more complex build scripts, it's a good idea to take an iterative approach and first migrate to windows-latest, make sure everything works, and then switch to ubuntu-latest.

Case sensitivity

As you know, on Linux, the file system is case-sensitive. So beware of files like Directory.Build.props. Yeah, Directory.build.props is something completely different.

Directory separator

Some of your code or unit tests may depend on the file system. In that case, make sure you assemble all paths using Path.Combine() or Path.DirectorySeparatorChar.

Quotes

Some shell commands require different syntax than on Windows. A nice example is dotnet nuget push which requires the first parameter to be enclosed in apostrophes on Linux to properly parse the wildcard when publishing multiple NuGet packages.

Running PowerShell

Yes, you can run PowerShell Core on Linux, which is absolutely brilliant if you're migrating from Windows and want to keep just one set of scripts!
The only thing you need to do is to specify pwsh as the type of your shell.

steps:
  - shell: pwsh
    env:
      SUPER_SECRET: ${{ secrets.SuperSecret }}
    run: |
      example-command "$env:SUPER_SECRET"

Just notice the different syntax for accessing environment variables - $SUPER_SECRET in Bash vs $env:SUPER_SECRET in PowerShell and don't mistake the pwsh moniker with powershell which is the non-Core version of PowerShell supported only on Windows.
See the shell platform compatibility matrix here.

WSL is your best friend

If you don't have the Windows Subsystem for Linux installed, or worse, you don't know what it is, I encourage you to head to the docs and be amazed. Being able to execute any Linux command on Windows is, in my opinion, one of the most brilliant engineering achievements of Microsoft in the past years.
Personally, I'm a fan of openSUSE but you can choose from many distros.

HttpClient

The HTTP request and response objects look slightly different when you serialize them on Windows and Linux. If, for instance, your unit tests heavily depend on serialized responses, you might want to address this issue or choose to run the workflow on windows-latest.

`.csproj` patching

AppVeyor offers features such as .csproj files patching and AssemblyInfo patching.
These features are not available in GitHub Actions and the easiest way to overcome this is to use parameters that the .NET CLI offers dotnet build -p:Version=1.2.3 and dotnet pack -p:PackageVersion=1.2.3 or dotnet pack -p:NuspecProperties="Version=1.2.3" if you use a .nuspec file.

This is how it can look like in a real workflow:

- name: Extract version from tag
  id: get_version
  uses: battila7/get-version-action@v2
- name: Build
  run: dotnet build /p:Version="${{ steps.get_version.outputs.version-without-v }}"
- name: Pack
  run: dotnet pack /p:PackageVersion="${{ steps.get_version.outputs.version-without-v }}"

Symbols

If you want to enable Source Link for the consumers of your NuGets, make sure you turn on deterministic builds using dotnet build /p:ContinuousIntegrationBuild=true.

Using custom actions

GitHub Actions use Semantic Versioning so you don't have to include the minor and patch version (codecov/codecov-action@v1.2.3) when referencing them. It's actually a good idea to use just the major version (codecov/codecov-action@v1) because you get any potential fixes free of charge.
You can use actions that are not published on the Marketplace. For instance, you can create your own actions and refer the them by gh_username/repo_with_action@version.

Creating custom actions

Two things I learned about creating custom GitHub Actions that you might find asking yourself too:

It's impossible to nest GitHub Actions (create GitHub Actions containing other GitHub Actions). However, according to this, it seems the topic of templating is about to be addressed this year.
Hosting multiple actions in a single repo IS possible. You then refer to such actions as gh_username/repo_with_actions/action1@version. I'm not sure if it's possible to publish them to the marketplace in this setup though.

Summary

And that's it. There we no other issues during the migration. That was a great surprise for me. Learning GitHub Actions and migrating most of the repos took me a single weekend. Everything was documented properly, I found many examples, and didn't run to any bugs!

Good job GitHub Actions!

Using Environment Protection Rules to Secure Secrets When Building External Forks with pull_request_target 🤐

Petr Švihlík — Fri, 05 Mar 2021 19:13:05 +0000

Building pull requests from forked repositories with GitHub Actions can be a bit tricky when it comes to secrets. As per the documentation, with the exception of GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository. This is to prevent the automatic execution of untrusted code that may be contained within the forked repo.
In other words, we can't use the pull_request trigger if there are secrets that need to be involved in the workflow.

Fortunately, pull_request_target comes to rescue.

...the pull_request_target event behaves in an almost identical way to the pull_request event with the same set of filters and payload. However, instead of running against the workflow and code from the merge commit, the event runs against the workflow and code from the base of the pull request. This means the workflow is running from a trusted source...

Ok, so now we have access to secrets but we're building the wrong code.

Apparently, there are some people who try to overcome this problem with the following code:

#INSECURE
steps:
- uses: actions/checkout@v2
  with:
    ref: ${{ github.event.pull_request.head.sha }} # Check out the code of the PR

This is highly discouraged and rightfully so, as it's insecure if no other security measures are taken.

In GitHub's own article Preventing pwn requests, the author - Jaroslav Lobačevski - suggests using the pull_request_target in combination with a condition checking whether the PR is labeled safe to test. Like this:

    jobs:
      build:
        name: Build and test
        runs-on: ubuntu-latest
        if: contains(github.event.pull_request.labels.*.name, 'safe to test')

This is a perfectly valid approach but I think I may have found a better and more convenient way of preventing unauthorized code execution during the build of forks.

Environment protection rules

Just a couple of months ago, GitHub introduced the Environment protection rules. The main intent of this feature is to protect environments during deployments by applying rules that will pause the execution of a workflow until given conditions are met - e.g. a human approval is given, the certain time elapsed, etc. But it can serve any general purpose. In our case, we'll use it to protect our repository secrets and to prevent the execution of untrusted code.

Protecting the build

Let's start with adding a dummy environment called "Integrate Pull Request" that will require human approval.

Our main build procedure will be associated with this environment and preceded by a dummy workflow step approve that will kick off the workflow and inform the author of the pull request that a review is necessary before proceeding any further.

on:
  pull_request_target:
    branches: [ master ]

jobs:
  approve: # First step
    runs-on: ubuntu-latest

    steps:
    - name: Approve
      run: echo For security reasons, all pull requests need to be approved first before running any automated CI.

  build: # Second step
    runs-on: ubuntu-latest

    needs: [approve] # Require the first step to finish
    environment:
      name: Integrate Pull Request # Our dummy environment
    steps:
    - ...

This way the workflow won't proceed until someone reviews the submitted code and therefore, we can safely check out the ${{ github.event.pull_request.head.sha }} in the next step and build it. So the build is executed using a trusted workflow from the base of the PR and the actual code of the PR.

How it works in practice

Someone submits a pull request and a workflow is triggered and immediately paused
The reviewer or a group of reviewers receive an e-mail notification
The reviewer clicks the link, navigates to the repo, verifies that the submitted PR doesn't contain any unwanted code, and finally gives an approval
The build proceeds
All approvals are audited

A few words on Codecov

While implementing this workflow, I ran into an issue where the Codecov action, similarly to the GitHub Checkout Action, is by default pointed to the PR's Base and needs to be overridden. This can be achieved by:

- name: Codecov
  uses: codecov/codecov-action@v1
  with:
    token: ${{ secrets.CODECOV_TOKEN }}
    override_pr: ${{ github.event.number }}
    override_commit: ${{ github.event.pull_request.head.sha }}

To get rid of the following warning message:

Issue detecting commit SHA. Please run actions/checkout with fetch-depth > 1 or set to 0

make sure to also set fetch-depth of the Checkout action to 2.

steps:
- uses: actions/checkout@v2      
  with:
    fetch-depth: 2

Note: I found an alternative approach using a conditional workflow step and a shell script. But overriding the commit SHA is far easier.

Summary

The advantage of this approach is that you can assign a group of reviewers who'll receive an email notification about the pending workflow and can review and approve it in a single click.
The process is, in my opinion, more transparent thanks to all events being logged and semantically more correct than using labels.

If you want to explore the whole workflow, feel free to check out my project WopiHost.

To learn more about the specifics of pull_request_target head to the documentation.

Reporting .NET 5 XUnit Code Coverage in Codecov via GitHub Actions and Coverlet

Petr Švihlík — Fri, 05 Mar 2021 14:56:10 +0000

You might remember my earlier post where I described how to set up code coverage reporting for .NET Core 3 using AppVeyor + OpenCover + Codecov.

Time has moved on, .NET 5 has arrived, and the best practices have changed, so let's see how to collect code coverage for .NET projects in 2021.

The stack

For some time, I have been intrigued by the coverlet.collector that you can find in any new XUnit .csproj. What it is? How can I use it? Can I replace OpenCover with it?

Also, I started noticing people migrating to GitHub Actions and I didn't have any experience with it and kinda felt that I'm missing the train. This observation has been confirmed recently in the 2020 State of Open Source Code Coverage report where GitHub Actions scored the #1 fastest growing CI used with Codecov (and also the #1 in absolute numbers).

I was thinking I could modernize the stack that I use for .NET projects both at work and for my personal projects. Of the three tools that I've been using previously, I'll be keeping just Codecov (because it's awesome ❤). Let's get to it.

Generating coverage

I like to use the dotnet CLI for as many tasks as possible. The good news is that coverlet offers deep integration with msbuild.
This is how you can generate code coverage for the whole solution:

dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

The command will generate coverage.opencover.xml files in all your test projects (if you run dotnet test on the solution level).

All you need to do to make the code coverage-related /p: switches work is to install also the coverlet.msbuild NuGet package. So your .csprojs will look something like this:

<PackageReference Include="coverlet.collector" Version="3.0.3">
  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
  <PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="coverlet.msbuild" Version="3.0.3">
  <PrivateAssets>all</PrivateAssets>
  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>

Make sure to leave in the /p:CoverletOutputFormat=opencover switch as this is the format that's accepted by Codecov.

For more advanced configuration, I recommend reading the documentation on Coverlet's integration with MSBuild. This is especially useful when your solution or projects don't stick to naming conventions and you need to do some extra filtering, etc.

Integrating into GitHub Actions

Plugging the dotnet test into the CI pipeline is just a matter of creating a new workflow step. No matter if you use the ubuntu-latest or windows-latest runner, it works flawlessly.

- name: Restore dependencies
  run: dotnet restore
- name: Build
  run: dotnet build --no-restore /p:ContinuousIntegrationBuild=true
- name: Test
  run: dotnet test --no-build --verbosity normal /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

Collecting coverage by Codecov

The last step is to upload the test coverage to Codecov. You can find an official Codecov GitHub Action on the marketplace which makes the process super easy - no need to install the Codecov CLI, etc.

Just add the following lines to your workflow:

- name: Codecov
  uses: codecov/codecov-action@v1

Codecov will search through the folder structure of your project and discover your coverage reports based on conventions. You don't even have to include the CODECOV_TOKEN for public repos.

Summary

As you can see, this approach is much more convenient than the one I was using before - downloading and installing tools, running PowerShell scripts, etc.

If you stick to standard .NET naming and folder structure conventions, setting up code coverage is actually very easy nowadays.

To see the whole workflow in action, head to some of our SDK repos. You can also explore how we do releases using GitHub Actions. All code is in the workflows folder.

In my next article, I'll cover how to secure secrets when building pull requests from forks.

Running .NET Core 3 XUnit Code Coverage in AppVeyor Using OpenCover and Codecov

Petr Švihlík — Sun, 01 Dec 2019 21:25:45 +0000

I love AppVeyor. It's my go-to CI/CD service for .NET projects. I use it for both personal and work open-source projects. I also love automation of all kinds so, among other things, I want my test coverage reports to be automatically generated with every build.

Recently, I started upgrading my project portfolio to .NET Core 3.0. With that, I also wanted to level-up my unit tests which are typically XUnit. I didn't want to introduce too much change to my code so, after some research, I decided to stick with the following setup: AppVeyor (which I've already been using) + Codecov (which we use for another project at work) + OpenCover.

Custom PowerShell coverage script 📜

The goal was to create a minimalistic, reusable piece of code (independent of environment settings, tool versions, etc.) that I could use across multiple projects. Here it goes:

coverage.ps1

The script can run locally and in AppVeyor with slightly different configurations. The most tricky part is:

$register = if ($ENV:APPVEYOR -eq $true ) { '-register' } else { '-register:user' }

When I finally made the script run on my local machine, it started failing in AppVeyor. It took me a few hours of digging and debugging to figure out that I can't use -register:user in AppVeyor. Fortunately, debugging is quite easy in AppVeyor as you can RDP to the build worker.

I highly recommend reading OpenCover's documentation. It'll help you understand the syntax of targetArgs, filter and register parameters.

Hooking the script into the CI pipeline 🔗

AppVeyor supports Chocolatey so I'm using cinst to install OpenCover and Codecov CLIs. Then I'm turning off the default test script and replacing it with my own which runs OpenCover to generate the coverage files, and finally, I'm calling Codecov CLI to upload the results.

Cool thing is that if you run codecov -f coverage.xml from AppVeyor you don't need an API key. It just works automagically ✨.

appveyor.yml

The result

I've got this nice sunburst chart indicating which files need attention:

I've also got the Codecov badge with % coverage:

Kentico / kontent-delivery-sdk-net

Kentico Kontent Delivery .NET SDK

Paradigm	Package	Downloads	Compatibility	Documentation
Async			`netstandard2.0`	📖
Reactive			`netstandard2.0`	📖

Summary

The Kentico Kontent Delivery .NET SDK is a client library that lets you easily retrieve content from Kentico Kontent.

Getting started

Installation via Package Manager Console in Visual Studio:

PM> Install-Package Kentico.Kontent.Delivery

Installation via .NET CLI:

> dotnet add package Kentico.Kontent.Delivery

Usage

To retrieve content from your Kentico Kontent projects, you'll be using an implementation of the IDeliveryClient interface. This is the main interface of the SDK. Here's how you can instantiate and use the Delivery client either with DI/IoC or without DI/IoC.

Use dependency injection (ideal for ASP.NET Core web apps)

Startup.cs

public void ConfigureServices(IServiceCollection services)
{
    services.AddDeliveryClient(Configuration);
}

HomeController.cs

public class HomeController
{
    private IDeliveryClient _client;

    public HomeController(IDeliveryClient deliveryClient)
    {
        _client = deliveryClient;
    }
}

In…

View on GitHub

And last but not least, a nice ASCII art in the AppVeyor log here:

So far, I have successfully tested this approach in two repos so I hope it'll work for you too. Any improvements are welcome!

Why I Won't Hire You - 5 Honest Tips to a Better CV

Petr Švihlík — Sun, 25 Aug 2019 22:10:06 +0000

Recently, I had the immense pleasure to go through several hundreds of CVs. I threw 95% of them to trash. Are you wondering how to get in the lucky 5%? Read on and follow the simple tips:

1. Keep it short

Your resume should fit on one page; two pages at maximum. If it's longer, it's not a sign of your overqualification but rather of the lack of communication skills.

2. Leave out irrelevant and obvious experience

Spare your future employer the details about your part-time jobs at school, unless, of course, it's the only experience you have. It really doesn't help if you tell me how delicious burgers you were preparing at McDonald's or how many graves you dug (true story btw).
If you really feel the urge to share such things somewhere, do it on LinkedIn.

If you are applying for a job in IT and include Word or Excel in your resume, congratulations, you've just ruled out yourself… Unless you are an Excel guru who can craft a multi-sheet tax calculation form.

Are you tempted to add "10 years experience in XML"? Perish the thought!

3. Don't try to fool the recipient

Do not try to name every technology you've ever come across…list things that you specialize in. I've seen elaborate tables and charts indicating how many years and months of experience a candidate has in each programming language.

Frankly, the numbers don't tell me much. I need to see you think and code.

Write honestly what kind of projects you've worked on, what was your contribution and please, be brief.

The fewer graphics, the better

Using pie charts, progress bars, or other sophisticated visuals to indicate the level of your skills is a big NO-NO. With the exception of people applying for a graphic designer or BI analyst jobs. Remember, it's a resume - not an infographic.

4. Make it personal

Do not broadcast your CV. By broadcasting a CV you are basically saying "I don't care where I will work. I don't want to excel in anything, I just want your money for whatever I'll be doing."

It shows a lack of motivation and disrespect to people reading your CV. You are wasting everybody's time.

Do proper research and craft a resume to fit the few carefully chosen potential employers.

5. Attach a cover letter when it makes sense

Are you moving to a different tech stack? Are you changing roles or field of expertise completely? Attach a cover letter and explain your motivation.

It's that simple. The less information you include and the fewer companies you address, the higher is the chance of you getting hired! Isn't that great? :D

Forem: Petr Švihlík

Lessons Learned: Migrating from AppVeyor to GitHub Actions

Linux vs. Windows

Case sensitivity

Directory separator

Quotes

Running PowerShell

WSL is your best friend

HttpClient

.csproj patching

Symbols

Using custom actions

Creating custom actions

Summary

Using Environment Protection Rules to Secure Secrets When Building External Forks with pull_request_target 🤐

Environment protection rules

Protecting the build

How it works in practice

A few words on Codecov

Summary

Reporting .NET 5 XUnit Code Coverage in Codecov via GitHub Actions and Coverlet

The stack

Generating coverage

Integrating into GitHub Actions

Collecting coverage by Codecov

Summary

Running .NET Core 3 XUnit Code Coverage in AppVeyor Using OpenCover and Codecov

Custom PowerShell coverage script 📜

coverage.ps1

Hooking the script into the CI pipeline 🔗

appveyor.yml

The result

Kentico / kontent-delivery-sdk-net

Kentico Kontent Delivery .NET SDK

Kentico Kontent Delivery .NET SDK

Summary

Getting started

Usage

Use dependency injection (ideal for ASP.NET Core web apps)

Why I Won't Hire You - 5 Honest Tips to a Better CV

1. Keep it short

2. Leave out irrelevant and obvious experience

3. Don't try to fool the recipient

The fewer graphics, the better

4. Make it personal

5. Attach a cover letter when it makes sense

`.csproj` patching