Forem: andrea

AWS profile chooser

andrea — Sat, 25 Jun 2022 13:37:17 +0000

Super short AWS hint.

Having different AWS profiles, I do not want to pick default one just because I forgot --profile command line option.

Just looking for a way of forcing me having to choose an AWS profile before issuing any command.

My solution was (for bash only):

Removing default profile in ~/.aws/credentials
No command will work from now on without a --profile or an AWS_PROFILE environment variable
Creating a one-liner alias for choosing the desired profile and setting $AWS_PROFILE environment variable.

An alias let me reuse the same shell I'm in.

alias aws-profile-chooser='x() { select prof in $( aws configure list-profiles ) ; do  export AWS_PROFILE="$prof"; break; done }; x'

(you can see the gist here)

This command will show me available profiles, and after choosing the right one, it will set AWS_PROFILE environment variable.

$ aws-profile-choose 
1) default
2) amplify-udemy
3) aws-lambda-deploy
4) test-aws
#?

From now on if I forget to choose a profile, any command will break.

Choosing and setting an AWS profile...is now a breeze! ;)

Using dotnet IConfiguration inside xUnit projects

andrea — Sun, 19 Jun 2022 15:57:32 +0000

In a .NET core XUnit project, configuration functionality is not readily available.

My need is using secrets inside UI tests (i.e. for signing in before testing the web app) without committing them into the repository.

I read different solutions and found many third party libraries but in the end I chose a simple approach, a few lines of code easily adaptable to different needs, still functional both for local development and CI/CD environments.

I created a simple github repo for demonstration purposes.

Basically there are 2 files:

appsettings.json - you can commit it, but without sensitive data. Here you put your config, using placeholders for sensitive data.
appsettings.local.json - you must not commit it, list it in your .gitignore file. Here you put you sensitive data, overriding placeholders from appsettings.json.

Then you have to copy those files to output directory, adding this to your .csproj:

    <ItemGroup Condition="'$(Configuration)' == 'Debug'">
        <None Update="appsettings.json" CopyToOutputDirectory="PreserveNewest" />
    </ItemGroup>
    <ItemGroup Condition="'$(Configuration)' == 'Debug'">
        <None Update="appsettings.local.json" CopyToOutputDirectory="PreserveNewest" />
    </ItemGroup>

Now the code.

Create an utilty class (TestConfigHelper) for setting up IConfiguration in every test (you can adjust this class as needed, an example will be reported in the following).

This is where you configure the Configuration service:

public static IConfigurationRoot GetIConfigurationRoot()
{
    return new ConfigurationBuilder()
        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
        .AddJsonFile("appsettings.local.json", optional: true, reloadOnChange: true)
        .AddEnvironmentVariables()
        .Build();
}

In this code we take out configuration from three places:

first appsetting.json
then appsetting.local.json
in the end from the environment

The order matters. appsetting.local.json will override appsetting.json, and environment variables will override everything.

In this way you can use appsetting.local.json for local development, but also inject sensitive data through environment variables in you CI/CD pipeline.

Your tests should inherit from a BaseTestClass which loads configuration invoking TestConfigHelper::GetIConfigurationRoot method.

And that's all. You can now access the configuration using the object returned from GetIConfigurationRoot in every test.

Bonus #1 - configuration class

I like creating a class representing my application configuration, and bind it to the configuration.
TestConfigHelper has an example of this. The code is like this (where MyAppConfig is the class representing the configuration, stored under "MyApp" root node in the configuration file):

public static MyAppConfig GetMyAppConfiguration()
{
    var configuration = new MyAppConfig();
    GetIConfigurationRoot().Bind("MyApp", configuration);
    return configuration;
}

Bonus #2: user secrets

Moreover, for local development, you can configure user secrets. This requires some more configuration.

Creating and setting a secret

dotnet user-secrets init
dotnet user-secrets set key value

Adding the user secrets key to the .csproj file:

<UserSecretsId>your-secret-id-here</UserSecretsId>

Configuring also Secrets in TestConfigHelper.cs, something like this:

public static IConfigurationRoot GetIConfigurationRoot()
{
    return new ConfigurationBuilder()
        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
        .AddJsonFile("appsettings.local.json", optional: true, reloadOnChange: true)
        .AddUserSecrets("e3dfcccf-0cb3-423a-b302-e3e92e95c128")         // <-- this is the new line !!
        .AddEnvironmentVariables()
        .Build();
}

QA - no excuses!

andrea — Mon, 24 Jan 2022 17:17:19 +0000

I made a very interesting journey into the world of Continuous Integration and QA. The result is a project based on Docker, Jenkins and some really interesting tools I've tried to put together.

Not only do I ask you to download and try it, but above all to recommend new tools to integrate to get the best pipeline ever!

This project is a starter kit for setting up a Jenkins docker installation for CI and QA.

I developed on Windows, with Docker Desktop with WSL 2 engine and VS Code, but with minor adjustments (before all slash notation for command line commands) it should run on Linux too.

Tools integrated

Tool	Usage
Docker	Jenkins is run inside Docker and pipelines run inside docker agents
Jenkins	The automation server
Jenkins Configuration as Code	Jenkins plugin for run an image pre-provisioned / pre-configured
Jenkins plugins	Many Jenkins plugins
SonarQube	Static code analyser
PostgreSQL	Database for storing Sonarqube results
xUnit.Net	Testing framework - used for Unit test, UI tests, Coverage
Selenium	Web browser automation for UI tests
k6	Load testing tool
Grafana	Visualization tool for analyzing load test results
influxdb	Database for storing k6 results used by k6 and Grafana

docker compose up

Project ships with some docker compose files for bringing up main parts of the system:

/docker-compose.yml
Brings up the core infrastructure:
- jenkins
- local docker registry (insecure version)
/sonarqube/docker-compose.yml
Brings up sonarqube infrastructure
- postregs database
- sonarqube server
/selenium/docker-compose.yml
Brings up selenium infrastructure
- selenium-hub
- chrome node
- firefox node
- edge node
/k6/docker-compose.yml
Brings up Grafana infrastructure
- influxdb:1.8
- Grafana with pre-provisioned dashboards

If you want the full system up&running with just one command you can issue (from the root of the repo):

docker compose -f .\docker-compose.yaml -f .\sonarqube\docker-compose.yaml -f .\k6\docker-compose.yaml up

You should see all containers coming up (following picture is taken from VS Code Docker extension):

Note: selenium containers are not run with this command because the grid will be automatically run on demand by the Jenkins before running UI tests.

Note: before running Sonarqube, be sure to meet Docker Host Requirements
Usually you should issue these command:

  sysctl -w vm.max_map_count=524288
  sysctl -w fs.file-max=131072`
  ulimit -n 131072
  ulimit -u 8192

If everything went well, you now have many systems ready:

URL	System
http://localhost:8181	Jenkins
http://localhost:9000	Sonarqube server - User: admin / Pasw: admin
http://localhost:4444	Selenium Grid (only available while UI tests runs)
http://localhost:3000	Grafana

⚠️ Security Notes ⚠️
This stack is not production ready:

For running docker agents from Jenkins, I exposed docker daemon on tcp://localhost:2375 without TLS locally
I relaxed some Jenkins CSP to correctly visualize k6 HTML reports. If you do not need this feature, feel free to remove hudson.model.DirectoryBrowserSupport.CSP directive inside /jenkins/Dockerfile file.
Docker registry is just meant for hosting local built images ( used as Jenkins Pipeline Docker agents) so it runs insecure (without certificates)

Jenkins

The first piece of the puzzle is Jenkins, the automation tool.
A custom docker image is built starting from the official one for minor adjustments:

disable wizard setup
setting up Jenkins Configuration as Code plugin directory
setting up pre-provisioned jobs
modifying Content Security Policy for supporting k6 HTML reports
adding docker and docker-compose to the image. This is useful when using the default agent, for running docker commands like bringing up a selenium grid
pre-provisioning the image with a bunch of plugins

Thanks to the JCasC and Seed Job plugins when we start Jenkins we will find all the job folder containing test jobs alredy in place:

Note: all the Jenkins configuration (JCasC and Jobs) is stored into separate files into /jenkins/casc_configs folder. You can add files here to automatically create new jobs or modify Jenkins configuration

Note: remember to approve scripts before runnig them otherwise they'll fail.
You can also edit and re-save job configuration for approving jobs.

Test job folder "docker":

Here you will find some tests for basic usage.

testing the default agent
testing a docker agent with an image from docker hub
locally building a test image and pushing it lo the local Docker registry
testing a docker agent with a custom local image pulled by the local registry (the one previously built)

All these pieces are used in following pipelines, so it seems a good idea to test the basic blocks

Test job folder "sonarqube":

In this folder you'll find jobs for executing static code analysis.

For this purpose I used Sonarqube. Sonarqube has a client/server architecture:

server - started with the docker compose command previously described
client - the scanner - analyses the project and sends data to the server

In Jenkins, I'm running dotnet core static analysis in two falvours:

with a netcore scanner built on the fly
with a netcore scanner pre-built, pulled from the local registry. This is meant to speed up this phase of the pipeline, using an agent with java and the netcore scanner already installed

I chose the very interesting pitstop netcore project to be analyzed.

Before running Sonarqube jobs, you need to create a Sonarqube token for the scanner to login into the server.

In Sonarqube - create a token for admin user
- http://localhost:9000, user: admin, password: admin
- Go to: Administration -> Security -> Users -> "Administrator" Tokens
In jenkins - config Sonarqube credential.
- JCasC has already provisioned a placeholder Sonarqube token, just update its password with the token just created.
- Go to Manage Jenkins -> Manage Credentials -> sonarqube-token-admin -> Update -> Change Password

These are the jobs in the Sonarqube folder:

01_build_netcore_sonarscanner_image_and_push_to_local_registry: Builds a netcore image with java & sonarscanner already installed and puh it to local registry
02_sonarscanner-in-docker-agent: Example of integrating jenkins - netcore - sonarqube After running this job login to Sonarqube http://localhost:9000 and you'll see:
- Static code analysis
- Code coverage
- Test results
03_sonarscanner-in-docker-agent-netcore-scanner: Like the job before but faster. This job uses a docker agent pre-configured for scanning net core projects.

In Sonarqube, the result should looks like this:

Test job folder "unit tests":

Here you will find a job example for running netcore unit tests with xUnit.

We again use the pitstop netcore project which ships with xUnit unit tests.

This job collects tests results and publishes results using xUnit Jenkins plugin.

This job also collects cove coverage and publishes results using cobertura Jenkins plugin

Results should look like this after running a couple of builds:

Test job folder "selenium":

The job runs xunit + selenium UI tests against a basic grid (3 browsers)

Note: selenium grid is automatically run on demand by the jenkins job and discarded when the pipeline ends.

You can follow tests progress pointing your browser to http://localhost:4444

Test job folder "k6":

k6 is a wonderful tool for performance and load testing.

In this job folder we try two integrations:

a pipeline running k6 load test and publishing HTML report directly in Jenkins.
After job runs, you should see a new report published like this:
a pipeline running k6 load test and publishing k6 metrics to Grafana

If you browse to the dashboard from Grafana ("Dashboards" -> "Browse" -> "k6 Load Testing Results") after you run the load test, you will see the dashboard populated like this:

HTML Results are obtained converting k6 output to HTML.
This is done using the k6-reporter project.
The HTML generated is not compliant with Jenkins Content Security Policy so I modified it for allowing correct visualization.
If you do not need this feature, feel free to remove hudson.model.DirectoryBrowserSupport.CSP directive inside /jenkins/Dockerfile file.

For visualizing k6 results into Grafana I chose influxdb+grafana integration.
Grafana is pre-provisioned with:

a datasource pointing to influxdb (version 1.8 because version 2.0 is not still supported by k6)
a k6 dashboard

Due to this issue it is not still possible to provision dashboard with variables in grafana.
So I modified the k6 dashbord replacing the variable "${DS_K6}" with the provisioned datasource UID: "influxdb-k6-uid".

Conclusions

I always take great inspiration from open-source projects.
This time I hope to be able to reciprocate.

This project is still a "work-in-progress",
please let me know which tools can be integrated to release more and more quality software!

Thank you for reading.