Forem: Edoardo Sanna

Track your coding progress on GitHub with a .NET Worker Service

Edoardo Sanna — Fri, 13 Jan 2023 16:04:59 +0000

Since I started pushing my code to GitHub, I really liked the Top Languages stats in the repository's properties (as well as the beautiful anuraghazra's github-readme-stats cards): I was just amazed that some kind of hidden mechanism was able to tell the language used in almost all the files in the repository.

As I later found out, GitHub uses the Linguist library to measure the amount of lines written in a specific language... which is still pretty magic 🪄.

🎯 Motivation

What surprised me is that there's no historical data about these statistics: I was wondering if I could produce a nice line chart race to show the progress I made along time on the languages I've been working on, similarly to the Top Programming Languages section of the Octoverse, maybe also adding some additional post-elaboration analytics, like

what's the fastest growing language
what's the language I've been working most lately
what's the least used language, or the 'sleeping' ones

⚠️ Warning: "Top Languages" don't indicate any skill level! But it's definitely interesting to use the amount of written lines of code as an approximate metric of progress, even for simple motivational purpose! Wouldn't it be nice to see a line chart showing the amazing progress you've made on all the languages across, let's say, 10-20 years?

👶🏻 Baby steps

Apparently the GitHub REST Repositories API only has a list request (GET /repos/{owner}/{repo}/languages, giving the current list of languages used in a specific repo, but there's no place in GitHub where such historical data are stored.

What if I use this listing feature to get a daily image of the languages used in all my repositories, and persist it on a historical table in a database?

Since I will use it to track my languages' progress, let's call it LangTracker.

Therefore, below I will try to:

set up the starter background service
add the authentication to GitHub API
connect the service to a database
deploy and run it

We will need the following ingredients:

🚛 Background Service

Since I'm learning .NET, I will create it as a background Worker Service.

Let's create the project using the worker template with the .NET CLI:

# Create new project from template
dotnet new worker --name LangTracker

# Create new solution
dotnet new sln --name LangTracker

# Add project to solution
dotnet sln ./LangTracker.sln add ./LangTracker.csproj

The basic Program.cs is quite simple:

// Program.cs
using LangTracker;
IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureServices(services =>
    {
        services.AddHostedService<Worker>();
    })
    .Build();
await host.RunAsync();

It just creates an host and registers the hosted service Worker in the Dependency Injection container. It then builds and starts the background service, and it finally runs the application.

The worker behavior is defined in the Worker.cs class, specifically in a while loop which runs indefinitely until a CancellationToken is hit:

// Worker.cs
namespace LangTracker;
public class Worker : BackgroundService
{
    private readonly ILogger<Worker> _logger;

    public Worker(ILogger<Worker> logger)
    {
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            // Do something
            // ...

            // Wait 1000 ms 
            await Task.Delay(1000, stoppingToken);
        }
    }
}

You may want to expand a little the execution frequency: to have a daily image, I set it to 24 hours:

await Task.Delay(1000*3600*24, stoppingToken); // 👈🏻 <-- or whatever you prefer

🗄️ Database

I expect each language record to be saved in a single table with a DateTime property and a Size property (i.e. size in KB).

For the sake of simplicity, I'll be using PostgreSQL as DBRMS.

To manage the database, we'll first install the required packages:

dotnet add package Microsoft.EntityFrameworkCore
dotnet add package Microsoft.EntityFrameworkCore.Tools
dotnet add package Npgsql.EntityFrameworkCore.PostgreSQL

Let's then get started by modeling our GithubLanguage record with the following class:

// Models/GitHubLanguage.cs
namespace LangTracker.Models
{
    public class GithubLanguage
    {
        [Key]
        public int Id { get; set; }
        public DateTime Date { get; set; }
        public string? Repo { get; set; }
        public string? Language { get; set; }
        public double Size { get; set; }
    }
}

We'll then scaffold a Database Context class with all the details of our database:

// Data/DbContext.cs
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    if (_configuration is not null)
    {
       optionsBuilder.UseNpgsql(_configuration.GetConnectionString("PostgreSQL"));
    }
    base.OnConfiguring(optionsBuilder);
}

And in mapping the model to the database:

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    // Map table names ("Languages")
    modelBuilder.Entity<GithubLanguage>().ToTable("Languages", "dbschema");
    modelBuilder.Entity<GithubLanguage>(entity =>
    {
        entity.HasKey(e => e.Id);
    });
    base.OnModelCreating(modelBuilder);
}

Remember to add the connection string in the appsettings.json file:

  "ConnectionStrings": {
    "PostgreSQL": "User ID=YOUR_POSTGRES_ID;Password=YOUR_POSTGRES_PASSWORD;Host=POSTGRES_HOSTNAME;Port=POSTGRES_PORT;Database=LangTracker;Pooling=true;"
  },

⚠️ PostgreSQL has some issues when saving datetimes instead of UTC timestamps, hence remember to add at the beginning of Program.cs:

// Pgsql-specific configuration for datetimes
AppContext.SetSwitch("Npgsql.EnableLegacyTimestampBehavior", true);
AppContext.SetSwitch("Npgsql.DisableDateTimeInfinityConversions", true);

Finally, let's create the database using Entity Framework Core's migrations, e.g. via .NET CLI:

# Create the first migration, called InitialCreate
dotnet ef migrations add InitialCreate

# Update the database
dotnet ef database update

💻 GitHub client

Let's proceed with getting an API personal access token from GitHub: you may generate one in your settings at "Personal Access Tokens".

Then test it via CLI, asking for instance the details of your user:

# bash
user=$"YOUR-GITHUB-USERNAME"
token=$"YOUR-GITHUB-PERSONAL-ACCESS-TOKEN"
curl -i -u "$user:$token" https://api.github.com/users/$user`

The switch -i will display the HTTP headers: notice the content-type header (it should be application/json) and the x-ratelimit-limit header (maximum amount of available request per hour, it should be 5000 for authenticated requests - usage is tracked by the x-ratelimit-remaining).

In order not to hardcode our GitHub username and token, we will save them as environment variables and pass them to our background service.

So now, back to our Worker, we'll have to inject the configuration :

public Worker(ILogger<Worker> logger, IConfiguration configuration) // 👈🏻 <-- add configuration here
{
    _logger = logger;
    _configuration = configuration; // 👈🏻 <-- and here
}

This loads all the .NET default configuration sources, which includes non-prefixed environment variables and user secrets.

Hence, by adding a new set of environment variables:

GITHUB_LOGIN=[YOUR-GITHUB-USERNAME-HERE]
GITHUB_TOKEN=[YOUR-GITHUB-TOKEN-HERE]

We will be able to retrieve them just by adding to our while loop:

// Read token from env variables
string login = _configuration["GITHUB_LOGIN"];
string token = _configuration["GITHUB_TOKEN"];

To interact with the GitHub API, let's install the Octokit.NET library with dotnet add package Octokit, then instantiate a client as simply as:

// Instantiate Github Client
var client = new GitHubClient(new ProductHeaderValue("lang-tracker"));

// Authenticate with token
var tokenAuth = new Credentials(login, token);
client.Credentials = tokenAuth;

Fetch data

Once the client is authenticated with a specific login on the GitHub API, we should be able to retrieve all the repositories for a specific user:

// Get all repos, public & private, from current login
var repos = await client.Repository.GetAllForCurrent();

Then, loop over the found repositories, saving the current image of all the languages' sizes in KB:

foreach (Repository repo in repos)
{
    // Filter away Company repos & forked repos
    if (repo.Owner.Login == login && !repo.Fork)
    {
    // All languages in current repo
        var langs = await client.Repository.GetAllLanguages(repo.Id);

        foreach (var lang in langs)
        {
            // New language record
        var githubLangRecord = new GithubLanguage
            {
                Date = DateTime.Now,
                Repo = repo.Name,
                Language = lang.Name,
                Size = lang.NumberOfBytes / 1024.00
            };
    }
    }
}

The code above creates a new GithubLanguage object, made of the retrieved language, the corresponding repository, the size in KB and it labels it with the current datetime.

I've neglected the Company's repositories and the forked ones, as I only want to keep track of my personal projects.

Save to database

Now, let's just add the operations required to save each record to the database:

foreach (var lang in langs)
{
    var githubLangRecord = new GithubLanguage
    {
        // ...
    };

    dbContext.Add(githubLangRecord); // 👈🏻 <-- Add entity
}
dbContext.SaveChanges(); // 👈🏻 <-- Save into db

🐋 Deployment

Now, I chose the Worker Service template because it can be easily deployed as a systemd daemon or a Windows service with a simple addition to Program.cs:

// Program.cs
using LangTracker;
IHost host = Host.CreateDefaultBuilder(args)
    .UseWindowsService() // 👈🏻 <-- here
    .UseSystemd() // 👈🏻 <-- and here
    .ConfigureServices(services =>
    {
        services.AddHostedService<Worker>();
    })
    .Build();
await host.RunAsync();

If you want to try it out, remember to install the packages:

dotnet add package Microsoft.Extensions.Hosting
dotnet add package Microsoft.Extensions.Hosting.Systemd
dotnet add package Microsoft.Extensions.Hosting.WindowsServices

Nevertheless, deploying as a containerized service is sooo much more convenient, as it gives me the chance to specify the installation instructions in a single docker-compose.yaml file and run it with a single command.

Therefore, we'll be using

the official ASP.NET Core runtime image as base image, and
the official .NET SDK image as the build image.

The Dockerfile is the standard one automatically built by Visual Studio:

# base image
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app

# build image: restore and build
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
COPY ["LangTracker.csproj", "."]
RUN dotnet restore "./LangTracker.csproj"
COPY . .
WORKDIR "/src/."
RUN dotnet build "LangTracker.csproj" -c Release -o /app/build

# publish image: publish
FROM build AS publish
RUN dotnet publish "LangTracker.csproj" -c Release -o /app/publish

# run image: run
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "LangTracker.dll"]

The next step is to provide a docker-compose.yaml recipe to start two container services:

langtracker_app containing the app (based on the Dockerfile above) and
langtracker_db containing the database (based on the official postgres image):

To ensure connectivity between the two containers, remember to:

Add the two environment variables POSTGRES_PASSWORD and POSTGRES_PORT, respectively the password for the postgres user and the local port to be mapped to the container's default PostgreSQL port 5432
Change your connection string's content in appsettings.json from Host=POSTGRES_HOSTNAME to Host=langtracker_db, taking advantage of the default bridge network driver providing automatic DNS resolution between containers.

Here's the docker-compose file I've been using:

services:
 db:
   image: postgres
   container_name: langtracker_db
   restart: always
   ports:
      - "${POSTGRES_PORT}:5432"
   environment:
     POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
   volumes:
     - postgres-data:/var/lib/postgresql/data
 app:
   container_name: langtracker_app
   build:
     context: .
     dockerfile: ./Dockerfile
   depends_on:
     - db
   environment:
     - GITHUB_LOGIN=${GITHUB_LOGIN}
     - GITHUB_TOKEN=${GITHUB_TOKEN}
volumes:
 postgres-data:

Let's just run it:

docker-compose up -d

Depending on your network, it may require a few minutes to download the base images and run the two containers, but in the end:

Now, when you access the database in your langtracker_db container (even with a simple sudo -u postgres psql -h localhost -p YOUR_POSTGRES_LOCAL_PORT), you may perform a nice aggregating query like:

select "Date"::date as Day, "Language", sum("Size") as TotalSizeKB from "dbschema"."Languages" 
group by "Date"::date, "Language" 
order by Day desc, TotalSizeKB desc;

Giving you the final result of your daily image of languages (below, a couple of days during last summer):

    day     |  Language  |  totalsizekb
------------+------------+----------------
 2022-07-20 | C#         |   166.75390625
 2022-07-20 | PowerShell |   166.16015625
 2022-07-20 | HTML       | 151.7119140625
 2022-07-20 | Python     |    94.58984375
 2022-07-20 | TSQL       |     23.9765625
 2022-07-20 | JavaScript |  23.9599609375
 2022-07-20 | CSS        |    9.599609375
 2022-07-20 | Shell      |     2.54296875
 2022-07-20 | Dockerfile |    2.412109375
 2022-07-19 | PowerShell |   166.16015625
 2022-07-19 | HTML       | 148.2822265625
 2022-07-19 | C#         | 143.2822265625
 2022-07-19 | Python     |    94.58984375
 2022-07-19 | TSQL       |     23.9765625
 2022-07-19 | JavaScript |  23.7392578125
 2022-07-19 | CSS        |   8.5712890625
 2022-07-19 | Shell      |     2.54296875
 2022-07-19 | Dockerfile |     1.56640625

Small and progressive steps in C# and HTML, apparently 😊...

We're good to go! 🎉✨ I'll just run it for a few months, then maybe I'll adjust it to track the language stats only once a month or so, and see the results.

💡 Next steps

The background service does what it's supposed to do, but it's way far from perfect. Here are some random ideas popping up:

🪲 Fixes:

If the only purpose is to show aggregated data, maybe it'd be a good idea to save in the database only the aggregated form
The service works from the moment you start it, but it cannot retrieve past data.
- Maybe we could replace the /repos/{owner}/{repo}/languages call with the /repos/{owner}/{repo}/commits request
  - Even the GitHub tree API looks interesting for this purpose
- Then retrieve all the files in each commit, and invoke the above-mentioned Linguist library to get the list of languages
- This looks like a very time consuming effort, as perfectly described by @ahegiy in its note on a GitLab issue on the same topic.
  - I particularly like the idea of doing this 'on demand' when asked by the user

🌼 Features:

Create a frontend application to show nice line chart (maybe with ChartJS?)
Explore the GitHub GrapQL API instead of the REST API!

Maybe I'll try to deep dive into one of these points in the next month or so, we'll see.

Meanwhile, have a great time and keep coding! 👍🏻

Setting up Windows virtual test environments with Vagrant

Edoardo Sanna — Thu, 19 Aug 2021 07:12:43 +0000

Introduction
Install and setup Vagrant
- Creating a new project
- Your first vagrant up
- Customizing your VM
Provisioning
Conclusion

Introduction: why and what

Recently we've been trying to implement a Continuous Delivery pipeline for our in-house .NET applications suite - you can't imagine how long and painful it is to install and configure a multi-tier application by hand 😓 ! Being a demanding task (a looot of scripting required!), we needed to be able to quickly provision and destroy several test environments (where 'environment', containers not being supported yet, is a whole Windows Virtual Machine).

The main idea was to create an easy, repeatable and trackable procedure. The best practices mentioned by Google's DevOps capabilities persuaded us to use Infrastructure as Code (IaC). IaC is basically code - therefore version-controlled with git - made of creation and configuration scripts to be fed to a proper virtualization engine.

Virtualbox, VMWare and Hyper-V all offer their own CLIs, but we'd like to use a provider-neutral tool. Hence we decided for Hashicorp's Vagrant, an open-source CLI-based VM manager already equipped with providers integrated with the main hypervisors.

This article is a quick start guide to create your first test environments using Vagrant.

Install and setup Vagrant

You can download Vagrant from the download page.

Once the installation is complete, you can verify it by opening your favorite shell (recommending Microsoft's open-source Windows Terminal), and running:



vagrant --version

Creating a new project

Vagrant depends on a specific location on your workstation to store all the configuration files of your project and the virtualized disks: therefore, you need to create your own MyVagrant\ folder and cd into it.

Once you're in it, run



vagrant init --minimal

to initialize your Vagrant project in the local folder. It will create a minimal configuration file called Vagrantfile.

The Vagrantfile is a Ruby-based script containing the instructions Vagrant will use to interact with the various virtualization providers in order to create the described infrastructure.

The minimal Vagrantfile created with the init command is pretty simple:



Vagrant.configure("2") do |config|
  config.vm.box = "base"
end

The Vagrant.configure block sets a specific "configurator" version: all the next instructions will need to be included in this block.

The central command (config.vm.box) specifies the "box" used to create your Virtual Machine. A "box") is a VM image - pretty much similar to the concept of Docker images - exportable from already created VMs or downloadable from the community's Vagrant Cloud repository.

Your first `Vagrant up`

If you try to spin up your VM with the initial Vagrantfile, Vagrant will return an error because it doesn't know any "base" box. You may replace that value with any box name available: in our example, we will try to boot a Windows Server machine, therefore we'll use the publicly available Stefan Scherer's Windows Server 2019 box to have a 180-days trial version of Windows Server 2019 Eval with Desktop.

In case you don't have a local copy of the box, Vagrant will automatically download it from the Vagrant Cloud before creating the VM.

To boot your VM, just run:



vagrant up

Since we didn't specify any virtualization provider, Vagrant will use Virtualbox as default: if you have a different hypervisor, make sure you edit the Vagrantfile with the appropriate settings. Remember then to add the --provider=PROVIDER_NAME (where PROVIDER_NAME is vmware_fusion, hyperv, vmware_desktop, docker, etc) to your vagrant up command.

Vagrant will take some minutes to download the image and boot up the VM, depending on the chosen OS. If your output includes:



Timed out while waiting for the machine to boot. This means that
Vagrant was unable to communicate with the guest machine within
the configured ("config.vm.boot_timeout" value) time period.

The VM has correctly started on Virtualbox nonetheless, but Vagrant won't be able to communicate with it and proceed with the following instructions, such as provisioning software. Therefore, we need to add in the first part of the Vagrantfile the following properties about the communicator (i.e. WinRM) that Vagrant will use to connect to our Windows operating system:



  # Additional parameters to communicate with Windows
  config.vm.boot_timeout = 60
  config.vm.communicator = "winrm"
  config.winrm.port = 55985

By opening the Virtualbox GUI you may find your new VM in the list: notice that, with a Vagrantfile as minimal as the one we provided, the Virtual Machine will have a combination of the provider's and Vagrant's default properties, such as

The default VM name chosen by Vagrant follows the format PROJECT-FOLDER_default_TIMESTAMP; you may want to customize it with a more descriptive hostname, such as web or db. ```cmd

==> default: Setting the name of the VM: MyVagrant_default_1629122422277_92221

* The VM will have a main network adapter ("Adapter 1") connected via NAT to the host:
```cmd


==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat

The network TCP port 22 on the host is mapped to TCP port 2222 on the guest: ```cmd

==> default: Forwarding ports...
default: 22 (guest) => 2222 (host) (adapter 1)


We'll see how to change the properties above in the following section.

## Customizing your VM <a name="customize"></a>

Customizing our newborn Virtual Machine requires a little bit of [additional configuration](https://www.vagrantup.com/docs/providers/virtualbox/configuration) in the Vagrantfile: some settings needs to be added in the  `config.vm` namespace of the Vagrantfile. 

All the code described below must be added inside the `Vagrant.configure("2") do |config|` section.

* Some properties can be changed by configuring the `provider`, such as the [VM's name](https://www.vagrantup.com/docs/providers/virtualbox/configuration#virtual-machine-name) or the [hypervisor's GUI](https://www.vagrantup.com/docs/providers/virtualbox/configuration#gui-vs-headless). Notice that some settings don't have a specific shortcut and they will need the [`customize`](https://www.vagrantup.com/docs/providers/virtualbox/configuration#vboxmanage-customizations) property:
```ruby


# Customization
config.vm.provider "virtualbox" do |v|
  v.name = "my_vm"    # Sets the new VM's name
  v.gui = true        # Enables the hypervisor's GUI
  v.memory = 2048     # Sets the VM's RAM
  v.customize ["modifyvm", :id, "--draganddrop", "hosttoguest"] # Enables drag-and-drop between host and guest
  v.customize ["modifyvm", :id, "--clipboard", "bidirectional"] # Enables a bidirectional clipboard between host and guest
end

Additional ports can be mapped between host and guest with the config.vm.network settings (additional details on networking can be found here) ```ruby

# Customization
config.vm.network :forwarded_port, guest: 80, host: 8080, id: "http" # Map host's port 8080 to guest's port 80


* By default, Vagrant will share (or ["sync"](https://www.vagrantup.com/docs/synced-folders)) your project directory - i.e. the directory with the Vagrantfile - to `/vagrant` on the guest. You can set additional synced folders with:
```ruby


  # Customization
  config.vm.synced_folder "app/wwwroot", "src/"  # Map the host's "src/" folder to the guest's "/app/wwwroot"

And many more!

After each change, remember to run vagrant validate to check the Vagrantfile for any syntax error.

If the validation is successful, run vagrant reload to reboot your VM and apply all the changes.

Provisioning

So now we have a ready VM, but what if we need to also install software or change the configuration during the boot process? This practice is called provisioning and it can be attached to the vagrant up command.

There are several provisioners integrated with Vagrant, such as Ansible, Puppet, Docker and even simply uploading files to the VM.

In our case, let's try to set up a development environment by installing a package manager (in our case Chocolatey) and then an IDE (what's better than Visual Studio Code?), and let's try to do it with the shell provisioner, i.e. uploading and running a shell script.

Create a scripts/ folder in your project and add the InstallChocolatey.ps1 PowerShell script (you may find it on the official Chocolatey install page, please be sure to read Chocolatey's disclaimer about downloading scripts from the Internet ❗) in it:

In the config block of your Vagrantfile, add



  # Provisioning
  config.vm.provision "shell", path: "scripts/InstallChocolatey.ps1"  # Run the external script to install Chocolatey
  config.vm.provision "shell", inline: "choco install vscode --yes"   # Run the inline script to install VSCode via Chocolatey

Now, if you vagrant reload your VM, the provisioning will be performed at the end of the boot. The first shell script (called from an external file whose path is provided) will download and install Chocolatey, the second inline script will force the download and installation of the Visual Studio Code's choco package from the Chocolatey community repository.

Conclusion

In just a few minutes, we were able to spin up a Virtual Machine with our desired operating system and with some software installed (a package manager and an IDE) and now we're ready to write code and start some tests! 🎉 🎊

Plus, our IaC files (i.e. the Vagrantfile and the corresponding PowerShell scripts) now can be stored in a git repository, thus taking advantage of all the source control features: basically we have a versioned infrastructure, repeatable and exportable in any settings.

Here you can find the code mentioned in this post.

See you next time! 👋

Deploying a PowerShell module with Chocolatey

Edoardo Sanna — Thu, 29 Jul 2021 08:49:22 +0000

Introduction
Create a Chocolatey extension
Installing the extension
Making the module available to the current PS session
Using the PS module
Conclusion

Introduction

During my journey towards automation, I had the chance to start using Chocolatey as the Windows package manager. Microsoft just started a new open-source project called winget, although it seems a bit in its early days to be already integrated into a production-like environment.

The deployment and configuration procedure of our .NET applications suite requires a lot of manual steps: we're talking about a complex N-tier architecture made of several IIS-based web applications, Windows services and WPF apps, etc. Any runbook or written guide may vary with the new releases and should be versioned into some automation instructions, hence the need to include the install scripts together with the binaries themselves.

Plus, the long-term maintenance of the applications also need the usage of operational tasks, often trivial (such as copying and pasting, repeating the same configuration for multiple instances of the same application, etc.) or just too repetitive to be performed manually without being intrinsically error-prone.

For both these reasons, we decided to implement an ad-hoc PowerShell module to fully manage our application suite. The PowerShell module should be used by the Chocolatey scripts when installing, upgrading or removing software, and by any operator wanting to perform maintenance tasks on the suite itself.

❗ Note: this post was inspired by Patrick Huber's guide on using PowerShellGet and Chocolatey to manage and deploy PowerShell modules, although some details may vary on the parameters of PowerShellGet.

Creating a Chocolatey extension

Chocolatey extensions are a way to make your customized PowerShell modules available to the Chocolatey scripts. More specifically, Chocolatey installs or upgrades or uninstalls the applications using PowerShell scripts (called ChocolateyInstall, ChocolateyUninstall and ChocolateyBeforeModify - here is a table of various conditions where these scripts are called) that may be extended with additional custom-made public and private functions.

In the following paragraphs we'll assume you already have such a PowerShell module. There are multiple helpful guides on the web on how to build a PowerShell module, although I strongly recommend the Rambling Cookie Monster's post about a proper PowerShell module structure.

Creating an extension is as simple as following the create instructions on the Chocolatey website: you simply need to put all your module's files and folders, except for the root level (which is usually called with the module's name), into a new folder called extension.

This is what your Chocolatey package folder would look like:

your-ps-module
├── extension
│   ├── functions
│   │   ├── public
|   |   |   └── your-public-function.ps1
|   |   └── private
|   |       └── your-private-function.ps1
│   ├── your-ps-module.psd1
|   ├── your-ps-module.psm1
|   ├── ChocolateyInstall.ps1
|   ├── ChocolateyBeforeModify.ps1
|   └── ChocolateyUninstall.ps1
└── your-ps-module.nuspec

Where you may recognize:

the .psm1 module script and the .psd1 module manifest file, containing respectively any instructions to be followed during the module import procedure and the module specifications. Although it's beyond the scope of this article, a typical PowerShell module script would look like:

# Module requirements
#Requires -Version 5.1
#Requires -RunAsAdministrator

# Export public functions
$PublicFunctionsFiles = [System.IO.Path]::Combine($PSScriptRoot, "Functions", "Public", "*.ps1")
Get-ChildItem -Path $PublicFunctionsFiles -Exclude *.tests.ps1, *profile.ps1 | ForEach-Object {
    try {
        . $_.FullName
        Write-Verbose "Exporting public function $($_.FullName)"
    }
    catch {
        Write-Warning "$($_.Exception.Message)"
    }
}

# Export private functions
$PrivateFunctionsFiles = [System.IO.Path]::Combine($PSScriptRoot, "Functions", "Private", "*.ps1")
Get-ChildItem -Path $PrivateFunctionsFiles -Exclude *.tests.ps1, *profile.ps1 | ForEach-Object {
    try {
        . $_.FullName
        Write-Verbose "Exporting private function $($_.FullName)"
    }
    catch {
        Write-Warning "$($_.Exception.Message)"
    }
}

the functions\ folder, hosting multiple public and private functions, exported in the shell session by the main .psm1 module script - as described in the previous point.
the Chocolatey*.ps1 scripts, with the install, upgrade and uninstall instructions when using Chocolatey
the .nuspec file, containing the Chocolatey package's specifications such as title, description, author, version, dependencies, etc. Remember that some items in the spec file are mandatory and needs to be filled: you may find a quick guide here. Below my example (notice the <files> section):

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2015/06/nuspec.xsd">
  <metadata>
    <id>your-ps-module.extension</id>
    <version>0.0.1</version>
    <title>your-ps-module</title>
    <authors>You, my friend</authors>
    <copyright>If any!</copyright>
    <summary>Your marvelous PowerShell module</summary>
    <description>A longer description of your marvelous PowerShell module</description>
    <dependencies>
      <dependency id="any-other-dependency" version="the.package.version"/>
    </dependencies>
  </metadata>
  <files>
    <!-- this section controls what actually gets packaged into the Chocolatey package -->
    <file src="extension\**" target="extension" />
  </files>
</package>

Installing the extension

So, if you have a PowerShell module already, you can quickly pack it into a Chocolatey extension, then put it in your package source and install it with Chocolatey like any other package by using:

choco install your-ps-module --yes --source your-source

Where the --yes options skips any question prompted to the user during the installation, and --source overrides the default source (i.e. the Chocolatey Community Repository) - unless you want to publish in it.
The details of the choco install command are available here.

What's the purpose of installing an extension? By definition, the extension extends the functions that are natively available in Chocolatey, by making available to the Chocolatey CLI the functions in your new module.

As described in the Chocolatey documentation,

Extensions allow you to package up PowerShell functions that you may reuse across packages as a package that other packages can use and depend on. This allows you to use those same functions as if they were part of Chocolatey itself. Chocolatey loads these PowerShell modules up as part of the regular module import load that it does for built-in PowerShell modules.

This means that now you can use your custom functions in any ChocolateyInstall.ps1, ChocolateyBeforeModify.ps1 or ChocolateyUninstall.ps1 script, for any package depending on this extension! 😄

Making the module available to the current PS session

So far, the PowerShell module we developed will only be available as a Chocolatey extension.

Our scenario is a bit different - the requirement is that the custom functions should be available not only to the Chocolatey scripts, but also to the current user opening a PowerShell session and starting any administration task on the applications... basically anything from retrieving data about the applications suite architecture (such as a list of running Windows services, database instance information, or the value of specific parameters in the configuration) to performing actual operational tasks (running a database backup, executing a query, changing a set of configuration values, upgrading applications, etc.).

Some of our public and private functions include Get-SuiteWindowsService, Set-WebApplicationPool, Invoke-SqlQuery and similar stuff, used in the daily administration of the software application suite: an example of a similar module is available here.

What if we want to distribute it to any admin/operator user who would like to take advantage of the custom PowerShell functions?

As described in Microsoft Docs' official documentation about installing PowerShell modules, you just need to add the module in all the paths specified by the $Env:PSModulePath environment variable.

Warning: not all these paths are directly usable, and sometimes they shouldn't be available for installing new modules. In the following example we'll only use the Program Files location ($Env:ProgramFiles\WindowsPowershell\Modules\), to make the module available to all user accounts on the computer.

From the ChocolateyInstall.ps1 script's point of view, the extension folder is reachable by using:

$extensionDir = Split-Path $MyInvocation.MyCommand.Definition

Therefore, we'll copy all the files in the module root folder (excluding for the Chocolatey scripts) with:

# ModuleName
$moduleName = 'your-ps-module'

# Source and destination variables
$destination = "$Env:ProgramFiles\WindowsPowerShell\Modules\$moduleName\"
$ExcludeFiles = $(Get-item "$extensionDir\*Chocolatey*.ps1").Name
$source = "$extensionDir\*"

# Copy-Item results differ depending on if destination exists or not, therefore we're creating it if non-existent
if (-not (Test-Path $destination)) { mkdir $destination | Out-Null }
Get-item $source | 
    Where-Object {$_.Name -notin $ExcludeFiles} | 
    Foreach-Object { Copy-Item $_ -Destination $destination -Force -Recurse }

Then, we'll need to import the module itself in the open shell session, so that we'll be able to immediately use it:

# Import module in current shell session
Import-Module your-ps-module

And finally, we want to make it available to any following session. This can be done by adding the same instructions to the current user's PowerShell $profile file, i.e. the settings that the shell should import at each startup. The following script creates the profile file if necessary and adds the import command:

# Create $profile file if it doesn't exist
if (!(Test-Path $profile)) { New-Item $profile -ItemType File}

# Add automatic Import-module to the $profile
if (!(Get-Content -Path $profile -Filter "your-ps-module")) {
    Write-Verbose "Adding automatic Import-Module in PowerShell profile file."
    Add-Content -Value "Import-Module your-ps-module" -Path $Profile
}

The other users of the computer may run the same script, or add the Import-module your-ps-module manually in their $profile file, in order to have it available when opening any new PowerShell session.

Using the PS module

...in Chocolatey

To recall your custom functions during the install, upgrade or uninstall procedure of other Chocolatey packages, you may just add the extension among your package dependencies. Open the spec file of the package whose ChocolateyInstall.ps1 is going to use those function, and paste in the <dependencies> section:

    <dependency id="your-ps-module.extension" version="0.0.1"/>   <!-- Replace '0.0.1' with your actual module version-->

Installing our package will return the following output:

PS C:\> $source = "Path/to/source"
PS C:\> choco install your-package --source $source --yes
Chocolatey v0.10.16-beta
2 validations performed. 1 success(es), 1 warning(s), and 0 error(s).

Validation Warnings:
 - A pending system reboot request has been detected, however, this is
   being ignored due to the current Chocolatey configuration.  If you
   want to halt when this occurs, then either set the global feature
   using:
     choco feature enable -name=exitOnRebootDetected
   or pass the option --exit-when-reboot-detected.

Installing the following packages:
your-package
By installing you accept licenses for the packages.

your-ps-module.extension v0.0.1
your-ps-module.extension package files install completed. Performing other installation steps.
Installing module your-ps-module to C:\Program Files\WindowsPowerShell\Modules ...
The PowerShell module your-ps-module was successfully installed!
 Installed/updated your-ps-module extensions.
 The install of your-ps-module.extension was successful.
  Software installed to 'C:\ProgramData\chocolatey\extensions\your-ps-module'

your-package v0.0.1
your-package package files install completed. Performing other installation steps.
...

As you may see, according to Chocolatey's logic, the dependencies (i.e. the your-ps-module extension) are installed before proceeding with the installation of the Chocolatey package.

...in a shell session

Once your package is installed, whenever you wish to call your custom functions when operating on your applications suite just invoke them on any shell session (we automatically imported the your-ps-module by using the $profile file: otherwise, just type Import-module your-ps-module).

For instance, you may want to know the list of all the functions included in the module and available to your current session:

Get-Command -Module your-ps-module

Or, since all the public and private functions were exported, you may just call them explicitely:

your-public-function

Conclusion

Since your-ps-module is now deployable as a Chocolatey package, we're able to take advantage of Chocolatey's package management logic, including versioning, dependencies management, repeatable and automatic install/upgrade/uninstall procedures, and so on.

The installed PowerShell module will also be available to any user willing to administrate the software application suite via PowerShell functions, which can be stored in an internal code repository and versioned, thus simplifying the long-term maintenance and operability.

Forem: Edoardo Sanna

Track your coding progress on GitHub with a .NET Worker Service

🎯 Motivation

👶🏻 Baby steps

🚛 Background Service

🗄️ Database

💻 GitHub client

Fetch data

Save to database

🐋 Deployment

💡 Next steps

Setting up Windows virtual test environments with Vagrant

Table of contents

Introduction: why and what

Install and setup Vagrant

Creating a new project

Your first `Vagrant up`

Provisioning

Conclusion

Deploying a PowerShell module with Chocolatey

Table of contents

Introduction

Creating a Chocolatey extension

Installing the extension

Making the module available to the current PS session

Using the PS module

...in Chocolatey

...in a shell session

Conclusion

Forem: Edoardo Sanna

Track your coding progress on GitHub with a .NET Worker Service

🎯 Motivation

👶🏻 Baby steps

🚛 Background Service

🗄️ Database

💻 GitHub client

Fetch data

Save to database

🐋 Deployment

💡 Next steps

Setting up Windows virtual test environments with Vagrant

Table of contents

Introduction: why and what

Install and setup Vagrant

Creating a new project

Your first Vagrant up

Provisioning

Conclusion

Deploying a PowerShell module with Chocolatey

Table of contents

Introduction

Creating a Chocolatey extension

Installing the extension

Making the module available to the current PS session

Using the PS module

...in Chocolatey

...in a shell session

Conclusion

Your first `Vagrant up`