Forem: Jeremy Wells

A headless Raspberry Pi workflow

Jeremy Wells — Tue, 18 Jan 2022 17:24:35 +0000

Photo by Quentin Schulz on Unsplash

It's always great to follow-up on a cool hobby IOT experiment, and I've collected a few Ardunios, Trinkets, and Raspberry Pis that are always waiting around for a free weekend. The Raspberry Pi can be a wonderfully versatile little computer that exposes pins for external controllers and sensors, but it could also be a simple $35 home server, or even a fully functioning Linux desktop computer.

My kids and I have set up our own Minecraft server, and I've made a little weather clock, and experimented with server deployment configurations and a home media streamer as well. Some more elaborate projects I'd like to try some day include a Raspberry Pi Kubernetes cluster or a Grateful Dead Time Machine. Since the whole OS lives on a Micro SD card, it's easy to keep a few and flash different OSes, swapping them out for different projects. In fact, with all of the intricate AWS and Azure learning I've had to do for my job, using a Raspberry Pi to test out provisioning a bare-bones server is a great way to step back and get a handle on what's actually happening under the hood when you're sending your artifacts off to some nebulous data center with a simple command.

One set of steps I always find myself looking up and repeating is for creating a Headless Raspberry Pi - just configure an OS on an SD card and turn it on, no keyboard, no monitor, just the machine.

You can certainly load a Raspberry Pi with its custom OS, a Linux Debian port called Raspbian, hook it up like a desktop computer and do all of the above. But for me, the most convenient way is to set it up on my local WiFi network and connect to it from the workstation where I do everything else, using my personal dev environment and code editors to also load programs onto the Raspberry Pi. That's where a headless, no-GUI Raspberry Pi is ideal.

In this article, I've combined steps from the tutorials I usually reference so that I've got some quick, easy steps to follow every time I put together a new headless Raspberry Pi from scratch.

Prereqisites

For this article, the only thing you'll need is a Raspberry Pi, the model doesn't matter. In fact, most of these steps are more related to provisioning a simple Linux server, so they could be applicable to several other hardware as well. You'll also need some sort of SD card reader on your desktop computer.

Creating your Raspberry Pi image

The first, quickesy way to set up a Raspberry Pi OS is using the Raspberry Pi Imager.

Download a copy
Meanwhile, plug in an SD card to whatever SD card reader you have.
You may want to re-format the SD card so that the OS image is freshly installed. On a Windows computer, I do this by finding the SD card drive in File Explorer under "This PC" and right-clicking it for its formatting options. Choose "FAT" as the file system type, and the default selections for everything else. Note, that any data already on the card will be erased.

Once the Raspberry Pi Imager is done downloading, open it up. You may have to give it administrative permissions on your computer.
You'll have two selections to make, 1. Choose OS and 2. Choose Storage.
Click "Choose OS" and select Raspberry Pi OS (Other), then Raspberry Pi OS Lite (32-bit). This version of the OS doesn't include a Desktop environment so it will only be controllable from the terminal.
Click "Choose Storage" and one of the options should be the drive for the SD card that you formatted earlier.
Lastly, click the "Write" button and the imager will do its work to load a boot-able OS onto the SD card. On my machine, this process takes about 5 minutes.

SSH and networking config

After the imaging is complete, you could eject the SD card and boot up the Raspberry Pi with a monitor, then see a Linux command-line interface. But since this is going to be a headless, peripheral-free setup we need to give the OS some extra information to enable it to connect to a local WiFi network and enable to us to connect it to it from a workstation.

If the SD card drive was ejected after the image was written, plug it back in and open it in File Explorer or Finder:

Add an empty file called ssh in the top directory of the SD card. The file shouldn't have an extension, or any content inside. This tells the OS to accept SSH connections when it boots.
With a text editor create a file called wpa_supplicant.conf. This is where you'll add information so it can connect to your local WiFi network on startup. Paste the following text, substituting your own Wifi name and password:



country=US
ctrl_interface=DIR=/var/run/wpa_supplicat GROUP=netdev
update_config=1

network={
    ssid="<YOUR WIFI NAME>"
    scan_ssid=1
    psk="<YOUR WIFI PASSWORD>"
    key_mgmt=WPA-PSK
}

Also save this file on the SD card in the root directory.

First Login

Your Raspberry Pi OS should be good to go. Plug the card into the Raspberry Pi and turn it on. It should only take a minute or two to boot up. Once it does, open a termninal on your workstation computer and make an SSH connection using the default login:



$ ssh pi@raspberrypi.local

The password will be raspberry. If you are successful, you'll see the command-line interface for the Raspberry Pi and you're ready to run commands.

The set of the commands in the terminal will look something like this:

The first thing I do is change the hostname. That way, if I run multiple Raspberry Pis on the same local Wifi, there's no confusion.

In the Raspberry Pi terminal, type $sudo raspi-config. This will bring up the configuration options.
- Select 1 System Options
- Select S4 Hostname and you'll be prompted to change the hostname. Now, when you SSH into the machine, it will be ssh pi@<NEW HOSTNAME>.local
- Exit the config screen and you can reboot if you'd like.

The next thing is to at the very least, change the password for the pi user. I like to replace the pi user with a brand new admin user. These are just regular Linux commands.

Change the password for pi just to be safe. Type $ passwd and follow the prompts.
Create a new user: $ sudo add user <USERNAME>. It will prompt you for a password and ask for some user profile information.
Then add your new user to the sudo group so that they can run adminstrative commands: ```

$ sudo usermod -aG sudo

 - You can switch to your new user: `$ su - <USERNAME>`
 - Or you can exit and reconnect with your new user:

$ sudo reboot
$ ssh @.local

 - Once your reconnected, you can optionally remove default the `pi` user.

$ sudo deluser -remove-home pi

  - And to safely turn off the Raspberry Pi:

$ sudo shutdown now


Now your Raspberry Pi is set up to run headlessly with and it's at least minimally secure.

## Editing code directly on the Pi

Now that you're Raspberry Pi is provisioned, you're ready to make it do some stuff. You'll need to add some code to run. By default, it should have a working Python environment. But how do you get the code over to the Pi? You could fire up a bare-bones vim or nano editor, but that could be a little painful compared to the well-accessorized code editor on your local workstation. 

One way to get around this is to set up your Raspberry Pi code as a remote Git repository, write the files in the local repository, then push them over. I used to do that, it's not too complicated, and it you have the advantage of having a local repo to store all of your code and projects and have them last while you change out and reconfigure multiple Raspberry Pi setups. [This article]() gives some great instructions on how to go about setting up your own remote Git repository.

An even easier solution if you're a VS Code user is, using a new plugin, you can access a directory on your Pi via SSH and edit files directly onto the Pi in your code editor. 

In VS Code, install the plugin [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh). Then type `CTRL/CMD + SHIFT + P` and type `Remote - SSH Connect to Host`. It will ask for an SSH url (exactly as above), prompt you for a password, then a new VS Code editor will open with access to the Raspberry Pi file system. 

And with all that, you've got an unburdened Raspberri Pi for all your home tinkering.

Dockerizing a Mongo database

Jeremy Wells — Mon, 22 Feb 2021 16:49:50 +0000

Photo by Salomé Guruli on Unsplash

Introduction

I tend to have about a dozen or so little tech projects loaded onto my machine at any given time - little web app ideas that I work on and stop, an occasional hackathon entry, something I've built working through a tutorial, a few experiments. I've gotten to really enjoy working with MongoDB over the past year, and I'll usually add that in as a persistence layer when appropriate. One thing I've realized working off of a local MongoDB development server is all of those databases can accumulate, making it a little scary to experiment with different server configurations and deployment setups for one app. Indeed, one of the challenges during local development is constantly changing and reconfiguring the database as the project evolves, and safely dropping, rebuilding, and reseeding databases.

One way I've found to add some flexibility to the database layer when developing locally is using Docker to run a database server that's dedicated to the project. By leveraging volumes with Docker, we can quickly configure, change, and migrate a database, making it easier to keep database configurations consistent in higher environments. Likewise, this setup can also simplify creating and tearing down pre-populated databases for automated tests.

In this article, I'll show you my Docker setup for container configurations and scripts to run a project's MongoDB server in a Docker container. We'll add some configuration files for a local database, then go on to seed some data, as well as simple commands to tear down and rebuild the database.

Prerequisites and loading the sample project

In order to test out the Docker scripts, you'll need Docker Desktop running on your local development machine. If you want to follow along with the sample repo, you'll also need Node.js installed. Lastly, a database browser tool like MongoDB Compass will be helpful for testing and inspecting the databases that we build.

To demonstrate, I've created a simple MEAN-stack repo and I'll show how to add scripts for containerizing the database. However, you could also easily carry out these steps in any other project that uses Mongo - I'll just be going over the Docker setup in this article. The example repo uses Angular, Express, and Typescript, but you don't need to be familiar with any of those to follow along.

I won't be explaining any MongoDB concepts, but that's not likely to get in the way of understanding the steps below.

I'll give a brief overview of some Docker terms and concepts so that the explanations are clear, but if you're still getting started with Docker, there are far better tutorials out there.

Now for loading the sample project if you're going to use it:

Clone the repo:



$ git clone https://github.com/jsheridanwells/MeanUrls.git
$ cd MeanUrls

The main branch of the repo is the complete example. There is also a branch called start which is the project without the setup. Check out the start branch and install the dependencies:



$ git checkout start
$ npm install

This example web app is an extremely simple URL shortener. There are a few npm scripts included to simplify running the project.

To compile and run the code for the API server:



$ npm run cp:www
$ npm run build:api
$ npm run start:api

You should see the Express app start up as expected, but then after a few seconds, an error will appear in the console because the application can't connect to a database server. We'll fix that below.

In a second terminal window, run the Angular client application:



$ npm run start

After the Node and Angular servers spin up, if you navigate to http://localhost:4200, you should see:

Again, you'll see some error messages in the browser console caused by not having a database available.

Stop both the API server and the Angular server (CTL+C).

Some terms and concepts

Before we set up our Docker configuration, I'd like to go over some concepts that will help clarify the steps we go through.

Docker is a platform for creating virtual environments called containers. I think of a container as a tiny, isolated server that is given just enough resources to run a single, specific application or a task. Docker containers run on a host machine which is a physical piece of hardware, e.g. my computer if I'm developing locally, or the physical server in a public deployment. Docker containers come from images which are basically a blueprint for creating the container: at a minimum, an image will describe an operating system to run in the container, and it will usually also have packages and libraries installed to help an application accomplish a task.

One of the most powerful aspects of Docker is that we can layer different pre-configured images, then add our own configurations and artifacts. Hundreds of different vendors and platforms make official Docker images available for their products - different Linux flavors, different Windows products, MySQL, Postgres, SQL Server - and indeed it's a great way to play with all sorts of different technologies. For our purposes, there is an official MongoDB image that we'll use to run a containerized database.

Once an image is defined, we can run the container on the host. When a container is run, usually we specify ports and network settings that are opened so that data can pass from the host machine to the container, or from container to container. We would also want to pass environment configuration settings to the container, for example a connection string so that the container can connect to an external database. Docker provides us with a variety of commands including one to run the container as a background process ("detached mode"), or with a command to be able to enter the container through the terminal and inspect its contents (docker exec, which we'll use later on).

One advantage of containers is that they are ephemeral - all you need is a command $ docker run <...> and the container will start with all of the components defined by its Docker image. When we don't need it, another command will make the container shut down and disappear. This helps developers to consistently deploy an application to different environments and to be sure it's most likely to run the exact same way. However, any data that is saved inside of the container will also disappear once the container is stopped. To handle persisted data, Docker uses volumes to connect data or files from the host machine to the container when it is running, as well as to write data from the container back to the host. There are two types: The first, bind mounts are directories that the developer manages directly. For example, if I want a directory of scripts available in the container, I can bind my local scripts directory in a project's source code to a directory in the container. The second type are called volumes and these are resources that are managed by Docker and live in a special location in the host machine's file system that we don't necessarily need to access directly. The advantage of using Docker volumes is that it gives us some special commands to create, manage, and dispose of persistent data that is consumed by a Docker container.

Lastly, Docker Compose is a feature that allows you to manage multiple Docker containers. Normally, it's best practice for a Docker container to do only one thing, so for a web application, I might have one container to serve static files publicly, one container to handle the web API, and a third container to run a database server. Docker Compose allows us to create a YAML file, normally named docker-compose.yaml, to define how multiple containers are configured and how they are allowed to communicate with each other.

So, taking all of these concepts together, we are going to use Docker to run our development database server in the following ways:

We'll use the official MongoDB image to create a container to have all of the functionality of a Mongo server without having to go through the cumbersome process of installing it on a development machine.
We'll bind mount a directory of startup scripts to a special directory on the Mongo container called /docker-entrypoint-initdb.d. Any bash scripts or Javascript files in this directory are immediately run when the container is launched. We'll use this to create a MongoDB database and an application user.
We'll define a Docker volume so that the Mongo container can persist data on our host machine. If we ever need to remove all of the saved data and start over, we'll have Docker commands available to do that.
Lastly, we'll use Docker Compose to define the Mongo container as a service. With a docker-compose.yaml file defined, an application service or other services can be added on for a more robust application.

For the rest of this article, I'll go through the steps for accomplishing all of this.

Environment variables

The first thing to do is establish the various values we'll need to build a MongoDB connection string and the names of the database and collection(s) for the project. The project contains the file env.sample with all of the values we'll need for this example:



MONGO_INITDB_ROOT_USERNAME=mongo_root
MONGO_INITDB_ROOT_PASSWORD=mongo_root()
APP_USER=app_user
APP_PWD=app_user()
DB_NAME=MeanUrls
DB_COLLECTION_NAME=Urls
MONGO_HOSTNAME=mongodb
MONGO_PORT=28017

These are the values that Docker will use to configure the server running in a Docker container. This example uses the following:

The root login and password for the database server.
An app user and password. These are the credentials the application itself will use, This helps to limit the application's access to just what it needs.
The name of the database and the name of a collection to start with.
The value MONGO_HOSTNAME is the name to access the database server in the Docker container from our host machine. This host name must match the name we give the container service that we'll set up later.
MONGO_PORT is the port that our application will use to access the database. MongoDB runs on port 27017 by convention; I like to change it to 28017 so I can tell it apart form any local instance of Mongo running on my machine, but that's just a personal preference.

Adjust these variables if necessary, then copy them as your actual .env file.



$ cp .env.sample .env

This is the file that we'll use to pass secret keys (e.g., the root password) to the Docker container and to the application. It can also be recreated for configuring secrets in a deployed application. As such, be careful not to check it into any version control.

If you look at the method buildMongoUrlin the file ./server/data/mongo.ts you can see where these values are used to build the Mongo connection string:



public static buildMongoUrl(config: MongoDbConfig): string {
    return 'mongodb://'
      + `${ config.appUser }:${ encodeURIComponent(config.appPassword) }`
      + `@${ config.hostName }:${ config.mongoPort }`
      + `/${ config.dbName }`;

Some setup scripts

The official MongoDB image lets us start up the database using either a Javascript file or a Bash script. We'll write some Javascript that will create the app user and the app database

We'll create some directories to hold our Mongo scripts:



$ mkdir -p ./scripts/mongo/init

And a couple of files:



$ touch ./scripts/mongo/init/{.dbshell,mongoInit.js}

The .dbshell file is just a blank placeholder file that's used to create a Linux user to run the scripts in the container. mongoInit.js is where we'll create our user and database by pasting the following:



// use shell command to save env variable to a temporary file, then return the contents.
// source: https://stackoverflow.com/questions/39444467/how-to-pass-environment-variable-to-mongo-script/60192758#60192758
function getEnvVariable(envVar, defaultValue) {
  var command = run("sh", "-c", `printenv --null ${ envVar } >/tmp/${ envVar }.txt`);
  // note: 'printenv --null' prevents adding line break to value
  if (command != 0) return defaultValue;
  return cat(`/tmp/${ envVar }.txt`)
}

// create application user and collection
var dbUser = getEnvVariable('APP_USER', 'app_user');
var dbPwd = getEnvVariable('APP_PWD', 'app_user()');
var dbName = getEnvVariable('DB_NAME', 'MeanUrls');
var dbCollectionName = getEnvVariable('DB_COLLECTION_NAME', 'Urls');
db = db.getSiblingDB(dbName);
db.createUser({
  'user': dbUser,
  'pwd': dbPwd,
  'roles': [
    {
      'role': 'dbOwner',
      'db': getEnvVariable('DB_NAME', 'MeanUrls')
    }
  ]
});

db.createCollection(dbCollectionName);

The code in this script runs in the Mongo shell, so you'll notice difference Mongo-specific objects are provided: e.g., db, run, cat.

The first function uses the mongo shell run command to retrieve environment variables (and I have to credit this Stack Overflow hack for help. Please let me know if there's a better way to do this).

Then, we create the database with db.getSiblingDB, a user with db.createUser, and a collection with db.createCollection.

Note that this script will only run when the container is initially started and bound to a Docker volume on the host, but not on any subsequent container startups bound to the same volume. This prevents any duplication or data collision when the container is shut down and restarted.

The next step is optional, but I want to also add some seed data so that I can test out the application right away. We'll create a new directory:



$ mkdir ./scripts/mongo/seed

And we'll download some data that I've already generated for the project using Mockaroo.



$ curl https://raw.githubusercontent.com/jsheridanwells/MeanUrls/main/scripts/mongo/seed/MOCK_DATA.json -o ./scripts/mongo/seed/MOCK_DATA.json

And we'll create a bash script that will be available in the container to seed the database if we choose to:



$ touch ./scripts/mongo/seed/mongo_seed.sh

Add these contents to the file:



#!/bin/bash
if [ -f "/MOCK_DATA.json" ]; then
  FILE="/MOCK_DATA.json"
elif [ -f "./MOCK_DATA.json" ]; then
  FILE="./MOCK_DATA.json"
else
  echo "Mock data file not found. Make sure container has a MOCK_DATA.json file for this script to work"
  exit 1
fi

mongoimport --host $MONGO_HOSTNAME \
  --authenticationDatabase $DB_NAME \
  --username $APP_USER --password $APP_PWD \
  --db $DB_NAME \
  --collection $DB_COLLECTION_NAME \
  --file $FILE --jsonArray

This script checks for a MOCK_DATA.json file, and if found, runs the mongoimport command with the environment settings we created earlier.

Creating our Mongo service

We've got our keys for creating our database and app user, and scripts to tell Docker how to set them up when creating the container. Now we need to run it.j

Normally, a Docker container is configured with a Dockerfile that is responsible for selecting a base image, building and copying artifacts from the project's source code, running any special commands or configurations, and a lot more, then the result is a new Docker image.

In our case, we're not changing any behaviors of the official MongoDB Docker image, or adding our own code, so we'll run the MongoDB container as-is. We'll use Docker Compose to configure and run the container as a service that's then available to our application running on the host machine.

Docker Compose is configured using a YAML file, conventionally named docker-compose.yaml. Let's create one at the root of the project:



$ touch ./docker-compose.yaml

Now paste the following:



version: "3"
services:
  mongodb:
    container_name: mean_urls_db
    image: mongo:latest
    volumes:
      - ./scripts/mongo/init/:/docker-entrypoint-initdb.d
      - ./scripts/mongo/init:/home/mongodb
      - ./scripts/mongo/seed/:/home/mongodb/seed
      - mean_urls_data:/data/db
    ports:
      - "28017:27017"
    environment:
      - MONGO_INITDB_ROOT_USERNAME=$MONGO_INITDB_ROOT_USERNAME
      - MONGO_INITDB_ROOT_PASSWORD=$MONGO_INITDB_ROOT_PASSWORD
      - APP_USER=$APP_USER
      - APP_PWD=$APP_PWD
      - DB_NAME=$DB_NAME
      - DB_COLLECTION_NAME=$DB_COLLECTION_NAME
      - MONGO_HOSTNAME=$MONGO_HOSTNAME
volumes:
  mean_urls_data:

Here's a walkthrough of what the yaml file is doing:

version: "3" is the version of Docker Compose we're using, the latest as I write this.
We've named the service mongodb. This will become the host name of the MongoDB server and must match MONGO_HOSTNAME in the .env file.
We'll build a container from the official MongoDB image: image: mongo:latest.
In the volumes list there are a few things going on: > - First, note that for each item, the syntax is such that the value to the left of the colon (:), pertains to the host machine, while to the right belongs to the container (./host-directory:/container-directory). > - We're mounting ./scripts/mongo/init to the directory on the container called /docker-entrypoint-initdb.d. As mentioned earlier, this is a special directory that runs any Javascript or Bash scripts inside of it when the container is first created. > - We'll also mount init to /home/mongodb on the container. This creates the Linux user that can run the scripts with the Mongo shell. > - The seed scripts will be copied over to /home/mongodb/seed. That way they're available to seed the database when the container is running if we want. > - The last item - mean_urls_data:/data/db will bind the Mongo database files as a Docker volume. This allows data to persist when the container is stopped and restarted. It lets us easily wipe out all of the Mongo data and start over when necessary.
Moving on, in ports, we're connecting our local 28017 port to the conventional Mongo port 27017 on the container.
The environment list will set all of the environment variables from .env in the container.
Lastly, volumes: mean_urls_data creates and names the Docker volume on the host machine and must match the last item of the volumes list set when defining the mongodb service.

Getting it up and running

It's time to test out the database now.

From the root of the project, use the docker-compose command to run the container:



$ docker-compose up

By default, docker-compose looks for a file in the working directory called docker-compose.yaml. If there are no errors, you'll see a mess of log entries in your terminal. Once it's settled, check for these lines near the top:



Creating network "meanurls_default" with the default driver
Creating volume "meanurls_mean_urls_data" with default driver
Creating mean_urls_db ... done

This shows that the network and the volume were created correctly.

Scanning through the logs, check for this entry:



mean_urls_db | Successfully added user: {
mean_urls_db |  "user" : "app_user",
mean_urls_db |  "roles" : [
mean_urls_db |          {
mean_urls_db |                  "role" : "dbOwner",
mean_urls_db |                  "db" : "MeanUrls"
mean_urls_db |          }
mean_urls_db |  ]
mean_urls_db | }

This helps confirm that mongoInit.js was run and that a user was created for the application.

Using MongoDB Compass, the mongo shell, or your database browser of choice, connect to the database using this connection string:



mongodb://app_user:app_user()@localhost:28017/MeanUrls

You should see that the MeanUrls database and the Urls collection were created.

Run the application again as shown above and navigate to http://localhost:4200. The application should be running in the browser without receving any errors from the API. Lastly, enter a URL to shorten in the application. The object should be saved and everything working as expected:

Seeding the database

The next step is optional, but there are times when it's helpful to start the database with some boilerplate data to test the components more authentically, or to see how the application handles a more realistic volume of data. The docker exec command lets us enter the Docker container from the terminal, navigate the file system on the container, and execute bash commands.

With the container running as in the previous step, execute the following command:



$ docker exec -it mean_urls_db bash

This will open a bash terminal running inside of the container.

If your container is configured as above, navigate to the directory where the seed scripts were mounted:



$ cd /home/mongodb/seed

Run the seed script:



$ bash mongo_seed.sh

If your output is similar to...



2021-02-17T20:46:39.307+0000    connected to: mongodb://localhost/
2021-02-17T20:46:39.309+0000    50 document(s) imported successfully. 0 document(s) failed to import.

... then you should be good to go.

Now, refresh the application in your web browser, or inspect the database with MongoDB Compass, and you should see that 50 fake documents have been added to the Urls collection.

To exit the container, type exit.

Cleanup and some helpful scripts

If I want to shut down the MongoDB server, I can run:



$ docker-compose down

This will stop and remove the Mongo container, but the volume where data is stored will persist when we start up the container again.

To remove the volume run:



$ docker volume rm meanurls_mean_urls_data

The next time I start the MongoDB container, the initialization scripts will run again.

If you're living dangerously, you can also put the two together:



$ docker-compose down -v

Lastly, in projects, I like to include all of the container startup and teardown scripts in the package.json for convenience:



{

  "...": "...",

  "scripts": {

    "mongo": "docker-compose up",

    "mongo:up": "docker-compose up",

    "mongo:down": "docker-compose down",

    "mongo:clean": "docker volume rm meanurls_mean_urls_data",

  }

Conclusion

And with that, we've created a MongoDB server that communicates with a Node.js application from a container. This gives our application a dedicated MongoDB instance that's easier to set up and tear down. We've also got a good foundation for getting configurations right in local development before replicating those consistently in higher environments. Lastly, we can re-configure and evolve the database server as the application changes without affecting other applications that would otherwise share the database server.

A Modern MEAN-stack with Angular and Typescript

Jeremy Wells — Thu, 21 Jan 2021 23:03:50 +0000

Photo by Tolu Olubode on Unsplash

Introduction

One of the advantages I love about using Angular is that the framework is truly "batteries included". From the application architecture, to configuration, to third-party libraries, to testing setup, to extra compilation tools, it's a set of really smart decisions that help get a fully featured browser application running quickly. For tasks like setting up scripts, compiling Typescript, CSS preprocessing, Webpack, and testing, the Angular CLI can save lots of tedious configuration.

Likewise, for independent projects, Node.js and Express can be great choices as they tend to be easy to deploy to a variety of platforms.

Combined with MongoDb for data persistence, these tools have long been known as the MEAN Stack (with AngularJS being the original A in MEAN), with "Javascript everywhere" being the unifying idea.

Over a few months of trying out different Angular-Express-With-Typescript setups, I've come up with a way to set up these kinds of projects using the Angular CLI as the starting point. After creating an Angular-based workspace, I can add an Express.js application written in Typescript, and configure the two to transpile Typescript files to a single deployable Node.js web app. With this set up, we can also use Nodemon and Concurrently to create a convenient dev workflow similar to what the Angular CLI provides with the ng serve --open command.

This article will be the first in a two-part series. In part one, we'll go through setting up and configuring Angular and Express. Then we'll use Nodemon, Concurrently, and live-server to handle compiling and refreshing the application when changes are made to the code base.

In another tutorial, I'll show how we can use Docker to serve the development database, a setup that's a little more flexible and convenient than running MongoDb locally (although that's perfectly fine too).

Prerequisites

This tutorial will assume at least some familiarity with Angular and the Angular CLI, Typescript and its CLI, and Express.

The following tools should be installed before starting (the links are to their respective "Getting Started" pages).

Node.js - I'm using version 14.15.1 as I write this.
Angular CLI - I'm using version 11. If you're a few version behind, these steps should still work.
Typescript - I'm using version 4.

We'll also install a couple of NPM tools globally, but I'll explain those as we come to them.

Set up an empty Angular project

The first step will be to use the Angular CLI to set up an Angular workspace so that we can take advantage of all of the framework's smart default configurations from the start.

Normally, we would use the ng new command to create the project which would scaffold a few application components and tests to get us going. However, in our first step, we're going to set up the workspace and the application separately.

Start with this command:

$ ng new NameApp --create-application=false --new-project-root=. --skip-install=true

Notice some new flags:

--create-application=false just sets up the workspace.
--new-project-root=. will help any configuration files (tsconfig.json, angular.json) find all of the locations in our project with minimal headaches.
--skip-install=true skips installing the node_modules packages. Since Angular comes with a ton of dependencies, we'll do our npm install all at once later on. This makes it easier to delete the entire project and start over if something doesn't turn out right.

Now we'll cd into the project directory and create the client application:

$ cd NameApp
$ ng generate application client --skip-install=true

You'll be prompted to select if you want to add routing to the project, and your preferred CSS library.

We just created the usual, scaffolded Angular directories in a client directory. Now we can keep separate directories for our client-side and server-side code.

Note that if you add --dry-run or -d to the end of both of these command, this runs the command without actually adding new files, allowing you to see how the project will be layed out first, which is very convenient for experimenting with unconventional setups.

If all the directories look correct, run npm install to install all of the Node.js packages.

With everything installed, run ng serve --open and test that the default application is working in a browser as expected.

Install some packages to support Express

Now that a basic browser application is working, we'll create an Express.js application that we'll write in Typescript. Everything will live in a directory called server and we'll structure it in a similar setup to a typical Javascript Express application.

In the project's root folder, we'll install the main dependencies we'll need:

$ npm install express
$ npm install --save-dev @types/express

We've added Express as our API server, and we've added the Typescript type definitions for Express.

Next, we'll set up some files and directories for the server-side code:

$ mkdir -p server/bin
$ touch server/app.ts
$ touch server/bin/www

All of the server-side code will live in server. The bin/www file is a typical entry point file for an Express app, and app.ts will be the root application file that will assemble all of the middleware for the API.

In an editor, open bin/www and paste the following:

#!/usr/bin/env node

/**
 * Module dependencies.
 */

const app = require('../app').default();
const debug = require('debug')('NameApp:server');
const http = require('http');
/**
 * Get port from environment and store in Express.
 */

const port = normalizePort(process.env.PORT || '3000');
app.set('port', port);

/**
 * Create HTTP server.
 */

const server = http.createServer(app);

/**
 * Listen on provided port, on all network interfaces.
 */

server.listen(port, () => console.log(`Application is listening on port ${ port }`));
server.on('error', onError);
server.on('listening', onListening);

/**
 * Normalize a port into a number, string, or false.
 */

function normalizePort(val) {
  const port = parseInt(val, 10);

  if (isNaN(port)) {
    // named pipe
    return val;
  }

  if (port >= 0) {
    // port number
    return port;
  }

  return false;
}

/**
 * Event listener for HTTP server "error" event.
 */

function onError(error) {
  if (error.syscall !== 'listen') {
    throw error;
  }

  const bind = typeof port === 'string'
    ? 'Pipe ' + port
    : 'Port ' + port;

  // handle specific listen errors with friendly messages
  switch (error.code) {
    case 'EACCES':
      console.error(bind + ' requires elevated privileges');
      process.exit(1);
      break;
    case 'EADDRINUSE':
      console.error(bind + ' is already in use');
      process.exit(1);
      break;
    default:
      throw error;
  }
}

/**
 * Event listener for HTTP server "listening" event.
 */

function onListening() {
  const addr = server.address();
  const bind = typeof addr === 'string'
    ? 'pipe ' + addr
    : 'port ' + addr.port;
  debug('Listening on ' + bind);
}

This is almost exactly what gets generated when scaffolding a typical Express application with javascript and it basically pulls in our application code to create an HTTP server in the Node.js runtime. This file will most likely stay unchanged throughout any project.

Next, open app.ts and we'll paste in a very minimal Express setup:

import * as express from 'express';
import { Express, Request, Response } from 'express';

export default function createApp(): Express {
    const app = express();
    app.get('/api/:name', async (req: Request, res: Response) => {
        const name = req.params.name;
        const greeting = { greeting: `Hello, ${ name }` };
        res.send(greeting);
    });
    return app;
}

Not much going on here. So far, what we can expect is that when we run the server, a GET request to /api/Artie will return Hello, Artie.

Now we need to process the Typescript file and output them as Javascript that the Node.js runtime can read.

Setting up a build artifacts directory

Our intention is to output all of the Javascript code to a ./dist directory in the root of the project. This is the directory that Angular normally compiles all of its browser code to, and it's already in the .gitignore that Angular created. We'll modify ./dist to instead end up with a unified deployment package once the Angular and Express code is all processed. At the end, all of our Typescript scripts will be output to the ./dist directory as follows:

/dist
 - /api
    - /bin/www
    - app.js
    # (... everything we made with Express ...)
 - /public
    # (... everything we made with Angular)

With the Angular code, we only need to make a small change to the angular.json file.

{
  // ...
  "projects": {
    "client": {
      "architect": {
        "build": {
          "options": {
            "outputPath": "dist/public",  // <--- CHANGE THIS PATH

The rest of the Angular configurations should be okay as they are.

For our server-side code, we'll add a separate tsconfig file in the ./server directory:

$ touch ./server/tsconfig.api.json

Add these values:

{
    "compilerOptions": {
      "baseUrl": "../",
      "module": "CommonJS",
      "resolveJsonModule": false,
      "esModuleInterop": false,
      "target": "ESNext",
      "outDir": "../dist/api",
      "sourceMap": true,
      "types": [
        "node"
      ],
      "experimentalDecorators": true,
      "emitDecoratorMetadata": true
    }
  }

The important settings to note is that the baseUrl value is still the root of the project so that it draws from the same node_modules library. Then outDir is set to the same ./dist directory where all of the compiled output goes.

With our first app.ts iteration written, and our configurations updated, we now need to transpile the file to Javascript and make sure that the bin/www file can load it. We'll do the following test:

Create a ./dist directory with an api/bin directory at the root of the project if there isn't one there already. Then copy the www file:

$ mkdir -p ./dist/api/bin
$ cp ./server/bin/www ./dist/api/bin

Now, with the Typescript compiler, we'll turn app.ts into Javascript output:

$ tsc -p ./server/tsconfig.api.json

Double check that it has been created ./dist/api/app.js.

Run the www with Node to see if the Express.js server runs and accepts a test GET request as expected:

$ node ./dist/api/bin/www

In another terminal:

$ curl http://localhost:3000/api/Artie

And we should see {"greeting" : "Hello, Artie"} returned to the terminal.

Setting up scripts for unified client and server-side development

At this point you should have gotten signs of life from both the Angular and the Express apps. Now we need to combine the two so that we can serve the entire application on a single port. To do this, we'll set up Angular to build to the ./dist/public, then set the Express server to serve the static files from that directory.

First, we'll set Express to serve static files from ./dist/public. Here is app.ts with those lines added:

import * as express from 'express';
import * as path from 'path';  // < -- add this
import { Express, Request, Response } from 'express';

export default function createApp(): Express {
    const app = express();
    const clientDir = path.join(__dirname, '../public');  // <-- add this 
    app.use(express.static(clientDir));                   // <-- and add this
    app.get('/api/:name', async (req: Request, res: Response) => {
        const name = req.params.name;
        const greeting = { greeting: `Hello, ${ name }` };
        res.send(greeting);
    });
    return app;
}

Note that the location of public is relative to the compiled app.js when it's in the ./dist/api directory.

Now, the following commands will 1) Build the static assets from Angular, 2) transpile the changes added to app.ts, and 3) serve the entire application from Express as before:

$ ng build
$ tsc -p ./server/tsconfig.api.json
$ node ./dist/api/bin/www

Navigate to http://localhost:3000 and you should see the default Angular page again. Make a GET request to http://localhost:3000/api/Oliver and you should get {"greeting" : "Hello, Oliver"} as the response.

Shut the server down and proceed.

With Express serving both the API and the static browser files, we'll add some scripts to the package.json file to make all of these steps more seamless and so the server can listen to file changes.

First, we'll need to install the following npm tools globally:

npm install --global nodemon - Nodemon is a development utility that will restart our API server whenever changes to the code are detected.
npm install --global concurrently - Concurrently is a tool that can run multiple npm processes in the same terminal, and it provides several options to deal with any of the processes failing. We'll use concurrently to watch and rebuild the client and server side code at the same time.

Now add the following scripts to package.json:

{
    // ...
    "scripts": {
        //...
        "clean": "rm -rf ./dist/api && rm -rf ./dist/public/",
        "cp:www": "mkdir -p ./dist/api/bin && cp ./server/bin/www ./dist/api/bin/",
        "dev": "concurrently -k \"tsc -p ./server/tsconfig.api.json -w\" \"cd ./dist/api && nodemon -r ./bin/www --watch\" \"ng build --watch\""
    }
}

Here's what they do:

$ npm run clean - will clean out the directories where the compiled output goes in case we need to make a fresh start.
$ npm run cp:www - This copies ./server/bin/www to its proper location.
$ npm run dev - Using Concurrently, we compile Typescript files every time there are changes, run the files in Node and watch for changes with Nodemon, then watch for changes to the Angular files and build those accordingly.

Run each of those scripts in order and you should get the same results as above when making requests to http://localhost:3000.

Bonus: Refresh the browser when client-side code changes

Unfortunately, one of the tradeoffs to using the above scripts instead of Angular's ng serve is that we'd have to manually refresh the browser each time we make changes. Configuring Express with a couple of npm packages - livereload and connect-livereload - can accomplish this in our current setup.

Install the packages as development dependencies:

$ npm install --save-dev livereload connect-livereload

In app.ts, import the libraries:

import * as livereload from 'livereload';
import * as connectLivereload from 'connect-livereload';

And underneath the line where the client directory is declared, paste the following:

  const app = express();
  const clientDir = path.join(__dirname, '../public');

  // In development, refresh Angular on save just like ng serve does
  let livereloadServer: any;
  if (process.env.NODE_ENV !== 'production') {
      livereloadServer = livereload.createServer();
      livereloadServer.watch(clientDir);
      app.use(connectLivereload());
      livereloadServer.once('connection', () => {
        setTimeout(() => livereloadServer.refresh('/'), 100);
      });
  }

In the code, we're creating a livereload server and setting it to listen to changes to the client directory. Then, connect-livereload provides middleware to the Express app that injects a bit of temporary code in our static files that makes the browser aware of any changes and refreshes accordingly.

Lastly, if your linter is giving you grief about not having type declarations for livereload and connect-livereload, you can add a type declarations file in the server directory:

$ touch ./server/decs.d.ts

And paste the following:

declare module 'livereload';
declare module 'connect-livereload';

I got this configuration mainly from this article which goes into much more detail about what is actually happening.

Putting it all together

As one final proof-of-life, let's get our Angular application to talk to the Express back end.

In the Angular app, open app.module.ts and paste the all of the following:

import { BrowserModule } from '@angular/platform-browser';
import { NgModule } from '@angular/core';
import { FormsModule } from '@angular/forms';
import { HttpClientModule } from '@angular/common/http';

import { AppComponent } from './app.component';

@NgModule({
  declarations: [
    AppComponent
  ],
  imports: [
    BrowserModule,
    FormsModule,
    HttpClientModule
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

The only difference is that we've added the FormsModule and the HttpClientModule.

Next open app.component.ts and replace the entire file with:

import { Component } from '@angular/core';
import { HttpClient } from '@angular/common/http';

@Component({
  selector: 'app-root',
  template: `
    <div class="app-container" style="width:20rem; margin: 2rem auto;">
      <div class="form-group" >
        <label for="name-input">Enter a name:</label>
        <input class="form-control" id="name-input" required [(ngModel)]="nameInput">
        <button class="btn btn-primary"(click)="greetMe()">Greet Me</button>
      </div>
      <div class="name-display">
        <p *ngIf="responseDisplay && responseDisplay.length > 0">
          {{ responseDisplay }}
        </p>
      </div>
    </div>
  `
})
export class AppComponent {

  constructor(private http: HttpClient) {  }

  nameInput: string = '';
  responseDisplay: string = '';

  greetMe(): void {
    this.http.get(`/api/${ this.nameInput }`)
      .subscribe((response: any) => this.responseDisplay = response.greeting);
  }
}

Optionally, you can add some basic Bootstrap so the result isn't hideous. In styles.css, add:

/* You can add global styles to this file, and also import other style files */
@import url('https://unpkg.com/bootstrap@3.3.7/dist/css/bootstrap.min.css');

Run the entire application again with $ npm run dev, and you should see a tiny form where you can send a name to the server, then get a greeting back.

Conclusion

The above steps should provide a good start to building out a fullstack application entirely in Typescript. By starting with Angular, we're bringing in a rich set of tools and configurations, then adding Express.js and some scripts for a convenient and automated development workflow.

In an upcoming tutorial, I'll show how to add in MongoDb (the M in MEAN) and how to use Docker to reduce some of the overhead in setting up a local database.

How to quickly mock an API server in a client-side project

Jeremy Wells — Thu, 14 Jan 2021 23:42:44 +0000

Photo by Lars Kienle on Unsplash

Introduction

In this post I'd like to demonstrate two simple tools that you can use to simulate fetching data from an API that can greatly facilitate frontend web development. This can be helpful for focusing on UI development without worrying about the server-side implementation of the project. It can also be helpful for setting up a proof-of-concept projects without having to worrying about protecting real data, or for setting up consistent fake data for end-to-end testing. The two tools I'll demonstrate are Mockaroo for generating sets of dummy data, and json-server for creating a fake API that a front-end application can make requests to.

Prerequisites

This tutorial is aimed squarely at folks getting familiar writing browser-side code with HTML, CSS, and Javascript. That said, these tools can help support more robust front-end projects as well. If you've got a bit of knowledge of HTML and Javascript and some familiarity with the command line, then you'll be able to get the tools in this article to work for you.

Before we start, you'll need the following installed:

Node.js. I'm using version 14 as I write this, but any LTS version should do.
Some way of serving static HTML and Javascript files locally. I use http-server and it's fast and really simple, but there are several other options.

To get started, create a tiny web project that'll consist of just an HTML page and a JSON file to will hold our fake data:

$ mkdir file_viewer
$ cd field_viewer
$ touch index.html db.json

These files can stay empty for now; We'll paste in a little bit of code at the end of this tutorial to get it all working.

Step one: Data modeling

I'm imagining a web application that allows users to upload files, then it shows a list of all the files that have been saved.
For this very simple example, we'll create a mock backend for an API that lists all of the file metadata (e.g., file name, file type, date created) after uploading.

Before generating mock data, we need to get some idea of what our data will look like, what properties each object will have when it's returned from the API. We'll need to know the properties we want and what primitive Javascript type they'll be. Our file metadata can look something like this:

id: number
fileName: string
fileType: string
createdDate: Date
email: string
active: boolean

An example json object with those properties might look like:

{
    "id": 30,
    "fileName": "The Chrysanthemum and the Dandelion",
    "fileType": ".pdf",
    "createdDate": "2020-11-18T01:36:46Z",
    "email": "kuni@garu.com",
    "active": true
}

Step two: Generate mock data with Mockaroo

Now that we have an idea of what our data looks like, we can use Mockaroo, a browser-based tool that can generate thousands of realistic looking records for us. Mockaroo has a variety of subscription-based utilities for creating mock APIs, but for this demonstration we'll use it for basic mock data generation which we can do for free.

The first part of Mockaroo is a form for adding properties to our mock data objects. Play around with the different data types that they offer and you'll see it can generate random data for a wide range of situations. Many of the data types have settings to further customize what gets generated. You can customize the behavior further by passing values through your own functions. Below, I'll create settings for each of the properties in our file metadata object, but feel free to add and change anything you find that's interesting. The rest of this tutorial will still work.

Below you can see the settings that I selected for the mock data object above:

id: For this field, I selected the Row Number data type which will number all of the records in order. This will help with fetching individual records from our fake API further on in this article.

fileName and email: For these two properties, Mockaroo can generate realistic random values by searching for and selecting the File Name and Email types respectively.

createdDate and active: For the createdDate, I can use Mockaroo's Datetime type and set it to return an ISO date which Javascript can parse, and I've set it to return dates from the past year. For the active property (which indicates if our file has been deleted or not), I can return a boolean.

fileType: For file types, I'd like my application to only allow Microsoft Word, Excel, and PDF files to be used, so in Mockaroo, I'll use a custom function to populate those values. Mockaroo allows us to write functions in Ruby to allow values to be transformed before returning. I'm going to click the fx button for the fileType row and add the following:

['.docx', '.xlsx', '.pdf'].sample

This will randomly select one item from that array and return it.

Now that my data objects are defined, I'll select # Rows: 100 and Format: JSON from the bottom of the form. I'll also select array and deselect include null values. Again, this is what my Mockaroo form looks like:

I'll click the Preview button to make sure my objects look like what I expect, then click Download Data to save it as a JSON file.

Serving the mock data with json-server

Now that we have some data, let's create a fake server that we can fetch it from.

json-server is a very light Node.js tool that can allow a frontend developer to define some API endpoints and sets of data that can be returned to a client application. This way, the mock data is completely separate from the client-side code and it can be switched out once the project is ready for a real API. In this article, we'll just set up the server and make a GET request to it, but json-server can also POST and save objects as well as more advanced uses.

First, install the json-server npm package globally:

$ npm install -g json-server

Open the file at the root of the FileViewer directory - ./db.json - that you created above. This will hold all of the json data:

As a quick test just to make sure it works, paste the following into db.json:

{
  "posts": [
    { "id": 1, "title": "json-server", "author": "typicode" }
  ],
  "comments": [
    { "id": 1, "body": "some comment", "postId": 1 }
  ],
  "profile": { "name": "typicode" }
}

This comes directly from the json-server documentation.

Run the json server with this command:

$ json-server --watch db.json

In a browser, paste in http://localhost:3000/posts/1 and it should return the first object in the posts array in our json file:

{ "id": 1, "title": "json-server", "author": "typicode" }

Now that we know the server is working, stop the server (CTRL+C).

Open db.json again and clear the contents. We'll define an endpoint for our mock data called files:

{
    "files": []
}

Open the JSON file that was downloaded from Mockaroo (it should be called MOCK_DATA.json). Copy all of the contents. Replace the empty "files" array with these contents so it looks something like this:

{
    "files": [
        {"id":1,"fileName":"nulla.xls","fileType":".xlsx","createdDate":"2020-02-27T05:10:27Z","email":"hweiner0@wikimedia.org","active":true},
        {"id":2,"fileName":"sapien urna.png","fileType":".xlsx","createdDate":"2020-12-20T05:12:00Z","email":"creding1@google.com.br","active":false},
        {"id":3,"fileName":"vel augue.ppt","fileType":".docx","createdDate":"2020-10-03T05:06:18Z","email":"ccrust2@theguardian.com","active":true},
        {"id":4,"fileName":"mauris morbi non.txt","fileType":".docx","createdDate":"2020-04-26T13:44:41Z","email":"nfolbig3@livejournal.com","active":true},
        {"id":5,"fileName":"sapien sapien non.ppt","fileType":".pdf","createdDate":"2020-04-16T17:56:06Z","email":"cdesimoni4@uol.com.br","active":true},

        // ...

    ]
}

Run json-server again and request the /files endpoint: http://localhost:3000/files.

It should load all 100 of the fake records that Mockaroo generated. Add an id number at the end of the url (http://localhost:3000/files/42) and it should return the one record with that id.

Finally, we can make an AJAX request from inside of our (tiny) web application to make sure it can connect with our front end code.

Open the index.html file that was created earlier and paste the following:

<!DOCTYPE html>
<html>
  <head>
    <title>File Viewer</title>
    <link href="css/bootstrap.min.css" rel="stylesheet">
  </head>
  <body>
    <h1>File Viewer</h1>

    <script>
        fetch('http://localhost:3000/files')
          .then(response => response.json())
          .then(data => console.log('Data from our mock backend: ', data));
    </script>
  </body>
</html>

Open a separate terminal window and cd into the FileViewer project created above. Start an HTTP server from this location (if you installed http-server, the command is just $ http-server --open).

In a browser, navigate to the http-server url (it would be http://localhost:8080 by default) and open a dev tools console. The page should have printed the mock data to the console.

Conclusion

In this tutorial, we walked through a very basic way to define data for a web application, generate hundreds of realistic records, then serve them from a mock API that can be consumed by the application. This is only a very bare minimum implementation of these steps. I highly advise going through the Mockaroo and json-server documentation for a wide variety of other uses and configurations that can suit your frontend development needs even more closely.

Configuration in ASP.NET Core Explained

Jeremy Wells — Wed, 26 Aug 2020 22:07:44 +0000

Photo by Alexey Ruban on Unsplash

A common consideration for working with an ASP.NET Core application is how to store and retrieve values in configuration settings, as well as how to change and update them as the application moves to higher environments. It's common to need sensitive values like passwords, API keys, and database connection strings in order to call out to other services, and these settings must be stored separately from the application's source code. To help manage this process, ASP.NET Core contains a set of configuration providers for retrieving and storing values, as well as a class to make complex settings available in the application as strongly-typed objects.

In this tutorial, I'll use an extremely simple web API to illustrate the different configuration providers and how one overrides the next as the application moves to different environments. We'll start with some boilerplate code, create a class to hold configuration settings, then load those settings from different locations in the file system. Finally, we'll move the application to a Docker container to show how settings can be updated in different environments.

Prerequisites

I'll assume you have some familiarity with ASP.NET Core applications, but this article will still stay at a beginner level. To build a copy of the example application and follow along, you will need:

Version 3.1 of the .NET Core SDK running on your development machine.
Docker Desktop running on your local machine. The version I'm using as of this writing is 19.03.12.
Postman, or an API tool of your choice, is recommended for making calls to the API and inspecting the results.
Lastly, to follow along with the examples, I've created a Github repo with a before and after branch. The following command will clone the repo:

$ git clone https://github.com/jsheridanwells/aspnetcore-config-example.git

To fetch the before and after branches, run:

$ git fetch origin before:before 
$ git fetch origin after:after

Then, $ git checkout <branch name> will take you to before or after.

Lastly, I tend to prefer writing .NET Core tutorials as OS-agnostically as possible, so I'll be using the command line with Bash for most of the steps, rather than Visual Studio. If you are using an IDE like Visual Studio or the Windows command line, there may be some differences from what you'll see here.

The boilerplate

Here are the important files we'll be working with in this project:

appsettings.json and appsettings.Development.json are two configuration files provided by ASP.NET core for configuration settings in different environments. The first file, appsettings.json contains any default settings, then any appsettings.{Environment Name}.json files override those settings. Since these files are stored at the root of the project directory, they should not contain any sensitive keys as these settings will usually checked into the project's version control system.
Program.cs is the main file that scaffolds an ASP.NET Core application and creates both the application and the host environment for the application. In this project, you'll notice a method called .ConfigureAppConfiguration that is commented out. Later in this tutorial, we'll uncomment the method and play around with the directives in there.
Startup.cs defines all the top-level settings for the application including configuration, dependency in injection, and the middlewares used by the application. In this class, we'll define the entry point for the app's configuration.
Controllers/ConfigController.cs is the single controller in the project with a single method called GetConfig that will be modified to return the values in our settings so that we can inspect the configuration values that are available in the application.
Lastly, Dockerfile will be used to build the application in a container and experiment with passing in configuration values that way. Note that we won't discuss containerizing applications in this tutorial, but you won't need to know much Docker for this demo.

If you run the project using the CLI...

 $ dotnet run

... and make a GET request to http://localhost:5000/config, you should get "nothing implemented yet" as your response.

Loading configuration in Startup.cs

Best practice for loading configuration settings is to create a simple class to map the configuration values, then adding them in the Startup class.

In this demo, our configuration will consist of two keys: Id will hold a Guid that we can pretend is a secret value like an API key, and Location where we'll write the location of the configuration object (e.g., appSettings, user secrets, environment variables etc.). At the root of the project, create a file called MyConfig.cs and add the following C# code:

using System;

namespace AspNetCoreConfigExample
{
    public class MyConfig
    {
        public Guid Id { get; set; }
        public string Location { get; set; }
    }
}

Now, whenever we need these values, we can reference the section as "MyConfig". In appsettings.json, the MyConfig section is set as follows:

"MyConfig": {
    "Id": "4c3c066c-928f-4de4-86b8-09365aed6a7c",
    "Location": "from appSettings.json"
}

In the Startup class, in the ConfigureServices method, we'll update the method as follows:

// Add configuration section here
var configs = Configuration.GetSection("MyConfig");
services.Configure<MyConfig>(configs);

Configuration.GetSection checks the configuration providers for any key named "MyConfig", then tries to map them to the MyConfig class.

You can check that the project compiles by running $ dotnet build.

Next, open ./Controllers/ConfigController.cs.

To make the configuration values available to the class, we'll add an IOptions<T> argument to the class constructor, and make the value of IOptions a private key.

private MyConfig _configs;
public ConfigController(IOptions<MyConfig> opts)
{
    _configs = opts.Value;
}

Finally, we'll update the GetConfig method to return whatever is in the _configs property:

[HttpGet]
public IActionResult GetConfig()
{
    // 3. return the config to inspect it
    var configs = _configs;
    return Ok(new { Result = "config values: ", configs });
}

Now, run the application using the .NET CLI:

$ dotnet run

Make a GET request to http://localhost:5000/config, and the response should come back as follows:

{
    "id": "4c3c066c-928f-4de4-86b8-09365aed6a7c",
    "location": "from appSettings.json"
}

From the above steps, we see how configuration is loaded through the Startup class, then made available to the application through the IOptions<T> class.

In the next steps, we'll add new configurations in different locations to demonstrate the hierarchy of providers in ASP.NET Core and how one provider can override the next.

The provider hierarchy

The default configuration setup for an ASP.NET Core app is hidden away in the Program class, in a method called .ConfigureWebHostDefaults. By default, the method checks for settings in a hierarchy of locations and if a matching configuration is found after a preceding one, settings from the latter are used.

According to this Microsoft blog, this is the order that providers are used:

The appsettings.json file
Any appsettings.{ ENVIRONMENT_NAME }.json files
User secrets from the .NET User Secrets Manager
Environment variables
Command line arguments

All of this happens under the hood, allowing us to write the configuration code once in Startup without having to add conditions for different environments. In the next few sections, we'll go through each of these providers, and at the end I'll show to override the default provider order for custom scenarios.

appsettings.json

When we ran the application in the last step, the configuration values that were returned came from the ./appsettings.json file. This can be overridden with an appsettings.{ ENVIRONMENT NAME }.json file. Add a file at the root of the project called appsettings.Development.json and paste in the following:

{
  "MyConfig": {
    "Id": "6e20863b-ae31-4e6c-8711-00177586bec0",
    "Location": "from appSettings.development.json"
  }
}

Run the server again and make another get request to http://localhost:5000/config and the response will now show the values from the Development settings, which is the default environment when running an ASP.NET Core app from a local host. This is useful for storing connection strings to local databases or any generic configuration values that can be safely stored in a source control repository.

To see how a different environment can override these settings, create another files called appsettings.Staging.json and paste the following:

{
    "MyConfig": {
        "Id": "bd304170-b899-42a9-9f9f-048e6e67a5dd",
        "Location": "from appSettings.Staging.json"
    }
}

We'll change the hosting environment to "Staging", then run set the application to run with that profile:

$ export ASPNETCORE_ENVIRONMENT="Staging"
$ dotnet run --launch-profile "Staging"

If you are using Windows, the first command is:

> set ASPNETCORE_ENVIRONMENT="Staging"

Now the request returns the settings from the Staging configuration.

User Secrets Manager

appsettings.json is okay for generic settings, but some other settings for development need to be stored separately from the source code. These could be API keys, passwords, settings for cloud infrastructure, or connections strings for remote databases. The .NET CLI provides a utility called the User Secrets Manager that can store and load values from a local development filesystem.

If you are using Visual Studio, you can right-click the name of the project in the Solution Explorer to get a user secrets dialog that makes managing these values automatic.

Using the .NET CLI, there are a few steps:

At the root of the project, initialize the User Secrets Manager: $ dotnet user-secrets init.
The project will need a user secrets ID which can be any string. This is added to the .csproj file. For this project, there is already a GUID added:

<!--AspNetCoreConfigExample.csproj-->
 <PropertyGroup>
<!--    ...-->
    <UserSecretsId>fa9be5ec-53e4-4275-8941-99cf83d8c344</UserSecretsId>
</PropertyGroup>

Using the cli, you can add new values for the Id and Location settings for the project:

 $ dotnet user-secrets set MyConfig:Id fa3e83d5-27e2-4a02-ae75-ad3763985290
 $ dotnet user-secrets set MyConfig:Location 'User Secrets Store'

Now, if you navigate to ~/.microsoft/usersecrets/<user_secrets_id>/secrets.json on your file system, you'll see a json file added that will map to the configuration settings as before. (Note: if you're using Windows, the path to the user secrets file is %APPDATA%\Microsoft\UserSecrets\<user_secrets_id>\secrets.json).

Run the application, make a request to /config again, and the values added with the User Secrets Manager have overridden the values from appsettings.json without having to modify any of the code.

Note that the User Secrets Manager is only for development purposes and should not be used in a production scenario since those values are stored on the host in plain text.

Environment variables, command line arguments, and container commands

We can also set MyConfig:Id and MyConfig:Location as environment variables and they will take precedent over the user secrets settings.

If you are using Linux or OSX, you can add these as environment variables:

$ export MyConfig__Id=52a8f62b-43fa-4da6-8a7e-7b788402aeed
$ export MyConfig__Location="Environment Variable"

In Windows, the equivalent is:

> set MyConfig:Id=52a8f62b-43fa-4da6-8a7e-7b788402aeed
> set MyConfig:Location="Environment Variable"

Once again, /config returns the environment variable settings.

We can also run the application with command-line arguments...

$ dotnet run MyConfig:Id=be4209bc-8162-4c77-9064-8dd912b2a413 MyConfig:Location='command line'

... and get the same result. These could be very useful when scripting the program in different environments like a CI/CD pipeline.

Where this really comes in handy is when building the application as a container image. In the next few steps, we'll Docker-ize the application three different ways. You won't need any deep Docker knowledge to do this as long as you have Docker Desktop installed and running on your development machine (check $ docker --version to be sure).

The AspNetCoreConfigExample project contains a simple Dockerfile that is basically the example provided in the Docker Documentation. It basically copies the code from the root directory and builds the executables, then moves the executables to a smaller environment with the .NET runtime to run the application.

On lines 17 and 18 of the Dockerfile, new environment variables are set which will be passed to the application from the container host:

ENV MyConfig__Id='73ce1157-c1b4-4d90-8669-2b33c86d7801'
ENV MyConfig__Location='from Dockerfile'

Use the following command to build an image from the Dockerfile:

$ docker build -t config-app .

Then run the image as a container on port :5000:

$ docker run -it -p 5000:5000 config-app

The /config response should now show values from the Dockerfile.

Lastly, a Docker container can be run with variables as command-line arguments to also keep these values separate from source code (or a container repository).

Run the container again with this command:

$ docker run -it -p 5000:5000 \
  -e MyConfig__Id='2021b838-6876-44b9-b123-1f760dc98aaa' \
  -e MyConfig__Location='from Docker CLI' \
  config-app

...and now you have settings from the command line.

Setting the configuration provider hierarchy

The default order of configuration providers that get called is logical for most situations, but ASP.NET Core also contains a scaffolding method for overriding the default order.

In this application, now that there are configuration settings stored in the user secrets file, those will be used over the asppsettings files. We can instead specify that we want appsettings.json to be used instead of user secrets.

In the sample project, in Program.cs there are five lines commented out that call a method called ConfigureAppConfiguration which takes a configuration builder as its second argument. The builder contains several methods for calling on configuration providers, and the settings from the last method to be called will override any previous settings. If you uncomment the lines you'll see the following:

.ConfigureAppConfiguration((hostContext, config) =>
{
    config.AddUserSecrets<MyConfig>();
    config.AddJsonFile("appsettings.json");
})

Run the application, and you'll see that the appsettings configuration gets returned. If you change the order of those methods, or add any of the other providers, different configuration settings will be given preference.

Conclusion

In this project, we saw some of the configuration providers that are available in an ASP.NET Core application and how those settings can be moved around and concealed as the application is deployed to higher environments. This gives several convenient strategies for setting up the application for a variety of deployment types.

TDD and Exception Handling with xUnit in ASP.NET Core

Jeremy Wells — Mon, 10 Aug 2020 22:13:46 +0000

Photo by Nicolas Thomas on Unsplash

Introduction and prerequisites

In this post, we're continuing our "walking skeleton" application where we build and deploy a minimal application with an ASP.NET Core WebApi and an Angular client. At this stage, the API is almost ready. We've got a controller that accepts a city location, a service that calls the third-party OpenWeatherMap API to return forecasts for that location, and in the last post we added the xUnit testing framework to describe the API. If you would like to start from the beginning, this is the first post{:class="no-target"}.

The goal of this series, and this application, is to create a bare-bones, testable, and deploy-able web application that can be used as a reference for starting similar projects. In each of these steps, I intend to describe the code we add in detail.

If you're starting the tutorial from this post, you can clone the following branch and continue modifying the code from there (note that you will need the .NET Core SDK installed on your machine):

$ git clone -b 3_adding-tests --single-branch git@github.com:jsheridanwells/WeatherWalkingSkeleton.git
$ cd WeatherWalkingSkeleton
$ dotnet restore

You'll also need to sign up for and register an OpenWeatherMap API key. This previous post{:class="no-target"} contains the steps for doing that.

Our TDD process

In the previous step, we started with one controller method that called one service method, then we set up a testing library using xUnit and Moq. With our testing framework ready to go, we'll use unit tests to guide some improvements to our API endpoint that fetches weather forecasts.

TDD stands for "Test-Driven Development", and it's a technique for ensuring that our code performs according to expectations, documenting the current expectations for code, and using tests to help ensure that changes aren't breaking prior functionality, especially when deploying code to higher environments. I won't discuss here the different types of testing, or the differing opinions on testing, but I've recently found this article which gives an excellent overview of the different software testing strategies and where they can fit into different kinds of projects.

Our test-driven development is going to follow a Red, Green, Refactor pattern:

Red: We will write a test and ensure that it fails. That way we're sure that our changes are actually bringing about the behavior we want, not some unanticipated side effect.
Green: We will modify our methods so that the tests pass.
Refactor: We will do any necessary refactoring to our changes so that the code is up to par, while making sure the test still passes.

Our changes

Right now, our API consists of one endpoint - GET http://localhost:5000/WeatherForecast/:location - that terminates at the WeatherForecastController and calls the Get method. Inside the Get method, the OpenWeatherService.GetFiveDayForecastAsync method is then called which returns a list of five forecasts for the next fifteen hours.

While manually testing this endpoint with Postman, and running our three current unit tests, proves that this indeed happens, our methods are very brittle right now. If an API consumer calls the endpoint without a location, or with a non-existent location, the unexpected result isn't handled. If we deploy the API to another environment without registering the OpenWeatherMap API key, we need to handle that failure as well in a way that communicates the problem to other developers. Also, the OpenWeatherMap API might itself fail and we need to be able to communicate the source of the problem. At the moment, if anything unexpected happens a long and unhelpful NullReferenceException is returned from the API.

Let's refactor the methods to handle the following scenarios:

One of our users sends a location that OpenWeatherAPI doesn't recognize: I would expect this to happen frequently, and it wouldn't be the result of any fault in the application, so to handle this we'll send back a helpful message to the user without throwing an exception.
The OpenWeatherMap API key is invalid: Right now, the application is running on our local machines with an API key configured. When we deploy to other environments, those servers will also need an API key to run. If the application gets deployed without one, or if the API key expires, we'll need to make that clear to any developers if OpenWeatherMap returns an unauthorized response.
OpenWeatherMap returns its own error: Since OpenWeatherMap is a third party, we cannot guarantee that it always functions within our own application as expected. If for some reason, a request to OpenWeatherMap fails, we need to handle that scenario as well.

Testing the service

We'll modify the OpenWeatherService class first. Open the corresponding unit test file: ./WeatherWalkingSkeletonTests/Services_Tests/OpenWeatherService_Tests.cs. Note, that in the previous post, we also created a static fixture class called OpenWeatherResponses that returns three simulated error responses from the OpenWeatherMap API: NotFoundResponse, UnauthorizedResponse, InternalErrorResponse. We'll use these responses to trigger the errors we could get from the third-party API.

In OpenWeatherService_Tests add the following tests:

[Fact]
public async Task Returns_OpenWeatherException_When_Called_With_Bad_Argument()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.NotFoundResponse,
        HttpStatusCode.NotFound);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await Assert.ThrowsAsync<OpenWeatherException>(() => sut.GetFiveDayForecastAsync("Westeros"));
    Assert.Equal(404, (int)result.StatusCode);
}

[Fact]
public async Task Returns_OpenWeatherException_When_Unauthorized()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.UnauthorizedResponse,
        HttpStatusCode.Unauthorized);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await Assert.ThrowsAsync<OpenWeatherException>(() => sut.GetFiveDayForecastAsync("Chicago"));
    Assert.Equal(401, (int)result.StatusCode);
}

[Fact]
public async Task Returns_OpenWeatherException_On_OpenWeatherInternalError()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.InternalErrorResponse,
        HttpStatusCode.InternalServerError);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await Assert.ThrowsAsync<OpenWeatherException>(() => sut.GetFiveDayForecastAsync("New York"));
    Assert.Equal(500, (int)result.StatusCode);
}

The tests follow the basic setup of the previous two tests, but we've configured the different possible error responses from the mock API. When OpenWeatherMap returns an unexpected result, we want our service to throw a custom exception called OpenWeatherException. This exception will communicate to the consuming class that the failure came from the third-party API.

If you run the test using your IDE's test runner, or using $ dotnet test in the terminal, we see our tests fail. We expected our custom exception and instead got a NullReferenceException since our service can't yet handle a response that it can't parse.

Open ./Api/Services/OpenWeatherService.cs and navigate to the GetFiveDayForecastAsync method. Going through the method line by line, we see the point where the method waits for a response from OpenWeatherMap:

var response = await client.GetAsync(url);

We'll check if the response is successful, and if it is then we'll deserialize the response as we were initially. If it's any other result, we'll build and throw an OpenWeatherException so the consuming class can respond accordingly. The if/else block will look like this (I'll copy the entire method further below):

if (response.IsSuccessStatusCode)
{
    // deserialize and return an OpenWeatherResponse
    var json = await response.Content.ReadAsStringAsync();
    var openWeatherResponse = JsonSerializer.Deserialize<OpenWeatherResponse>(json);
    foreach (var forecast in openWeatherResponse.Forecasts)
    {
        forecasts.Add(new WeatherForecast
        {
            Date = new DateTime(forecast.Dt),
            Temp = forecast.Temps.Temp,
            FeelsLike = forecast.Temps.FeelsLike,
            TempMin = forecast.Temps.TempMin,
            TempMax = forecast.Temps.TempMax,
        });
    } 

    return forecasts;
}
else
{
    // build an exception with information from the third-party API
    throw new OpenWeatherException(response.StatusCode, "Error response from OpenWeatherApi: " + response.ReasonPhrase);
}

The exception will contain the OpenWeatherMap HTTP status code and a simple message, then a consuming class can create logic based on that information.

Below is what the entire GetFiveDayFirecastAsync method should look like:

public async Task<List<WeatherForecast>> GetFiveDayForecastAsync(string location, Unit unit = Unit.Metric)
{
    string url = BuildOpenWeatherUrl("forecast", location, unit);
    var forecasts = new List<WeatherForecast>();

    var client = _httpFactory.CreateClient("OpenWeatherClient");
    var response = await client.GetAsync(url);

    if (response.IsSuccessStatusCode)
    {
        var json = await response.Content.ReadAsStringAsync();
        var openWeatherResponse = JsonSerializer.Deserialize<OpenWeatherResponse>(json);
        foreach (var forecast in openWeatherResponse.Forecasts)
        {
            forecasts.Add(new WeatherForecast
            {
                Date = new DateTime(forecast.Dt),
                Temp = forecast.Temps.Temp,
                FeelsLike = forecast.Temps.FeelsLike,
                TempMin = forecast.Temps.TempMin,
                TempMax = forecast.Temps.TempMax,
            });
        } 

        return forecasts;
    }
    else
    {
        throw new OpenWeatherException(response.StatusCode, "Error response from OpenWeatherApi: " + response.ReasonPhrase);
    } 
}

Run the tests again and they should all pass. At this point, we've done the Red and Green steps of the test process. I'll leave it up to you to find any opportunities for refactoring this method or letting it go as is.

Testing the controller

Our service can now graceful indicate if there was a failed response from the OpenWeatherMap API. Now we need our controller communicate these exceptions back to our API's consuming clients.

Going back to our original three scenarios, the controller could respond to each in the following ways:

If OpenWeatherMap couldn't recognize the location, the controller can return a 400 BadRequest response and let the consumer know the name of the location that failed. Also, if the request is made without a location, we should return a 400 before even calling the service.
If the OpenWeatherMap returns an Unauthorized response, it's due to an invalid API key and for this project that's likely from a bad configuration. We'll return a 500 Internal Server Error with the message from the OpenWeatherMap API which will indicate if the request was not authorized.
If there is any other error, we'll return another 500 response with the message from OpenWeatherMap. Lastly, we'll also return a 500 response for any other exception that is thrown within the application.

The responses above lead to three tests that we'll add to ./Tests/Controllers_Tests/:

[Fact]
public async Task Returns_400_Result_When_Missing_Location()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.NotFoundResponse);
    var service = new OpenWeatherService(opts, clientFactory);
    var sut = new WeatherForecastController(new NullLogger<WeatherForecastController>(), service);

    var result = await sut.Get(String.Empty) as ObjectResult;

    Assert.Equal(400, result.StatusCode);
}

[Fact]
public async Task Returns_BadRequestResult_When_Location_Not_Found()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.NotFoundResponse,
        HttpStatusCode.NotFound);
    var service = new OpenWeatherService(opts, clientFactory);
    var sut = new WeatherForecastController(new NullLogger<WeatherForecastController>(), service);

    var result = await sut.Get("Westworld") as ObjectResult;

    Assert.Contains("not found", result.Value.ToString());
    Assert.Equal(400, result.StatusCode);
}

[Fact]
public async Task Returns_OpenWeatherException_When_Unauthorized()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.UnauthorizedResponse,
        HttpStatusCode.Unauthorized);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await Assert.ThrowsAsync<OpenWeatherException>(() => sut.GetFiveDayForecastAsync("Chicago"));
    Assert.Equal(401, (int)result.StatusCode);
}

[Fact]
public async Task Returns_500_When_Api_Returns_Error()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.UnauthorizedResponse,
        HttpStatusCode.Unauthorized);
    var service = new OpenWeatherService(opts, clientFactory);
    var sut = new WeatherForecastController(new NullLogger<WeatherForecastController>(), service);

    var result = await sut.Get("Rio de Janeiro") as ObjectResult;

    Assert.Contains("Error response from OpenWeatherApi: Unauthorized", result.Value.ToString());
    Assert.Equal(500, result.StatusCode); 
}

If we run them, they should fail.

We'll open the class under test in ./Api/Controllers/WeatherForecastController.cs and find the Get() method. Add the following as the first step of the method to check if there is a usable location value with the request:

[HttpGet]
public async Task<IActionResult> Get(string location, Unit unit = Unit.Metric)
{
   if (string.IsNullOrEmpty(location))
       return BadRequest("location parameter is missing");
   // [ ... ] 
}

Now, three of the four new tests should be failing.

For the rest of the tests, we can get them to pass by returning a 400 Bad Request result if OpenWeatherMap can't find the location, or returning a 500 Internal Server Error for any other reason, along with a helpful message. Also, we can wrap our logic in a try/catch block that will handle an OpenWeatherException as indicated above, then handle any other exception. The updated Get() method can now look like this:

[HttpGet]
public async Task<IActionResult> Get(string location, Unit unit = Unit.Metric)
{
    if (string.IsNullOrEmpty(location))
        return BadRequest("location parameter is missing");
    try
    {
        var forecast = await _weatherService.GetFiveDayForecastAsync(location, unit);
        return Ok(forecast);
    }
    catch (OpenWeatherException e)
    {
        if (e.StatusCode == HttpStatusCode.NotFound)
            return BadRequest($"Location: \"{ location }\" not found.");
        else
            return StatusCode(500, e.Message);
    }
    catch (Exception e)
    {
        return StatusCode(500, e.Message);
    }
}

Run the tests and we're successful if we now have nine passing tests in the collection. As before, if you would like to experiment with other ways to handle exceptions from the OpenWeatherService, you can now refactor the method secured with its corresponding tests.

Conclusion

In this tutorial, we started with a project with an API endpoint that could handle a "happy path," but could not meaningfully handle exceptions. We came up with three possible exception scenarios, then used test-driven development for describing the desired behavior for our classes, making changes until the tests passed. We now have a more robust example ASP.NET Core project. In the next tutorials, we will Docker-ize the API to support complimentary development, then scaffold an Angular project to serve as a client.

TDD and Exception Handling with xUnit in ASP.NET Core

Jeremy Wells — Mon, 10 Aug 2020 19:48:13 +0000

Photo by Nicolas Thomas on Unsplash

Introduction and prerequisites

If you're starting the tutorial from this post, you can clone the following branch and continue modifying the code from there (note that you will need the .NET Core SDK installed on your machine):

$ git clone -b 3_adding-tests --single-branch git@github.com:jsheridanwells/WeatherWalkingSkeleton.git
$ cd WeatherWalkingSkeleton
$ dotnet restore

You'll also need to sign up for and register an OpenWeatherMap API key. This previous post contains the steps for doing that.

Our TDD process

Our test-driven development is going to follow a Red, Green, Refactor pattern:

Red: We will write a test and ensure that it fails. That way we're sure that our changes are actually bringing about the behavior we want, not some unanticipated side effect.
Green: We will modify our methods so that the tests pass.
Refactor: We will do any necessary refactoring to our changes so that the code is up to par, while making sure the test still passes.

Our changes

Let's refactor the methods to handle the following scenarios:

One of our users sends a location that OpenWeatherAPI doesn't recognize: I would expect this to happen frequently, and it wouldn't be the result of any fault in the application, so to handle this we'll send back a helpful message to the user without throwing an exception.
The OpenWeatherMap API key is invalid: Right now, the application is running on our local machines with an API key configured. When we deploy to other environments, those servers will also need an API key to run. If the application gets deployed without one, or if the API key expires, we'll need to make that clear to any developers if OpenWeatherMap returns an unauthorized response.
OpenWeatherMap returns its own error: Since OpenWeatherMap is a third party, we cannot guarantee that it always functions within our own application as expected. If for some reason, a request to OpenWeatherMap fails, we need to handle that scenario as well.

Testing the service

In OpenWeatherService_Tests add the following tests:

[Fact]
public async Task Returns_OpenWeatherException_When_Called_With_Bad_Argument()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.NotFoundResponse,
        HttpStatusCode.NotFound);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await Assert.ThrowsAsync<OpenWeatherException>(() => sut.GetFiveDayForecastAsync("Westeros"));
    Assert.Equal(404, (int)result.StatusCode);
}

[Fact]
public async Task Returns_OpenWeatherException_When_Unauthorized()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.UnauthorizedResponse,
        HttpStatusCode.Unauthorized);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await Assert.ThrowsAsync<OpenWeatherException>(() => sut.GetFiveDayForecastAsync("Chicago"));
    Assert.Equal(401, (int)result.StatusCode);
}

[Fact]
public async Task Returns_OpenWeatherException_On_OpenWeatherInternalError()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.InternalErrorResponse,
        HttpStatusCode.InternalServerError);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await Assert.ThrowsAsync<OpenWeatherException>(() => sut.GetFiveDayForecastAsync("New York"));
    Assert.Equal(500, (int)result.StatusCode);
}

var response = await client.GetAsync(url);

if (response.IsSuccessStatusCode)
{
    // deserialize and return an OpenWeatherResponse
    var json = await response.Content.ReadAsStringAsync();
    var openWeatherResponse = JsonSerializer.Deserialize<OpenWeatherResponse>(json);
    foreach (var forecast in openWeatherResponse.Forecasts)
    {
        forecasts.Add(new WeatherForecast
        {
            Date = new DateTime(forecast.Dt),
            Temp = forecast.Temps.Temp,
            FeelsLike = forecast.Temps.FeelsLike,
            TempMin = forecast.Temps.TempMin,
            TempMax = forecast.Temps.TempMax,
        });
    } 

    return forecasts;
}
else
{
    // build an exception with information from the third-party API
    throw new OpenWeatherException(response.StatusCode, "Error response from OpenWeatherApi: " + response.ReasonPhrase);
}

The exception will contain the OpenWeatherMap HTTP status code and a simple message, then a consuming class can create logic based on that information.

Below is what the entire GetFiveDayFirecastAsync method should look like:

public async Task<List<WeatherForecast>> GetFiveDayForecastAsync(string location, Unit unit = Unit.Metric)
{
    string url = BuildOpenWeatherUrl("forecast", location, unit);
    var forecasts = new List<WeatherForecast>();

    var client = _httpFactory.CreateClient("OpenWeatherClient");
    var response = await client.GetAsync(url);

    if (response.IsSuccessStatusCode)
    {
        var json = await response.Content.ReadAsStringAsync();
        var openWeatherResponse = JsonSerializer.Deserialize<OpenWeatherResponse>(json);
        foreach (var forecast in openWeatherResponse.Forecasts)
        {
            forecasts.Add(new WeatherForecast
            {
                Date = new DateTime(forecast.Dt),
                Temp = forecast.Temps.Temp,
                FeelsLike = forecast.Temps.FeelsLike,
                TempMin = forecast.Temps.TempMin,
                TempMax = forecast.Temps.TempMax,
            });
        } 

        return forecasts;
    }
    else
    {
        throw new OpenWeatherException(response.StatusCode, "Error response from OpenWeatherApi: " + response.ReasonPhrase);
    } 
}

Testing the controller

Our service can now graceful indicate if there was a failed response from the OpenWeatherMap API. Now we need our controller communicate these exceptions back to our API's consuming clients.

Going back to our original three scenarios, the controller could respond to each in the following ways:

If OpenWeatherMap couldn't recognize the location, the controller can return a 400 BadRequest response and let the consumer know the name of the location that failed. Also, if the request is made without a location, we should return a 400 before even calling the service.
If the OpenWeatherMap returns an Unauthorized response, it's due to an invalid API key and for this project that's likely from a bad configuration. We'll return a 500 Internal Server Error with the message from the OpenWeatherMap API which will indicate if the request was not authorized.
If there is any other error, we'll return another 500 response with the message from OpenWeatherMap. Lastly, we'll also return a 500 response for any other exception that is thrown within the application.

The responses above lead to three tests that we'll add to ./Tests/Controllers_Tests/:

[Fact]
public async Task Returns_400_Result_When_Missing_Location()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.NotFoundResponse);
    var service = new OpenWeatherService(opts, clientFactory);
    var sut = new WeatherForecastController(new NullLogger<WeatherForecastController>(), service);

    var result = await sut.Get(String.Empty) as ObjectResult;

    Assert.Equal(400, result.StatusCode);
}

[Fact]
public async Task Returns_BadRequestResult_When_Location_Not_Found()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.NotFoundResponse,
        HttpStatusCode.NotFound);
    var service = new OpenWeatherService(opts, clientFactory);
    var sut = new WeatherForecastController(new NullLogger<WeatherForecastController>(), service);

    var result = await sut.Get("Westworld") as ObjectResult;

    Assert.Contains("not found", result.Value.ToString());
    Assert.Equal(400, result.StatusCode);
}

[Fact]
public async Task Returns_OpenWeatherException_When_Unauthorized()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.UnauthorizedResponse,
        HttpStatusCode.Unauthorized);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await Assert.ThrowsAsync<OpenWeatherException>(() => sut.GetFiveDayForecastAsync("Chicago"));
    Assert.Equal(401, (int)result.StatusCode);
}

[Fact]
public async Task Returns_500_When_Api_Returns_Error()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.UnauthorizedResponse,
        HttpStatusCode.Unauthorized);
    var service = new OpenWeatherService(opts, clientFactory);
    var sut = new WeatherForecastController(new NullLogger<WeatherForecastController>(), service);

    var result = await sut.Get("Rio de Janeiro") as ObjectResult;

    Assert.Contains("Error response from OpenWeatherApi: Unauthorized", result.Value.ToString());
    Assert.Equal(500, result.StatusCode); 
}

If we run them, they should fail.

[HttpGet]
public async Task<IActionResult> Get(string location, Unit unit = Unit.Metric)
{
   if (string.IsNullOrEmpty(location))
       return BadRequest("location parameter is missing");
   // [ ... ] 
}

Now, three of the four new tests should be failing.

[HttpGet]
public async Task<IActionResult> Get(string location, Unit unit = Unit.Metric)
{
    if (string.IsNullOrEmpty(location))
        return BadRequest("location parameter is missing");
    try
    {
        var forecast = await _weatherService.GetFiveDayForecastAsync(location, unit);
        return Ok(forecast);
    }
    catch (OpenWeatherException e)
    {
        if (e.StatusCode == HttpStatusCode.NotFound)
            return BadRequest($"Location: \"{ location }\" not found.");
        else
            return StatusCode(500, e.Message);
    }
    catch (Exception e)
    {
        return StatusCode(500, e.Message);
    }
}

Conclusion

Testing an ASP.NET Core Service with xUnit

Jeremy Wells — Mon, 03 Aug 2020 21:00:31 +0000

Photo by Joyce McCown on Unsplash

Introduction and prerequisites

This post is part of an ongoing series where we build a "walking skeleton" application using ASP.NET Core and Angular as well as other technologies for deployment and testing. By now, our application is a minimally functional web API that organizes and returns weather data from a location. In this post, we will use xUnit to test how a service handles calls to a third-party API, then how a controller responds to a successful response. In the next post, we'll then use those tests to scaffold some exception handling that's missing from our classes right now. As in other posts, the aim of this article is to go through the steps with detailed explanations.

If you would like to start from the beginning, this post will introduce the project and walk you up to this point. If you're just here for a walkthrough of testing with xUnit, then you can:

Begin by cloning the project up to this point and cd-ing into it:

$ git clone -b 2_adding-async --single-branch git@github.com:jsheridanwells/WeatherWalkingSkeleton.git
$ cd WeatherWalkingSkeleton

Register for an OpenWeatherMap API key,
And add the API key to the secrets store for this project:

$ dotnet user-secrets set "OpenWeather:ApiKey" "<YOUR-API-KEY>" --project ./Api/WeatherWalkingSkeleton.csproj

Test that the web API is working properly up to now:

$ dotnet restore
$ dotnet run --project Api/WeatherWalkingSkeleton.csproj

... and in another terminal, make a request to make sure a result comes out:

$ curl -k https://localhost:5001/weatherforecast?location=detroit

If you get what looks like an array of weather forecasts, then you are good to go.

Testing Goals

For this stage of the project, we will add some tests for two of the classes that we've built so far: the OpenWeatherService and the WeatherForecastController. This is to establish a pattern of tests that describe the code, and as the application grows in complexity, we'll be sure new changes won't break prior functionality.

I've read lots of opinions on software testing from engineers much more experienced than me - from strictly adhering to test-driven development to strategies that might be more pragmatic. The strategy I've decided on is to test how the OpenWeatherService responds to different possible responses from the third-party API. Since we do not want to call the actual OpenWeatherMap API, we will set up a substitute class where we can simulate the responses. After writing tests for the service, we'll then set up the service with the WeatherForecastController to test that data is returned properly from the API. From there, we'll have a small set of tests that describe the classes and can run in the future to keep any new work from breaking the prior work.

To the steps to get there are:

Set up our test project with the xUnit and Moq libraries.
Write tests to describe the classes' current functionality.

Creating a test project

The dotnet CLI contains a template for adding a xUnit test project, as well as templates for the nUnit and MSTest libraries. Let's create that project:

$ dotnet new xunit -o Tests -n WeatherWalkingSkeleton.Tests

Next, we add the test project to the WeatherWalkingSkeleton solution:

$ dotnet sln WeatherWalkingSkeleton.sln add ./Tests/WeatherWalkingSkeleton.Tests.csproj

Then, to see that the tests are discoverable, run:

$ dotnet test

If everything is working alright, you'll see the results of one passing fake test.

If you are using Visual Studio, there is a built-in test explorer that will provide a UI for running and debugging tests. You can get a similar set of functionality with VS Code using the .NET Core Test Explorer plugin. Otherwise, running $ dotnet test from the command line will suffice for this tutorial.

We need to add a reference to our test project so that it can access the classes from the API library under test:

$ dotnet add Tests/WeatherWalkingSkeleton.Tests.csproj reference Api/WeatherWalkingSkeleton.csproj

Lastly, we'll add a few directories and test classes to the testing library. It will take on a similar structure to the API project so that it will be easier to compare a class to its tests. Let's add directories for any controller and service classes:

$ mkdir ./Tests/Controllers_Tests ./Tests/Services_Tests

Then we'll add the test classes. You can either add the files via command line or scaffold a class with the IDE you're using:

$ touch ./Tests/{/Controllers_Tests/WeatherForecastController_Tests.cs,/Services_Tests/OpenWeatherService_Tests.cs}

We'll also create an Infrastructure directory for any fixtures or utilities needed to support the test classes:

$ mkdir ./Tests/Infrastructure

Lastly, the fake example test can be removed:

$ rm ./Tests/UnitTest1.cs

Setting up test fixtures

The OpenWeatherService will be the trickier class to test because it creates an HttpClient object to make calls to a third-party API. We want to test how it handles different kinds of responses from the API, but we don't want to actually make those requests. We'll have to simulate the kinds of responses the API might return. Also, our service class uses an IOptions object in order to extract a secret API key to use. We don't want any API keys to appear in our code, and in fact it's not really important whether we have a real API key or not, so we'll have to create a service to test with an alternate IOptions object.

So here's a strategy for describing and testing the OpenWeatherService:

Create an empty IOptions<OpenWeather> object to inject into the service.
Create a mock HttpClient that can return simulated responses.
Handle the "happy path" scenario - how does the service return a successful response from the API?

1. Setting up an Options Builder

In the Infrastructure directory, add a class called OptionsBuilder.cs. This will be a static class, and so far all we need it to do is to return an Options object with one of the OpenWeatherMap configuration objects as its value:

using Microsoft.Extensions.Options;
using WeatherWalkingSkeleton.Config;

namespace WeatherWalkingSkeleton.Tests.Infrastructure
{
    public static class OptionsBuilder
    {
        public static IOptions<OpenWeather> OpenWeatherConfig()
        {
            return Options.Create<OpenWeather>(new OpenWeather { ApiKey = "00000"});
        }
    }
}

Not much happening here, but we've got a passable object to build a test OpenWeatherService

2. Setting up an HttpClient Builder

This one is going to be more involved. Our service gets instantiated with an IHttpClientFactory and we call a CreateClient method from the object to get an HttpClient. This strategy is a workaround because we cannot mock an HttpClient directly.

To assist in mocking the objects, we'll add a very common Nuget package called Moq:

$ dotnet add Tests/WeatherWalkingSkeleton.Tests.csproj package Moq

In Infrastructure, create a ClientBuilder.cs class, also a static class. It will have a static method called OpenWeatherClientFactory. In order to make the method more versatile, we'll give it two arguments: StringContent content will be the simulated response from the API and HttpStatusCode statusCode will be HTTP response code, e.g. 200, 400, 401. I'll paste the entire class directly below, then explain each part:

using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using Moq;
using Moq.Protected;

namespace WeatherWalkingSkeleton.Tests.Infrastructure
{
    public static class ClientBuilder
    {
        public static IHttpClientFactory OpenWeatherClientFactory(StringContent content, HttpStatusCode statusCode = HttpStatusCode.OK)
        {
            var handler = new Mock<HttpMessageHandler>();
            handler.Protected()
                .Setup<Task<HttpResponseMessage>>(
                    "SendAsync",
                    ItExpr.IsAny<HttpRequestMessage>(),
                    ItExpr.IsAny<CancellationToken>()
                )
                .ReturnsAsync(new HttpResponseMessage
                {
                    StatusCode = statusCode,
                    Content = content
                });
            var client = new HttpClient(handler.Object);
            var clientFactory = new Mock<IHttpClientFactory>();
            clientFactory.Setup(_ => _.CreateClient(It.IsAny<string>()))
                .Returns(client);
            return clientFactory.Object;
        }
    }
}

HttpClient uses an object internally called HttpMessageHandler to make the actual HTTP request. We will mock it using the Moq library:

var handler = new Mock<HttpMessageHandler>();

The handler has a method called SendAsync that is called to send the request, so we will use Moq to set up the response that we want:

handler.Protected()
.Setup<Task<HttpResponseMessage>>(
    "SendAsync",
    ItExpr.IsAny<HttpRequestMessage>(),
    ItExpr.IsAny<CancellationToken>()
)
.ReturnsAsync(new HttpResponseMessage
{
    StatusCode = statusCode,
    Content = content
});

With our fake message handler, we'll create a real HttpClient object:

var client = new HttpClient(handler.Object);

And then we'll create a mock IHttpClientFactory that returns our HttpClient

var client = new HttpClient(handler.Object);
var clientFactory = new Mock<IHttpClientFactory>();
clientFactory.Setup(_ => _.CreateClient(It.IsAny<string>()))
    .Returns(client);
return clientFactory.Object;

3. Generating Controlled Third-Party API Responses

The last piece of infrastructure we'll need is a static class that can return some canned responses that sort of look like the responses that would come back from the OpenWeatherMap API. From testing the API in Postman, we can see that a successful response returns an array of objects that resemble the project's WeatherForecast class with an HTTP status of 200.

We can also predict a few other scenarios:

If the resource is called with a missing or invalid API key, we get a 401 status with "Invalid Api key".
If the resource is called without a valid city name, we get a 404 status with "city not found".
And, I can't be sure from testing what an error from the third-party server looks like, based on convention we'll guess that it's a 500 HTTP status with "Internal Error." as the message.

Add a file to the ./Tests/Infrastructure directory called OpenWeatherResponses.cs. Then, paste the following code which will create the canned responses for our mock HTTP factory to return:

using System.Collections.Generic;
using System.Net.Http;
using System.Text.Json;
using WeatherWalkingSkeleton.Models;

namespace WeatherWalkingSkeleton.Tests.Infrastructure
{
    public static class OpenWeatherResponses
    {
        public static StringContent OkResponse => BuildOkResponse();
        public static StringContent UnauthorizedResponse => BuildUnauthorizedResponse();
        public static StringContent NotFoundResponse => BuildNotFoundResponse();
        public static StringContent InternalErrorResponse => BuildInternalErrorResponse();

        private static StringContent BuildOkResponse()
        {
            var response = new OpenWeatherResponse
            {
                Forecasts = new List<Forecast>
                {
                    new Forecast{ Dt = 1594155600, Temps = new Temps { Temp = (decimal)32.93 } }
                }
            };
            var json = JsonSerializer.Serialize(response);
            return new StringContent(json);
        }

        private static StringContent BuildUnauthorizedResponse()
        {
            var json = JsonSerializer.Serialize(new { Cod = 401, Message = "Invalid Api key." });
            return new StringContent(json);
        }

        private static StringContent BuildNotFoundResponse()
        {
            var json = JsonSerializer.Serialize(new { Cod = 404, Message = "city not found" });
            return new StringContent(json);
        }

        private static StringContent BuildInternalErrorResponse()
        {
            var json = JsonSerializer.Serialize(new {Cod = 500, Message = "Internal Error."});
            return new StringContent(json);
        }
    }
}

Testing the Service

Now that we can control the response we get when pretending to call the OpenWeatherMap API, we'll set up some tests to describe the OpenWeatherService.

So far, the class contains one method: GetFiveDayForecastAsync. We expect it to return a list of WeatherForecast objects. In the future, we'll need to update this method to handle any errors that get returned from the API, but for now the test will just describe what the method is supposed to do. Also, note that the class contains a private method called BuildOpenWeatherUrl. Since it's a private method, we can't test it in isolation, but since we know that GetFiveDayForecastAsync depends on it, any failures from the private method will be indicated when testing the public method.

Here are a couple of responses that we can expect from GetFiveDayForecastAsync:

It will return a list of WeatherForecast objects, no other objects.
The Date and the Temp properties that come from the API response will be the same when the WeatherForecast list is created.

Add a test file in the .Tests/Services_Tests directory:

$ touch ./Tests/Services_Tests/OpenWeatherService_Tests.cs

The class, with all of the using statements should start like this:

using System;
using System.Collections.Generic;
using System.Net;
using System.Threading.Tasks;
using WeatherWalkingSkeleton.Infrastructure;
using WeatherWalkingSkeleton.Models;
using WeatherWalkingSkeleton.Tests.Infrastructure;
using Xunit;

namespace WeatherWalkingSkeleton.Services
{
    public class OpenWeatherService_Tests
    {

    }
}

Inside the service, let's add two methods for each of the descriptions we want to provide. Here are the methods:

[Fact]
public async Task Returns_A_WeatherForecast()
{

}

[Fact]
public async Task Returns_Expected_Values_From_the_Api()
{

}

Note that the [Fact] annotation allows a test explorer to find and run any test methods.

Now we'll add code to the first method. Since in each test, we'll need to create a OpenWeatherService, we'll generate IOptions<OpenWeather> and IHttpClientFactory objects using the fixtures created above, then create an OpenWeatherService named sut for "system under test". With the service instantiated, we'll call GetFiveDayForecastAsync. On the last line, the Assert class from xUnit is used to test that the method is returning the type of object that we expect:

[Fact]
public async Task Returns_A_WeatherForecast()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.OkResponse);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await sut.GetFiveDayForecastAsync("Chicago");

    Assert.IsType<List<WeatherForecast>>(result);
}

The second test is set up exactly the same way, but in this test we're seeing if we find the same Date and Temp values that we loaded OpenWeatherResponses.BuildOkResponse() with earlier:

[Fact]
public async Task Returns_Expected_Values_From_the_Api()
{
    var opts = OptionsBuilder.OpenWeatherConfig();
    var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.OkResponse);
    var sut = new OpenWeatherService(opts, clientFactory);

    var result = await sut.GetFiveDayForecastAsync("Chicago");

    Assert.Equal(new DateTime(1594155600), result[0].Date);
    Assert.Equal((decimal)32.93, result[0].Temp);
}

Now run the tests in the IDE test explorer, or in the command line terminal ($ dotnet test) to make sure they pass.

Testing the Controller

Now we'll add a file for running some controller tests:

$ touch ./Tests/Services_Tests/WeatherForecastController_Tests.cs

Just like the service, so far our WeatherForecastController that consumes the OpenWeatherService just has one method called Get for returning the result of the service. While our previous tests were strictly isolated unit tests - the API and the HTTP client were mocked - I'd like the controller tests to be more of an integration test: I'd like to know how it responds to different results from the OpenWeatherService that it depends on. If we make a change to the OpenWeatherService that could break the WeatherForecastController, we wouldn't know it if we were mocking the service in these tests. So in our tests, we'll build an OpenWeatherService with the API response that we expect, then build the controller with that.

Our WeatherForecastController requires an ILogger<WeatherForecastController> in the constructor. Luckily, the Microsoft.Extensions.Logging library that it the interface comes from also has a class called NullLogger<T> that lets us just pass in an empty logger since logging has nothing to do with the functionality that we're testing. The setup for creating the controller as our system under test is as follows (Note, I'll copy in the full method further down):

var opts = OptionsBuilder.OpenWeatherConfig();
var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.OkResponse);
var service = new OpenWeatherService(opts, clientFactory);
var sut = new WeatherForecastController(new NullLogger<WeatherForecastController>(), service);

After the controller is created we will await the result and convert it to an OkObjectResult that will contain the API response to evaluate:

var result = await sut.Get("Chicago") as OkObjectResult;

Lastly, we'll make sure that a successful response from the OpenWeatherService results in a "success" response from the controller, along with the List<WeatherForecast> object that we're expecting:

Assert.IsType<List<WeatherForecast>>(result.Value);
Assert.Equal(200, result.StatusCode);

Altogether, the WeatherForecastController_Tests class should look like this:

using System.Collections.Generic;
using System.Net;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging.Abstractions;
using WeatherWalkingSkeleton.Controllers;
using WeatherWalkingSkeleton.Models;
using WeatherWalkingSkeleton.Services;
using WeatherWalkingSkeleton.Tests.Infrastructure;
using Xunit;

namespace WeatherWalkingSkeleton.Tests.Controllers_Tests
{
    public class WeatherForecastController_Tests
    {
        [Fact]
        public async Task Returns_OkResult_With_WeatherForecast()
        {
            var opts = OptionsBuilder.OpenWeatherConfig();
            var clientFactory = ClientBuilder.OpenWeatherClientFactory(OpenWeatherResponses.OkResponse);
            var service = new OpenWeatherService(opts, clientFactory);
            var sut = new WeatherForecastController(new NullLogger<WeatherForecastController>(), service);

            var result = await sut.Get("Chicago") as OkObjectResult;

            Assert.IsType<List<WeatherForecast>>(result.Value);
            Assert.Equal(200, result.StatusCode);
        }
    }
}

Run the tests again, and we should have three total successful tests.

Conclusion

That's far enough for now. From this tutorial, we were able to install a test library for an ASP.NET Core WebApi project. We also created some initial infrastructure to control the dependencies that we are not testing as well as create a mock version of a third-party API so that we can control the different responses it might give. We used this to evaluate successful responses from our service, then to evaluate the same responses in the controller that consumes the service.

In the next tutorial, we'll start a TDD process for adding exception handling functionality to the controller and service methods.

References

These posts proved especially helpful in figuring out how to use HttpClient in tests.

Consuming third-party APIs in ASP.NET Core

Jeremy Wells — Mon, 29 Jun 2020 20:29:04 +0000

Photo by Wim van 't Einde on Unsplash

Introduction

This post is the second phase of work on a walking skeleton application, and part of a series where we build out and deploy a minimal-but-functional web application built with the ASP.NET Core and Angular frameworks. In the introductory post, I explained the intent of this series in more detail and set up and toured the boilerplate code for an ASP.NET Core WebApi application. In the next post, we made some initial configuration, and built out a controller and a service to be able to return data from a third-party API. Here, we will make our service a little more robust. We will refactor this service class to make it more testable and to take advantage of async features in C#.

Prequisites

This article will be a continuation of Part 0 and Part 1 of this series. My goal for this series is to walk through a sample application while explaining the hows and whys of the ASP.NET Core framework in detail. If you are looking for a solution for consuming a third-party API in an ASP.NET Core application, going through this tutorial on its own should suffice. If you are looking for a more fundamental understanding of the framework, starting from the beginning may be a better bet.

For this tutorial, you will need:

Git, Postman, the .NET Core SDK, and an IDE or text editor that can work comfortably with C# code. Visual Studio Code with these plugins is a serviceable free setup. Part 0 of this series goes through this setup in more detail.
An OpenWeatherMap API key. This is the third-party API that we'll be consuming in this example application. in Part 1, I walk through getting the API key and using the User Secrets Manager in .NET Core to store the key in a file system.
The starting point for the example code used in this application. You can clone the starting branch for this repository with this command:

$ git clone -b 1_aspnetcore_webapi_setup --single-branch git@github.com:jsheridanwells/WeatherWalkingSkeleton.git

As in the previous articles, if this is your first time building a web API, or if you need an introduction to how to go about it in ASP.NET Core, this article will provide a detailed explanation of the steps and the features of this particular framework. I won't go into the C# language syntax, but the Microsoft documentation has a great primer on the language itself.

Initial Walkthrough

If you've cloned the application from our starting point in the previous section, and if you've registered an API key for the OpenWeatherMap service, from the root directory of the project, use these steps to verify that the application is running:

$ dotnet run --project Api/WeatherWalkingSkeleton.csproj

If the output is error-message free, similar to the following...

info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://localhost:5000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /Users/YOU-USER-NAME/workspace/projects/Portfolio/BlogTutorials/WeatherWalkingSkeleton/Api

... then the application is working properly.

Open Postman, and make a GET request to this URL : https://localhost:5001/WeatherForecast?location=chicago. (You can substitute any other city for Chicago).

If the response is an array of temperature forecast objects, then your OpenWeatherMap API key is configured properly. If not, you may want to go back to Part 1 of this series to check over the User Secrets Manager setup. Alternately, if you just want to hard-code the API key and skip saving it to your file system, I'll show you the place to do that further below.

The file that we will be refactoring the most in this tutorial is located at ./Api/Services/OpenWeatherService.cs. Open that now in an IDE or text editor. It contains a method called GetFiveDayForecast:

public List<WeatherForecast> GetFiveDayForecast(string location, Unit unit = Unit.Metric)
{
    string url = $"https://api.openweathermap.org/data/2.5/forecast?q={ location }&appid={ _openWeatherConfig.ApiKey }&units={ unit }";
    var forecasts = new List<WeatherForecast>();
    using (HttpClient client = new HttpClient())
    {
        var response = client.GetAsync(url).Result;
        var json = response.Content.ReadAsStringAsync().Result;
        var openWeatherResponse = JsonSerializer.Deserialize<OpenWeatherResponse>(json);
        foreach (var forecast in openWeatherResponse.Forecasts)
        {
            forecasts.Add(new WeatherForecast
            {
                Date = new DateTime(forecast.Dt),
                Temp = forecast.Temps.Temp,
                FeelsLike = forecast.Temps.FeelsLike,
                TempMin = forecast.Temps.TempMin,
                TempMax = forecast.Temps.TempMax,
            });
        }
    }

    return forecasts;
}

The method takes in the name of a location and a unit of measurement, defaulting to metric.
A URL is built using these parameters and an API key. Note that if you want to skip the configuration step from the previous tutorial, you can copy paste your OpenWeatherMap API key here, and the method should work.
An HttpClient object is built to return the data.
HttpClient makes the call to the API, reads the result as a JSON string, then deserializes the desired values.
Finally, a list of weather forecasts is built and returned.

This method will work as is, but it has a few code smells. First, the service class will be difficult to unit test because HttpClient is instantiated directly in the method. A test that triggers the method will make an actual call to the API which is undesirable for several reasons including inconsistent test results if the API happens to fail. We can take advantage of dependency injection to make the HttpClient object more versatile.

Another problem is that we are calling two asynchronous methods from HttpClient, but we are still using them synchronously. As written, GetAsync and ReadAsStringAsync will block the thread that is running this process until the two methods resolve. For what we're doing right now, this doesn't make a difference, but if we imagine a more complex version of this method - one that fetches data from several sources or processes large amounts of data - then the results of this method would not be performant nor consistent. We want to change our method to take advantage of asynchronous features that are available in C#.

Finally, while not necessarily a code smell, we can have the URL built by another method. This could make the URL-building DRY-er and more versatile as other methods get added to this class in the future.

Creating a URL-builder

Writing a separate URL method will be the easiest step in refactoring. When we break down the structure of the OpenWeatherMap url, its parts are primarily domain + version number + a resource + query parameters + the API key. Let's create a method where we can at least specify the resource and the query parameters when requesting a URL.

Below the GetFiveDayForecast method, add the following private method:

private string BuildOpenWeatherUrl(string resource, string location, Unit unit)
{
    return $"https://api.openweathermap.org/data/2.5/{resource}" +
           $"?appid={_openWeatherConfig.ApiKey}" +
           $"&q={location}" +
           $"&units={unit}";
}

And in GetFiveDayForecast, change the hard-coded url to the result of the private method:

public async Task<List<WeatherForecast>> GetFiveDayForecastAsync(string location, Unit unit = Unit.Metric)
{
    string url = BuildOpenWeatherUrl("forecast", location, unit);
    // [...]
}

Nothing major there, but at least it moves a big ugly string out of the method and makes it more readable.

Injecting IHttpClientFactory

For us to unit test our OpenWeatherMapService class in the future, it would be better to create a mock of HttpClient that will return canned responses that look like OpenWeatherMap responses without actually using the real API. We want our tests to describe the service independently of the API.

However, while HttpClient is a very common way to make HTTP requests, it is an older .NET library and as such it is not really designed to be easily mocked in unit tests. Luckily, .NET Core provides us with an interface called IHttpClientFactory which solves a number of issues with making a WebApi application more stable and scalable. Specifically for us, IHttpFactory will create an HttpClient for us instead of us directly instantiating it in our method. That way, in future unit tests, we can easily inject a pseudo-HttpClient.

First, we will add IHttpClientFactory as one of the service's class constructor parameters and hold it as a private attribute:

public class OpenWeatherService : IOpenWeatherService
{
    private readonly OpenWeather _openWeatherConfig;
    private readonly IHttpClientFactory _httpFactory;

    public OpenWeatherService(IOptions<OpenWeather> opts, IHttpClientFactory httpFactory)
    {
        _openWeatherConfig = opts.Value;
        _httpFactory = httpFactory;
    }
    // [...]
}

Before moving on, we'll need to declare an instance of an IHttpClientFactory class in our Startup class. From the root directory, open ./Api/Startup.cs and add the following line to the ConfigureServices method:

public class Startup
{
   // [...] 
    public void ConfigureServices(IServiceCollection services)
    {
        // [...] 
        services.AddHttpClient();
    }
}

The documentation describes a number of possibilities for setting up an IHttpClientFactory, but for now this will do what our code is already doing.

Back to the OpenWeatherService, we'll replace the code in the block that starts with using (HttpClient client = new HttpClient()) with the following:

var client = _httpFactory.CreateClient();
var response = await client.GetAsync(url);
var json = await response.Content.ReadAsStringAsync();
var openWeatherResponse = JsonSerializer.Deserialize<OpenWeatherResponse>(json);
foreach (var forecast in openWeatherResponse.Forecasts)
{
    forecasts.Add(new WeatherForecast
    {
        Date = new DateTime(forecast.Dt),
        Temp = forecast.Temps.Temp,
        FeelsLike = forecast.Temps.FeelsLike,
        TempMin = forecast.Temps.TempMin,
        TempMax = forecast.Temps.TempMax,
    });
}

Now we've separated creating the HttpClient object from our own class.

At this point, the method will show errors because of the lines starting with var response and var json. WE have added the await keyword so that response and json represent the result of the two tasks that were called. In order to run this properly, we'll need to convert GetFiveDayWeatherForecast to an async method.

Converting to async

We're halfway through refactoring, but we've still got our more complicated tasks ahead. Now we'll convert GetFiveDayWeatherForecast so that it runs asynchronously.

Normally a program written in C# runs sequentially, line by line, and each subsequent step waits for the preceding step. However, some operations that require separate processes - eg. reading from a database, reading from a file system - will cause the process that is executing to stop until other operations can finish. C# provides us with features to better manage these processes so that some might run in parallel and the results returned in order. This article provides a detailed description of async methods in C# that is well worth reading. In C#, asynchronous methods are typically managed as tasks that return a result after completion. We use the await keyword to make sure that we have the result of the process before performing any logic on it.

To transform GetGiveDayWeatherForecast into an async method, first we need to change its return type from a list of weather forecasts, to a task that will result in this list. While, we're at it, we'll rename the method GetFiveDayWeatherForecastAsync which is a convention in C# that indicates to anyone writing a consuming class that the method can be await-ed. We'll change the method in the IOpenWeatherService interface and the OpenWeatherService class.

public interface IOpenWeatherService
{
    Task<List<WeatherForecast>> GetFiveDayForecastAsync(string location, Unit unit = Unit.Metric);
}

public class OpenWeatherService : IOpenWeatherService
{
    // [...]
    public async Task<List<WeatherForecast>> GetFiveDayForecastAsync(string location, Unit unit = Unit.Metric)
    // [...]
}

We'll also need to change the reference to the method in the WeatherForecastController class located in ./Api/Controllers/WeatherForecastController.cs. Since we will be awaiting the result of GetFiveDayForecast, the controller method has to be async as well:

public class WeatherForecastController : ControllerBase
{
    // [...]
    [HttpGet]
    public async Task<IActionResult> Get(string location, Unit unit = Unit.Metric)
    {
        var forecast = await _weatherService.GetFiveDayForecastAsync(location, unit);
        return Ok(forecast);
    }
}

You may want to compile the code ($ dotnet build) to make sure these changes are correct before moving on.

Back in OpenWeatherService, we can add await to the two async methods that are called from HttpClient:

public async Task<List<WeatherForecast>> GetFiveDayForecastAsync(string location, Unit unit = Unit.Metric)
{
    // [...]
    var response = await client.GetAsync(url);
    // [...]
    var json = await response.Content.ReadAsStringAsync();

This ensures that the method waits until there's a response from the HTTP client, then waits for the result to be encoded. If our method was also getting data from another source, or performing some other intensive operation, it could do that at the same time. As it is, our method isn't performing differently, but using async features where possible is still usually best practice.

Run the application and test https://localhost:5001/WeatherForecast?location=chicago again to make sure no errors were introduced. If everything is still working as it did originally, then the refactoring is working.

Summary

The changes in this step haven't been very drastic, but they at least establish some more sustainable patterns as we continue to build out this application. Now that our classes are more testable and using async methods when possible, the next step is to set up unit tests and use those tests to support adding better exception handling.

Setting up an N-Tier ASP.NET Core App

Jeremy Wells — Mon, 22 Jun 2020 18:42:08 +0000

Photo by Elena Koycheva on Unsplash

Introduction

This is article #1 in a series of tutorials that walks through building and hosting an Angular and ASP.NET Core web application. The application will have extremely minimal functionality - a Walking Skeleton - but it can serve as a template for building out functionality in more useful projects.

In the previous article, I gave a more detailed overview of this series and walked through preparing a development environment for .NET Core. I also created and explained the default boilerplate code that comes from creating an ASP.NET Core WebApi application. For a little more background, I suggest reading through that article first before stepping through this tutorial.

Prerequisites

For this tutorial, I'll assume you'll aren't yet familiar with ASP.NET Core, but you have the .NET Core SDK and an IDE or text editor set up and ready to go, If not, the previous article will go through that step. If this is your first time building a server-side application, or if you're familiar with a comparable framework like Rails or Django, but haven't tried one from the .NET Core family, then hopefully this article will clearly introduce you to the process with this framework. I won't go into the syntax of the C# language (for that, I recommend Microsoft's introduction), but I'll explain the steps of building our application in detail.

N-Tier architecture

A common design pattern for web applications is called the N-Tier Pattern, where n is the number of layers of the application. Wikipedia describes one N-Tier scenario as a presentation tier, a logic tier, and a data tier:

Source: Wikipedia

An important feature of this pattern is that a lower layer is not aware of any higher layers and changes to a higher layer do not affect lower layers. This pattern can present itself as simply a separation between client-side code, a service layer that contains the business logic, and a data-access layer for interfacing with a database. And this Microsoft Azure article describes a much more scaled version of this pattern that separates the tiers into a number of physical processes:

Source: Microsoft

In our very simple application, the presentation layer will be an Angular client that we'll build in a subsequent article. Our controller will represent a web tier and in this tutorial, we'll build and secure a service as a business-logic tier that will call an external API.

The result will be an API endpoint that returns the next 5 temperatures in a forecast by location.

Here are the steps to get there:

Sign up for and retrieve an API key to use the OpenWeatherMap service.
Secure the API key so that it can be used in our application.
Build a service that handles requests to the OpenWeatherMap API.
Modify the controller to work with the service, then test it in Postman.

Going through these steps will demonstrate how ASP.NET Core uses dependency injection, protects sensitive values such as API keys and database connection strings, and how it handles exceptions.

The starting point for this code can be found at this repo. Using Git, you can clone the repo locally using:



$ git clone -b 0_GettingStarted --single-branch git@github.com:jsheridanwells/WeatherWalkingSkeleton.git

Then restore the project:



$ dotnet restore

Getting the OpenWeatherMap API key

Instead of returning random objects in the boilerplate code, we'll return values from a live API. The OpenWeatherMap API is an easy way to incorporate third-party data when testing out a project. You'll need to create an account, then get an API key to be able to make requests from their service.

Go to https://OpenWeatherMap.org/api and click "Sign in". Create an account if you haven't before and log in.
If you are at home.openweathermap.org, at the top of the page is a nav item called "API keys". Click that and you'll arrive here.
Click the "Generate" button. Name the key and save it.
The table will now list your API key. We'll copy this in the next step so that it's available in our application.

At this point, we can test out the OpenWeatherMap API to get an idea of the data structure it returns. Postman is a great tool for this. You can download it from here if you don't have it yet.

We'll test out the 5-day Weather Forecast endpoint. There are a variety of ways to query this resource: We'll query it by city name.

According to the documentation, the structure for the url is



api.openweathermap.org/data/2.5/forecast?q={ city name }&appid={ your api key }

We'll set that up in Postman by entering the url up to the resource name (forecast), then entering a city name and our api key in the query parameters table below. Also, you can add a units parameter with a value of either metric or imperial, otherwise the temperatures will be returned in Kelvin. Since I'm writing from the United States, I've opted for imperial.

If everything is set up correctly, your response will be an array of 40 temperature objects for whatever city was selected. We'll keep Postman open so that we can use the URL that was formed when we create a service in our web API to make this request.

Securing our API key

Although I'm having a hard time imagining what mischief could be made with my OpenWeatherMap API key, it's still best practice to store the actual value separately from the source code. In a real project, there will be all kinds of secret keys, passwords, and database connections strings - and these values would be different from environment to environment - so here we'll save the API key in our file system, then bring it into our application configuration. The User Secrets API that comes with the dotnet SDK is an ideal tool for this.

To accomplish this, we'll:

Create a class to help inject our key in the places where we need it.
Save our key to the file system using the dotnet cli.
Bring our key into the configuration schema in our project's Startup.cs class.

First, we'll create a class called OpenWeatherMap and give it one property: ApiKey:



$ mkdir Config
$ touch Config/OpenWeather.cs

(Note: if you're using Visual Studio, you can create this file using the Solution Explorer)

Add these contents to OpenWeather.cs:

When the dotnet CLI saves secrets for a project, it's in a directory structured as follows:



$ ~/.microsoft/usersecrets/<USER-SECRETS-ID>/secrets.json

The USER-SECRETS-ID is saved in the .csproj file at the root of the project. Any string will work as a user secrets ID, but for this project we'll use a GUID: 65988f0a-26ed-44ef-8749-f86a2f5c18a9 (you can also generate your own GUID if you prefer).

Open WeatherWalkingSkeleton.csproj and add the UserSecretID:

Next, run the following command, replacing YOUR-API-KEY with the API key you generated when signing up for the OpenWeatherMap service:



$ dotnet user-secrets set "OpenWeather:ApiKey" "YOUR-API-KEY"

If you're successful, you should see output like this:



Successfully saved OpenWeather:ApiKey = YOUR-API-KEY to the secret store.

Now that the key value is stored in our usersecrets/ directory, we need to bring it into the application. This is done in the Startup class by calling a method from the Configuration object, then adding it to the application's service collection:

Now the API key is available in the application when we use the OpenWeatherMap API. We'll be able to confirm that later when we build a service to call the API.

Classes for mapping the OpenWeatherMap API response

Our next step is to create a service to sit between the WeatherForecastController and the OpenWeatherMap API. Ideally, a controller's only responsibilities are to route requests to the right services and to return either success or error responses. All the business logic should be handled at the lower levels.

Before implementing the actual service though, we'll need to create a class to deserialize the response we get from the OpenWeatherMap API. We'll also create a class for our own API response that will organize the data the way we would like to present it.

First, delete the default ./WeatherForecast.cs class that was created when the project was bootstrapped.

We'll create a Models/ directory, with two classes: a new and improved WeatherForecast which will be the object that gets returned from the WeatherForecastController, and an OpenWeatherResponse which we'll use to deserialize the JSON data returned from OpenWeatherMap.



$ mkdir Models
$ touch Models/{WeatherForecast.cs,OpenWeatherResponse.cs}

After inspecting the response from OpenWeatherMap in Postman,...

...we see an array called list. I've decided that I'd like our API to return the date and time, temperature, the "feels like" temperature, as well as the min and max temperatures for each item in that array. We'll add those properties to the WeatherForecast model.

Feel free to experiment and extract different kinds of values from this response as you follow along.

This class is pretty straightforward, but extracting these values from the OpenWeatherMap response, a rather complex JSON object, will take more work. The response is organized as follows:

The root object contains an array property called list.
Each item in list contains a Unix timestamp called dt and an object called main.
main then holds the different temperatures that we want.

To deserialize this response into a C# object, we'll create three classes, and leverage a library called System.Text.Json which is included in ASP.NET Core as of version 3x.

Add the following to OpenWeatherResponse.cs:

We'll name the root object OpenWeatherResponse. The System.Text.Json library provides a data annotation called JsonPropertyName which allows us to indicate the json property that we're extracting these values from. This way we can take an array originally called list and name it something more meaningful in this context: Forecasts.

We'll create a Forecast class that will hold the dt and main properties from the API response:

By taking the values from main and renaming them Temps, our own code will be easier to understand.

Below the Forecast class, we'll add the final class called Temps to indicate which temperature values to include. As before, we can use JsonPropertyName to name our properties with the conventional C#-style casing.

Now that a strategy for handling the API data is in place, the next step is to call the API with a service.

Setting up and injecting an ASP.NET Core service

As mentioned earlier, ASP.NET Core uses dependency injection as a primary design consideration, and we'll see how this works here as we implement a service that calls the OpenWeatherMap API and returns the data as a WeatherForecast object. Our service will be called OpenWeatherService with a method called GetFiveDayForecast. The method will take a location and a unit of measurement to use when calling the API. The service will be represented in other classes as an interface called IOpenWeatherService.

We'll build it in such a way that we can verify that the service is wired up correctly in the application first before getting into actual functionality: We'll create the service class, create the interface, register the types in Startup, then inject them into the WeatherForecastController. After these steps, we'll implement the actual GetFiveDayForecast method.

Create a new file to hold the service and the interface:



$ mkdir Services
$ touch Services/OpenWeatherService.cs

Normally, we would have the interface and the class in separate files, but since this class will only have one method right now, we'll keep them together. Add the following to create the IOpenWeatherService interface:

Our interface defines a method that accepts a location and a unit of measurement, and we've restricted it to the three acceptable options using an enum. The method will return a list of objects of the WeatherForecast type that we defined earlier.

Below the interface we'll add a service class to implement the method:

So far, the only thing the method does is raise an exception to say it's not ready yet. We'll leave it this way for now so that we can register it in the Startup class and inject it into the controller. Then we can do a quick test with the controller to make sure the method is getting called. After that, we'll build out the method.

Open Startup.cs again and find the ConfigureServices method. Below the line where we brought in the API key, add the following line:

services.AddScoped is the method telling the application to instantiate an OpenWeatherService object whenever another class depends on the IOpenWeatherService interface. ASP.NET Core provides three methods for registering dependencies: AddSingleton which initializes an object once during the application lifecycle, AddScoped which keeps the same object available during a single request before disposing it, and AddTransient which provides a new instance every time it is injected. The Microsoft docs explain service lifetimes in more detail.

Next, we'll modify the WeatherForecastController to return data from the OpenWeatherService instead of random values.

First, we will inject IOpenWeatherService in the class constructor and make it available as a private property called _weatherService.

Next, we'll change the Get method to use the data returned from the service.

Anything else in the controller can be deleted.

Instead of producing values directly in the controller, we're returning the result of data produced elsewhere, so our return type will be IActionResult. We're receiving an object we'll call forecast from the OpenWeatherService, and we'll pass it on in the body of a "Success" response: Ok(). Also, for now, I've hard-coded "Chicago" as the argument for GetFiveDayForecast to that everything will compile. Later, we'll modify Get again to accept query parameters.

Let's do a quick test to make sure that OpenWeatherService.GetFiveDayForecast() is being called within the application. Run the application ($ dotnet run) and call the WeatherForecastController from Postman or a browser (https://localhost:5001/WeatherForecast). The response we expect looks like this:

Since the stack trace is throwing a NotImplementedException called from the OpenWeatherService in our WeatherForecastController, everything is working as expected. Now, we'll make our five-day forecast method actually work.

Implementing our HTTP service

Open OpenWeatherService.cs back up. The plan for our GetFiveDayForecast method is for it to do what we were doing before in Postman: make a GET request to the OpenWeatherMap API and map the values we want as a list of WeatherForecast objects.

The first thing to do is to build a URL so that it looks like the URL that was successful for us when testing the API in Postman, e.g.:



api.openweathermap.org/data/2.5/forecast?q=Chicago&appid=YOUR-API-KEY8&units=imperial

First, we need to extract the API key from the configuration of our application. At the top of the class we'll create a constructor that injects an instance of IOptions:

Using the OpenWeather type, we can deserialize the value of our API key from the local environment without having to write it anywhere in the code. We'll store it as a private property called _openWeatherConfig. With the API key, along with the location and unit of measurement coming in as method arguments, we have the values needed to build the URL. We'll do that on the first line of the GetFiveDayForecast method:

If you are handy with the debugger in the IDE that you're using, you should set a breakpoint here and inspect the value of _openWeatherConfig.ApiKey.

Next we'll create a list of forecasts that the method will return:

In the middle is where we add the logic to:

Make a GET request to the OpenWeatherMap API,
Deserialize the response,
And use the response to build the list of WeatherForecast object we're returning.

That implementation can be described as follows:

The entire method should now look like this:

Start the application again, make the same API request to /WeatherForecast and we should now see actual data.

The final bit of functionality we'll need is to update the controller so that we can also send location and unit of measurement parameters with our requests. Back in WeatherForecast.cs, we'll add two arguments to the Get method and use those when we call GetFiveDayForecast:

The ASP.NET Core ControllerBase class has several ways to extract parameters and data from HTTP requests. Here, if we simply add arguments to a controller method, and add the matching names in the URL...



https://localhost:5001/WeatherForecast?location=london&unit=kelvin

...the controller will bring them in as parameters.

Run the application again, make a POSTMAN request at the above url. If everything is great, you should see a response like this:

Summary

Now that we've got an API of our own returning real data, now is a good stopping point before building in a little more infrastructure in this project. We'll do that in future tutorials.

Using the above steps, we were able to conceal a secret API key using .NET Core's user secrets. We built out a service class to pull data from an external resource and modified a controller class so that it is limited to simply relaying requests and responses. We also saw how the Startup class in an ASP.NET Core application sets up dependency injection and configures services using values from the host environment. With this basic functionality in place, we'll be able to explore some of the other necessary components to deploy a working walking skeleton and have a solid reference for putting together our real projects.

To reference the code that was created in this tutorial, see the 1_aspnetcore_webapi_setup branch in its Github repo.

Starting an Asp.Net Core Walking Skeleton

Jeremy Wells — Thu, 11 Jun 2020 19:02:34 +0000

Photo by diana on Unsplash

Introduction

I first learned about the concept of a "walking skeleton" project when I was following Michael Hartl's excellent Ruby on Rails tutorial. One of my favorite learning tools was an introductory chapter that builds out the simplest possible iteration of a Rails project from start to deployment in order to see the process of developing a web application end-to-end. If I can go from an initial CLI command to getting a page to say "Hello" back to me from a public web server, then I can be sure I will greatly minimize the future troubleshooting that comes with taking an application from local development to production environments - a solid skeleton for building out real features.

With the walking skeleton idea in mind, this post will be the beginning of a series of tutorials taking an extremely simple ASP.NET Core web application and building it out - from client, to server, to database, to tests, to deployment - with simple renderings of each necessary component. I intend for this project as a place to experiment with the new tools and techniques I learn along the way, and a reference for going back after I've long forgotten about what I did. Throughout, I hope this will serve as a clear introductory tutorial for beginning web developers looking to pick up on the offerings from the .NET ecosystem.

As of .NET Core version 3, the scaffold code for an ASP.NET Core WebApi project produces a simple endpoint that returns a set of temperatures. The Weather Walking Skeleton will take this scaffold and turn it into an API that accepts a location and returns the expected temperature for the next five days. Not an interesting requirement at all, but it provides a simple unit of functionality that can be tested throughout the layers of a web application.

Project overview and prerequisites

For now, here is the plan for the project:

An ASP.NET Core WebApi application will accept the name of a location from a client and return the expected temperatures for the next five days from the OpenWeatherMap API.
A suite of unit tests using xUnit.
An Angular client.
A CI/CD pipeling using the tools available from Azure DevOps.
Deployment to an AWS EC2 instance,

These are the current technologies that I use in my job, and as I explore others, I'll add them to this stack as I go.

This project will live at this Github repo. I'll commit example code for every tutorial to its own branch, and the master branch will be the latest working version of the project.

These tutorials are intended for beginning web developers who want to get an initial understanding of the different technologies I'll use. I'll assume no prior experience with .NET or .NET Core, but I won't go into the syntactic features of the C# language. For that, I highly recommend Microsoft's own intro tutorials.

The last prerequisite is to have the .NET Core SDK installed on your machine. As of this writing, the latest version is 3.1 available for Windows, Mac OSX, and Linux. Visual Studio is the standard IDE for writing .NET applications, and if you're using a Windows machine, then downloading the free Visual Studio Community edition is a great option. On a Mac, using Visual Studio Code with a few plugins is also a viable free option (this is how I'm writing these tutorials). I'll try to be as platform- and IDE-agnostic as I can.

Scaffolding and picking through an ASP.NET Core WebApi

I'm assuming now you've got the .NET Core SDK installed and an IDE or text editor to view it with. We will use the command line to scaffold a WebApi project:

And these are the files that are produced:

You should cd into the directory now: $ cd WeatherWalkingSkeleton.

One last initial step - if you want to use Git for source control, you should add a .gitignore file with these contents. This is a general list of paths to exclude from source control that will work with most project types and IDEs.

For the rest of this tutorial, we'll go through the important files in this project, then run the project to make sure it's working.

File walkthrough

What the .NET Core CLI has given us so far is very basic boilerplate code that returns a list of weather temperatures from a single endpoint, just enough for us to see that we have working WebApi project. The three important files that make this happen are Program.cs, Startup.cs, and WeatherForecastController.cs.

A Program class with a Main() method is the entry point for any executable application written in C#, and it shows us that at its core a WebApi project is a console application. In our template file, the Main() method calls CreateHostBuilder() which puts together all the default settings for running a web server on our development machine:

In .NET Core version 3, the default functionality for creating the web server is pretty well hidden, but as the project grows, and we set up a more targeted deployment, we add that configuration here.

In CreateHostBuilder(), we reference a class called Startup which is found in the Startup.cs file. This is the second entry class for our web application. While Program configures the web server, Startup configures the application itself. Startup consists of two methods: ConfigureServices() and Configure().

ConfigureServices() puts together the various services and configuration settings that make up the application. Arguably one of the most important design features of an ASP.NET Core application is the way it facilitates dependency injection. All of that is ultimately implemented in this method.

The Configure() method sets up the various middleware components that each HTTP request will go through. Functionality like CORS, authentication, and exception handling will be set up here.

Our last important class lives in Controllers/WeatherForecastController.cs. Our WeatherForecastController class inherits from the ControllerBase class, and it is decorated with the [ApiController] annotation which makes its methods available as API endpoints:

The [Route("[controller]")] annotation defines the route based on the name of the controller class, so here the route is /WeatherForecast. If I had the same decoration on a controller class called UsersController, the route would be /Users.

Then, within the controller class, we've decorated a method called Get() with the [HttpGet] annotation. With the route and the action declarations, we've set up the endpoint so that a GET request to /Users will return the output of the UserController.Get() method.

Now we'll make sure this default scaffold can run locally and that there's no problem with the development environment. From your project directory ($ cd WeatherWalkingSkeleton, if you haven't already), in your terminal run the command: $ dotnet run.

If everything's good, it will tell you the project is running on ports 5000 and 5001...

..., and if you make a GET request to https://localhost:5001/WeatherForecast...

...you'll see the return of the WeatherForecastController.Get() method:

[
  {
    "date": "2020-05-26T13:06:20.464957-05:00",
    "temperatureC": -9,
    "temperatureF": 16,
    "summary": "Warm"
  },
  {
    "date": "2020-05-27T13:06:20.464998-05:00",
    "temperatureC": -20,
    "temperatureF": -3,
    "summary": "Scorching"
  },
  {
    "date": "2020-05-28T13:06:20.464999-05:00",
    "temperatureC": 49,
    "temperatureF": 120,
    "summary": "Balmy"
  },
// [...]
]

Summary

We've just gotten an overview of this walking skeleton tutorial idea, then went on to scaffold and examine the basic parts of an ASP.NET Core WebApi application. The basic entry classes - Program and Startup - help configure and launch the application on a server, while the Controller classes help define the public endpoints for the application. Lastly, we used two commands to scaffold the application and to run it locally in order to verify our local development environment is working properly before adding to the project. In the next article, I'll walk through some initial configuration in our Startup class and adding a service to return a real weather forecast from our controller.

A dive into D3

Jeremy Wells — Mon, 01 Jun 2020 20:10:14 +0000

Photo by Mika Baumeister on Unsplash

D3, or Data-Driven Documents, is still a top choice for data visualization in the browser. Part of its power is that it isn't merely a charts library, but rather a way to add and manipulate DOM elements in response to data.

As these examples show, this provides seemingly limitless possibilities for original ways to illustrate with data as well as creative coding.

However, all of that power comes with a steep learning curve, and D3 requires some investment for it to become a reliable tool. If your use case is simply to add standard data charts to a front-end application, Chart.Js (as well as several commercial products) can be a much friendlier solution. But if you are interested in original ways of presenting particularly large, unique, or complex data sets, then a dive into D3 is certainly worth the effort.

This tutorial will serve as a jumping off point. We will build a simple SPA that illustrates the forecast for a location over the next five days with a line chart. While not a particularly exciting result, my intent is to show some fo the fundamental ways D3 uses data to manipulate the DOM. This can serve as a jumping off point for exploring further with this tool, and for understanding the code in the numerous D3 examples that are out there. Note that to build the line chart for this tutorial, I borrowed heavily from this example.

0. Prerequisites

This tutorial is designed for any beginner to advanced front-end developer looking for an introduction to setting up some standard D3 methods. I'll walk through some basic steps for creating a line graph with data from the OpenWeatherMap API, describing what we're tasking D3 to do in each step. This tutorial assumes familiarity with basic front-end concepts like javascript functions, AJAX calls, and using jQuery to make selecting DOM elements easier. To set up and run the boilerplate code, you'll need at least a little familiarity with Github and running a local HTTP server (installing http-server globally is an easy option).

1. Tour of the boilerplate

The Dive-Into-D3 Github repo has got some boilerplate files to get us started. The master branch contains our starting point. If you want to just see the result, you can switch to the final branch. Either clone the repo ($ git clone git@github.com:jsheridanwells/dive-into-d3.git) or download and unpack the zip. Run your HTTP server from the project's root directory ($ http-server -o if you're using http-server).

Navigating to the project's index.html file (http://localhost:8080 for me), we can see a very basic nav bar that lets a user enter a US zip code (I've hard coded 60606 to make manual testing easier). Since I'm writing from the US, I'm using zip codes for locations and temperatures in Fahrenheit, but the code can be adjusted to suit other locales with minimal effort.

Below will be a space to show a weather chart when that gets built. If you click the submit button, nothing will happen and you'll see a 404 error in the browser's console, but if you've gotten this far, then the boilerplate code is working on your machine as expected. In the next step, we'll register for an OpenWeatherMap api key and make the error go away.

Before we start adding to the code, here's an inventory of our repo so far:

index.html is a basic Bootstrap starter page with a navbar called #nav for entering a location, and a viewport called #chart for displaying the result.
ajax.js contains a function called getWeather that retrieves data from the OpenWeatherMap api. If you don't want to go through with registering and setting up the api, there is a commented out version of the same function that just pulls some sample data from a local json file. If you want to skip step 2, you can comment out the first getWeather, uncomment the second one, and everything should still work.
line-chart.js contains a function called drawChart. This is the file we'll be adding code to in order to create our line chart in the steps below.
main.js is what gets controlled from the DOM. There is a jQuery listener that calls our getWeather function and then calls drawChart to render the data. There is also a helper function called sortTimeTemps that processes the OpenWeatherMap json result so it's easier to work with in the D3 functions.
main.css contains any styles we'll need. I won't go into how D3 leverages CSS to style the chart, but you can certainly play with the appearance of the line chart from here on your own.

You wont' have to modify any of the files above except line-chart-.js for this tutorial.

2. Getting an OpenWeatherMap api key

The OpenWeatherMap api is an option I like for testing out a new tool using data from an external source. Walk through their sign up process, then you can generate your own API key that's found at this url.

With your api key, we'll do a quick and dirty command to create an api key file that will be ignored in source control, but threaded into the rest of the application:

When you see a new javascript file called apiKey.js with the following contents, you should be ready:

Refresh your application and if everything is successful, it should call out to the OpenWeatherMap API, and you'll see different cities in the result box when you enter different US zip codes. Likewise, if you open up the console, you should see a response from the OpenWeatherMap api.

Note that this is not a great way to protect an api key for an app in production, but for now this will keep the key out of source control.

From here, we'll open up the line-chart.js file and start adding to the drawChart function. Note that we'll be creating a really long function in kind of an imperative style as a way to make the steps for building a D3 illustration more explicit.

3. Setting up an SVG

We'll start our chart by setting up an SVG element that will render our illustration. An SVG is a vector-based image format that uses plain text to determine how all the shapes and colors of an image are placed. This is as opposed to a raster-based image format like JPEG or PNG that creates an image by setting the color of each pixel in an area. Here is a great overview of SVGs and their use cases. Also, this article goes over drawing with SVGs and D3 which is also very helpful.

In the drawChart function, we'll begin by setting up the height, width, and margins of our chart so that it fits within the space we've allocated to it on the page.

The first set of variables - margin, width, and height - are set using the height of the browser window and whatever heights the #chart and #nav divs happen to be. This assures us that our chart will fit where it's supposed to be no matter the size of the browser window.

The second variable - svg - comes from D3 selecting the #chart div, and appending an svg element. The height and width are SVG attributes that are set using the height, width, and margin variables that we came up with in the first step. An empty <g> element is added to the SVG. <g> is an element to "group" the other child elements that will be added to the SVG later. Lastly, we're setting the transform attribute to shift the SVG so that there's some space between the SVG and the edge of the <div> that it lives in.

Now, reload the page, click the Submit button, and you should see a rectangular outline in the #chart area. Inside that div, an empty SVG should be rendered:

4. Setting the X and Y axes

Next, we'll set the scale for the X and Y axes. The Y axis of our chart will be the temperatures over the next five days while the X axis will be the dates and times.

We'll add the next two steps to the drawChart function:

Our data - called chartData - will be coming in as an array of objects with two keys: temp_max and time, looking like this:

The yScale is set with D3's scaleLinear method. This method basically maps the lowest and highest values of a dataset to the height of the Y axis in our chart. The domain method tells us to take the lowest to highest temp_max values in our array, and map it to our range which is 0 to the height of our chart (plus 10 to give a little space over the highest point that our line will reach).

xScale is working similarly, setting a scale to our time values and mapping it from 0 to the width of the chart. Creating the scale with scaleTime() rather than scaleLinear() gives us access to special time-based values, while the latter function returns a linear series of integers.

The scales are mapping objects that will be used in the next steps, so they don't produce anything in our HTML yet.

5. Adding points for the line

We'll come back to another SVG method, this time to place dots across the chart for each time and temperature intersection. Let's add this to drawChart:

This is where D3 can be tricky to reason out. It starts with selectAll, which says to select elements with the CSS class .dot, but we don't have elements with that class until further down the chain (I'm thinking of it like the way SELECT comes first in a SQL query before you indicate the table to select from).

We're declaring that our data is the chartData we're sending into the function. enter signals that new elements need to be added to the SVG, and append('circle') indicates what elements to add. It will add one circle for each item in the chartData array (If you're confused, and I don't blame you, this explanation of enter and exit in D3 is helpful).

Now, each circle will have the CSS class .dot (selected in the first step). The final SVG attributes that we are adding are 'cx' and 'cy'which are the circle's X and Y axis locations, then r which is the radius of the circle. Notice that the xScale and yScale functions from earlier are used to take the time and temperature values and map them to an x,y location for each circle.

If we refresh our page and submit another zip code, we'll see a series of dots across the line chart area:

Likewise, all of our circles will have been added to the SVG:

6. Connecting the points

Next, we'll draw a line to connect the circles:

d3.line() returns a function that will output the X,Y coordinates for our line to follow, and we're indicating that the time and temp_max properties will determine those values. The output of the d3.line() function is an SVG path string: a string of commands for a path to follow.

To actually draw the line, we'll add a <path> element to the SVG:

We are appending the path to the SVG, indicating that chartData is our data, and using the output of line to set the path's d attribute (apparently d stands for data). Lastly, we are adding a CSS class called 'line' that sets the appearance of the line. In main.css, this is how I styled the line in the boilerplate, but play around with it as you'd like:

Now if you refresh the browser and submit another zip code, a line will connect the dots:

7. Adding the Labels

Our line doesn't make much sense without axis labels. We will append two more groups (<g>) to our SVG to indicate the time of day for every six hours on the X axis:

First we push the object to the bottom of the SVG with a translate command, then we use the mapping in the xScale function to determine the values, while the ticks set the intervals.

The append command for the Y-axis is a little bit simpler to add, but more involved under the hood:

For each item in the array that yScale outputs, D3 is adding a vertical align, then an SVG <text> element at every interval set by the yScale array. The D3 Axis Methods are worth getting more familiar with as they save a lot of tedious work with setting up axis labels.

A final missing piece is that our viewer has no idea what our Y-axis numbers represent. Our last touch is to add a 'Temperature (F)' label. We'll extend the last append function further:

The append('text') and everything that follows is simply manually setting the attributes of an additional SVG <text> element.

And now we've got our line chart:

Conclusion

And with the above function, you've got an SVG that dynamically reacts to different data sets that are sent to it. By manipulating SVG elements, setting scales, and leveraging some of D3's extensive functions for manipulating and rendering data, you can see the level of detail that can be applied to creating visualizations with raw data. We get a sense of the process of setting up a D3 visualization that we can experiment with further, as well as study the thousands of other impressive D3 data illustration and creative coding that are out there.

Forem: Jeremy Wells

A headless Raspberry Pi workflow

Photo by Quentin Schulz on Unsplash

Prereqisites

Creating your Raspberry Pi image

SSH and networking config

First Login

Dockerizing a Mongo database

Photo by Salomé Guruli on Unsplash

Introduction

Prerequisites and loading the sample project

Some terms and concepts

Environment variables

Some setup scripts

Creating our Mongo service

Getting it up and running

Seeding the database

Cleanup and some helpful scripts

Conclusion

A Modern MEAN-stack with Angular and Typescript

Photo by Tolu Olubode on Unsplash

Introduction

Prerequisites

Set up an empty Angular project

Install some packages to support Express

Setting up a build artifacts directory

Setting up scripts for unified client and server-side development

Bonus: Refresh the browser when client-side code changes

Putting it all together

Conclusion

How to quickly mock an API server in a client-side project

Photo by Lars Kienle on Unsplash

Introduction

Prerequisites

Step one: Data modeling

Step two: Generate mock data with Mockaroo

Serving the mock data with json-server

Conclusion

Configuration in ASP.NET Core Explained

Photo by Alexey Ruban on Unsplash

Prerequisites

The boilerplate

Loading configuration in Startup.cs

The provider hierarchy

appsettings.json

User Secrets Manager

Environment variables, command line arguments, and container commands

Setting the configuration provider hierarchy

Conclusion

Further reading

TDD and Exception Handling with xUnit in ASP.NET Core

Photo by Nicolas Thomas on Unsplash

Introduction and prerequisites

Our TDD process

Our changes

Testing the service

Testing the controller

Conclusion

TDD and Exception Handling with xUnit in ASP.NET Core

Photo by Nicolas Thomas on Unsplash

Introduction and prerequisites

Our TDD process

Our changes

Testing the service

Testing the controller

Conclusion

Testing an ASP.NET Core Service with xUnit

Photo by Joyce McCown on Unsplash

Introduction and prerequisites

Testing Goals

Creating a test project

Setting up test fixtures

1. Setting up an Options Builder

2. Setting up an HttpClient Builder

3. Generating Controlled Third-Party API Responses

Testing the Service

Testing the Controller

Conclusion

References

Consuming third-party APIs in ASP.NET Core