Forem: Daniel Blackwell

Nuxt.js and Vercel Functions without using `vercel dev`

Daniel Blackwell — Mon, 12 Apr 2021 14:06:04 +0000

I'd recently being having issues using Vercel's serverless functions with SSG. Although I haven't been able to fix that just yet, I realised I could run vercel functions without using the dev proxy.

NOTE: I wasn't able to consistently get the correct data from the body of the request. Not sure why, but res.read() was skipping the first line of the request body when toString() was called. This means with my current code ONLY GET requests are supported. Shouldn't be hard to get it working with POST if you can spare the troubleshooting time.

Add `@vercel/node`

You will need @vercel/node either globally or locally. You don't need to use it directly, but the file we are copying will reference back to this package.

Nuxt `serverMiddleware` to the rescue.

In production, Vercel uses serverless functions to handle the calls to the functions found at /api. These functions are not available using the standard yarn dev.

serverMiddleware allows any call to the dev server be intercepted.

For us, that means allowing requests to captured and redirected to Vercel functions directly. The only issue there is pass over the request and response in a way that the functions expect.

The @vercel/node package has the file @vercel\node\dist\helpers.js which contains the js that normally runs via the vercel dev command.

Putting it together

`/middleware-server/vercel.ts`

vercel.ts is the serverMiddleware handler that intercepts the calls to any url under /api.
The request and response have properties added to become a VercelResponse and a VercelRequest.
The api function is loaded using a require, but the require cache is cleared of the module first to make sure only the latest code is used.

// To convert to JS, remove the type imports and the type declarations
// e.g. : VercelRequest 
import { ServerMiddleware } from '@nuxt/types'
import { VercelResponse, VercelRequest, VercelApiHandler } from '@vercel/node'

const helpers = require('./helpers'); 

const vercel: ServerMiddleware = async (req, res, next) => {

    // const body = req.read(); - this should work, but it's being very strange. Leaving for now.
    const body = Buffer.from("");

    const apiNameRegex = /\/([\w-]+).*/g

    const matches = apiNameRegex.exec(req.url || "");

    if (!matches || !matches[1]) {
        res.statusCode = 404;
        res.end()
        return;
    }

    const apiName = matches[1];
    const moduleName = '../api/' + apiName;

    // allow dynamic code change reloads - possibly detect changes and remove that way
    const requestedApi = requireReload(moduleName).default as VercelApiHandler;

    const vercelRequest: VercelRequest = helpers.httpToVercelRequest(req, body);
    const vercelResponse: VercelResponse = helpers.httpToVercelResponse(req, res);

    try {
        requestedApi(vercelRequest, vercelResponse)
    } catch (err) {
        console.error(err)
        res.statusCode = 500;
        res.end()
    }
}

/// https://stackoverflow.com/a/58645175/59532
function requireReload(moduleName: string) {

    let mp = require.resolve(moduleName)
    if (require.cache[mp]) {
        delete require.cache[mp]
        console.log(`[clear] module: ${mp}`)
    }
    return require(moduleName);
}


export default vercel;

`/middleware-server/helper.js`

helper.js is a copy of the file @vercel\node\dist\helpers.js, but some extra code at 795 to allow access to the surrounding methods.

// The file is a copy of @vercel\node\dist\helpers.js
// ADD THIS CODE AT LINE 795

function httpToVercelRequest (req, body) {
    setLazyProp(req, 'cookies', getCookieParser(req));
    setLazyProp(req, 'query', getQueryParser(req));
    setLazyProp(req, 'body', getBodyParser(req, body));
    return req;
  }

function httpToVercelResponse(req, res) {
    res.status = statusCode => status(res, statusCode);
    res.redirect = (statusOrUrl, url) => redirect(res, statusOrUrl, url);
    res.send = body => send(req, res, body);
    res.json = jsonBody => json(req, res, jsonBody);
    return res;
  }

  exports.httpToVercelRequest = httpToVercelRequest;
  exports.httpToVercelResponse = httpToVercelResponse;

Add the serverMiddleware to the nuxt.config.js

`nuxt.config.js`

  // Added property:
  serverMiddleware: [
    { path: '/api', handler: '~/middleware-server/vercel' },
  ]

Conclusion

When not running vercel dev, any calls to a url starting with /api should now be handled by nuxt, and will be redirected to the appropriate api function.

If running with vercel dev, or on a server, then the functions will be handled by Vercel, as it was before.

If you are able to get the body part working, drop the solution back here. It would be interesting to see what the issue was.

The new .Net 5 Console Logging registration

Daniel Blackwell — Fri, 22 Jan 2021 12:44:01 +0000

.Net 5 has changed the way you can set up the console for logging.

You may have received this message on your change to .Net 5:

ConsoleLoggerOptions.TimestampFormat has been deprecated. Please use ConsoleFormatterOptions.TimestampFormat instead.

The builder.UseConsole() now looks like this under the covers:

// Microsoft.Extensions.Logging.ConsoleLoggerExtensions
public static ILoggingBuilder AddConsole(this ILoggingBuilder builder)
{
    builder.AddConfiguration();
    builder.AddConsoleFormatter<JsonConsoleFormatter, JsonConsoleFormatterOptions>();
    builder.AddConsoleFormatter<SystemdConsoleFormatter, ConsoleFormatterOptions>();
    builder.AddConsoleFormatter<SimpleConsoleFormatter, SimpleConsoleFormatterOptions>();
    builder.Services.TryAddEnumerable(ServiceDescriptor.Singleton<ILoggerProvider, ConsoleLoggerProvider>());
    LoggerProviderOptions.RegisterProviderOptions<ConsoleLoggerOptions, ConsoleLoggerProvider>(builder.Services);
    return builder;
}

So if you are wanting to change formatting, you can replace the services.AddConsole() with these extension methods:

services.AddLogging(opt =>
{
    // Provided by Microsoft.Extensions.Logging.Console
    opt.AddSimpleConsole(options => options.TimestampFormat = "[HH:mm:ss] ");
    opt.AddJsonConsole(options => { });
    opt.AddSystemdConsole(options => { });
});

Run a cheap Postgres / PostGIS server on a Digital Ocean droplet for USD$6 per month

Daniel Blackwell — Sat, 09 Jan 2021 12:44:47 +0000

The following shouldn't be used for anything where data security is a concern. This is a proof of concept only.

Before you go down this route, just know Digital Ocean offers managed database solutions for about $15 per month which offers far better availability, data resiliency, security and pretty much everything else.

Sure, having a database run on a small droplet means the performance isn't great, but sometimes you need to run something on the cheap. This is a way to do just that.

Create a droplet

https://cloud.digitalocean.com/droplets

Image <- Ubuntu (I used the 20.04 LTS x64)
Shared CPU <- Basic Plan
Block Storage <- 10GB (larger volumes cost more)
Data center <- somewhere close to your server
Host name <- something useful
Cert / Password <- Password is easier for most people

You can leave everything else as it is.

SSH to the droplet

Username: root
Password: [created in previous step]
Host/IP: Should be showing in the droplet overview.

Install Postgres / Postgis

apt-get update -y && apt-get upgrade -y && \
apt-get install postgis # or postgres if that is all you need

CHANGE THE POSTGRES PASSWORD!

sudo -u postgres psql # Run psql with the postgres user
# In psql, issue the change password command \password
 \password postgres
# Enter new password:
# Enter it again:

Edit the configs

Stop Postgres

systemctl stop postgresql

Find the version of postgres you are running

ls /etc/postgresql  # lists the version folders
# "12"

Find the `/mnt/` point of your block volume

ls /mnt
# volume_lon1_01 or similar

Move the postgres data

This post was useful for pointers on moving the data.

rsync -av /var/lib/postgresql /mnt/[BLOCK VOLUME]/

Edit `/etc/postgresql/[VERSION]/main/pg_hba.conf`

Change:

# IPv4 local connections:
host    all             all             127.0.0.1/32               md5

to:

# IPv4 local connections:
host    all             all             0.0.0.0/0               md5

Edit `/etc/postgresql/[VERSION]/main/postgresql.conf`

Change:

listen_addresses = 'localhost' # listening address
data_directory = '/var/lib/postgresql/12/main' # data

to:

listen_addresses = '*' # listen to all 
data_directory = '/mnt/[BLOCK VOLUME]/postgresql/[VERSION]/main' # use block storage

Start Postgres

 systemctl start postgresql

Finally, configure the droplet firewall

Never open a database to the open web without securing it first!

https://cloud.digitalocean.com/networking/firewalls

Create a firewall
Add the Postgres port to the firewall
- Incoming TCP Port 5432 from All Sources

Setting up Entity Framework to work with geographical distances

Daniel Blackwell — Sat, 14 Nov 2020 19:43:10 +0000

This is a post about a simple solution to a simple problem:

Find distances between things in a database using .net core

Here's how I did it.

The requirements:

.Net Core 3.1 with Entity Framework

This is the base stack.

A Linux database

I couldn't use MSSQL as I'm restricted to Linux based docker images.

Server evaluated geographical distances

I need to find the distances between two points on the database side. Over-returning results and then filtering isn't ideal for anything.

The solution

Database - PostGIS a.k.a Postgres + GIS

It took a few goes (MySQL, Sqlite and then Postgres), but Postgres was the winner in getting things to work actually work and easily work with distance.
npgsql.org has very good documentation and the steps are easy to follow.

Using `docker compose`

version: '3.1'
services:
  db:
    image: postgis/postgis
    restart: always
    environment:
      POSTGRES_PASSWORD: mysecretpassword_shhhh
    ports:
     - 5432:5432
  adminer:
    image: adminer
    restart: always
    ports:
      - 8080:8080

The connection string for this setup is:
"Server=localhost;Database=mydatabase;Uid=postgres;Pwd=mysecretpassword_shhhh;"

Entity Framework and NetTopologySuite

Install Packages

NetTopologySuite <- "NTS" Provides Points and other spatial types.
Microsoft.EntityFrameworkCore
Microsoft.EntityFrameworkCore.Proxies <- Lazy loading
Npgsql.EntityFrameworkCore.PostgreSQL
Npgsql.EntityFrameworkCore.PostgreSQL.NetTopologySuite <- "NTS " Adapter

Use Points for your locations

If you are using Lat/Lon co-ordinates, then the points should be stored in Postgres as geoGRAPHY and not geoMETRY (Capitalisation is for emphasis only).
There is supposedly an extra CPU cost to calculating queries against Geographic data, but it's negligible.

using Point = NetTopologySuite.Geometries.Point;
public class MyEntity
{ 
   [Key] public int Id { get; set; }

   // Set the column type either here or in the context. 
   // NOTE: The type is geoGRAPHY not geoMETRY
   [Column(TypeName="geography")] 
   [Required] public Point Location { get; set; }
}

Setup your EF database context

The postgis extension is required for geometry and geographical types and is installed per database. The call to HasPostgresExtension ensures that this is installed.

public class MyPostgresContext: MyDatabaseContext
{
public PostgresMyContext(DbContextOptions options) : base(options){}
protected override void OnModelCreating(ModelBuilder builder)
{
base.OnModelCreating(builder);
builder.HasPostgresExtension("postgis");

// Equivalent to using [Column(TypeName="geography")] 
// NOTE: The type is geoGRAPHY not geoMETRY
builder.Entity<MyEntity>().Property(b => b.Location).HasColumnType("geography (point)");
}

And in your Startup.cs:

public void ConfigureServices(IServiceCollection services)
{
services
 .AddDbContext<MyDatabaseContext, MyPostgresContext>(builder =>
 {
     builder.UseNpgsql(CONNECTION_STRING, x => x.UseNetTopologySuite());
     builder.EnableDetailedErrors();
     builder.UseLazyLoadingProxies();     
 }) 
}

Query all the things

With this all in place, you can use the methods that deal with distance, or many other things.

var distanceInMetres = 1000; // 1 km
var entities = context.MyEntities.Select(x => x.Location.Distance(new Point(1,1)) <= distanceInMetres) .ToArray();

Hope this helps!

Passthrough JWT Authentication using Firebase & SignalR (.Net Core)

Daniel Blackwell — Mon, 13 Jul 2020 12:57:41 +0000

Why? No backchannel validation

We can validate JWT tokens without any communication with the issuer using public keys.
This means we can know that everything that is provided in the JWT is valid without a callout to somewhere else.

Pros

Many times faster than calling out to a backchannel service
As there are no backchannel requests, no API limits can be hit.

Neutral

Payload can be any size up to 7KB

Cons

Token cannot be revoked once created; token can only expire.

My code for the .Net Validation Setup is available here: https://gist.github.com/deeja/c67e6027ca37a8d6a367b8b8bf86d5c6
It should be a guide only! There needs to be work put in to make it production ready.

Firebase Authentication

When logged on with Firebase Authentication, the client is provided a Json Web Token (JWT). In the case of Firebase, these can be validated using publically shared x509 certificates.

Gettting the token

There isn't much in this post around setting up and using Firebase Auth client-side as that's not what this is supposed to be.

If you are wanting to use Firebase, I recommend following a tutorial or two, then come back to this post.

Post Login

After login, you will need to exchange your ID token for a JWT token.

JS API "Firebase Auth: getIdTokenResult()"
REST "Exchange a refresh token for an ID token"

If you are using Nuxt.js, here is a Firebase plugin that uses the @nuxt/firebase module

The Token

The Firebase JWT looks a bit like this:

eyJhbGciOiJSUzI1NiIsImtpZCI6IjIxODQ1OWJiYTE2NGJiN2I5MWMzMjhmODkxZjBiNTY1M2UzYjM4YmYiLCJ0eXAiOiJKV1QifQ.eyJuYW1lIjoiU3RldmUgTWNRdWVlbiIsInBpY3R1cmUiOiJodHRwczovL2xoMy5nb29nbGV1c2VyY29udGVudC5jb20vYS0vYXBpY3VybCIsImlzcyI6Imh0dHBzOi8vc2VjdXJldG9rZW4uZ29vZ2xlLmNvbS9bUFJPSkVDVC1JRF0iLCJhdWQiOiJbUFJPSkVDVC1JRF0iLCJhdXRoX3RpbWUiOjE1OTU1NjM2NzAsInVzZXJfaWQiOiJbVVNFUiBJRF0iLCJzdWIiOiJbVVNFUiBJRCBBR0FJTl0iLCJpYXQiOjE1OTQ2Mzc2NTksImV4cCI6MTU5NDY0MTI1OSwiZW1haWwiOiJbRU1BSUxdIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImZpcmViYXNlIjp7ImlkZW50aXRpZXMiOnsiZ29vZ2xlLmNvbSI6WyI5ODI3MzQ1OTc4MzQ1MDIzNDU5OCJdLCJlbWFpbCI6WyJbRU1BSUxdIl19LCJzaWduX2luX3Byb3ZpZGVyIjoiZ29vZ2xlLmNvbSJ9fQ.Q8p3zArOtkqcnNlNhBfdU7Bo8vtW5ML-D019lsRJTFe_hj65bNqbLyVU1BRhTsTS87DyQlA-acXmY22i5mS-vzhZcRXzoV-gkAn8Zy1xUprp7kh6he8uiIK5EoO4045e-gGFR8z3AqgpW-ZetCRT0gejq_q9mSg6cyz0UP7RCVXXyFns-RhU4gk_r7HzIclFGfPIEqabYuufJQZ_-Hv_do3gUt5BljfqAwAsSB6V8oxTfSxfqI_IBMiyU-Lxa-nCwt_S0kLWueIUUhsdkkHy2NSp4Y2EqLPtIUeWEq8EMbVfCoMKLD_TVGEk3NRPMcPQNC6CTpLUuQgpxFCaIcPXVw

Which splits into three parts, delimited by .:

Header
Payload
Signature

Header

Algorithm, Key ID and Type.

Firebase uses the RS256 encryption method.
The Key ID references public/shared keys at https://www.googleapis.com/robot/v1/metadata/x509/securetoken@system.gserviceaccount.com

{
  "alg": "RS256",
  "kid": "218459bba164bb7b91c328f891f0b5653e3b38bf",
  "typ": "JWT"
}

Payload:

General info and claims

{
  "name": "Steve McQueen",
  "picture": "https://lh3.googleusercontent.com/a-/apicurl",
  "iss": "https://securetoken.google.com/[PROJECT-ID]",
  "aud": "[PROJECT-ID]",
  "auth_time": 1595563670,
  "user_id": "[USER ID]",
  "sub": "[USER ID AGAIN]",
  "iat": 1594637659,
  "exp": 1594641259,
  "email": "[EMAIL]",
  "email_verified": true,
  "firebase": {
    "identities": {
      "google.com": [
        "98273459783450234598"
      ],
      "email": [
        "[EMAIL]"
      ]
    },
    "sign_in_provider": "google.com"
  }
}

Validation Signature

The signature is a verification token generated using Google's private keys, which can be verified using the public / shared keys.

For more info on how this is done, check out https://jwt.io/

SignalR

https://dotnet.microsoft.com/apps/aspnet/signalr
SignalR is a websockets framework that works "natively" with .Net.
The connections are made to "Hubs", and those "Hubs" co-ordinate responses based on messages and events.

SignalR JS Client

The SignalR JS client gets the JWT via a factory method on the HubConnectionBuilder.

An interesting thing is that SignalR doesn't appear to support the Bearer [My Token] Authorization header.
Instead, the token is added as a query sting with the name access_token

import { HubConnectionBuilder, LogLevel } from "@microsoft/signalr";

// using a delegate function as the factory
const getMyJwtToken = () => { /* return the token from somewhere */};

const connection = new HubConnectionBuilder()
    .withUrl(connectionUrl, {accessTokenFactory: getMyJwtToken })
    .withAutomaticReconnect()
    .configureLogging(LogLevel.Information)
    .build();

SignalR .Net Host / Server

The Host is a bit more complicated. The code for this is available on my gist https://gist.github.com/deeja/c67e6027ca37a8d6a367b8b8bf86d5c6

I will go over some of the details here.

ValidIssuer - Set to "https://securetoken.google.com/[PROJECT ID]"
Audience - Set to the PROJECT ID
AddAuthentication(JwtBearerDefaults.AuthenticationScheme).AddJwtBearer(...) - Allow use of JWT
Events.OnMessageReceived - Get the query string access_token and reassign to context.Token for handling.
OnChallenge, OnAuthenticationFailed, OnForbidden, OnTokenValidated - Use these for debugging
TokenValidationParameters - Validate everything
IssuerSigningKeyResolver = manager.GetCertificate - Set the Certificate manager to be the delegated supplier of Security keys
AddCors UseCors - Required for SignalR

`CertificateManager.cs`

As the Google public certificates can change, these need to periodically refreshed. For this I have added a CertificateManager to the gist which holds a task called _backgroundRefresher

private readonly Task _backgroundRefresher;

 public CertificateManager()
        {
            _backgroundRefresher = Task.Run(async () =>
            {
                while (true)
                {
                    await RefreshTokens();
                    await Task.Delay(1000 * 60 * CertificateFetchIntervalMinutes);
                }
            });
        }

Certificates are hydrated from the provided JSON

 var wc = new WebClient();
                var jsonString = await wc.DownloadDataTaskAsync(_googleCertUrl);
                var keyDictionary = await JsonSerializer.DeserializeAsync<Dictionary<string, string>>(new MemoryStream(jsonString));
                _certificates = keyDictionary.ToDictionary(pair => pair.Key, pair => new X509SecurityKey(new X509Certificate2(Encoding.ASCII.GetBytes(pair.Value)), pair.Key));

GetCertificate is the member that was delegated to handle the request for Certificates in the JwtBearer options.

 public IEnumerable<SecurityKey> GetCertificate(string token, SecurityToken securityToken, string kid, TokenValidationParameters validationParameters)
        {
            _lock.EnterReadLock();
            var x509SecurityKeys = _certificates.Where((pair, i) => pair.Key == kid).Select(pair => pair.Value).ToArray(); // toArray() should be called collapse expression tree
            _lock.ExitReadLock();
            return x509SecurityKeys;
        }

Nuxt.js Environment Variables (without dotenv)

Daniel Blackwell — Sun, 24 May 2020 17:16:04 +0000

This article covers Nuxt for versions 2.12 and below.
For 2.13 and above, you can now use $config with publicRuntimeConfig and privateRuntimeConfig

Nuxt's process.env works in mysterious ways. Hopefully this demystifies some of it. I'm going to cover the dotenv module in another post.

Webpack replaces process.env.YOUR_VARIABLE
Access variables using context.env
Automatic variables using NUXT_ENV_
Declared variables using nuxt.config.js

Webpack replaces `process.env.YOUR_VARIABLE` client-side

Imagine I set a system environment variable NUXT_ENV_COOL_WORD with the value jabberwocky.
When the client side JS is generated, the literal string process.env.NUXT_ENV_COOL_WORD is replaced, by Webpack, with 'jabberwocky' in the generated client JS.
This code snippet will work client side:

<template>
    <pre>
    {{processEnv}}
    </pre>
</template>
<script>
export default { 
  data() { return { processEnv: process.env.NUXT_ENV_COOL_WORD } }
}
</script>

with the generated client JS containing

    var o = {data: function() {
            return { processEnv: "jabberwocky" };
          }},

The following example doesn't won't work on the client side in the standard setup:

<template>
    <pre>Will be empty: {{processEnv.NUXT_ENV_COOL_WORD}}</pre>
</template>
<script> 
// DOESN'T WORK ON THE CLIENT RENDERED HTML
// BUT DOES ON SERVER RENDERED HTML
export default { data() { return { processEnv: process.env }}
</script>

The value of NUXT_ENV_COOL_WORD will show when the server side html is loaded, which then immediately disappears as the client side SPA JS kicks in. Although the string is not replaced on the server side generation, all environment variables are available server side.

Access variables client-side with `context.env`

Although direct access through process.env doesn't work client-side, there is still a way to use the variables.
The Nuxt.js context is populated with the declared and automatic variables.

Example of the generated code:

   app.context = {
      isStatic: process.static,
      isDev: true,
      isHMR: false,
      app,

      payload: context.payload,
      error: context.error,
      base: '/',
      env: {"NUXT_ENV_COOL_WORD":"this is pretty cool"}
    }

context is provided to:

asyncData
fetch
middleware
plugins
nuxtServerInit (server only)

example.vue

<template>
   <pre>
     async: {{myAsyncEnv}}
     fetch: {{myFetchEnv}}
   </pre>
</template>

<script>
export default {
  middleware: ["my-middleware"], 
  asyncData({env}){
    return { myAsyncEnv: env.NUXT_ENV_COOL_WORD }
  },
  fetch({env}){
    this.myFetchEnv = env.NUXT_ENV_COOL_WORD
  }
}
</script>

my-middleware.js

export default function({ env }) {
  console.log(env.NUXT_ENV_COOL_WORD);
}

Automatic `env` variables using `NUXT_ENV_`

The Nuxt.js ENV page states

If you define environment variables starting with NUXT_ENV_ in the build phase (f.ex. NUXT_ENV_COOL_WORD=freezing nuxt build, they'll be automatically injected into the process environment.

If there is a environment variable that starts with NUXT_ENV_, then that key AND value will be included in the generated JS files, and will be available from process.env on both the server and client side.

If there is no NUXT_ENV_, then the variable will not be included in the client side JS automatically; and therefore:

CAN'T be used Client side,
but CAN still be used Server side

This can be tested by:

Adding NUXT_ENV_COOL_WORD to your System Environment variables,
Restarting just to be sure it's on all processes
Test using the working example above

There is a way to get this to work using the dotenv module. That will be in another post.

Declared `env` variables using the `nuxt.config.js`

Within nuxt.config.js, one can define (pseudo) environment variables.
These can be used client and server side without issues, as well as to pass along environment variables that don't have the NUXT_ENV_ prefix

export default {
/* ... */
  env: {
    NUXT_ENV_COOL_WORD: "a default value that can be overridden",
    SOME_OTHER_VALUE: "can't be overridden - missing NUXT_ENV_ prefix",
    CONDITIONAL: process.env.CONDITIONAL || "some default value" //if can't use NUXT_ENV_
    SOME_VARIABLE: process.env.SOME_VARIABLE // Ensure variable reaches the client
  }
}

These variables are also available on the context.env

Variables with the NUXT_ENV_ are OVERRIDDEN by environment variables of the same name. This is very useful for development.