Forem: Afonso Barracha

Go for Node developers: creating an IDP from scratch - Set-up

Afonso Barracha — Sat, 19 Apr 2025 03:44:46 +0000

Series Intro

Nowadays, due to performance constraints a lot of companies are moving away from NodeJS to Go for their network and API stacks. This series is for developers interest in making the jump from Node.js to Go.

On this series, I will teach the process of building an IDP (identity provider), a project I started to create a comunity driven self-hosted opensource version of clerk.

Series Requirements

This series requires some basic knowledge of Go, as it will teach you how to structure a Go API, and how to transfer Node.js to Go skills. Before starting this series, I recommend reading the following two books to gain familiarity with the Go programming language:

Go Fundamentals: an introduction to the go programming language.
Network Programming with Go: an introduction to Network Programming, from network nodes and TCP to HTTP fundamentals.

Or if you're in a hurry just take a tour of Go in the official website.

Topics to cover

This series will be divided in 6 parts:

Project structure and Database design: I'll lay the foundation by setting up the project structure and designing the database schema for the IDP.
Local and External Authentication with JWTs: I will touch on how I implemented local authentication using JSON Web Tokens (JWTs) Bearer tokens (RFC-6750) for our Accounts (tenants) including external auth (RFC-6749).
Production mailer: to send emails we will create our own queue using redis and the "net/smtp" package from the standard library.
Component testing with docker-compose: I'll touch on how to ensure the reliability of our API by writing endpoint tests using Docker Compose and Go's standard net/http and testing packages.
Apps and account mapping: add multiple apps support for each account.
Deploying our API: We will deploy our App to a VPS (virtual private server) using coolify.

Intro

A well-organized project structure is a requirement for building maintainable and scalable APIs. In this article, we'll explore how to structure a Go API using a very common pattern in the Node.JS world, using the Model, Service, Controller (MSC) architecture.

We will leverage the following stack:

Fiber: Go framework inspired in express.
PostgreSQL with SQLC: SQLC is a Go code generator with type safety and compile-time checks for our SQL queries.
Redis with fiber's Redis abstraction: a caching storage abstraction for fiber using Redis.

We will also design the database schema and create the necessary migrations to set up our data layer.

Packages structure

For the structure of our code we'll levarage the go-blueprint CLI tool made by the youtuber Melky.

To start the project install and initialize the project:

$ go install github.com/melkeydev/go-blueprint@latest
$ makedir devlogs && cd devlogs
$ go-blueprint create --name idp --framework fiber --driver postgres --git commit

This will lead to an initial file structure like:

C:/Users/user/IdeProjects/devlogs/idp:
├─── cmd
│      ├─── api
│      │      main.go
├─── internal
│      ├─── database
│      │      database.go
│      ├─── server
│      │      server.go
│      │      routes.go
│      │      routes_test.go
| ...

While the name of the module will be devlogs/idp, this is a bit far of what we want, which would be github.com/your-username/devlogs/idp. So update the module name to github.com/your-username/devlogs/idp and run:

$ go mod tidy

And move the files around for a MSC (Model, Service, Controller) architecture:

C:/Users/user/IdeProjects/devlogs/idp:
├─── cmd
│      ├─── api
│      │      main.go
├─── internal
│      ├─── config
│      │      config.go
│      │      encryption.go
│      │      logger.go
│      │      oauth.go
│      │      rate_limiter.go
│      │      tokens.go
│      │      ...
│      ├─── controllers
│      │      ├─── bodies
│      │      │      common.go
│      │      │      ...
│      │      ├─── params
│      │      │      common.go
│      │      │      ...
│      │      ├─── paths
│      │      │      common.go
│      │      │      ...
│      │      auth.go
│      │      controllers.go
│      │      ...
│      ├─── exceptions
│      │      controllers.go
│      │      services.go
│      ├─── providers
│      │      ├─── cache
│      │      │      cache.go
│      │      │      ...
│      │      ├─── database
│      │      │      database.go
│      │      │      ...
│      │      ├─── mailer
│      │      │      mailer.go
│      │      │      ...
│      │      ├─── oauth
│      │      │      oauth.go
│      │      │      ...
│      │      ├─── tokens
│      │      │      tokens.go
│      │      │      ...
│      │      ├─── encryption
│      │      │      encryption.go
│      │      │      ...
│      ├─── database
│      │      database.go
│      ├─── server
│      │      ├─── routes
│      │      │      routes.go
│      │      │      ...
│      │      logger.go
│      │      server.go
│      │      routes.go
│      ├─── services
│      │      ├─── dtos
│      │      │      common.go
│      │      │      ...
│      ├─── utils
│      │      ...
| ...

Most logic will live on the internal folder, where the main structure is as follows:

Config: centralizes and loads all environment configurations.
Server: contains the Fiber instance initialization and endpoints routing
- /routes: specifies the routes for a given controller method.
- logger.go: builds the default configuration for the structure logger.
- routes.go: has the main method to register all routes RegisterFiberRoutes.
- server.go: builds a FiberServer instance with the fiber.App instance and routes router.
Controllers: handle incoming HTTP requests, process them using services, and return the appropriate HTTP responses.
- /bodies: specifies the controllers bodies.
- /params: specifies the controllers URL and Query parameters.
- /paths: where the routes path constants are defined so they can be easily shared.
Services: where most of our business logic and interactions with external providers lives.
- /dtos: where our data transfer objects are defined,
Providers: where we have our external providers such as Data stores, JWTs, etc.
- /cache: Redis storage interactions.
- /database: PostgreSQL interactions and model structs implementations.
- /mailer: connection to our mailing queue.
- /oauth: oauth interactions with external authentication providers.
- /tokens: jwt signing and verifying.
- /encryption: envelope encryption logic provider.

Configuration

As most APIs, the configuration will come from environment variables, we can load them with the os package from the standard library.

For local development we can load the environment variables from a .env file, by installing the following package:

$ go get github.com/joho/godotenv

This variables most of the time act as secrets, hence we will use a OOP style encapsulation with them where all members of configurations structs are private and immutable, while you can get their values with getters.

Our IDP will have main configurations groups apart from the base config:

Logger: whether debug is active and the env to chose whether we want a text or JSON handler.
Tokens: the private/public keys pairs and TTL for signing and verifying JWTs.
OAuth: collection of client ids and secrets for each of the external authentication providers.
Rate Limiter: specifies the max number of request withing a window that an IP can make.
Encryption: list of KEK (Key encryption keys) provided to the environment.

Logger

On the internal/config directory create a logger.go file with the following struct and New function builder:

package config

type LoggerConfig struct {
    isDebug     bool
    env         string
    serviceName string
}

func NewLoggerConfig(isDebug bool, env, serviceName string) LoggerConfig {
    return LoggerConfig{
        isDebug:     isDebug,
        env:         env,
        serviceName: serviceName,
    }
}

func (l *LoggerConfig) IsDebug() bool {
    return l.isDebug
}

func (l *LoggerConfig) Env() string {
    return l.env
}

func (l *LoggerConfig) ServiceName() string {
    return l.serviceName
}

NOTE: for most methods in Go it is recommended using pointer receivers. Pointers are just address pointing to the underlying memory of the struct, but for simplicity you can think of them as pass by reference instead of pass by value.

Tokens

JWT have 3 main parts that need to be provided by the environment:

Public Key: the key that will be used to verify the token.
Private Key: the key that will be used to sign the token.
Time to live: the TTL in seconds of the token.

And the service will have 7 different JWTs with different key pairs:

Access: for the access token.
Account Credentials: for machine to machine access.
Refresh: for the refresh token to refresh the access token on client to machine access.
Confirmation: for the email confirmation token (JWT for confirmation can be overkill, however it saves on ram memory by not saving the hashed email in cache).
Reset: for email resetting.
OAuth: for a temporary authorization header for the token exchange.
Two Factor: for a temporary authorization header for passing the two factor code.

Single Configuration

Create a tokens.go file on the /internal/config directory with the struct, new method and getters:

package config

type SingleJwtConfig struct {
    publicKey  string
    privateKey string
    ttlSec     int64
}

func NewSingleJwtConfig(publicKey, privateKey string, ttlSec int64) SingleJwtConfig {
    return SingleJwtConfig{
        publicKey:  publicKey,
        privateKey: privateKey,
        ttlSec:     ttlSec,
    }
}

func (s *SingleJwtConfig) PublicKey() string {
    return s.publicKey
}

func (s *SingleJwtConfig) PrivateKey() string {
    return s.privateKey
}

func (s *SingleJwtConfig) TtlSec() int64 {
    return s.ttlSec
}

Tokens Configuration

Now do the same as previously but for each token type:

package config

// ...

type TokensConfig struct {
    access             SingleJwtConfig
    accountCredentials SingleJwtConfig
    refresh            SingleJwtConfig
    confirm            SingleJwtConfig
    reset              SingleJwtConfig
    oAuth              SingleJwtConfig
    twoFA              SingleJwtConfig
}

func NewTokensConfig(
    access SingleJwtConfig,
    accountCredentials SingleJwtConfig,
    refresh SingleJwtConfig,
    confirm SingleJwtConfig,
    oAuth SingleJwtConfig,
    twoFA SingleJwtConfig,
) TokensConfig {
    return TokensConfig{
        access:      access,
        accountCredentials: accountCredentials,
        refresh:     refresh,
        confirm:     confirm,
        oAuth:       oAuth,
        twoFA:       twoFA,
    }
}

func (t *TokensConfig) Access() SingleJwtConfig {
    return t.access
}

func (t *TokensConfig) AccountCredentials() SingleJwtConfig {
    return t.accountKeys
}

func (t *TokensConfig) Refresh() SingleJwtConfig {
    return t.refresh
}

func (t *TokensConfig) Confirm() SingleJwtConfig {
    return t.confirm
}

func (t *TokensConfig) Reset() SingleJwtConfig {
    return t.reset
}

func (t *TokensConfig) OAuth() SingleJwtConfig {
    return t.oAuth
}

func (t *TokensConfig) TwoFA() SingleJwtConfig {
    return t.twoFA
}

OAuth

External authentication providers have two environment variables each:

Client ID: the identifier of the app on the external IDP
Client Secret: a secure secret to fetch the user info from the code exchange.

And we will add support for the 5 main western ones:

GitHub
Google
Facebook
Apple
Microsoft

Single OAuth provider configuration

Create a oauth.go file on the /internal/config directory with the struct, new method and getters:

package config

type OAuthProviderConfig struct {
    clientID     string
    clientSecret string
}

func NewOAuthProvider(clientID, clientSecret string) OAuthProviderConfig {
    return OAuthProviderConfig{
        clientID:     clientID,
        clientSecret: clientSecret,
    }
}

func (o *OAuthProviderConfig) ClientID() string {
    return o.clientID
}

func (o *OAuthProviderConfig) ClientSecret() string {
    return o.clientSecret
}

OAuth providers configuration

Create a struct, new method and getter for each provider:

package config

// ...

type OAuthProvidersConfig struct {
    gitHub    OAuthProviderConfig
    google    OAuthProviderConfig
    facebook  OAuthProviderConfig
    apple     OAuthProviderConfig
    microsoft OAuthProviderConfig
}

func NewOAuthProviders(gitHub, google, facebook, apple, microsoft OAuthProviderConfig) OAuthProvidersConfig {
    return OAuthProvidersConfig{
        gitHub:    gitHub,
        google:    google,
        facebook:  facebook,
        apple:     apple,
        microsoft: microsoft,
    }
}

func (o *OAuthProvidersConfig) GitHub() OAuthProviderConfig {
    return o.gitHub
}

func (o *OAuthProvidersConfig) Google() OAuthProviderConfig {
    return o.google
}

func (o *OAuthProvidersConfig) Facebook() OAuthProviderConfig {
    return o.facebook
}

func (o *OAuthProvidersConfig) Apple() OAuthProviderConfig {
    return o.apple
}

func (o *OAuthProvidersConfig) Microsoft() OAuthProviderConfig {
    return o.microsoft
}

Rate Limiter

The rate limiter has only two options:

Max: the number of maximum number of requests per window size.
Expiration Seconds: the window size in seconds.

Just create the rate_limiter.go file on the /internal/config directory as follows:

package config

type RateLimiterConfig struct {
    max    int64
    expSec int64
}

func NewRateLimiterConfig(max, expSec int64) RateLimiterConfig {
    return RateLimiterConfig{
        max:    max,
        expSec: expSec,
    }
}

func (r *RateLimiterConfig) Max() int64 {
    return r.max
}

func (r *RateLimiterConfig) ExpSec() int64 {
    return r.expSec
}

Encryption

Finaly to configure the KEKs for each encryption space:

Accounts: encryption of secrets for TOTP two factor auth.
Apps: encryption for the APPs private keys.
Users: encryption of secrets for TOTP two factor auth.

As well as old keys for easy keys rotation.

Add them to encryption.go file:

package config

import "encoding/json"

type EncryptionConfig struct {
    accountSecret string
    appSecret     string
    userSecret    string
    oldSecrets    []string
}

func NewEncryptionConfig(accountSecret, appSecret, userSecret, oldSecrets string) EncryptionConfig {
    var secretSlice []string
    if err := json.Unmarshal([]byte(oldSecrets), &secretSlice); err != nil {
        panic(err)
    }

    return EncryptionConfig{
        accountSecret: accountSecret,
        appSecret:     appSecret,
        userSecret:    userSecret,
        oldSecrets:    secretSlice,
    }
}

func (e *EncryptionConfig) AccountSecret() string {
    return e.accountSecret
}

func (e *EncryptionConfig) AppSecret() string {
    return e.appSecret
}

func (e *EncryptionConfig) UserSecret() string {
    return e.userSecret
}

func (e *EncryptionConfig) OldSecrets() []string {
    return e.oldSecrets
}

Base config

On the base config you just need to add the environment variables in an array, as well as getters for each config parameter of the struct:

package config

import (
    "log/slog"
    "os"
    "strconv"
    "strings"

    "github.com/google/uuid"
    "github.com/joho/godotenv"
)

type Config struct {
    port                 int64
    maxProcs             int64
    databaseURL          string
    redisURL             string
    frontendDomain       string
    backendDomain        string
    cookieSecret         string
    cookieName           string
    emailPubChannel      string
    encryptionSecret     string
    serviceID            uuid.UUID
    loggerConfig         LoggerConfig
    tokensConfig         TokensConfig
    oAuthProvidersConfig OAuthProvidersConfig
    rateLimiterConfig    RateLimiterConfig
    encryptionConfig     EncryptionConfig
}

func (c *Config) Port() int64 {
    return c.port
}

func (c *Config) MaxProcs() int64 {
    return c.maxProcs
}

func (c *Config) DatabaseURL() string {
    return c.databaseURL
}

func (c *Config) RedisURL() string {
    return c.redisURL
}

func (c *Config) FrontendDomain() string {
    return c.frontendDomain
}

func (c *Config) BackendDomain() string {
    return c.backendDomain
}

func (c *Config) CookieSecret() string {
    return c.cookieSecret
}

func (c *Config) CookieName() string {
    return c.cookieName
}

func (c *Config) EmailPubChannel() string {
    return c.emailPubChannel
}

func (c *Config) EncryptionSecret() string {
    return c.encryptionSecret
}

func (c *Config) ServiceID() uuid.UUID {
    return c.serviceID
}

func (c *Config) LoggerConfig() LoggerConfig {
    return c.loggerConfig
}

func (c *Config) TokensConfig() TokensConfig {
    return c.tokensConfig
}

func (c *Config) OAuthProvidersConfig() OAuthProvidersConfig {
    return c.oAuthProvidersConfig
}

func (c *Config) RateLimiterConfig() RateLimiterConfig {
    return c.rateLimiterConfig
}

func (c *Config) EncryptionConfig() EncryptionConfig {
    return c.encryptionConfig
}

var variables = [40]string{
    "PORT",
    "ENV",
    "DEBUG",
    "SERVICE_NAME",
    "SERVICE_ID",
    "MAX_PROCS",
    "DATABASE_URL",
    "REDIS_URL",
    "FRONTEND_DOMAIN",
    "BACKEND_DOMAIN",
    "COOKIE_SECRET",
    "COOKIE_NAME",
    "RATE_LIMITER_MAX",
    "RATE_LIMITER_EXP_SEC",
    "EMAIL_PUB_CHANNEL",
    "JWT_ACCESS_PUBLIC_KEY",
    "JWT_ACCESS_PRIVATE_KEY",
    "JWT_ACCESS_TTL_SEC",
    "JWT_ACCOUNT_CREDENTIALS_PUBLIC_KEY",
    "JWT_ACCOUNT_CREDENTIALS_PRIVATE_KEY",
    "JWT_ACCOUNT_CREDENTIALS_TTL_SEC",
    "JWT_REFRESH_PUBLIC_KEY",
    "JWT_REFRESH_PRIVATE_KEY",
    "JWT_REFRESH_TTL_SEC",
    "JWT_CONFIRM_PUBLIC_KEY",
    "JWT_CONFIRM_PRIVATE_KEY",
    "JWT_CONFIRM_TTL_SEC",
    "JWT_RESET_PUBLIC_KEY",
    "JWT_RESET_PRIVATE_KEY",
    "JWT_RESET_TTL_SEC",
    "JWT_OAUTH_PUBLIC_KEY",
    "JWT_OAUTH_PRIVATE_KEY",
    "JWT_OAUTH_TTL_SEC",
    "JWT_2FA_PUBLIC_KEY",
    "JWT_2FA_PRIVATE_KEY",
    "JWT_2FA_TTL_SEC",
    "ACCOUNT_SECRET",
    "APP_SECRET",
    "USER_SECRET",
    "OLD_SECRETS",
}

var optionalVariables = [17]string{
    "GITHUB_CLIENT_ID",
    "GITHUB_CLIENT_SECRET",
    "GOOGLE_CLIENT_ID",
    "GOOGLE_CLIENT_SECRET",
    "FACEBOOK_CLIENT_ID",
    "FACEBOOK_CLIENT_SECRET",
    "APPLE_CLIENT_ID",
    "APPLE_CLIENT_SECRET",
    "MICROSOFT_CLIENT_ID",
    "MICROSOFT_CLIENT_SECRET",
    "OLD_JWT_ACCESS_PUBLIC_KEY",
    "OLD_JWT_ACCOUNT_CREDENTIALS_PUBLIC_KEY",
    "OLD_JWT_REFRESH_PUBLIC_KEY",
    "OLD_JWT_CONFIRM_PUBLIC_KEY",
    "OLD_JWT_RESET_PUBLIC_KEY",
    "OLD_JWT_OAUTH_PUBLIC_KEY",
    "OLD_JWT_2FA_PUBLIC_KEY",
}

var numerics = [11]string{
    "PORT",
    "MAX_PROCS",
    "JWT_ACCESS_TTL_SEC",
    "JWT_ACCOUNT_CREDENTIALS_TTL_SEC",
    "JWT_REFRESH_TTL_SEC",
    "JWT_CONFIRM_TTL_SEC",
    "JWT_RESET_TTL_SEC",
    "JWT_OAUTH_TTL_SEC",
    "JWT_2FA_TTL_SEC",
    "RATE_LIMITER_MAX",
    "RATE_LIMITER_EXP_SEC",
}

func NewConfig(logger *slog.Logger, envPath string) Config {
    err := godotenv.Load(envPath)
    if err != nil {
        logger.Error("Error loading .env file")
    }

    variablesMap := make(map[string]string)
    for _, variable := range variables {
        value := os.Getenv(variable)
        if value == "" {
            logger.Error(variable + " is not set")
            panic(variable + " is not set")
        }
        variablesMap[variable] = value
    }

    for _, variable := range optionalVariables {
        value := os.Getenv(variable)
        variablesMap[variable] = value
    }

    intMap := make(map[string]int64)
    for _, numeric := range numerics {
        value, err := strconv.ParseInt(variablesMap[numeric], 10, 0)
        if err != nil {
            logger.Error(numeric + " is not an integer")
            panic(numeric + " is not an integer")
        }
        intMap[numeric] = value
    }

    env := variablesMap["ENV"]
    return Config{
        port:            intMap["PORT"],
        maxProcs:        intMap["MAX_PROCS"],
        databaseURL:     variablesMap["DATABASE_URL"],
        redisURL:        variablesMap["REDIS_URL"],
        frontendDomain:  variablesMap["FRONTEND_DOMAIN"],
        backendDomain:   variablesMap["BACKEND_DOMAIN"],
        cookieSecret:    variablesMap["COOKIE_SECRET"],
        cookieName:      variablesMap["COOKIE_NAME"],
        emailPubChannel: variablesMap["EMAIL_PUB_CHANNEL"],
        serviceID:       uuid.MustParse(variablesMap["SERVICE_ID"]),
        loggerConfig: NewLoggerConfig(
            strings.ToLower(variablesMap["DEBUG"]) == "true",
            env,
            variablesMap["SERVICE_NAME"],
        ),
        tokensConfig: NewTokensConfig(
            NewSingleJwtConfig(
                variablesMap["JWT_ACCESS_PUBLIC_KEY"],
                variablesMap["JWT_ACCESS_PRIVATE_KEY"],
                variablesMap["OLD_JWT_ACCESS_PUBLIC_KEY"],
                intMap["JWT_ACCESS_TTL_SEC"],
            ),
            NewSingleJwtConfig(
                variablesMap["JWT_ACCOUNT_CREDENTIALS_PUBLIC_KEY"],
                variablesMap["JWT_ACCOUNT_CREDENTIALS_PRIVATE_KEY"],
                variablesMap["OLD_JWT_ACCOUNT_CREDENTIALS_PUBLIC_KEY"],
                intMap["JWT_ACCOUNT_CREDENTIALS_TTL_SEC"],
            ),
            NewSingleJwtConfig(
                variablesMap["JWT_REFRESH_PUBLIC_KEY"],
                variablesMap["JWT_REFRESH_PRIVATE_KEY"],
                variablesMap["OLD_JWT_REFRESH_PUBLIC_KEY"],
                intMap["JWT_REFRESH_TTL_SEC"],
            ),
            NewSingleJwtConfig(
                variablesMap["JWT_CONFIRM_PUBLIC_KEY"],
                variablesMap["JWT_CONFIRM_PRIVATE_KEY"],
                variablesMap["OLD_JWT_CONFIRM_PUBLIC_KEY"],
                intMap["JWT_CONFIRM_TTL_SEC"],
            ),
            NewSingleJwtConfig(
                variablesMap["JWT_RESET_PUBLIC_KEY"],
                variablesMap["JWT_RESET_PRIVATE_KEY"],
                variablesMap["OLD_JWT_RESET_PUBLIC_KEY"],
                intMap["JWT_RESET_TTL_SEC"],
            ),
            NewSingleJwtConfig(
                variablesMap["JWT_OAUTH_PUBLIC_KEY"],
                variablesMap["JWT_OAUTH_PRIVATE_KEY"],
                variablesMap["OLD_JWT_OAUTH_PUBLIC_KEY"],
                intMap["JWT_OAUTH_TTL_SEC"],
            ),
            NewSingleJwtConfig(
                variablesMap["JWT_2FA_PUBLIC_KEY"],
                variablesMap["JWT_2FA_PRIVATE_KEY"],
                variablesMap["OLD_JWT_2FA_PUBLIC_KEY"],
                intMap["JWT_2FA_TTL_SEC"],
            ),
        ),
        oAuthProvidersConfig: NewOAuthProviders(
            NewOAuthProvider(variablesMap["GITHUB_CLIENT_ID"], variablesMap["GITHUB_CLIENT_SECRET"]),
            NewOAuthProvider(variablesMap["GOOGLE_CLIENT_ID"], variablesMap["GOOGLE_CLIENT_SECRET"]),
            NewOAuthProvider(variablesMap["FACEBOOK_CLIENT_ID"], variablesMap["FACEBOOK_CLIENT_SECRET"]),
            NewOAuthProvider(variablesMap["APPLE_CLIENT_ID"], variablesMap["APPLE_CLIENT_SECRET"]),
            NewOAuthProvider(variablesMap["MICROSOFT_CLIENT_ID"], variablesMap["MICROSOFT_CLIENT_SECRET"]),
        ),
        rateLimiterConfig: NewRateLimiterConfig(
            intMap["RATE_LIMITER_MAX"],
            intMap["RATE_LIMITER_EXP_SEC"],
        ),
        encryptionConfig: NewEncryptionConfig(
            variablesMap["ACCOUNT_SECRET"],
            variablesMap["APP_SECRET"],
            variablesMap["USER_SECRET"],
            variablesMap["OLD_SECRETS"],
        ),
    }
}

Providers

With the config done, we can start creating the providers:

cache: Redis cache;
database: PostgreSQL database;
mailer: Redis PubSub publisher for emails (the queue will be built on a subsquent article);
oauth: external authentication providers;
tokens: jwt signing and verifying;
encrytion: the provider for envolope encryption (more of an isolation of logic than a proper provider. Encryption logic should always be isolated in production)

Utilities

Observasibility of APIs is important for production debugging, so lets create an utility to build a structure logger.

To make it easy to locate the function and logic, we need to pass the package location, method, and layer. Create the utility on the root utils package. Create a logger.go file and add the following code:

package utils

import (
    "log/slog"
)

type LogLayer = string

const (
    ControllersLogLayer LogLayer = "controllers"
    ServicesLogLayer    LogLayer = "services"
    ProvidersLogLayer   LogLayer = "providers"
)

type LoggerOptions struct {
    Layer     string
    Location  string
    Method    string
    RequestID string
}

func BuildLogger(logger *slog.Logger, opts LoggerOptions) *slog.Logger {
    return logger.With(
        "layer", opts.Layer,
        "location", opts.Location,
        "method", opts.Method,
        "requestId", opts.RequestID,
    )
}

Cache

Our cache implementation will just extend the fiber Redis Storage abstraction. On the internal/providers create a /cache directory and add cache.go file with the following struct and new function:

package cache

import (
    "context"
    "log/slog"

    fiberRedis "github.com/gofiber/storage/redis/v3"
    "github.com/redis/go-redis/v9"
)

const logLayer string = "cache"

type Cache struct {
    logger  *slog.Logger
    storage *fiberRedis.Storage
}

func NewCache(logger *slog.Logger, storage *fiberRedis.Storage) *Cache {
    return &Cache{
        logger:  logger,
        storage: storage,
    }
}

With three common methods to reset the cache, get the redist client and ping the cache:

package cache

// ...

func (c *Cache) ResetCache() error {
    return c.storage.Reset()
}

func (c *Cache) Client() redis.UniversalClient {
    return c.storage.Conn()
}

func (c *Cache) Ping(ctx context.Context) error {
    return c.Client().Ping(ctx).Err()
}

Database

The database set-up is a bit more complex. We will write both our migrations and queries in SQL and use migrate to migrate our DB changes, and SQLC to generate safe Go SQL query codes.

Start by installing the necessary dependencies:

$ go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
$ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

And create a config file for SQLC as sqlc.yaml on the idp root folder:

version: "2"
sql:
  - schema: "internal/providers/database/migrations"
    queries: "internal/providers/database/queries"
    engine: "postgresql"
    gen:
      go:
        package: "database"
        out: "internal/providers/database"
        sql_package: "pgx/v5"
        emit_empty_slices: true
        overrides:
          - db_type: "timestamptz"
            go_type: "time.Time"
          - db_type: "uuid"
            go_type: "github.com/google/uuid.UUID"

Database Schema

Since we are creating an IDP we will have three main entitites:

Accounts (also known as tenants): the main users/admins that can create APPs that provide authentication to their own services.
Apps: the authentication provider configuration for a given service or services.
Users: users that have registered for the account's apps.

This leads to the following somewhat complex datamodel:

Initial Migration

With migrate installed run the following command:

$ migrate create --ext sql --dir ./internal/providers/database/migration create_initial_schema

This will generate two files:

YYYYMMDDHHMMSS_create_initial_schema.up.sql
YYYYMMDDHHMMSS_create_initial_schema.down.sql

To the up migration copy the following SQL:

CREATE TABLE "accounts" (
  "id" serial PRIMARY KEY,
  "first_name" varchar(50) NOT NULL,
  "last_name" varchar(50) NOT NULL,
  "username" varchar(109) NOT NULL,
  "email" varchar(250) NOT NULL,
  "password" text,
  "version" integer NOT NULL DEFAULT 1,
  "is_confirmed" boolean NOT NULL DEFAULT false,
  "two_factor_type" varchar(5) NOT NULL DEFAULT 'none',
  "created_at" timestamp NOT NULL DEFAULT (now()),
  "updated_at" timestamp NOT NULL DEFAULT (now())
);

CREATE TABLE "account_totps" (
  "id" serial PRIMARY KEY,
  "account_id" integer NOT NULL,
  "url" varchar(250) NOT NULL,
  "secret" text NOT NULL,
  "dek" text NOT NULL,
  "recovery_codes" jsonb NOT NULL,
  "created_at" timestamp NOT NULL DEFAULT (now()),
  "updated_at" timestamp NOT NULL DEFAULT (now())
);

CREATE TABLE "account_credentials" (
  "id" serial PRIMARY KEY,
  "account_id" integer NOT NULL,
  "scopes" jsonb NOT NULL,
  "client_id" varchar(22) NOT NULL,
  "client_secret" text NOT NULL,
  "created_at" timestamp NOT NULL DEFAULT (now()),
  "updated_at" timestamp NOT NULL DEFAULT (now())
);

CREATE TABLE "auth_providers" (
  "id" serial PRIMARY KEY,
  "email" varchar(250) NOT NULL,
  "provider" varchar(10) NOT NULL,
  "created_at" timestamp NOT NULL DEFAULT (now()),
  "updated_at" timestamp NOT NULL DEFAULT (now())
);

CREATE TABLE "apps" (
  "id" serial PRIMARY KEY,
  "account_id" integer NOT NULL,
  "name" varchar(50) NOT NULL,
  "client_id" varchar(22) NOT NULL,
  "client_secret" text NOT NULL,
  "dek" text NOT NULL,
  "callback_uris" varchar(250)[] NOT NULL DEFAULT '{}',
  "logout_uris" varchar(250)[] NOT NULL DEFAULT '{}',
  "user_scopes" jsonb NOT NULL DEFAULT '{ "email": true, "name": true }',
  "app_providers" jsonb NOT NULL DEFAULT '{ "email_password": true }',
  "id_token_ttl" integer NOT NULL DEFAULT 3600,
  "jwt_crypto_suite" varchar(7) NOT NULL DEFAULT 'ES256',
  "created_at" timestamp NOT NULL DEFAULT (now()),
  "updated_at" timestamp NOT NULL DEFAULT (now())
);

CREATE TABLE "app_keys" (
  "id" serial PRIMARY KEY,
  "app_id" integer NOT NULL,
  "account_id" integer NOT NULL,
  "name" varchar(10) NOT NULL,
  "jwt_crypto_suite" varchar(7) NOT NULL,
  "public_key" jsonb NOT NULL,
  "private_key" text NOT NULL,
  "created_at" timestamp NOT NULL DEFAULT (now()),
  "updated_at" timestamp NOT NULL DEFAULT (now())
);

CREATE TABLE "users" (
  "id" serial PRIMARY KEY,
  "account_id" integer NOT NULL,
  "email" varchar(250) NOT NULL,
  "password" text,
  "version" integer NOT NULL DEFAULT 1,
  "two_factor_type" varchar(5) NOT NULL DEFAULT 'none',
  "user_data" jsonb NOT NULL DEFAULT '{}',
  "created_at" timestamp NOT NULL DEFAULT (now()),
  "updated_at" timestamp NOT NULL DEFAULT (now())
);

CREATE TABLE "user_totps" (
  "id" serial PRIMARY KEY,
  "user_id" integer NOT NULL,
  "url" varchar(250) NOT NULL,
  "secret" text NOT NULL,
  "dek" text NOT NULL,
  "recovery_codes" jsonb NOT NULL,
  "created_at" timestamp NOT NULL DEFAULT (now()),
  "updated_at" timestamp NOT NULL DEFAULT (now())
);

CREATE TABLE "user_auth_providers" (
  "id" serial PRIMARY KEY,
  "user_id" integer NOT NULL,
  "email" varchar(250) NOT NULL,
  "provider" varchar(10) NOT NULL,
  "account_id" integer NOT NULL,
  "created_at" timestamp NOT NULL DEFAULT (now()),
  "updated_at" timestamp NOT NULL DEFAULT (now())
);

CREATE TABLE "blacklisted_tokens" (
  "id" uuid PRIMARY KEY,
  "expires_at" timestamp NOT NULL,
  "created_at" timestamp NOT NULL DEFAULT (now())
);

CREATE UNIQUE INDEX "accounts_email_uidx" ON "accounts" ("email");

CREATE UNIQUE INDEX "accounts_username_uidx" ON "accounts" ("username");

CREATE UNIQUE INDEX "accounts_totps_account_id_uidx" ON "account_totps" ("account_id");

CREATE UNIQUE INDEX "account_credentials_client_id_uidx" ON "account_credentials" ("client_id");

CREATE INDEX "account_credentials_account_id_idx" ON "account_credentials" ("account_id");

CREATE INDEX "auth_providers_email_idx" ON "auth_providers" ("email");

CREATE UNIQUE INDEX "auth_providers_email_provider_uidx" ON "auth_providers" ("email", "provider");

CREATE INDEX "apps_account_id_idx" ON "apps" ("account_id");

CREATE UNIQUE INDEX "client_id_uidx" ON "apps" ("client_id");

CREATE INDEX "app_keys_app_id_idx" ON "app_keys" ("app_id");

CREATE INDEX "app_keys_account_id_idx" ON "app_keys" ("account_id");

CREATE UNIQUE INDEX "app_keys_name_app_id_uidx" ON "app_keys" ("name", "app_id");

CREATE UNIQUE INDEX "users_account_id_email_uidx" ON "users" ("account_id", "email");

CREATE INDEX "users_account_id_idx" ON "users" ("account_id");

CREATE UNIQUE INDEX "user_totps_user_id_uidx" ON "user_totps" ("user_id");

CREATE INDEX "user_auth_provider_email_idx" ON "user_auth_providers" ("email");

CREATE INDEX "user_auth_provider_user_id_idx" ON "user_auth_providers" ("user_id");

CREATE UNIQUE INDEX "user_auth_provider_account_id_provider_uidx" ON "user_auth_providers" ("email", "account_id", "provider");

CREATE INDEX "user_auth_provider_account_id_idx" ON "user_auth_providers" ("account_id");

ALTER TABLE "account_totps" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;

ALTER TABLE "account_credentials" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;

ALTER TABLE "auth_providers" ADD FOREIGN KEY ("email") REFERENCES "accounts" ("email") ON DELETE CASCADE ON UPDATE CASCADE;

ALTER TABLE "apps" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;

ALTER TABLE "app_keys" ADD FOREIGN KEY ("app_id") REFERENCES "apps" ("id") ON DELETE CASCADE;

ALTER TABLE "app_keys" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;

ALTER TABLE "users" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;

ALTER TABLE "user_totps" ADD FOREIGN KEY ("user_id") REFERENCES "users" ("id") ON DELETE CASCADE;

ALTER TABLE "user_auth_providers" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;

ALTER TABLE "user_auth_providers" ADD FOREIGN KEY ("user_id") REFERENCES "users" ("id") ON DELETE CASCADE;

While on the down drop all tables if they exist:

DROP TABLE IF EXISTS "user_auth_providers";
DROP TABLE IF EXISTS "user_totps";
DROP TABLE IF EXISTS "users";
DROP TABLE IF EXISTS "app_keys";
DROP TABLE IF EXISTS "apps";
DROP TABLE IF EXISTS "auth_providers";
DROP TABLE IF EXISTS "account_credentials";
DROP TABLE IF EXISTS "account_totps";
DROP TABLE IF EXISTS "accounts";
DROP TABLE IF EXISTS "blacklisted_tokens";

SQLC Code generation

To set up the SQLC generated code we need to start by writting a query on the queries folder. Create a accounts.sql file in the queries directory and add the logic to insert an account:

-- name: CreateAccountWithPassword :one
INSERT INTO "accounts" (
    "first_name",
    "last_name",
    "username",
    "email", 
    "password"
) VALUES (
    $1, 
    $2, 
    $3,
    $4,
    $5
) RETURNING *;

SQLC knows what to return and what to call to the go method by the code comment on top of the SQL code.

Then in the terminal run the following command:

$ sqlc generate

This will generate the following files:

models.go
db.go
accounts.sql.go

These files are auto-generated and auto-updated when you run the sqlc generate command, hence on a new database.go create the logic to connect to the SQLC generated code:

package database

import (
    "context"

    "github.com/jackc/pgx/v5"
    "github.com/jackc/pgx/v5/pgxpool"

    "github.com/tugascript/devlogs/idp/internal/exceptions"
)

type Database struct {
    connPool *pgxpool.Pool
    *Queries
}

func NewDatabase(connPool *pgxpool.Pool) *Database {
    return &Database{
        connPool: connPool,
        Queries:  New(connPool),
    }
}

func (d *Database) BeginTx(ctx context.Context) (*Queries, pgx.Tx, error) {
    txn, err := d.connPool.BeginTx(ctx, pgx.TxOptions{
        DeferrableMode: pgx.Deferrable,
        IsoLevel:       pgx.ReadCommitted,
        AccessMode:     pgx.ReadWrite,
    })

    if err != nil {
        return nil, nil, err
    }

    return d.WithTx(txn), txn, nil
}

func (d *Database) FinalizeTx(ctx context.Context, txn pgx.Tx, err error, serviceErr *exceptions.ServiceError) {
    if serviceErr != nil || err != nil {
        if err := txn.Rollback(ctx); err != nil {
            panic(err)
        }
        return
    }
    if commitErr := txn.Commit(ctx); commitErr != nil {
        panic(commitErr)
    }
    if p := recover(); p != nil {
        if err := txn.Rollback(ctx); err != nil {
            panic(err)
        }
        panic(p)
    }
}

func (d *Database) RawQuery(ctx context.Context, sql string, args []interface{}) (pgx.Rows, error) {
    return d.connPool.Query(ctx, sql, args...)
}

func (d *Database) RawQueryRow(ctx context.Context, sql string, args []interface{}) pgx.Row {
    return d.connPool.QueryRow(ctx, sql, args...)
}

func (d *Database) Ping(ctx context.Context) error {
    return d.connPool.Ping(ctx)
}

Encryption

Encryption is less of a provider, but more of a logic isolation, hence we'll still load it as a provider.

Utilities

For each secret (or key), we will need to derive a KID (Key ID), this is common logic, so in the internal directory lets create a utils package and on a jwk.go folder add the following files:

encoders.go:

package utils

import (
    "math/big"
)

func Base62Encode(bytes []byte) string {
    var codeBig big.Int
    codeBig.SetBytes(bytes)
    return codeBig.Text(62)
}

jwk.go:

package utils

import (
    "crypto/sha256"
)

func ExtractKeyID(keyBytes []byte) string {
    hash := sha256.Sum256(keyBytes)
    return Base62Encode(hash[:12])
}

secrets.go:

  package utils

  import (
      "crypto/rand"
      "encoding/base32"
      "encoding/base64"
      "encoding/hex"
      "math/big"
  )

  func generateRandomBytes(byteLen int) ([]byte, error) {
      b := make([]byte, byteLen)

      if _, err := rand.Read(b); err != nil {
          return nil, err
      }

      return b, nil
  }

  func GenerateBase64Secret(byteLen int) (string, error) {
      randomBytes, err := generateRandomBytes(byteLen)
      if err != nil {
          return "", err
      }

      return base64.RawURLEncoding.EncodeToString(randomBytes), nil
  }

  func DecodeBase64Secret(secret string) ([]byte, error) {
      decoded, err := base64.RawURLEncoding.DecodeString(secret)
      if err != nil {
          return nil, err
      }

      return decoded, nil
  }

  func GenerateBase62Secret(byteLen int) (string, error) {
      randomBytes, err := generateRandomBytes(byteLen)
      if err != nil {
          return "", err
      }

      randomInt := new(big.Int).SetBytes(randomBytes)
      return randomInt.Text(62), nil
  }

  func GenerateBase32Secret(byteLen int) (string, error) {
      randomBytes, err := generateRandomBytes(byteLen)
      if err != nil {
          return "", err
      }

      return base32.StdEncoding.WithPadding(base32.NoPadding).EncodeToString(randomBytes), nil
  }

  func GenerateHexSecret(byteLen int) (string, error) {
      randomBytes, err := generateRandomBytes(byteLen)
      if err != nil {
          return "", err
      }

      return hex.EncodeToString(randomBytes), nil
  }

Provider

Now with the utilites done, create the following providers/encryption package:

package encryption

import (
    "encoding/base64"
    "log/slog"

    "github.com/tugascript/devlogs/idp/internal/config"
    "github.com/tugascript/devlogs/idp/internal/utils"
)

const logLayer string = "encryption"

type Secret struct {
    kid string
    key []byte
}

type Encryption struct {
    logger           *slog.Logger
    accountSecretKey Secret
    appSecretKey     Secret
    userSecretKey    Secret
    oldSecrets       map[string][]byte
    backendDomain    string
}

func decodeSecret(secret string) Secret {
        // Decode the Base64 code to bytes
    decodedKey, err := base64.StdEncoding.DecodeString(secret)
    if err != nil {
        panic(err)
    }

    return Secret{
                // Generate a Key ID per secret
        kid: utils.ExtractKeyID(decodedKey),
        key: decodedKey,
    }
}

func NewEncryption(
    logger *slog.Logger,
    cfg config.EncryptionConfig,
    backendDomain string,
) *Encryption {
    oldSecretsMap := make(map[string][]byte)
    for _, s := range cfg.OldSecrets() {
        ds := decodeSecret(s)
        oldSecretsMap[ds.kid] = ds.key
    }

    return &Encryption{
        logger:           logger,
        accountSecretKey: decodeSecret(cfg.AccountSecret()),
        appSecretKey:     decodeSecret(cfg.AppSecret()),
        userSecretKey:    decodeSecret(cfg.UserSecret()),
        oldSecrets:       oldSecretsMap,
        backendDomain:    backendDomain,
    }
}

DEK Generation

DEKs (Date Encryption Keys) need to be generated by the system, and are saved in the database, for this reason, create a file called dek.go and add the following logic:

DEK generation & Encryption:

package encryption

import (
    "fmt"

    "github.com/tugascript/devlogs/idp/internal/utils"
)

const dekLocation string = "dek"

func generateDEK(keyID string, key []byte) ([]byte, string, error) {
    base64DEK, err := utils.GenerateBase64Secret(32)
    if err != nil {
        return nil, "", err
    }

    encryptedDEK, err := utils.Encrypt(base64DEK, key)
    if err != nil {
        return nil, "", err
    }

    dek, err := utils.DecodeBase64Secret(base64DEK)
    if err != nil {
        return nil, "", err
    }

    return dek, fmt.Sprintf("%s:%s", keyID, encryptedDEK), nil
}

// ...

func reEncryptDEK(isOldKey bool, dek, key []byte) (string, error) {
    if !isOldKey {
        return "", nil
    }

    return utils.Encrypt(base64.RawURLEncoding.EncodeToString(dek), key)
}

DEK decryption:

package encryption

import (
    "context"
    "encoding/base64"
    "errors"
    "fmt"
    "log/slog"
    "strings"

    "github.com/tugascript/devlogs/idp/internal/utils"
)

// ...

type decryptDEKOptions struct {
    storedDEK  string
    secret     *Secret
    oldSecrets map[string][]byte
}

func decryptDEK(
    logger *slog.Logger,
    ctx context.Context,
    opts decryptDEKOptions,
) ([]byte, bool, error) {
    dekID, encryptedDEK, err := splitDEK(opts.storedDEK)
    if err != nil {
        logger.ErrorContext(ctx, "Failed to split DEK", "error", err)
        return nil, false, err
    }

    key := opts.secret.key
    oldKey := dekID != opts.secret.kid
    if oldKey {
        var ok bool
        key, ok = opts.oldSecrets[dekID]
        if !ok {
            logger.ErrorContext(ctx, "DEK key ID not found")
            return nil, false, errors.New("secret key not found")
        }
    }

    base64DEK, err := utils.Decrypt(encryptedDEK, key)
    if err != nil {
        logger.ErrorContext(ctx, "Failed to decrypt DEK", "error", err)
        return nil, false, err
    }

    dek, err := utils.DecodeBase64Secret(base64DEK)
    if err != nil {
        logger.ErrorContext(ctx, "Failed to decode DEK", "error", err)
        return nil, false, err
    }

    return dek, oldKey, nil
}

Methods to decrypt for Account, App & User:

// ...

func (e *Encryption) decryptAccountDEK(ctx context.Context, requestID, storedDEK string) ([]byte, bool, error) {
    logger := utils.BuildLogger(e.logger, utils.LoggerOptions{
        Layer:     logLayer,
        Location:  dekLocation,
        Method:    "decryptAccountDEK",
        RequestID: requestID,
    })
    logger.DebugContext(ctx, "Decrypting Account DEK...")
    return decryptDEK(logger, ctx, decryptDEKOptions{
        storedDEK:  storedDEK,
        secret:     &e.accountSecretKey,
        oldSecrets: e.oldSecrets,
    })
}

func (e *Encryption) decryptAppDEK(ctx context.Context, requestID, storedDEK string) ([]byte, bool, error) {
    logger := utils.BuildLogger(e.logger, utils.LoggerOptions{
        Layer:     logLayer,
        Location:  dekLocation,
        Method:    "decryptAppDEK",
        RequestID: requestID,
    })
    logger.DebugContext(ctx, "Decrypting App DEK...")
    return decryptDEK(logger, ctx, decryptDEKOptions{
        storedDEK:  storedDEK,
        secret:     &e.appSecretKey,
        oldSecrets: e.oldSecrets,
    })
}

func (e *Encryption) decryptUserDEK(ctx context.Context, requestID, storedDEK string) ([]byte, bool, error) {
    logger := utils.BuildLogger(e.logger, utils.LoggerOptions{
        Layer:     logLayer,
        Location:  dekLocation,
        Method:    "decryptUserDEK",
        RequestID: requestID,
    })
    logger.DebugContext(ctx, "Decrypting User DEK...")
    return decryptDEK(logger, ctx, decryptDEKOptions{
        storedDEK:  storedDEK,
        secret:     &e.userSecretKey,
        oldSecrets: e.oldSecrets,
    })
}

Mailer

The mailer will just be a redis publisher, that will publish messages to the email service, hence create a mailer package with the following mailer.go file:

package mailer

import (
    "context"
    "encoding/json"
    "log/slog"

    "github.com/tugascript/devlogs/idp/internal/utils"

    "github.com/redis/go-redis/v9"
)

const logLayer string = "mailer"

type email struct {
    To      string `json:"to"`
    Subject string `json:"subject"`
    Body    string `json:"body"`
}

type EmailPublisher struct {
    client         redis.UniversalClient
    pubChannel     string
    frontendDomain string
    logger         *slog.Logger
}

func NewEmailPublisher(
    client redis.UniversalClient,
    pubChannel,
    frontendDomain string,
    logger *slog.Logger,
) *EmailPublisher {
    return &EmailPublisher{
        client:         client,
        pubChannel:     pubChannel,
        frontendDomain: frontendDomain,
        logger:         logger,
    }
}

type PublishEmailOptions struct {
    To        string
    Subject   string
    Body      string
    RequestID string
}

func (e *EmailPublisher) publishEmail(ctx context.Context, opts PublishEmailOptions) error {
    logger := utils.BuildLogger(e.logger, utils.LoggerOptions{
        Layer:     logLayer,
        Location:  "mailer",
        Method:    "PublishEmail",
        RequestID: opts.RequestID,
    })
    logger.DebugContext(ctx, "Publishing email...")

    message, err := json.Marshal(email{
        To:      opts.To,
        Subject: opts.Subject,
        Body:    opts.Body,
    })
    if err != nil {
        logger.ErrorContext(ctx, "Failed to marshal email", "error", err)
        return err
    }

    return e.client.Publish(ctx, e.pubChannel, string(message)).Err()
}

It will consume the UniversalClient from the cache, and publish to a given channel.

OAuth

For OAuth 2.0 we can just use the golang.org/x/oauth2 package, just run:

$ go get golang.org/x/oauth2

Scopes

Each external provider has their own scopes, lets start by creating a scopes struct:

package oauth

// ...

type oauthScopes struct {
    email    string
    profile  string
    birthday string
    location string
    gender   string
}

And for each provider lets create a global scope variable:

apple.go:

package oauth

// ...

var appleScopes = oauthScopes{
    email:   "email",
    profile: "name",
}

facebook.go:

package oauth

// ...

var facebookScopes = oauthScopes{
    email:    "email",
    profile:  "public_profile",
    birthday: "user_birthday",
    location: "user_location",
    gender:   "gender",
}

github.go:

package oauth

// ...

var gitHubScopes = oauthScopes{
    email:   "user:email",
    profile: "read:user",
}

google.go:

package oauth

// ...

var googleScopes = oauthScopes{
    email:    "https://www.googleapis.com/auth/userinfo.email",
    profile:  "https://www.googleapis.com/auth/userinfo.profile",
    birthday: "https://www.googleapis.com/auth/user.birthday.read",
    location: "https://www.googleapis.com/auth/user.addresses.read",
    gender:   "https://www.googleapis.com/auth/user.gender.read",
}

microsoft.go:

package oauth

// ...

var microsoftScopes = oauthScopes{
    email:   "User.Read",
    profile: "User.ReadBasic.All",
}

Provider

Start by creating an oauth2.Config struct for each external provider, using existing defaults when possible:

package oauth

import (
    "log/slog"

    "golang.org/x/oauth2"
    "golang.org/x/oauth2/facebook"
    "golang.org/x/oauth2/github"
    "golang.org/x/oauth2/google"
    "golang.org/x/oauth2/microsoft"

    "github.com/tugascript/devlogs/idp/internal/config"
)

// ...

type Config struct {
    Enabled bool
    oauth2.Config
}

type Providers struct {
    gitHub    Config
    google    Config
    facebook  Config
    apple     Config
    microsoft Config
    logger    *slog.Logger
}

func NewProviders(
    log *slog.Logger,
    githubCfg,
    googleCfg,
    facebookCfg,
    appleCfg,
    microsoftCfg config.OAuthProviderConfig,
) *Providers {
    return &Providers{
        gitHub: Config{
            Config: oauth2.Config{
                ClientID:     githubCfg.ClientID(),
                ClientSecret: githubCfg.ClientSecret(),
                Endpoint:     github.Endpoint,
                Scopes:       []string{gitHubScopes.email},
            },
            Enabled: githubCfg.Enabled(),
        },
        google: Config{
            Config: oauth2.Config{
                ClientID:     googleCfg.ClientID(),
                ClientSecret: googleCfg.ClientSecret(),
                Endpoint:     google.Endpoint,
                Scopes:       []string{googleScopes.email},
            },
            Enabled: googleCfg.Enabled(),
        },
        facebook: Config{
            Config: oauth2.Config{
                ClientID:     facebookCfg.ClientID(),
                ClientSecret: facebookCfg.ClientSecret(),
                Endpoint:     facebook.Endpoint,
                Scopes:       []string{facebookScopes.email},
            },
            Enabled: facebookCfg.Enabled(),
        },
        apple: Config{
            Config: oauth2.Config{
                ClientID:     appleCfg.ClientID(),
                ClientSecret: appleCfg.ClientSecret(),
                Endpoint: oauth2.Endpoint{
                    AuthURL:  "https://appleid.apple.com/auth/authorize",
                    TokenURL: "https://appleid.apple.com/auth/token",
                },
                Scopes: []string{appleScopes.email},
            },
            Enabled: appleCfg.Enabled(),
        },
        microsoft: Config{
            Config: oauth2.Config{
                ClientID:     microsoftCfg.ClientID(),
                ClientSecret: microsoftCfg.ClientSecret(),
                Endpoint:     microsoft.AzureADEndpoint("common"),
                Scopes:       []string{microsoftScopes.email},
            },
            Enabled: microsoftCfg.Enabled(),
        },
        logger: log,
    }
}

For getting the token we will need to pass the correct Scopes, this touches on a concept in go that is passing a value as a value (or copy) of the underlying resource. Since we dynamically add scopes we create a copy of the config each time.

Getting the access token:

package oauth

import (
    "context"
    // ...

    "github.com/tugascript/devlogs/idp/internal/exceptions"
)


// ...

func mapScopes(scopes []Scope, oas oauthScopes) []string {
    scopeMapper := make(map[string]bool)

    for _, s := range scopes {
        switch s {
        case ScopeBirthday:
            scopeMapper[oas.birthday] = true
        case ScopeGender:
            scopeMapper[oas.gender] = true
        case ScopeLocation:
            scopeMapper[oas.location] = true
        case ScopeProfile:
            scopeMapper[oas.location] = true
        }
    }

    mappedScopes := make([]string, 0, len(scopeMapper))
    for k := range scopeMapper {
        if k != "" {
            mappedScopes = append(mappedScopes, k)
        }
    }

    return mappedScopes
}

// Here we pass by value so we don't update the base configuration
func appendScopes(cfg Config, scopes []string) Config {
    cfg.Scopes = append(cfg.Scopes, scopes...)
    return cfg
}

// Here we pass by value so we don't update the base configuration
func getConfig(cfg Config, redirectURL string, oas oauthScopes, scopes []Scope) Config {
    cfg.RedirectURL = redirectURL

    if scopes != nil {
        return appendScopes(cfg, mapScopes(scopes, oas))
    }

    return cfg
}

type getAccessTokenOptions struct {
    logger      *slog.Logger
    cfg         Config
    redirectURL string
    oas         oauthScopes
    scopes      []Scope
    code        string
}

func getAccessToken(ctx context.Context, opts getAccessTokenOptions) (string, *exceptions.ServiceError) {
    opts.logger.DebugContext(ctx, "Getting access token...")

    if !opts.cfg.Enabled {
        opts.logger.DebugContext(ctx, "OAuth config is disabled")
        return "", exceptions.NewNotFoundError()
    }

    cfg := getConfig(opts.cfg, opts.redirectURL, opts.oas, opts.scopes)
    token, err := cfg.Exchange(ctx, opts.code)
    if err != nil {
        opts.logger.ErrorContext(ctx, "Failed to exchange the code for a token", "error", err)
        return "", exceptions.NewUnauthorizedError()
    }

    opts.logger.DebugContext(ctx, "Access token exchanged successfully")
    return token.AccessToken, nil
}

Getting the authorization URL:

package oauth

import (
    "context"

    // ...
    "github.com/tugascript/devlogs/idp/internal/utils"
)

type getAuthorizationURLOptions struct {
    logger      *slog.Logger
    redirectURL string
    cfg         Config
    oas         oauthScopes
    scopes      []Scope
}

func getAuthorizationURL(
    ctx context.Context,
    opts getAuthorizationURLOptions,
) (string, string, *exceptions.ServiceError) {
    opts.logger.DebugContext(ctx, "Getting authorization url...")

    if !opts.cfg.Enabled {
        opts.logger.DebugContext(ctx, "OAuth config is disabled")
        return "", "", exceptions.NewNotFoundError()
    }

    state, err := utils.GenerateHexSecret(16)
    if err != nil {
        opts.logger.ErrorContext(ctx, "Failed to generate state", "error", err)
        return "", "", exceptions.NewServerError()
    }

    cfg := getConfig(opts.cfg, opts.redirectURL, opts.oas, opts.scopes)
    url := cfg.AuthCodeURL(state)
    opts.logger.DebugContext(ctx, "Authorization url generated successfully")
    return url, state, nil
}

Getting the user data:

package oauth

import (
    "context"
    "errors"
    "io"
    "log/slog"
    "net/http"

    // ...

    "github.com/tugascript/devlogs/idp/internal/config"
    "github.com/tugascript/devlogs/idp/internal/exceptions"
    "github.com/tugascript/devlogs/idp/internal/utils"
)

func getUserResponse(logger *slog.Logger, ctx context.Context, url, token string) ([]byte, int, error) {
    logger.DebugContext(ctx, "Getting user data...", "url", url)

    logger.DebugContext(ctx, "Building user data request")
    req, err := http.NewRequest(http.MethodGet, url, nil)
    if err != nil {
        logger.ErrorContext(ctx, "Failed to build user data request")
        return nil, 0, err
    }

    req.Header.Set("Accept", "application/json")
    req.Header.Set("Authorization", "Bearer "+token)

    logger.DebugContext(ctx, "Requesting user data...")
    res, err := http.DefaultClient.Do(req)
    if err != nil {
        logger.ErrorContext(ctx, "Failed to request the user data")
        return nil, 0, err
    }

    if res.StatusCode != http.StatusOK {
        logger.ErrorContext(ctx, "Responded with a non 200 OK status", "status", res.StatusCode)
        return nil, res.StatusCode, errors.New("status code is not 200 OK")
    }

    logger.DebugContext(ctx, "Reading the body")
    body, err := io.ReadAll(res.Body)
    if err != nil {
        logger.ErrorContext(ctx, "Failed to read the body", "error", err)
        return nil, 0, err
    }
    defer func() {
        if err := res.Body.Close(); err != nil {
            logger.ErrorContext(ctx, "Failed to close response body", "error", err)
        }
    }()

    return body, res.StatusCode, nil
}

type UserLocation struct {
    City    string
    Region  string
    Country string
}

type UserData struct {
    Name       string
    FirstName  string
    LastName   string
    Username   string
    Picture    string
    Email      string
    Gender     string
    Location   UserLocation
    BirthDate  string
    IsVerified bool
}

type ToUserData interface {
    ToUserData() UserData
}

type extraParams struct {
    params string
}

func (p *extraParams) addParam(prm string) {
    if p.params != "" {
        p.params = p.params + "," + prm
        return
    }

    p.params += prm
}

func (p *extraParams) isEmpty() bool {
    return p.params == ""
}

Common options between all providers:

package oauth

// ...

type AccessTokenOptions struct {
    RequestID   string
    Code        string
    RedirectURL string
    Scopes      []Scope
}

type AuthorizationURLOptions struct {
    RequestID   string
    RedirectURL string
    Scopes      []Scope
}

type UserDataOptions struct {
    RequestID string
    Token     string
    Scopes    []Scope
}

Tokens

Keys Algorithms

When choosing the algorithm of the key pairs to sign and verify JWTs we need to choose the best ones with the best efficient and security ratio in mind.

By my own research the most balanced algorithm would be EdDSC (Edwards-curve Digital Signature Algorithm) with the Ed25519 signature scheme, however this algorithm is not part of the base JWT RFC-7519 standard, only the RFC-8037 which is not widly supported.

Hence for simplicity and compatibility sake for all tokens that require their public keys to be distributed will use the recommended ECDSA (Elliptic Curve Digital Signature Algorithm) algorithm with the P-256 signature scheme.

Utilities

For sharing and saving keys we need to convert them to JWKs so create an encode and decode functions for both Ed25519 and P256 on the utils/jwk.go file:

package utils

import (
    "crypto/ecdsa"
    "crypto/ed25519"
    "crypto/elliptic"
    "crypto/sha256"
    "encoding/base64"
    "fmt"
    "math/big"
)

type Ed25519JWK struct {
    Kty    string   `json:"kty"`     // Key Type (OKP for Ed25519)
    Crv    string   `json:"crv"`     // Curve (Ed25519)
    X      string   `json:"x"`       // Public Key
    Use    string   `json:"use"`     // Usage (e.g., "sig" for signing)
    Alg    string   `json:"alg"`     // Algorithm (EdDSA for Ed25519)
    Kid    string   `json:"kid"`     // Key AccountID
    KeyOps []string `json:"key_ops"` // Key Operations
}

type P256JWK struct {
    Kty    string   `json:"kty"`     // Key Type (EC for Elliptic Curve)
    Crv    string   `json:"crv"`     // Curve (P-256)
    X      string   `json:"x"`       // X Coordinate
    Y      string   `json:"y"`       // Y Coordinate
    Use    string   `json:"use"`     // Usage (e.g., "sig" for signing)
    Alg    string   `json:"alg"`     // Algorithm (ES256 for P-256)
    Kid    string   `json:"kid"`     // Key AccountID
    KeyOps []string `json:"key_ops"` // Key Operations
}

// Because of Apple
type RS256JWK struct {
    Kty    string   `json:"kty"`
    Kid    string   `json:"kid"`
    Use    string   `json:"use"`
    Alg    string   `json:"alg"`
    N      string   `json:"n"`
    E      string   `json:"e"`
    KeyOps []string `json:"key_ops,omitempty"`
}

const (
    kty    string = "OKP"
    crv    string = "Ed25519"
    use    string = "sig"
    alg    string = "EdDSA"
    verify string = "verify"

    p256Kty string = "EC"
    p256Crv string = "P-256"
)

// ...

func EncodeEd25519Jwk(publicKey ed25519.PublicKey, kid string) Ed25519JWK {
    return Ed25519JWK{
        Kty:    kty,
        Crv:    crv,
        X:      base64.RawURLEncoding.EncodeToString(publicKey),
        Use:    use,
        Alg:    alg,
        Kid:    kid,
        KeyOps: []string{verify},
    }
}

func DecodeEd25519Jwk(jwk Ed25519JWK) (ed25519.PublicKey, error) {
    publicKey, err := base64.RawURLEncoding.DecodeString(jwk.X)
    if err != nil {
        return nil, err
    }

    return publicKey, nil
}

func EncodeP256Jwk(publicKey *ecdsa.PublicKey, kid string) P256JWK {
    return P256JWK{
        Kty:    p256Kty,
        Crv:    p256Crv,
        X:      base64.RawURLEncoding.EncodeToString(publicKey.X.Bytes()),
        Y:      base64.RawURLEncoding.EncodeToString(publicKey.Y.Bytes()),
        Use:    use,
        Alg:    alg,
        Kid:    kid,
        KeyOps: []string{verify},
    }
}

func DecodeP256Jwk(jwk P256JWK) (ecdsa.PublicKey, error) {
    x, err := base64.RawURLEncoding.DecodeString(jwk.X)
    if err != nil {
        return ecdsa.PublicKey{}, err
    }

    y, err := base64.RawURLEncoding.DecodeString(jwk.Y)
    if err != nil {
        return ecdsa.PublicKey{}, err
    }

    return ecdsa.PublicKey{
        Curve: elliptic.P256(),
        X:     new(big.Int).SetBytes(x),
        Y:     new(big.Int).SetBytes(y),
    }, nil
}

func DecodeRS256Jwk(jwk RS256JWK) (*rsa.PublicKey, error) {
    nBytes, err := base64.RawURLEncoding.DecodeString(jwk.N)
    if err != nil {
        return nil, err
    }
    n := new(big.Int).SetBytes(nBytes)

    eBytes, err := base64.RawURLEncoding.DecodeString(jwk.E)
    if err != nil {
        return nil, err
    }
    e := big.NewInt(0).SetBytes(eBytes).Int64()

    if e <= 0 {
        return nil, fmt.Errorf("invalid RSA exponent")
    }

    return &rsa.PublicKey{N: n, E: int(e)}, nil
}

Provider

For each key we need a key pair, and a reference to a previous public key for keys rotation, create the tokens/tokens.go package:

package tokens

import (
    "crypto/ecdsa"
    "crypto/ed25519"
    "crypto/x509"
    "encoding/pem"

    "github.com/golang-jwt/jwt/v5"

    "github.com/tugascript/devlogs/idp/internal/config"
    "github.com/tugascript/devlogs/idp/internal/utils"
)

type PreviousPublicKey struct {
    publicKey ed25519.PublicKey
    kid       string
}

type TokenKeyPair struct {
    publicKey  ed25519.PublicKey
    privateKey ed25519.PrivateKey
    kid        string
}

type TokenSecretData struct {
    curKeyPair TokenKeyPair
    prevPubKey *PreviousPublicKey
    ttlSec     int64
}

// ...

type PreviousEs256PublicKey struct {
    publicKey *ecdsa.PublicKey
    kid       string
}

type Es256TokenKeyPair struct {
    privateKey *ecdsa.PrivateKey
    publicKey  *ecdsa.PublicKey
    kid        string
}

type Es256TokenSecretData struct {
    curKeyPair Es256TokenKeyPair
    prevPubKey *PreviousEs256PublicKey
    ttlSec     int64
}

Now each key is encoded as a PEM in the environment, hence we need to decode the x509 certificates for:

Ed25519 keys:

package tokens

// ...

func extractEd25519PublicKey(publicKey string) (ed25519.PublicKey, string) {
    publicKeyBlock, _ := pem.Decode([]byte(publicKey))
    if publicKeyBlock == nil || publicKeyBlock.Type != "PUBLIC KEY" {
        panic("Invalid public key")
    }

    publicKeyData, err := x509.ParsePKIXPublicKey(publicKeyBlock.Bytes)
    if err != nil {
        panic(err)
    }

    publicKeyValue, ok := publicKeyData.(ed25519.PublicKey)
    if !ok {
        panic("Invalid public key")
    }

    return publicKeyValue, utils.ExtractKeyID(publicKeyValue)
}

func extractEd25519PrivateKey(privateKey string) ed25519.PrivateKey {
    privateKeyBlock, _ := pem.Decode([]byte(privateKey))
    if privateKeyBlock == nil || privateKeyBlock.Type != "PRIVATE KEY" {
        panic("Invalid private key")
    }

    privateKeyData, err := x509.ParsePKCS8PrivateKey(privateKeyBlock.Bytes)
    if err != nil {
        panic(err)
    }

    privateKeyValue, ok := privateKeyData.(ed25519.PrivateKey)
    if !ok {
        panic("Invalid private key")
    }

    return privateKeyValue
}

func extractEd25519PublicPrivateKeyPair(publicKey, privateKey string) TokenKeyPair {
    pubKey, kid := extractEd25519PublicKey(publicKey)
    return TokenKeyPair{
        publicKey:  pubKey,
        privateKey: extractEd25519PrivateKey(privateKey),
        kid:        kid,
    }
}

func newTokenSecretData(
    publicKey,
    privateKey,
    previousPublicKey string,
    ttlSec int64,
) TokenSecretData {
    curKeyPair := extractEd25519PublicPrivateKeyPair(publicKey, privateKey)

    if previousPublicKey != "" {
        pubKey, kid := extractEd25519PublicKey(previousPublicKey)
        return TokenSecretData{
            curKeyPair: curKeyPair,
            prevPubKey: &PreviousPublicKey{publicKey: pubKey, kid: kid},
            ttlSec:     ttlSec,
        }
    }

    return TokenSecretData{
        curKeyPair: curKeyPair,
        ttlSec:     ttlSec,
    }
}

Es256 keys:

package tokens

// ...

func extractEs256KeyPair(privateKey string) Es256TokenKeyPair {
    privateKeyBlock, _ := pem.Decode([]byte(privateKey))
    if privateKeyBlock == nil || privateKeyBlock.Type != "PRIVATE KEY" {
        panic("Invalid private key")
    }

    privateKeyData, err := x509.ParsePKCS8PrivateKey(privateKeyBlock.Bytes)
    if err != nil {
        privateKeyData, err = x509.ParseECPrivateKey(privateKeyBlock.Bytes)
        if err != nil {
            panic(err)
        }
    }

    privateKeyValue, ok := privateKeyData.(*ecdsa.PrivateKey)
    if !ok {
        panic("Invalid private key")
    }

    publicKeyValue, err := x509.MarshalPKIXPublicKey(&privateKeyValue.PublicKey)
    if err != nil {
        panic(err)
    }

    return Es256TokenKeyPair{
        privateKey: privateKeyValue,
        publicKey:  &privateKeyValue.PublicKey,
        kid:        utils.ExtractKeyID(publicKeyValue),
    }
}

func extractEs256PublicKey(publicKey string) (*ecdsa.PublicKey, string) {
    publicKeyBlock, _ := pem.Decode([]byte(publicKey))
    if publicKeyBlock == nil || publicKeyBlock.Type != "PUBLIC KEY" {
        panic("Invalid public key")
    }

    publicKeyData, err := x509.ParsePKIXPublicKey(publicKeyBlock.Bytes)
    if err != nil {
        panic(err)
    }

    pubKey, ok := publicKeyData.(*ecdsa.PublicKey)
    if !ok {
        panic("Invalid public key")
    }

    publicKeyValue, err := x509.MarshalPKIXPublicKey(pubKey)
    if err != nil {
        panic(err)
    }

    return pubKey, utils.ExtractKeyID(publicKeyValue)
}

// ...

func newEs256TokenSecretData(privateKey, previousPublicKey string, ttlSec int64) Es256TokenSecretData {
    curKeyPair := extractEs256KeyPair(privateKey)

    if previousPublicKey != "" {
        prevPubKey, kid := extractEs256PublicKey(previousPublicKey)
        return Es256TokenSecretData{
            curKeyPair: curKeyPair,
            prevPubKey: &PreviousEs256PublicKey{publicKey: prevPubKey, kid: kid},
            ttlSec:     ttlSec,
        }
    }

    return Es256TokenSecretData{
        curKeyPair: curKeyPair,
        ttlSec:     ttlSec,
    }
}

Finally finish by creating the full provider:

package tokens

// ...

type Tokens struct {
    frontendDomain         string
    backendDomain          string
    accessData             Es256TokenSecretData
    accountCredentialsData Es256TokenSecretData
    refreshData            TokenSecretData
    confirmationData       TokenSecretData
    resetData              TokenSecretData
    oauthData              TokenSecretData
    twoFAData              TokenSecretData
    jwks                   []utils.P256JWK
}

func NewTokens(
    accessCfg,
    accountCredentialsCfg,
    refreshCfg,
    confirmationCfg,
    resetCfg,
    oauthCfg,
    twoFACfg config.SingleJwtConfig,
    frontendDomain,
    backendDomain string,
) *Tokens {
    accessData := newEs256TokenSecretData(
        accessCfg.PrivateKey(),
        accessCfg.PreviousPublicKey(),
        accessCfg.TtlSec(),
    )
    accountKeysData := newEs256TokenSecretData(
        accountCredentialsCfg.PrivateKey(),
        accountCredentialsCfg.PreviousPublicKey(),
        accountCredentialsCfg.TtlSec(),
    )

    jwks := []utils.P256JWK{
        utils.EncodeP256Jwk(accountKeysData.curKeyPair.publicKey, accountKeysData.curKeyPair.kid),
        utils.EncodeP256Jwk(accessData.curKeyPair.publicKey, accessData.curKeyPair.kid),
    }

    if accountKeysData.prevPubKey != nil {
        jwks = append(jwks, utils.EncodeP256Jwk(
            accountKeysData.prevPubKey.publicKey,
            accessData.prevPubKey.kid,
        ))
    }
    if accessData.prevPubKey != nil {
        jwks = append(jwks, utils.EncodeP256Jwk(
            accessData.prevPubKey.publicKey,
            accessData.prevPubKey.kid,
        ))
    }

    return &Tokens{
        accessData:             accessData,
        accountCredentialsData: accountKeysData,
        refreshData: newTokenSecretData(
            refreshCfg.PublicKey(),
            refreshCfg.PrivateKey(),
            refreshCfg.PreviousPublicKey(),
            refreshCfg.TtlSec(),
        ),
        confirmationData: newTokenSecretData(
            confirmationCfg.PublicKey(),
            confirmationCfg.PrivateKey(),
            confirmationCfg.PreviousPublicKey(),
            confirmationCfg.TtlSec(),
        ),
        resetData: newTokenSecretData(
            resetCfg.PublicKey(),
            resetCfg.PrivateKey(),
            resetCfg.PreviousPublicKey(),
            resetCfg.TtlSec(),
        ),
        oauthData: newTokenSecretData(
            oauthCfg.PublicKey(),
            oauthCfg.PrivateKey(),
            oauthCfg.PreviousPublicKey(),
            oauthCfg.TtlSec(),
        ),
        twoFAData: newTokenSecretData(
            twoFACfg.PublicKey(),
            twoFACfg.PrivateKey(),
            twoFACfg.PreviousPublicKey(),
            twoFACfg.TtlSec(),
        ),
        frontendDomain: frontendDomain,
        backendDomain:  backendDomain,
        jwks:           jwks,
    }
}

func (t *Tokens) JWKs() []utils.P256JWK {
    return t.jwks
}

Error Handling

Error handling in Go works differently from most languages, instead of throwing and catching exceptions, values are return as a pointer (error as values) to the error.

This means for ease of use, we need to map/coerce the errors to a known value on the service layer, and map them to an error response on the controller layer.

Service Errors

Create an expections directory on the internal folder for the errors mapping.

Format

The service error is gonna have a standard error class, with a type/code, and a message, but for simplicity we won't add a details slice field.

Create the exceptions/services.go file for the exceptions package:

package exceptions

type ServiceError struct {
    Code    string
    Message string
}

func NewError(code string, message string) *ServiceError {
    return &ServiceError{
        Code:    code,
        Message: message,
    }
}

// To make `ServiceError` an error type interface
func (e *ServiceError) Error() string {
    return e.Message
}

Codes and messages

We need to standerdize the code and message based on HTTP status codes:

package exceptions

// ...

const (
    CodeValidation           string = "VALIDATION"
    CodeConflict             string = "CONFLICT"
    CodeInvalidEnum          string = "INVALID_ENUM"
    CodeNotFound             string = "NOT_FOUND"
    CodeUnknown              string = "UNKNOWN"
    CodeServerError          string = "SERVER_ERROR"
    CodeUnauthorized         string = "UNAUTHORIZED"
    CodeForbidden            string = "FORBIDDEN"
    CodeUnsupportedMediaType string = "UNSUPPORTED_MEDIA_TYPE"
)

const (
    MessageDuplicateKey string = "Resource already exists"
    MessageNotFound     string = "Resource not found"
    MessageUnknown      string = "Something went wrong"
    MessageUnauthorized string = "Unauthorized"
    MessageForbidden    string = "Forbidden"
)

func NewNotFoundError() *ServiceError {
    return NewError(CodeNotFound, MessageNotFound)
}

func NewValidationError(message string) *ServiceError {
    return NewError(CodeValidation, message)
}

func NewServerError() *ServiceError {
    return NewError(CodeServerError, MessageUnknown)
}

func NewConflictError(message string) *ServiceError {
    return NewError(CodeConflict, message)
}

func NewUnsupportedMediaTypeError(message string) *ServiceError {
    return NewError(CodeUnsupportedMediaType, message)
}

func NewUnauthorizedError() *ServiceError {
    return NewError(CodeUnauthorized, MessageUnauthorized)
}

func NewForbiddenError() *ServiceError {
    return NewError(CodeForbidden, MessageForbidden)
}

Coercion

For the PostgreSQL errors we can coerce them using a mapper:

package exceptions

import (
    "errors"

    "github.com/jackc/pgx/v5"
    "github.com/jackc/pgx/v5/pgconn"
)

// ...

func FromDBError(err error) *ServiceError {
    if errors.Is(err, pgx.ErrNoRows) {
        return NewError(CodeNotFound, MessageNotFound)
    }

    var pgErr *pgconn.PgError
    if errors.As(err, &pgErr) {
        switch pgErr.Code {
        case "23505":
            return NewError(CodeConflict, MessageDuplicateKey)
        case "23514":
            return NewError(CodeInvalidEnum, pgErr.Message)
        case "23503":
            return NewError(CodeNotFound, MessageNotFound)
        default:
            return NewError(CodeUnknown, pgErr.Message)
        }
    }

    return NewError(CodeUnknown, MessageUnknown)
}

Controllers Error

ServiceError need to be mappend to a JSON Response, however there are also other types of error responses: body validation and oauth validation.

Error Response

Create the controllers.go and add the error response, which is the same as ServiceError but JSON parsable:

package exceptions 

// ...

const (
    StatusConflict     string = "Conflict"
    StatusInvalidEnum  string = "BadRequest"
    StatusNotFound     string = "NotFound"
    StatusServerError  string = "InternalServerError"
    StatusUnknown      string = "InternalServerError"
    StatusUnauthorized string = "Unauthorized"
    StatusForbidden    string = "Forbidden"
    StatusValidation   string = "Validation"
)

type ErrorResponse struct {
    Code    string `json:"code"`
    Message string `json:"message"`
}

func NewErrorResponse(err *ServiceError) ErrorResponse {
    switch err.Code {
    case CodeServerError:
        return ErrorResponse{
            Code:    StatusServerError,
            Message: err.Message,
        }
    case CodeConflict:
        return ErrorResponse{
            Code:    StatusConflict,
            Message: err.Message,
        }
    case CodeInvalidEnum:
        return ErrorResponse{
            Code:    StatusInvalidEnum,
            Message: err.Message,
        }
    case CodeNotFound:
        return ErrorResponse{
            Code:    StatusNotFound,
            Message: err.Message,
        }
    case CodeValidation:
        return ErrorResponse{
            Code:    StatusValidation,
            Message: err.Message,
        }
    case CodeUnknown:
        return ErrorResponse{
            Code:    StatusUnknown,
            Message: err.Message,
        }
    case CodeUnauthorized:
        return ErrorResponse{
            Code:    StatusUnauthorized,
            Message: StatusUnauthorized,
        }
    case CodeForbidden:
        return ErrorResponse{
            Code:    StatusForbidden,
            Message: StatusForbidden,
        }
    default:
        return ErrorResponse{
            Code:    StatusUnknown,
            Message: err.Message,
        }
    }
}

Validation Response

To validation the body inputs we will use the go validator package, start by installing it:

$ go get github.com/go-playground/validator/v10

Services

Services is where most of the server business logic is located, lets create the services/services.go package, the Services struct needs to encapsulate all providers, and be the only layer talking directly to them.

package services

import (
    "log/slog"

    "github.com/tugascript/devlogs/idp/internal/providers/cache"
    "github.com/tugascript/devlogs/idp/internal/providers/database"
    "github.com/tugascript/devlogs/idp/internal/providers/encryption"
    "github.com/tugascript/devlogs/idp/internal/providers/mailer"
    "github.com/tugascript/devlogs/idp/internal/providers/oauth"
    "github.com/tugascript/devlogs/idp/internal/providers/tokens"
)

type Services struct {
    logger         *slog.Logger
    database       *database.Database
    cache          *cache.Cache
    mail           *mailer.EmailPublisher
    jwt            *tokens.Tokens
    encrypt        *encryption.Encryption
    oauthProviders *oauth.Providers
}

func NewServices(
    logger *slog.Logger,
    database *database.Database,
    cache *cache.Cache,
    mail *mailer.EmailPublisher,
    jwt *tokens.Tokens,
    encrypt *encryption.Encryption,
    oauthProv *oauth.Providers,
) *Services {
    return &Services{
        logger:         logger,
        database:       database,
        cache:          cache,
        mail:           mail,
        jwt:            jwt,
        encrypt:        encrypt,
        oauthProviders: oauthProv,
    }
}

We also need a helper to build the logger, and some common constant, create a helpers.go file:



import (
    "log/slog"

    "github.com/tugascript/devlogs/idp/internal/utils"
)

const (
    AuthProviderEmail     string = "email"
    AuthProviderGoogle    string = "google"
    AuthProviderGitHub    string = "github"
    AuthProviderApple     string = "apple"
    AuthProviderMicrosoft string = "microsoft"
    AuthProviderFacebook  string = "facebook"

    TwoFactorNone  string = "none"
    TwoFactorEmail string = "email"
    TwoFactorTotp  string = "totp"
)

func (s *Services) buildLogger(requestID, location, function string) *slog.Logger {
    return utils.BuildLogger(s.logger, utils.LoggerOptions{
        Layer:     utils.ServicesLogLayer,
        Location:  location,
        Method:    function,
        RequestID: requestID,
    })
}

Health

For our first service, we will create the health endpoint for our API, this service only needs to ping PostgreSQL and ValKey.

Create the health.go file:

package services

import (
    "context"

    "github.com/tugascript/devlogs/idp/internal/exceptions"
)

const healthLocation string = "health"

func (s *Services) HealthCheck(ctx context.Context, requestID string) *exceptions.ServiceError {
    logger := s.buildLogger(requestID, healthLocation, "HealthCheck")
    logger.InfoContext(ctx, "Performing health check...")

    if err := s.database.Ping(ctx); err != nil {
        logger.ErrorContext(ctx, "Failed to ping database", "error", err)
        return exceptions.NewServerError()
    }
    if err := s.cache.Ping(ctx); err != nil {
        logger.ErrorContext(ctx, "Failed to ping cache", "error", err)
        return exceptions.NewServerError()
    }

    logger.InfoContext(ctx, "Service is healthy")
    return nil
}

Controllers

Controllers are where we map our services to the correct HTTP status response.

Helpers

For the controllor, error handling and logging is the same for all routes, hence create a logger builder and error handling functions on a helpers.go file.

package controllers

import (
    "errors"
    "fmt"
    "log/slog"

    "github.com/go-playground/validator/v10"
    "github.com/gofiber/fiber/v2"
    "github.com/google/uuid"

    "github.com/tugascript/devlogs/idp/internal/exceptions"
    "github.com/tugascript/devlogs/idp/internal/utils"
)

func (c *Controllers) buildLogger(
    requestID,
    location,
    method string,
) *slog.Logger {
    return utils.BuildLogger(c.logger, utils.LoggerOptions{
        Layer:     utils.ControllersLogLayer,
        Location:  location,
        Method:    method,
        RequestID: requestID,
    })
}

func logRequest(logger *slog.Logger, ctx *fiber.Ctx) {
    logger.InfoContext(
        ctx.UserContext(),
        fmt.Sprintf("Request: %s %s", ctx.Method(), ctx.Path()),
    )
}

func getRequestID(ctx *fiber.Ctx) string {
    return ctx.Get("requestid", uuid.NewString())
}

func logResponse(logger *slog.Logger, ctx *fiber.Ctx, status int) {
    logger.InfoContext(
        ctx.UserContext(),
        fmt.Sprintf("Response: %s %s", ctx.Method(), ctx.Path()),
        "status", status,
    )
}

func validateErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, location string, err error) error {
    logger.WarnContext(ctx.UserContext(), "Failed to validate request", "error", err)
    logResponse(logger, ctx, fiber.StatusBadRequest)

    var errs validator.ValidationErrors
    ok := errors.As(err, &errs)
    if !ok {
        return ctx.
            Status(fiber.StatusBadRequest).
            JSON(exceptions.NewEmptyValidationErrorResponse(location))
    }

    return ctx.
        Status(fiber.StatusBadRequest).
        JSON(exceptions.ValidationErrorResponseFromErr(&errs, location))
}

func validateBodyErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, err error) error {
    return validateErrorResponse(logger, ctx, exceptions.ValidationResponseLocationBody, err)
}

func validateURLParamsErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, err error) error {
    return validateErrorResponse(logger, ctx, exceptions.ValidationResponseLocationParams, err)
}

func validateQueryParamsErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, err error) error {
    return validateErrorResponse(logger, ctx, exceptions.ValidationResponseLocationQuery, err)
}

func serviceErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, serviceErr *exceptions.ServiceError) error {
    status := exceptions.NewRequestErrorStatus(serviceErr.Code)
    resErr := exceptions.NewErrorResponse(serviceErr)
    logResponse(logger, ctx, status)
    return ctx.Status(status).JSON(&resErr)
}

func oauthErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, message string) error {
    resErr := exceptions.NewOAuthError(message)
    logResponse(logger, ctx, fiber.StatusBadRequest)
    return ctx.Status(fiber.StatusBadRequest).JSON(&resErr)
}

func parseRequestErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, err error) error {
    logger.WarnContext(ctx.UserContext(), "Failed to parse request", "error", err)
    logResponse(logger, ctx, fiber.StatusBadRequest)
    return ctx.
        Status(fiber.StatusBadRequest).
        JSON(exceptions.NewEmptyValidationErrorResponse(exceptions.ValidationResponseLocationBody))
}

Health

The health controller is pretty simple, since it will only consume the service HealtCheck method directly:

package controllers

import "github.com/gofiber/fiber/v2"

func (c *Controllers) HealthCheck(ctx *fiber.Ctx) error {
    requestID := getRequestID(ctx)
    logger := c.buildLogger(requestID, "health", "HealthCheck")
    logRequest(logger, ctx)

    if serviceErr := c.services.HealthCheck(ctx.UserContext(), requestID); serviceErr != nil {
        return serviceErrorResponse(logger, ctx, serviceErr)
    }

    return ctx.SendStatus(fiber.StatusOK)
}

Paths

On the controllers folder create a paths package with the health.go path:

package paths

const Health string = "/health"

Server

The server is where we will initialize the Fiber App, build Services, Controllers and load common middleware.

Routes

Start by creating a server/routes and add routes.go file with the Routes struct:

package routes

import (
    "github.com/tugascript/devlogs/idp/internal/controllers"
)

type Routes struct {
    controllers *controllers.Controllers
}

func NewRoutes(ctrls *controllers.Controllers) *Routes {
    return &Routes{controllers: ctrls}
}

Now create a router method for the health endpoint on health.go:

package routes

import (
    "github.com/gofiber/fiber/v2"

    "github.com/tugascript/devlogs/idp/internal/controllers/paths"
)

func (r *Routes) HealthRoutes(app *fiber.App) {
    app.Get(paths.Health, r.controllers.HealthCheck)
}

Plus also create a common.go with the route Group for API version 1:

package routes

import "github.com/gofiber/fiber/v2"

const V1Path string = "/v1"

func v1PathRouter(app *fiber.App) fiber.Router {
    return app.Group(V1Path)
}

Server Instance

Server instance is where we hook up most of our logic, hence it will be a big method where you take the config as a parameter and initialize everything.

On the server.go file inside the server directory, and create the FiberServer instance:

package server

import (
    // ...

    // ...
    "github.com/tugascript/devlogs/idp/internal/server/routes"
    // ...
)

type FiberServer struct {
    *fiber.App
    routes *routes.Routes
}

And load everything in the New function that takes the context.Context, the struter logger (*slog.Logger) and the configuration (config.Config):

package server

import (
    "context"
    "log/slog"
    "time"

    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/cors"
    "github.com/gofiber/fiber/v2/middleware/encryptcookie"
    "github.com/gofiber/fiber/v2/middleware/helmet"
    "github.com/gofiber/fiber/v2/middleware/limiter"
    "github.com/gofiber/fiber/v2/middleware/requestid"
    fiberRedis "github.com/gofiber/storage/redis/v3"
    "github.com/google/uuid"
    "github.com/jackc/pgx/v5/pgxpool"

    "github.com/tugascript/devlogs/idp/internal/config"
    "github.com/tugascript/devlogs/idp/internal/controllers"
    "github.com/tugascript/devlogs/idp/internal/providers/cache"
    "github.com/tugascript/devlogs/idp/internal/providers/database"
    "github.com/tugascript/devlogs/idp/internal/providers/encryption"
    "github.com/tugascript/devlogs/idp/internal/providers/mailer"
    "github.com/tugascript/devlogs/idp/internal/providers/oauth"
    "github.com/tugascript/devlogs/idp/internal/providers/tokens"
    "github.com/tugascript/devlogs/idp/internal/server/routes"
    "github.com/tugascript/devlogs/idp/internal/server/validations"
    "github.com/tugascript/devlogs/idp/internal/services"
)

// ...

func New(
    ctx context.Context,
    logger *slog.Logger,
    cfg config.Config,
) *FiberServer {
    logger.InfoContext(ctx, "Building redis storage...")
    cacheStorage := fiberRedis.New(fiberRedis.Config{
        URL: cfg.RedisURL(),
    })
    cc := cache.NewCache(
        logger,
        cacheStorage,
    )
    logger.InfoContext(ctx, "Finished building redis storage")

    logger.InfoContext(ctx, "Building database connection pool...")
    dbConnPool, err := pgxpool.New(ctx, cfg.DatabaseURL())
    if err != nil {
        logger.ErrorContext(ctx, "Failed to connect to database", "error", err)
        panic(err)
    }
    db := database.NewDatabase(dbConnPool)
    logger.InfoContext(ctx, "Finished building database connection pool")

    logger.InfoContext(ctx, "Building mailer...")
    mail := mailer.NewEmailPublisher(
        cc.Client(),
        cfg.EmailPubChannel(),
        cfg.FrontendDomain(),
        logger,
    )
    logger.InfoContext(ctx, "Finished building mailer")

    logger.InfoContext(ctx, "Building JWT token keys...")
    tokensCfg := cfg.TokensConfig()
    jwts := tokens.NewTokens(
        tokensCfg.Access(),
        tokensCfg.AccountCredentials(),
        tokensCfg.Refresh(),
        tokensCfg.Confirm(),
        tokensCfg.Reset(),
        tokensCfg.OAuth(),
        tokensCfg.TwoFA(),
        cfg.FrontendDomain(),
        cfg.BackendDomain(),
    )
    logger.InfoContext(ctx, "Finished building JWT tokens keys")

    logger.InfoContext(ctx, "Building encryption...")
    encryp := encryption.NewEncryption(logger, cfg.EncryptionConfig(), cfg.BackendDomain())
    logger.InfoContext(ctx, "Finished encryption")

    logger.InfoContext(ctx, "Building OAuth provider...")
    oauthProvidersCfg := cfg.OAuthProvidersConfig()
    oauthProviders := oauth.NewProviders(
        logger,
        oauthProvidersCfg.GitHub(),
        oauthProvidersCfg.Google(),
        oauthProvidersCfg.Facebook(),
        oauthProvidersCfg.Apple(),
        oauthProvidersCfg.Microsoft(),
    )
    logger.InfoContext(ctx, "Finished building OAuth provider")

    logger.InfoContext(ctx, "Building services...")
    newServices := services.NewServices(
        logger,
        db,
        cc,
        mail,
        jwts,
        encryp,
        oauthProviders,
    )
    logger.InfoContext(ctx, "Finished building services")

    logger.InfoContext(ctx, "Loading validators...")
    vld := validations.NewValidator(logger)
    logger.InfoContext(ctx, "Finished loading validators")

    server := &FiberServer{
        App: fiber.New(fiber.Config{
            ServerHeader: "idp",
            AppName:      "idp",
        }),
        routes: routes.NewRoutes(controllers.NewControllers(
            logger,
            newServices,
            vld,
            cfg.FrontendDomain(),
            cfg.BackendDomain(),
            cfg.CookieName(),
        )),
    }

    logger.InfoContext(ctx, "Loading middleware...")
    server.Use(helmet.New())
    server.Use(requestid.New(requestid.Config{
        Header: fiber.HeaderXRequestID,
        Generator: func() string {
            return uuid.NewString()
        },
    }))
    rateLimitCfg := cfg.RateLimiterConfig()
    server.Use(limiter.New(limiter.Config{
        Max:               int(rateLimitCfg.Max()),
        Expiration:        time.Duration(rateLimitCfg.ExpSec()) * time.Second,
        LimiterMiddleware: limiter.SlidingWindow{},
        Storage:           cacheStorage,
    }))
    server.Use(encryptcookie.New(encryptcookie.Config{
        Key: cfg.CookieSecret(),
    }))
    server.App.Use(cors.New(cors.Config{
        AllowOrigins:     "*",
        AllowMethods:     "GET,POST,PUT,DELETE,OPTIONS,PATCH,HEAD",
        AllowHeaders:     "Accept,Authorization,Content-Type",
        AllowCredentials: false, // credentials require explicit origins
        MaxAge:           300,
    }))
    logger.Info("Finished loading common middlewares")

    return server
}

Registering the routes

Just create a routes.go file inside the server directory with a RegisterFiberRoutes method:

package server

func (s *FiberServer) RegisterFiberRoutes() {
    s.routes.HealthRoutes(s.App)
}

Logging

For logging we will need to create a default logger initially then add the configuration when they are loaded on startup, on a logger.go file:

package server

import (
    "log/slog"
    "os"

    "github.com/tugascript/devlogs/idp/internal/config"
)

func DefaultLogger() *slog.Logger {
    return slog.New(slog.NewJSONHandler(
        os.Stdout,
        &slog.HandlerOptions{
            Level: slog.LevelInfo,
        },
    ))
}

func ConfigLogger(cfg config.LoggerConfig) *slog.Logger {
    logLevel := slog.LevelInfo

    if cfg.IsDebug() {
        logLevel = slog.LevelDebug
    }

    logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
        Level: logLevel,
    }))

    if cfg.Env() == "production" {
        logger = slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
            Level: logLevel,
        }))
    }

    return logger.With("service", cfg.ServiceName())
}

Running the app

On the cmd/api folder's main.go file just set-up everything and register the routes:

package main

import (
    "context"
    "fmt"
    "log/slog"
    "os/signal"
    "runtime"
    "syscall"
    "time"

    "github.com/tugascript/devlogs/idp/internal/config"
    "github.com/tugascript/devlogs/idp/internal/server"
)

func gracefulShutdown(
    logger *slog.Logger,
    fiberServer *server.FiberServer,
    done chan bool,
) {
    // Create context that listens for the interrupt signal from the OS.
    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer stop()

    // Listen for the interrupt signal.
    <-ctx.Done()

    logger.InfoContext(ctx, "shutting down gracefully, press Ctrl+C again to force")

    // The context is used to inform the server it has 5 seconds to finish
    // the request it is currently handling
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if err := fiberServer.ShutdownWithContext(ctx); err != nil {
        logger.ErrorContext(ctx, "Server forced to shutdown with error", "error", err)
    }

    logger.InfoContext(ctx, "Server exiting")

    // Notify the main goroutine that the shutdown is complete
    done <- true
}

func main() {
    logger := server.DefaultLogger()
    ctx := context.Background()
    logger.InfoContext(ctx, "Loading configuration...")
    cfg := config.NewConfig(logger, "./.env")

    logger = server.ConfigLogger(cfg.LoggerConfig())
    logger.InfoContext(ctx, "Setting GOMAXPROCS...", "maxProcs", cfg.MaxProcs())
    runtime.GOMAXPROCS(int(cfg.MaxProcs()))
    logger.InfoContext(ctx, "Finished setting GOMAXPROCS")

    logger.InfoContext(ctx, "Building server...")
    server := server.New(ctx, logger, cfg)
    logger.InfoContext(ctx, "Server built")

    server.RegisterFiberRoutes()

    // Create a done channel to signal when the shutdown is complete
    done := make(chan bool, 1)

    go func() {
        err := server.Listen(fmt.Sprintf(":%d", cfg.Port()))
        if err != nil {
            logger.ErrorContext(ctx, "http server error", "error", err)
            panic(fmt.Sprintf("http server error: %s", err))
        }
    }()

    // Run graceful shutdown in a separate goroutine
    go gracefulShutdown(logger, server, done)

    // Wait for the graceful shutdown to complete
    <-done
    logger.InfoContext(ctx, "Graceful shutdown complete.")
}

Conclusion

In this first article, we introduced the fiber framework, a express inspired go library, as well as how to set up our API with a MSC (model, service, controller) architecture with a centralized configuration.

This code is based on the current ongoing project found on the devlogs repository.

About the Author

Hey there! I am Afonso Barracha, your go-to econometrician who found his way into the world of back-end development with a soft spot for GraphQL. If you enjoyed reading this article, why not show some love by buying me a coffee?

Lately, I have been diving deep into more advanced subjects. As a result, I have switched from sharing my thoughts every week to posting once or twice a month. This way, I can make sure to bring you the highest quality content possible.

Do not miss out on any of my latest articles – follow me here on dev, LinkedIn or Instagram to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

NestJS Authentication with OAuth2.0: Adding Mobile Apps Support

Afonso Barracha — Sat, 24 Aug 2024 03:56:41 +0000

Series Intro

This series will cover the full implementation of OAuth2.0 Authentication in NestJS for the following types of APIs:

Express REST API;
Fastify REST API;
Apollo GraphQL API.

And it is divided in 5 parts:

Configuration and operations;
Express Local OAuth REST API;
Fastify Local OAuth REST API;
Apollo Local OAuth GraphQL API;
Adding External OAuth Providers to our API;

Lets start an extra 6th part.

Tutorial Intro

Originally this series was made in a way that only works for Web Apps that use web browsers, however nowadays most people use smartphones to interact with the internet. For the most part, mobile apps do not support HTTP-only cookies; hence, we need to return information about the tokens, including the refresh token and the access token validity period, for better storage and management.

On this tutorial we will make the necessary changes to our API to make it mobile friendly.

Returning tokenType, expiresIn and refreshToken with the authentication response.
Add optional body with refreshToken parameter to the logout and refresh endpoints.
Add a token endpoint for external providers to remove the need for HTTP only cookies.

Local Mobile friendly API

For local authentication we need to make two changes:

Update the response to have tokenType, expiresIn and refreshToken.
Allow the user to pass the refreshToken on the logout and refresh access endpoints.

Updated Response Mapper

Start by fixing the IAuthResponse interface and AuthResponseMapper class:

src/auth/interfaces/auth-response.interface.ts:

// ...

export interface IAuthResponse {
  // ...
  // New fields
  refreshToken: string;
  tokenType: string;
  expiresIn: number;
}

src/auth/mappers/auth-response.mapper.ts:

// ...

export class AuthResponseMapper implements IAuthResponse {
  // ...
  // New fields

  @ApiProperty({
    description: 'Refresh token',
    example:
  'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c',
    type: String,
  })
  public readonly refreshToken: string;

  @ApiProperty({
    description: 'Token type',
    example: 'Bearer',
    type: String,
  })
  public readonly tokenType: string;

  @ApiProperty({
    description: 'Expiration period in seconds',
    example: 3600,
    type: Number,
  })
  public readonly expiresIn: number;

  constructor(values: IAuthResponse) {
    Object.assign(this, values);
  }

  public static map(result: IAuthResult): AuthResponseMapper {
    return new AuthResponseMapper({
      user: AuthResponseUserMapper.map(result.user),
      accessToken: result.accessToken,
      refreshToken: result.refreshToken,
      tokenType: 'Bearer',
      expiresIn: result.expiresIn,
    });
  }
}

Services Updates

To get the access token validity period we need to create a getter for it on jwt.service.ts:

@Injectable()
export class JwtService {
  // ...

  public get accessTime(): number {
    return this.jwtConfig.access.time;
  }
}

Then on the auth.service.ts update all methods that return Promise<IAuthResult> with the new response format:

@Injectable()
export class AuthService {
  // ...

  public async confirmEmail(
    dto: ConfirmEmailDto,
    domain?: string,
  ): Promise<IAuthResult> {
    // ...
    return {
      user,
      accessToken,
      refreshToken,
      expiresIn: this.jwtService.accessTime,
    };
  }

  // ...

  public async signIn(dto: SignInDto, domain?: string): Promise<IAuthResult> {
    // ...

    return {
      user,
      accessToken,
      refreshToken,
      expiresIn: this.jwtService.accessTime,
    };
  }

  // ...

  public async refreshTokenAccess(
    refreshToken: string,
    domain?: string,
  ): Promise<IAuthResult> {
    // ...

    return {
      user,
      accessToken,
      refreshToken: newRefreshToken,
      expiresIn: this.jwtService.accessTime,
    };
  }

  // ...

  public async updatePassword(
    userId: number,
    dto: ChangePasswordDto,
    domain?: string,
  ): Promise<IAuthResult> {
    // ...
    return {
      user,
      accessToken,
      refreshToken,
      expiresIn: this.jwtService.accessTime,
    };
  }
}

Refresh token in the body

Now for /refresh-access and /logout we need to be able to pass the refreshToken both from the body and from the cookie.

Start by creating a new refresh-access.dto.ts:

import { ApiProperty } from '@nestjs/swagger';
import { IsJWT, IsOptional, IsString } from 'class-validator';

export abstract class RefreshAccessDto {
  @ApiProperty({
    description: 'The JWT token sent to the user email',
    example:
      'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c',
    type: String,
  })
  @IsOptional()
  @IsString()
  @IsJWT()
  public refreshToken?: string;
}

And update the AuthController's refreshTokenFromReq method to take the dto:

// ...
import { RefreshAccessDto } from './dtos/refresh-access.dto';
// ...

@ApiTags('Auth')
@Controller('api/auth')
@UseGuards(FastifyThrottlerGuard)
export class AuthController {
  // ...

  private refreshTokenFromReq(
    req: FastifyRequest,
    dto?: RefreshAccessDto,
  ): string {
    const token: string | undefined = req.cookies[this.cookieName];

    if (isUndefined(token) || isNull(token)) {
      if (!isUndefined(dto?.refreshToken)) {
        return dto.refreshToken;
      }

      throw new UnauthorizedException();
    }

    const { valid, value } = req.unsignCookie(token);

    if (!valid) {
      throw new UnauthorizedException();
    }

    return value;
  }

  // ...
}

Finally, pass the dto to both endpoints:

// ...

@ApiTags('Auth')
@Controller('api/auth')
@UseGuards(FastifyThrottlerGuard)
export class AuthController {
  // ...

  @Public()
  @Post('/refresh-access')
  @ApiOkResponse({
    type: AuthResponseMapper,
    description: 'Refreshes and returns the access token',
  })
  @ApiUnauthorizedResponse({
    description: 'Invalid token',
  })
  @ApiBadRequestResponse({
    description:
      'Something is invalid on the request body, or Token is invalid or expired',
  })
  public async refreshAccess(
    @Req() req: FastifyRequest,
    @Res() res: FastifyReply,
    @Body() refreshAccessDto?: RefreshAccessDto,
  ): Promise<void> {
    const token = this.refreshTokenFromReq(req, refreshAccessDto);
    const result = await this.authService.refreshTokenAccess(
      token,
      req.headers.origin,
    );
    this.saveRefreshCookie(res, result.refreshToken)
      .status(HttpStatus.OK)
      .send(AuthResponseMapper.map(result));
  }

  @Post('/logout')
  @ApiOkResponse({
    type: MessageMapper,
    description: 'The user is logged out',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body',
  })
  @ApiUnauthorizedResponse({
    description: 'Invalid token',
  })
  public async logout(
    @Req() req: FastifyRequest,
    @Res() res: FastifyReply,
    @Body() refreshAccessDto?: RefreshAccessDto,
  ): Promise<void> {
    const token = this.refreshTokenFromReq(req, refreshAccessDto);
    const message = await this.authService.logout(token);
    res
      .clearCookie(this.cookieName, { path: this.cookiePath })
      .header('Content-Type', 'application/json')
      .status(HttpStatus.OK)
      .send(message);
  }

  // ...
}

External Mobile friendly API

To make our external OAuth mobile friendly we need to add an internal token endpoint for an internal code exchange.

Exchange Code Generation

First, we need to decide on the format of the code. A good, secure, and URL-friendly random string is a base62-encoded UUID, so that is what we will use. Add a static method to the Oauth2Service to generate base62 UUIDs:

// ...
import { v4 } from 'uuid';
// ...

@Injectable()
export class Oauth2Service {
  private static readonly BASE62 =
    '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
  private static readonly BIG62 = BigInt(Oauth2Service.BASE62.length);
  // ...

  private static generateCode(): string {
    let num = BigInt('0x' + v4().replace(/-/g, ''));
    let code = '';

    while (num > 0) {
      const remainder = Number(num % Oauth2Service.BIG62);
      code = Oauth2Service.BASE62[remainder] + code;
      num = num / Oauth2Service.BIG62;
    }

    return code.padStart(22, '0');
  }

  // ...
}

NOTE: this is a standard base 62 encoding that you can find by searching stackoverflow.

New Callback Logic

Since the callback now needs to work for both mobile and web, we need to redirect with Accept status code to the device.

Start by renaming the login method to callback on the Oauth2Service and create a callback-result.interface.ts file:

export interface ICallbackResult {
  readonly code: string;
  readonly accessToken: string;
  readonly expiresIn: number;
}

Next, update the callback method to cache the code and return both the code and accessToken:

// ...

@Injectable()
export class Oauth2Service {
  // ...

  private static getOAuthCodeKey(code: string): string {
    return `oauth_code:${code}`;
  }

  // ...

  public async callback(
    provider: OAuthProvidersEnum,
    email: string,
    name: string,
  ): Promise<ICallbackResult> {
    // Get or create a user for the given provider
    const user = await this.usersService.findOrCreate(provider, email, name);

    // Generate and cache the code with the user email
    const code = Oauth2Service.generateCode();
    await this.commonService.throwInternalError(
      this.cacheManager.set(
        Oauth2Service.getOAuthCodeKey(code),
        user.email,
        this.jwtService.accessTime * 1000,
      ),
    );

    // Generate an access token to authenticate on the token endpoint
    const accessToken = await this.jwtService.generateToken(
      user,
      TokenTypeEnum.ACCESS,
    );
    return {
      code,
      accessToken,
      expiresIn: this.jwtService.accessTime,
    };
  }

  // ...
}

Lastly, update the Oauth2Controller to redirect with the correct URL Search params. Rename the loginAndRedirect method to callbackAndRedirect, and add the new business logic:

// ...

@ApiTags('Oauth2')
@Controller('api/auth/ext')
@UseGuards(FastifyThrottlerGuard)
export class Oauth2Controller {
  // ... 

  private async callbackAndRedirect(
    res: FastifyReply,
    provider: OAuthProvidersEnum,
    email: string,
    name: string,
  ): Promise<FastifyReply> {
    // Get the code, access token and expiration
    const { code, accessToken, expiresIn } = await this.oauth2Service.callback(
      provider,
      email,
      name,
    );

    // Build the query params
    const urlSearchParams = new URLSearchParams({
      code,
      accessToken,
      tokenType: 'Bearer',
      expiresIn: expiresIn.toString(),
    });

    // Redirect to the front-end with the encoded query params
    return res
      .status(HttpStatus.FOUND) // Note that now we use `FOUND` instead of `Redirect` as the user still needs to do an extra request
      .redirect(`${this.url}/auth/callback?${urlSearchParams.toString()}`);
  }
}

Token Endpoint

At long last, to be able to get a refreshToken the we need to build a POST /api/auth/ext/token endpoint.

Start by creating a token method on the Oauth2Service:

// ...

@Injectable()
export class Oauth2Service {
  // ...

  public async token(code: string, userId: number): Promise<IAuthResult> {
    // Check if the code is cached
    const codeKey = Oauth2Service.getOAuthCodeKey(code);
    const email = await this.commonService.throwInternalError(
      this.cacheManager.get<string>(codeKey),
    );

    // Return 401 UNAUTHORIZED if code is invalid or expired
    if (!email) {
      throw new UnauthorizedException();
    }

    // Delete the code so it can't be used again
    await this.commonService.throwInternalError(this.cacheManager.del(codeKey));
    // Find the user by the cached email
    const user = await this.usersService.findOneByEmail(email);


    // Check if the authenticated user is the owner of the code
    if (user.id !== userId) {
      throw new UnauthorizedException();
    }

    // Return new access and refresh token
    const [accessToken, refreshToken] =
      await this.jwtService.generateAuthTokens(user);
    return {
      user,
      accessToken,
      refreshToken,
      expiresIn: this.jwtService.accessTime,
    };
  }

  // ...
}

Afterward, create a dto for this endpint token.dto.ts:

import { ApiProperty } from '@nestjs/swagger';
import { IsString, IsUrl, Length } from 'class-validator';

export abstract class TokenDto {
  @ApiProperty({
    description: 'The Code to exchange for a token',
    example: '5WA0R4DVyWThKFnc73z7nT',
    minLength: 1,
    maxLength: 22,
    type: String,
  })
  @IsString()
  @Length(1, 22)
  public code: string;

  @ApiProperty({
    description: 'Redirect URI that was used to get the token',
    example: 'https://example.com/auth/callback',
    type: String,
  })
  @IsString()
  @IsUrl()
  public redirectUri: string;
}

Finally, create the new endpoint on the Oauth2Controller:

// ...
import { TokenDto } from './dtos/token.dto';
// ...

@ApiTags('Oauth2')
@Controller('api/auth/ext')
@UseGuards(FastifyThrottlerGuard)
export class Oauth2Controller {
  // ...

  @Post('token')
  @ApiResponse({
    description: "Returns the user's OAuth 2 response",
    status: HttpStatus.OK,
  })
  @ApiUnauthorizedResponse({
    description: 'Code or redirectUri is invalid',
  })
  public async token(
    @CurrentUser() userId: number,
    @Body() tokenDto: TokenDto,
    @Res() res: FastifyReply,
  ): Promise<void> {
    // Check if the URI is correct
    if (tokenDto.redirectUri !== this.url + '/auth/callback') {
      throw new UnauthorizedException();
    }

    // Login the user and return the auth response
    const result = await this.oauth2Service.token(tokenDto.code, userId);
    return res
      .cookie(this.cookieName, result.refreshToken, {
        secure: !this.testing,
        httpOnly: true,
        signed: true,
        path: this.cookiePath,
        expires: new Date(Date.now() + this.refreshTime * 1000),
      })
      .header('Content-Type', 'application/json')
      .status(HttpStatus.OK)
      .send(AuthResponseMapper.map(result));
  }

  // ...
}

NOTE: if you really want to be strict you could also enforce the content-type to be application/x-www-form-urlencoded. Personally, I think allowing json as well makes the API friendlier.

Conclusion

This is the true end of my OAuth 2.0 series with NestJS, with this series you have learnt how to create a production grade OAuth 2.0 microservice using NodeJS that supports both web and mobile apps.

The full source code can be found on this repo.

About the Author

Do not miss out on any of my latest articles – follow me here on dev and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

Rust Keyword Extraction: Creating the YAKE! algorithm from scratch

Afonso Barracha — Sat, 27 Apr 2024 10:04:51 +0000

Disclosure: The material in this Tutorial has not been reviewed, endorsed, or approved of by the Rust Foundation. For more information on the Rust Foundation Trademark Policy, click here.

Series Intro

You know, like many of you, I was blown away by Chat GPT, especially GPT-4, which reignited my love for machine learning after a two-year break. But once I was back, I hit the same issue that made me quit: slow training times.

I tried unsupervised NLP models and keyword extraction, but even those were too slow for a simple VPS. That is when I realised Rust and Actix-Web could be the answer.

Therefore, let's explore Rust for NLP together and dive into the awesome world of text mining and keyword extraction.

What will I cover in this series

I will mainly cover 3 algorithms:

TF-IDF: Term Frequency-Inverse Document Frequency algorithm converts raw text into insightful numerical data.
Co-occurrence Matrix: the Co-occurrence Matrix covers underlying patterns and associations in a language by examining word pairs and building semantic networks in textual data.
RAKE: Rapid Automatic Keyword Extraction Algorithm identifies crucial words and phrases to enhance text summarization, topic modeling, and information retrieval in any text-based project.

Introduction

Although I was not expecting to implement in this series, in this article we will dive into YAKE! (Yet Another Keyword Extraction Algorithm). This powerful unsupervised algorithm lets us extract meaningful keywords without relying on dictionaries or massive datasets, based on the following YAKE! Collection-Independent Automatic Keyword Extractor paper.

NOTE: due to my recent implementation of YAKE! (within the past week of this publication), I am covering this algorithm before RAKE to prioritize discussing its implementation details.

Components

The algorithm consists of six main components:

Text pre-processing: preparing the text for analysis.
Feature extraction: identifying important characteristics of words.
Individual term scoring: assigning a score to each potential keyword.
Candidate keyword list generation: creating a list of potential keywords.
Data deduplication: removing redundant keywords.
Ranking: ordering keywords by their computed scores.

Scoring

Each individual term is ranked based on the following characteristics:

Casing: whether the term has uppercased versions;
Word Positional: the term's position within the document (e.g., appearing at the beginning);
Word Frequency: how often the term appears in the document;
Word Relatedness to Context: how strongly the term is related to surrounding words (i.e., does it frequently appear alongside other significant words?);
Word DifSentence: the number of different sentences in which the term appears.

The individual term score is calculated using this formula:

H = (Relatedness * Position) / (Casing + (Frequency / Relatedness) + (DifSentence / Relatedness))

After calculating the n-gram keyword, its score will be determined as follows:

S(kw) = Π(H) / TF(kw)(1 + Σ(H))

Since higher scores indicate less important keywords, the final keyword score is:

Score = 1 / S(kw)

Implementation

For a more programmatically efficient approach, we will slightly deviate from the order presented in the original paper. Our implementation will involve the following steps:

Text pre-processing and tokenization:
- Prepare the text for analysis (e.g., lowercasing, removing punctuation, etc.);
- Divide the text into individual tokenized sentences.
Candidate selection:
- Candidate creation: for an appropriate n-gram size, create a hashmap to store candidate keywords;
- Deduplication: use a separate hashmap to track unique candidate words, preventing redundancy.
Context building:
- Occurrence tracking: create a hashmap to store stemmed word forms, along with their casing information and locations (sentences) within the text;
- Left/Right context: employ a window size and a sliding window approach to build a hashmap that captures stemmed words and their left and right neighboring words.
Feature extraction and individual term scoring:
- Calculate the relevant features (Casing, Word Position, Word Frequency, Word Relatedness to Context, Word DifSentence) for each candidate word;
- Apply the provided formula to compute individual word scores.
Keyword scoring:
- Utilize the calculated individual word scores and provided formulas to determine the final score for each keyword.
Ranking with similar candidates elimination:
- We only eliminate similar candidates when getting the N best, as it is an expensive O(log(n) * n) operation.

Set-up

Create a library lib.rs if you don't have one, and create a folder named yake with a mod.rs file inside.

And add the following two crates:

$ cargo add unicode-segmentation
$ cargo add regex

Inside of the module file add the following scaffold:

use std::collections::HashMap;

pub struct Yake {
    keyword_rank: HashMap<String, f32>,
    term_rank: HashMap<String, f32>,
    size: usize,
    threshold: f32,
}

Text pre-processing and tokenization

Create two files, one for pre-processing and one for building sentences (do not forget to add them to the mod file).

text_pre_processor.rs:

use std::borrow::Cow;

use regex::Regex;

// Create a regex for removing special white spaces (tabs, new lines, etc)
fn get_space_regex() -> Option<Regex> {
    Regex::new(r"[\n\t\r]").ok()
}

// Export a struct to encapsulate the logic
pub struct TextPreProcessor;

impl<'a> TextPreProcessor {
    pub fn process_text(text: &'a str) -> Cow<'a, str> {
        let space_regex = get_space_regex();
        // 1. Trim whitespaces from both sides of the text
        let trimmed_text = text.trim();

        if let Some(regex) = space_regex {
            // 2. Remove special whitespaces
            regex.replace_all(trimmed_text, " ")
        } else {
            // This should never happen but we do not want the system to panic
            trimmed_text.into()
        }
    }
}

sentences_builder.rs:

use std::borrow::Cow;

use regex::Regex;
use unicode_segmentation::UnicodeSegmentation;

fn get_special_char_regex() -> Option<Regex> {
    Regex::new(r"('s|,|\.|\s)").ok()
}

// The struct for a tokenized sentence
pub struct Sentence<'a> {
    // Cased words
    pub words: Vec<Cow<'a, str>>,
    // Lowercased words
    pub stemmed: Vec<String>,
    // Length of the sentence
    pub length: usize,
}

impl<'a> Sentence<'a> {
    pub fn new(s: &'a str, special_char_regex: &Option<Regex>) -> Self {
        let words = s
            .split_word_bounds()
            .filter_map(|w| {
                // Trim the word to remove whitespaces
                let trimmed = w.trim();

                if trimmed.is_empty() {
                    return None;
                }

                // Remove special characters from strings
                if let Some(regex) = special_char_regex {
                    let value = regex.replace_all(trimmed, "");

                    if value.is_empty() {
                        return None;
                    }

                    Some(value)
                } else {
                    Some(trimmed.into())
                }
            })
            .collect::<Vec<Cow<'a, str>>>();
    Self {
            stemmed: words
                .iter()
                .map(|w| w.to_lowercase())
                .collect::<Vec<String>>(),
            length: words.len(),
            words,
        }
    }
}

// Struct to encapsolate the logic
pub struct SentencesBuilder;

impl<'a> SentencesBuilder {
    // Split the pre-processed text into tokenized sentences
    pub fn build_sentences(text: &'a str) -> Vec<Sentence<'a>> {
        let special_char_regex = get_special_char_regex();
        // Use unicode sentences to get the text individual sentences
        text.unicode_sentences()
            .map(|s| Sentence::new(s.trim(), &special_char_regex))
            .collect()
    }
}

Candidate selection

Since candidate selection and context building depend on a loop through all the sentences, create a file for all its logic candidate_selection_and_context_builder.rs, which will have the logic for:

Candidates selection;
Deduplication map creation.

Plus the context builder logic as well.

Candidate Selection

Start by creating the Candidate struct with both:

lexical_form (stemmed form) field;
surface_forms the way the candidate appears in the text field.

pub struct Candidate<'a> {
    // Stemmed form
    pub lexical_form: Vec<&'a str>,
    // All the ways that stemmed form appears in the text
    pub surface_forms: Vec<Vec<&'a str>>,
}

impl<'a> Candidate<'a> {
    // A helper to create a new candidate
    pub fn new(stems: Vec<&'a str>) -> Self {
        Self {
            lexical_form: stems,
            surface_forms: Vec::new(),
        }
    }

    // A helper to mutate the candidate variations
    fn add(&mut self, words: Vec<&'a str>) {
        self.surface_forms.push(words);
    }
}

type Candidates<'a> = HashMap<String, Candidate<'a>>;

To make a candidate we need to have take every right-to-left combination of n-gram values within each sentence. Hence the code needs to do the following:

Find combinations of words within each sentence: we look at different n-gram groupings of words to see if they could be meaningful keywords;
Filters out irrelevant candidates: We skip candidates with things like punctuation, common words ("the", "and", etc.), and numbers to focus on the most important terms;
Tracks unique candidates: Ensures we only keep one version of a potential keyword, even if it appears in slightly different forms (each new form is added to the Candidate.surface_forms).

use std::{
    cmp::min,
    collections::{HashMap, HashSet, VecDeque},
};

use unicode_segmentation::UnicodeSegmentation;

use super::sentences_builder::Sentence;

// ...

fn is_punctuation(word: &str, punctuation: &HashSet<&str>) -> bool {
    word.is_empty() || ((word.graphemes(true).count() == 1) && punctuation.contains(word))
}

fn is_invalid_word(word: &str, punctuation: &HashSet<&str>, stop_words: &HashSet<&str>) -> bool {
    is_punctuation(word, punctuation) || stop_words.contains(word) || word.parse::<f32>().is_ok()
}

pub struct CandidateSelectionAndContextBuilder;

impl<'a> CandidateSelectionAndContextBuilder {
    pub fn select_candidates_and_build_context(
        sentences: &'a [Sentence],
        ngram: usize,
        window_size: usize,
        stop_words: HashSet<&'a str>,
        punctuation: HashSet<&'a str>,
    ) -> (
        // Candidate Selection
        Candidates<'a>,
        // ...
    ) {
        // Iterate through every sentence
        sentences.iter().enumerate().fold(
            (
                Candidates::new(),
                // ...
            ),
            |(mut candidates), (i, sentence)| {
                sentence.words.iter().enumerate().fold(
                    VecDeque::<(&str, usize)>::with_capacity(window_size + 1),
                    |mut buffer, (j, w1)| {
                    // Iterate through each n-gram combination
                    // Ensure n-grams don't extend beyond the end of the sentence
                    (j + 1..=min(j + ngram, sentence.length)).for_each(|k: usize| {
                        // Take the stems for the candidate, this will later for the unique key
                        let stems = sentence.stemmed[j..k]
                            .iter()
                            .map(|s| s.as_str())
                            .collect::<Vec<&'a str>>();
                        // Filter out punctuation, stop words and numbers
                        if stems
                            .iter()
                            .any(|w| is_invalid_word(&w, &punctuation, &stop_words))
                        {
                            return;
                        }

                        let words = sentence.words[j..k]
                            .iter()
                            .map(|s| s.as_ref())
                            .collect::<Vec<&'a str>>();
                        // Join the stems into a unique candidate key
                        let key = stems.join(" ");
                        let entry = match candidates.get_mut(&key) {
                            Some(entry) => entry,
                            None => {
                                // ...
                                candidates.entry(key).or_insert(Candidate::new(stems))
                            }
                        };
                        entry.add(words);
                    });
                });
            (candidates)
        })
    }
}

Explanation: We do 3 iterations:

Iterate through every sentence;
Iterate through every word in a sentence;
Iterate through the next n-gram - 1 words after the current word.

Deduplication map creation

Since our algorithm prioritizes combinations of words (n-grams) over single words, we need a way to track how often individual words appear within important multi-word keywords. This is where the deduplication map comes in.

Add a new HashMap to select_candidates_and_build_context:

// ...
type DedupMap<'a> = HashMap<&'a str, f32>;

// ...

impl<'a> CandidateSelectionAndContextBuilder {
    pub fn select_candidates_and_build_context(
        // ...
    ) -> (
        // Candidate Selection
        Candidates<'a>,
        DedupMap<'a>,
        // ...
    ) {
        // Iterate through every sentence
        sentences.iter().enumerate().fold(
            (
                Candidates::new(),
                DedupMap::new(),
                // ...
            ),
            |(mut candidates, mut dedup_map), (i, sentence)|  {
                // ... 
            (candidates, dedup_map)
        })
    }
}

And add the stems words into the map if they have a length greater than 1:

// ...

type DedupMap<'a> = HashMap<&'a str, f32>;

// ...

impl<'a> CandidateSelectionAndContextBuilder {
    pub fn select_candidates_and_build_context(
        // ...
    ) -> (
        // Candidate Selection
        Candidates<'a>,
        DedupMap<'a>,
        // ...
    ) {
        // Iterate through every sentence
        sentences.iter().enumerate().fold(
            (
                Candidates::new(),
                DedupMap::new(),
                // ...
            ),
            |(mut candidates, mut dedup_map), (i, sentence)|  {
                sentence.words.iter().enumerate().fold(
                    VecDeque::<(&str, usize)>::with_capacity(window_size + 1),
                    |mut buffer, (j, w1)| {
                        // ...

                        let entry = match candidates.get_mut(&key) {
                            Some(entry) => entry,
                            None => {
                                // Check if stems has multiple words
                                if stems.len() > 1 {
                                    // Iterate through every word 
                                    // and add them or increment it
                                    stems.iter().for_each(|w| {
                                        let entry = dedup_map.entry(*w).or_insert(0.0);
                                        *entry += 1.0;
                                    });
                                }

                                candidates.entry(key).or_insert(Candidate::new(stems))
                            }
                        };
                        entry.add(words);
                    });
                });
            (candidates, dedup_map)
        })
    }
}

Explanation: Each time we find a potential keyword with multiple words, we increment the count of its individual parts in the dedup_map.

Context building

Building the term context is divided in two steps:

Left/Right context building;
Occurrences tracking.

Left/Right context building

To understand the relationships between words, we need to identify which words frequently appear to the left and right of other important terms. This is where the LeftRightContext map comes in.

Sliding Window: To build these contexts, we will use a sliding window technique. Imagine it as a queue that moves along the sentence, capturing a fixed size (window_size) of words at a time.

Add a new HashMap that tracks the left and right neighbours of a term:

// ...
pub type LeftRightContext<'a> = HashMap<&'a str, (Vec<&'a str>, Vec<&'a str>)>;

// ...

impl<'a> CandidateSelectionAndContextBuilder {
    pub fn select_candidates_and_build_context(
        // ...
    ) -> (
        // Candidate Selection
        Candidates<'a>,
        DedupMap<'a>,
        // Context Builder
        // ...
        LeftRightContext<'a>,
    ) {
        // Iterate through every sentence
        sentences.iter().enumerate().fold(
            (
                Candidates::new(),
                DedupMap::new(),
                // ...
                LeftRightContext::new(),
            ),
            |(mut candidates, mut dedup_map, mut lr_contexts), (i, sentence)|  {
                // ... 
            (candidates, dedup_map, lr_contexts)
        })
    }
}

Create a buffer that tracks the current words for a given window size:

// ...

impl<'a> CandidateSelectionAndContextBuilder {
    pub fn select_candidates_and_build_context(
        // ...
    ) -> (
        // Candidate Selection
        Candidates<'a>,
        DedupMap<'a>,
        // Context Builder
        // ...
        LeftRightContext<'a>,
    ) {
        sentences.iter().enumerate().fold(
            (
                Candidates::new(),
                DedupMap::new(),
                // ...
                LeftRightContext::new(),
            ),
            |(mut candidates, mut dedup_map, mut occurrences, mut lr_contexts), (i, sentence)| {
                sentence.words.iter().enumerate().fold(
                    VecDeque::<(&str, usize)>::with_capacity(window_size + 1),
                    |mut buffer, (j, w1)| {
                        // ...

                        // Context Building
                        let key1 = sentence.stemmed[j].as_str();
                        let w1_str = w1.as_ref();

                        // ...

                        // map through the sliding window
                        buffer.iter().for_each(|(w2, k)| {
                            // add left context
                            let entry_1 =
                                lr_contexts.entry(key1).or_insert((Vec::new(), Vec::new()));
                            entry_1.0.push(*w2);
                            // add right context
                            let entry_2 = lr_contexts
                                .entry(sentence.stemmed[*k].as_str())
                                .or_insert((Vec::new(), Vec::new()));
                            entry_2.1.push(w1_str);
                        });

                        buffer.push_back((w1_str, j));

                        // Keep the slidding window at the window size
                        if buffer.len() > window_size {
                            buffer.pop_front();
                        }

                        buffer
                    },
                );
                (candidates, dedup_map, lr_contexts)
            },
        )
    }
}

Explanation: Each time the window moves, we record the terms appearing to the left and right of the central term.

Occurrences tracking

To understand the significance of a term, we need to know in which sentences it appears and the different forms it takes (e.g., capitalized, lowercase). This is where the Occurrences map plays a role.

Create a new HashMap that tracks the term format and its sentence index:

// ...
pub type Occurrences<'a> = HashMap<&'a str, Vec<(&'a str, usize)>>;

// ...

impl<'a> CandidateSelectionAndContextBuilder {
    pub fn select_candidates_and_build_context(
        // ...
    ) -> (
        // Candidate Selection
        Candidates<'a>,
        DedupMap<'a>,
        // Context Builder
        Occurrences<'a>,
        LeftRightContext<'a>,
    ) {
        sentences.iter().enumerate().fold(
            (
                Candidates::new(),
                DedupMap::new(),
                Occurrences::new(),
                LeftRightContext::new(),
            ),
            |(mut candidates, mut dedup_map, mut occurrences, mut lr_contexts), (i, sentence)| {
                // ...

                (candidates, dedup_map, occurrences, lr_contexts)
            },
        )
    }
}

Filter out invalid words before adding terms to the occurrences:

// ...
pub type Occurrences<'a> = HashMap<&'a str, Vec<(&'a str, usize)>>;

// ...

impl<'a> CandidateSelectionAndContextBuilder {
    pub fn select_candidates_and_build_context(
        // ...
    ) -> (
        // Candidate Selection
        Candidates<'a>,
        DedupMap<'a>,
        // Context Builder
        Occurrences<'a>,
        LeftRightContext<'a>,
    ) {
        sentences.iter().enumerate().fold(
            (
                Candidates::new(),
                DedupMap::new(),
                Occurrences::new(),
                LeftRightContext::new(),
            ),
            |(mut candidates, mut dedup_map, mut occurrences, mut lr_contexts), (i, sentence)| {
                sentence.words.iter().enumerate().fold(
                    VecDeque::<(&str, usize)>::with_capacity(window_size + 1),
                    |mut buffer, (j, w1)| {

                        // Context Building
                        let key1 = sentence.stemmed[j].as_str();
                        let w1_str = w1.as_ref();

                        // Remove invalid words
                        if !is_invalid_word(key1, &punctuation, &stop_words) {
                            // Add all valid occurrences
                            let entry = occurrences.entry(key1).or_default();
                            entry.push((w1_str, i));
                        }

                        // ...
                    },
                );
                (candidates, dedup_map, occurrences, lr_contexts)
            },
        )
    }
}

Explanation: Each time we encounter a valid term, we add it to the Occurrences map along with the specific form it has in the sentence and the sentence index.

Feature extraction and individual term scoring

Feature extraction and individual term scoring, as the name entails is composed of two steps, that follow the following formulas.

Feature extraction:
- Explanation:
  - Casing: does the term appear capitalized or in uppercase?
  - Frequency: how often the normalized term occurs in the text.
  - Position: where the term tends to appear within sentences, earlier appearances are more important.
  - Relatedness: how strongly the term is connected to other significant words in the text. Does the term act as a stop word?
  - Difference: does the term appear in a variety of sentences?
- Formulas:
  - Casing: MAX(TF_U, TF_C) / (1 + log(TF))
    - TF_U: the uppercased term frequency;
    - TF_C: the capitalized term frequency;
    - TF: term frequency.
  - Frequency: TF / (MEAN(TOTAL_TF) + STD(TOTAL_TF))
    - TOTAL_TF: the term frequency of all words.
  - Position: LOG(3 + MEDIAN(IDX))
    - IDX: sentences indexes
  - Relatedness: 1 + (((WDL / WIL) + (WDR / WIR)) * (TF / MAX(TF_TOTAL))
    - WDL: word degree information left, or unique terms directly neighboring the central term on the left;
    - WIL: word interaction strengh left, sum of edges neighboring the central term on the left;
    - WDR: same as WDL but for the right side;
    - WIR: same as WIL but for the right side.
  - Difference: UNIQUE(IDX) / LEN(IDX)
Individual term scoring:
- Explanation: we combine the individual scores into a single score that determines the term inverse importance.
- Formula: (Relatedness * Position) / (Casing + (Frequency / relatedness) + (Difference / Relatedness))

These formulas are derived from the original YAKE! python implementation found here.

Set up

Create a new file named feature_extraction.rs and add a struct to encapsulate this step logic (score_words):

use std::collections::{HashMap, HashSet};

use unicode_segmentation::UnicodeSegmentation;

use super::candidate_selection_and_context_builder::{LeftRightContext, Occurrences};

pub struct FeatureExtractor;

impl<'a> FeatureExtractor {
    pub fn score_words(
        occurrences: Occurrences<'a>,
        contexts: LeftRightContext<'a>,
        sentences_len: f32,
    ) -> HashMap<&'a str, f32> {
        // ...
    }
}

Before scoring each score, we need to calculate the TOTAL_TF, its max, mean and standard deviation, as it depends on all occurrences:

// ...

impl<'a> FeatureExtractor {
    pub fn score_words(
        occurrences: Occurrences<'a>,
        contexts: LeftRightContext<'a>,
        sentences_len: f32,
    ) -> HashMap<&'a str, f32> {
        // Calculate aggregate statistics across all occurrences
        // Calculate the total and max term frequency
        let (tf_total, tf_max_u) = occurrences.values().fold((0, 0), |(tf_total, tf_max), v| {
            let length = v.len();
            (tf_total + length, tf_max.max(length))
        });
        let tf_max = tf_max_u as f32;

        // Calculate the mean of term frequencies
        let tf_mean = tf_total as f32 / (occurrences.len() as f32 + f32::EPSILON);

        // Calculate standard deviation of term frequencies
        let tf_std = occurrences
            .values()
            .fold(0.0_f32, |a, v| a + (v.len() as f32 - tf_mean).powi(2))
            .sqrt();

        // ...
    }
}

Feature extraction

Casing:

impl<'a> FeatureExtractor {
    pub fn score_words(
        // ...
    ) -> HashMap<&'a str, f32> {
        // ...

        occurrences
            .into_iter()
            .map(|(word, occurrence)| {
                // The term frequency is the number of its occurrences
                let tf = occurrence.len() as f32;

                let (tf_upper, tf_capitalized) = occurrence.iter().fold(
                    (0.0_f32, 0.0_f32),
                    |(tf_upper, tf_capitalized), (w, _)| {
                        (
                            // Check if word is all uppercased and has a length greater than 1
                            tf_upper
                                + if w.graphemes(true).count() > 1
                                    && &w.to_uppercase().as_str() == w
                                {
                                    1.0
                                } else {
                                    0.0
                                },
                             // Check if word starts with an uppercase character and contains any lowercased character
                            tf_capitalized
                                + if w.chars().next().unwrap_or(' ').is_uppercase()
                                    && (w.graphemes(true).count() == 1
                                        || w.chars().skip(1).any(|c| c.is_lowercase()))
                                {
                                    1.0
                                } else {
                                    0.0
                                },
                        )
                    },
                );

                // Apply the `MAX(TF_U, TF_C) / (1 + log(TF))` formula
                let casing = tf_upper.max(tf_capitalized) / (1.0 + tf.ln());
                // ...
            })
            .collect()
    }
}

Frequency:

impl<'a> FeatureExtractor {
    pub fn score_words(
        // ...
    ) -> HashMap<&'a str, f32> {
        // ...

        occurrences
            .into_iter()
            .map(|(word, occurrence)| {
                // The term frequency is the number of its occurrences
                let tf = occurrence.len() as f32;

                // ...

                // Apply the `TF / (MEAN(TOTAL_TF) + STD(TOTAL_TF))` formula
                let frequency = tf / (tf_mean + tf_std + f32::EPSILON);

                // ...
            })
            .collect()
    }
}

Position:

impl<'a> FeatureExtractor {
    pub fn score_words(
        // ...
    ) -> HashMap<&'a str, f32> {
        // ...

        occurrences
            .into_iter()
            .map(|(word, occurrence)| {
                // ...

                // Take the tf in usize
                let occ_len = occurrence.len();
                // Calculate the median sentence index
                let median = if occ_len == 0 {
                    0.0
                } else if occ_len == 1 {
                    occurrence[0].1 as f32
                } else if occ_len % 2 == 0 {
                    occurrence[occ_len / 2].1 as f32
                } else {
                    let mid = occ_len / 2;
                    (occurrence[mid].1 + occurrence[mid - 1].1) as f32 / 2.0
                };

                // Apply the `LOG(3 + MEDIAN(IDX))` formula
                let position = (3.0 + median).ln().ln();

                // ...
            })
            .collect()
    }
}

Relatedness:

impl<'a> FeatureExtractor {
    pub fn score_words(
        // ...
    ) -> HashMap<&'a str, f32> {
        // ...

        occurrences
            .into_iter()
            .map(|(word, occurrence)| {
                // ...

                // Get the context
                let left_right_context = contexts
                    .get(word)
                    .map(|(v1, v2)| (v1.as_slice(), v2.as_slice()))
                    .unwrap_or((&[], &[]));

                // Calculate the `WDL` by creating a hashset of the left neighbours
                let left_context_unique = left_right_context
                    .0
                    .iter()
                    .copied()
                    .collect::<HashSet<&str>>();

                // Calculate `(WDL / WIL)`
                let wl = if !left_context_unique.is_empty() {
                    left_context_unique.len() as f32
                        / (left_right_context.0.len() as f32 + f32::EPSILON)
                } else {
                    0.0_f32
                };

                // Calculate the `WDR` by creating a hashset of the rigth neighbours
                let right_context_unique = left_right_context
                    .1
                    .iter()
                    .copied()
                    .collect::<HashSet<&str>>();

                // Calculate `(WDL / WIL)`
                let wr = if !right_context_unique.is_empty() {
                    right_context_unique.len() as f32
                        / (left_right_context.1.len() as f32 + f32::EPSILON)
                } else {
                    0.0_f32
                };

                // Apply the `1 + (((WDL / WIL) + (WDR / WIR)) * (TF / MAX(TF_TOTAL))` formula
                let relatedness = 1.0 + ((wl + wr) * (tf / tf_max));

                // ...
            })
            .collect()
    }
}

Difference:

impl<'a> FeatureExtractor {
    pub fn score_words(
        // ...
    ) -> HashMap<&'a str, f32> {
        // ...

        occurrences
            .into_iter()
            .map(|(word, occurrence)| {
                // ...

                // Calculate the unique sentences by creating a hashmap
                let unique_sentences = occurrence
                    .iter()
                    .map(|occ| occ.1)
                    .collect::<HashSet<usize>>();
                // Apply the `UNIQUE(IDX) / LEN(IDX)` formula
                let different = unique_sentences.len() as f32 / (sentences_len + f32::EPSILON);

                // ...
            })
            .collect()
    }
}

Individual term scoring

For this step we just need to calculate the inverse score by using all the previous calculated features in the previous formula: (Relatedness * Position) / (Casing + (Frequency / relatedness) + (Difference / Relatedness)).

impl<'a> FeatureExtractor {
    pub fn score_words(
        // ...
    ) -> HashMap<&'a str, f32> {
        // ...

        occurrences
            .into_iter()
            .map(|(word, occurrence)| {
                // ...

                (
                    word,
                    (relatedness * position)
                        / (casing + (frequency / relatedness) + (different / relatedness)),
                )
            })
            .collect()
    }
}

Keyword scoring

After calculating scores for individual terms, we combine them to determine the importance of multi-word keywords (n-grams). YAKE! uses the following formula to calculate a keyword's "inverse score":

S(kw) = Π(H) / TF(kw)(1 + Σ(H))

Where the equation is composed by:

S(kw): the keyword's inverse score (lower scores indicate more important keywords);
Π(H): the product of individual term scores within the keyword;
TF(kw): the total frequency of the keyword in the text;
Σ(H): the sum of individual term scores within the keyword.

We will also score the indevidual terms just by inversing the inverse score:

W = 1 / H

This section will be divided in three steps:

Keyword scoring;
Term scoring;
Building YAKE!.

Set up

Create a new file named yake_logic.rs and add a struct to encapsulate all functions necessary to create Yake:

YakeLogic struct and build_yake:

use std::collections::{HashMap, HashSet};

use super::{
    candidate_selection_and_context_builder::{Candidate, CandidateSelectionAndContextBuilder},
    feature_extraction::FeatureExtractor,
    sentences_builder::SentencesBuilder,
    text_pre_processor::TextPreProcessor,
};

pub struct YakeLogic;

impl YakeLogic {
    pub fn build_yake(
        text: &str,
        stop_words: HashSet<&str>,
        punctuation: HashSet<&str>,
        ngram: usize,
        window_size: usize,
    ) -> (HashMap<String, f32>, HashMap<String, f32>) {
        // ... 
    }

    // ...
}

score_candidates:

impl YakeLogic {
    // ...

    fn score_candidates<'a>(
        candidates: HashMap<String, Candidate<'a>>,
        dedup_hashmap: HashMap<&'a str, f32>,
        word_scores: &HashMap<&'a str, f32>,
    ) -> HashMap<String, f32> {
        // ...
    }

    // ...
}

score_terms:

impl YakeLogic {
    // ...

    fn score_terms(word_scores: HashMap<&str, f32>) -> HashMap<String, f32> {
        // ...
    }
}

Keyword scoring

Scoring the keyword comprises of 3 steps:

Iterate throuch each candidate;
Calculate the score, and inverse it;
Normalize the score by the max score.

impl YakeLogic {
    // ...

    fn score_candidates<'a>(
        candidates: HashMap<String, Candidate<'a>>,
        dedup_hashmap: HashMap<&'a str, f32>,
        word_scores: &HashMap<&'a str, f32>,
    ) -> HashMap<String, f32> {
        let candidates_len = candidates.len();
        // Iterate throuch each candidate
        let (vec_scores, max) = candidates.into_iter().fold(
            (
                Vec::<(String, f32)>::with_capacity(candidates_len),
                f32::EPSILON,
            ),
            |(mut acc, max), (k, pc)| {
                // Calculate the product and sum
                let (prod, sum) = pc.lexical_form.iter().fold(
                    // Penalize single words that are used in other keywords
                    (*dedup_hashmap.get(k.as_str()).unwrap_or(&1.0), 0.0),
                    |acc, w| {
                        let weight = *word_scores.get(*w).unwrap_or(&0.0);

                        (acc.0 * weight, acc.1 + weight)
                    },
                );
                let tf = pc.surface_forms.len() as f32;
                let sum = if sum == -1.0 { 1.0 - f32::EPSILON } else { sum };
                // Apply the `S(kw) = Π(H) / TF(kw)(1 + Σ(H))` formula
                let score = prod / (tf * (1.0 + sum));

                // Since the less is better, inverse the score
                let inverse_score = 1.0 / score;
                acc.push((k, inverse_score));
                (acc, max.max(inverse_score))
            },
        );

        // Normalize the scores
        vec_scores
            .into_iter()
            .map(|(k, v)| (k, v / max))
            .collect::<HashMap<String, f32>>()
    }

    // ...
}

Term scoring

Like the "Keyword scoring", term scoring comprises of 3 steps:

Iterate throuch each term;
Inverse the inverse term score;
Normalize the score by the max score.

impl YakeLogic {
    // ...

    fn score_terms(word_scores: HashMap<&str, f32>) -> HashMap<String, f32> {
        let word_scores_len = word_scores.len();
        // Iterate throuch each term
        let (vec_scores, max) = word_scores.into_iter().fold(
            (
                Vec::<(String, f32)>::with_capacity(word_scores_len),
                f32::EPSILON,
            ),
            |(mut acc, max), (k, score)| {
                // Inverse the score
                let inverse_score = 1.0 / score;
                acc.push((k.to_string(), inverse_score));
                (acc, max.max(inverse_score))
            },
        );

        // Normalize the scores
        vec_scores
            .into_iter()
            .map(|(k, v)| (k, v / max))
            .collect::<HashMap<String, f32>>()
    }
}

Building yake

Putting the previous sections together, we can just build the algorithm model:

impl YakeLogic {
    pub fn build_yake(
        text: &str,
        stop_words: HashSet<&str>,
        punctuation: HashSet<&str>,
        ngram: usize,
        window_size: usize,
    ) -> (HashMap<String, f32>, HashMap<String, f32>) {
        // Preprocess the input text
        let text = TextPreProcessor::process_text(text);

        // Split the text into sentences
        let sentences = SentencesBuilder::build_sentences(&text);

        // Identify keywords and build terms context
        let (candidates, dedup_hashmap, occurrences, lr_contexts) =
            CandidateSelectionAndContextBuilder::select_candidates_and_build_context(
                &sentences,
                ngram,
                window_size,
                stop_words,
                punctuation,
            );

        // Calculate feature scores for individual terms
        let word_scores =
            FeatureExtractor::score_words(occurrences, lr_contexts, sentences.len() as f32);

        // Score keywords and individual terms
        (
            Self::score_candidates(candidates, dedup_hashmap, &word_scores),
            Self::score_terms(word_scores),
        )
    }

    // ...
}

The build_yake function would be represent by this flow:

Ranking with similar candidates elimination

By reading the paper we see that:

In the fifth step, we eliminate similar candidates coming from the previous steps. For this, we use the Levenshtein distance

Hence we need to implement the Levenshtein distance algorithm before ranking the keywords.

Levenshtein distance

The Levenshtein distance algorithm measures the "edit distance" between two words. In more technical terms, it calculates the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one string into another.

use std::cmp::{max, min};

use unicode_segmentation::UnicodeSegmentation;

fn calculate_distance(str1: &str, str2: &str) -> usize {
    // Return early with minimal computation
    if (str1.is_empty() && str2.is_empty()) || str1 == str2 {
        return 0;
    }

    // Split string slices into user-perceived characters
    let graphemes1 = str1.graphemes(true);
    let graphemes2 = str2.graphemes(true);

    // Create a table to store transformations
    let len = graphemes2.clone().count() + 1;
    let mut prev_row = (0..len).collect::<Vec<usize>>();

    let last_row = graphemes1
        .enumerate()
        .fold(prev_row.clone(), |row, (i, char1)| {
            let mut new_row = vec![i + 1; len];
            graphemes2.clone().enumerate().for_each(|(j, char2)| {
                // Calculating the cost of converting substrings
                // The cost is 0 if the characters are the same, otherwise 1 (for a substitution)
                let cost = if char1 == char2 { 0 } else { 1 };
                new_row[j + 1] = min(row[j + 1] + 1, min(new_row[j] + 1, row[j] + cost));
            });
            prev_row = row;
            new_row
        });

    last_row[len - 1]
}

pub struct Levenshtein<'a>(&'a str, &'a str, usize);

impl<'a> Levenshtein<'a> {
    pub fn new(str1: &'a str, str2: &'a str) -> Self {
        Self(str1, str2, calculate_distance(str1, str2))
    }

    pub fn ratio(&self) -> f32 {
        let max_len = max(
            self.0.graphemes(true).count(),
            self.1.graphemes(true).count(),
        );
        // Calculate the percentage of change between the two string slices
        1.0 - (self.2 as f32 / max_len as f32)
    }
}

Ranking keywords/terms

Ranking keywords is composed of two steps:

Sorting the keywords/terms maps;
Building the ranking vector by removing similar candidates.

On the mod.rs file add the following functions:

sort_ranked_map:

fn sort_ranked_map<'a>(map: &'a HashMap<String, f32, RandomState>) -> Vec<(&'a String, &'a f32)> {
    let mut map_values = map.iter().collect::<Vec<(&'a String, &'a f32)>>();
    map_values.sort_by(|a, b| {
        let order = b.1.partial_cmp(a.1).unwrap_or(Ordering::Equal);

        if order == Ordering::Equal {
            return a.0.cmp(b.0);
        }

        order
    });
    map_values
}

build_ranked_scores:

fn build_ranked_scores(vec: &mut Vec<(String, f32)>, word: &str, score: f32, threshold: f32) {
    if vec
        .iter()
        .any(|(w, _)| Levenshtein::new(w, word).ratio() >= threshold)
    {
        return;
    }
    vec.push((word.to_string(), score));
}

Putting all together

Implement a new function on the Yake struct:

// ...
mod levenshtein;
// ...

pub struct Yake {
    keyword_rank: HashMap<String, f32>,
    term_rank: HashMap<String, f32>,
    size: usize,
    threshold: f32,
}

impl Yake {
    pub fn new(
        text: &str, 
        stop_words: &'a [String],
        puctuation: &'a [String],
        threshold: f32,
        ngram: usize,
        window_size: usize,
    ) -> Self {
        let (keyword_rank, term_rank) = YakeLogic::build_yake(
            text,
            stop_words.iter().map(|s| s.as_str()).collect(),
            match puctuation {
                Some(p) => p.iter().map(|s| s.as_str()).collect(),
                None => PUNCTUATION.iter().copied().collect(),
            },
            ngram,
            window_size,
        );
        Self {
            size: keyword_rank.len(),
            keyword_rank,
            term_rank,
            threshold,
        }
    }

    // ...
}

And add a function to get the ranked keyword scores:

impl Yake {
    // ...

    pub fn get_ranked_keyword_scores(&self, n: usize) -> Vec<(String, f32)> {
        // Calculate the capacity
        let capacity = min(self.size, n);
        let result = sort_ranked_map(&self.keyword_rank).into_iter().try_fold(
            Vec::<(String, f32)>::with_capacity(capacity),
            |mut acc, (word, score)| {
                // Short-circuit and exit if Vec reached capacity
                if acc.len() == capacity {
                    return Err(acc);
                }
                build_ranked_scores(&mut acc, word, *score, self.threshold);
                Ok(acc)
            },
        );

        match result {
            Ok(v) => v,
            Err(v) => v,
        }
    }
}

Conclusion

In this article we explored the YAKE! algorithm, a versatile and efficient unsupervised keyword extraction technique. We delved into its implementation, covering the following steps:

Pre-processing: preparing the text for analysis;
Candidate selection and context building: identifying potential keywords and their relationships;
Feature extraction and scoring: calculating features to determine word and keyword importance;
Keyword ranking and deduplication: sorting keywords by their calculated scores and eliminating redundant candidates.

All the code discussed in this article can be accessed through this repository. For integration with existing projects consider using keyword_extraction crate available on crates.io.

About the Author

Hey there! I am Afonso Barracha, a back-end developer with a soft spot for GraphQL. If you enjoyed reading this article, why not show some love by buying me a coffee?

Do not miss out on any of my latest articles – follow me here on dev and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

Rust Keyword Extraction: Creating the Co-Occurrence Matrix algorithm from scratch

Afonso Barracha — Sat, 17 Feb 2024 03:21:28 +0000

Disclosure: The material in this Tutorial has not been reviewed, endorsed, or approved of by the Rust Foundation. For more information on the Rust Foundation Trademark Policy, click here.

Series Intro

I tried unsupervised NLP models and keyword extraction, but even those were too slow for a simple VPS. That is when I realised Rust and Actix-Web could be the answer.

Therefore, let's explore Rust for NLP together and dive into the awesome world of text mining and keyword extraction.

What will I cover in this series

I will mainly cover 3 algorithms:

TF-IDF: Term Frequency-Inverse Document Frequency algorithm converts raw text into insightful numerical data.
Co-occurrence Matrix: the Co-occurrence Matrix covers underlying patterns and associations in a language by examining word pairs and building semantic networks in textual data.
RAKE: Rapid Automatic Keyword Extraction Algorithm identifies crucial words and phrases to enhance text summarization, topic modeling, and information retrieval in any text-based project.

Introduction

Before we dig into implementing the Co-Occurence Matrix algorithm, it's necessary to grasp its concept. This algorithm generates a matrix detailing the frequency of word pairs occurring together within a specified context size across one or multiple documents.

Steps to implement Co-Occurrence Matrix

Note: if you're short on time and can't read the entire article, all the code is available in this repository. Also, consider buying me a coffee if you find my content helpful.

The Co-Occurrence Matrix algorithm is straightforward to implement to some extent:

Begin by creating a hashmap to index all words;
Calculate the matrix:
- Split each document into vectors of individual words;
- Iterate over each word to compute the context;
- Increment the cor-occurrence count for each word index;
- Normalize the matrix by dividing each value by the maximum value found in the matrix.

These steps outline a basic approach to constructing a Co-Occurrence Matrix, an invaluable tool in text analysis and natural language processing for understanding the relationships between words in a corpus.

Implementation

Start by creating a file called co_occurence.rs and add the following struct:

use std::collections::HashMap;

pub struct CoOccurrence {
    matrix: Vec<Vec<f32>>,
    words: Vec<String>,
    words_indexes: HashMap<String, usize>,
}

Helper functions

Create a function to create the word index, where it takes a reference to the vector of words, enumerating through the words and assigning its index as the numeric key of the word:

// ...

fn create_words_indexes(words: &[String]) -> HashMap<String, usize> {
    words
        .iter()
        .enumerate()
        .map(|(i, w)| (w.to_string(), i))
        .collect::<HashMap<String, usize>>()
}

Create a function to calculate the context by determining the window range:

use std::{collections::HashMap, ops::Range};
// ...

fn get_window_range(window_size: usize, index: usize, words_length: usize) -> Range<usize> {
    let window_start = index.saturating_sub(window_size);
    let window_end = (index + window_size + 1).min(words_length);
    window_start..window_end
}

Creating the Matrix

To create the matrix, we need the following parameters:

documents: the documents to be analyzed;
word_indexes: the hash map of the word indexes;
length: the number of words passed to the matrix;
window_size: the size of the context.

Create the function to initialize the matrix with zeros::

// ...

fn get_matrix(
    documents: &[String],
    words_indexes: &HashMap<String, usize>,
    length: usize,
    window_size: usize,
) -> Vec<Vec<f32>> {
    let mut matrix = vec![vec![0.0_f32; length]; length];
    // ...
}

Iterate through each document and split it into words:

// ...

fn get_matrix(
    documents: &[String],
    words_indexes: &HashMap<String, usize>,
    length: usize,
    window_size: usize,
) -> Vec<Vec<f32>> {
    // ...


    documents.iter().for_each(|doc| {
        let doc_words = doc.split_whitespace().collect::<Vec<&str>>();
        // ...
    });
}

Note: we separate the document's words into a distinct variable instead of using flat_map directly because we require access to the entire vector of words. This access is essential for calculating the window range (context) around each word.

Enumerate through the words to get the range:

// ...

fn get_matrix(
    documents: &[String],
    words_indexes: &HashMap<String, usize>,
    length: usize,
    window_size: usize,
) -> Vec<Vec<f32>> {
    // ...

    documents.iter().for_each(|doc| {
        // ...

        doc_words
            .iter()
            .enumerate()
            .filter_map(|(i, w)| words_indexes.get(*w).map(|first_index| (i, *first_index)))
            .for_each(|(i, first_index)| {
                get_window_range(window_size, i, doc_words.len());
            });
    });
}

Filter and map the range to get the other word's indexes:

// ...

fn get_matrix(
    documents: &[String],
    words_indexes: &HashMap<String, usize>,
    length: usize,
    window_size: usize,
) -> Vec<Vec<f32>> {
    // ...

    documents.iter().for_each(|doc| {
        // ...

        doc_words
            // ...
            .for_each(|(i, first_index)| {
                get_window_range(window_size, i, doc_words.len())
                    .filter_map(|j| {
                         // If indexes are the same skip
                        if i == j {
                            return None;
                        }

                        doc_words
                             // Tries to get Option<&&str> of the word by index
                            .get(j)
                             // Check if the word is part of the matrix
                            .and_then(|other_word| words_indexes.get(*other_word));
            });
    });
}

For each index calculate the value of the matrix:

fn get_matrix(
    documents: &[String],
    words_indexes: &HashMap<String, usize>,
    length: usize,
    window_size: usize,
) -> Vec<Vec<f32>> {
    // ...

    documents.iter().for_each(|doc| {
        // ...

        doc_words
            // ...
            .for_each(|(i, first_index)| {
                get_window_range(window_size, i, doc_words.len())
                    // ...
                    .for_each(|other_index| {
                        matrix[first_index][*other_index] += 1.0;
                    });
            });
    });
}

We still miss the normalization, add a max variable at the top:

fn get_matrix(
    documents: &[String],
    words_indexes: &HashMap<String, usize>,
    length: usize,
    window_size: usize,
) -> Vec<Vec<f32>> {
    // ...
    let mut max = 0.0_f32;

   // ...
}

Then calculate it on every matrix iteration:

fn get_matrix(
    documents: &[String],
    words_indexes: &HashMap<String, usize>,
    length: usize,
    window_size: usize,
) -> Vec<Vec<f32>> {
    // ...

    documents.iter().for_each(|doc| {
        // ...

        doc_words
            // ...
            .for_each(|(i, first_index)| {
                get_window_range(window_size, i, doc_words.len())
                    // ...
                    .for_each(|other_index| {
                        // ...
                        let current = matrix[first_index][*other_index];

                        if current > max {
                            max = current;
                        }
                    });
            });
    });
}

Normalize the matrix by dividing each value by the maximum value found, and return it:

fn get_matrix(
    documents: &[String],
    words_indexes: &HashMap<String, usize>,
    length: usize,
    window_size: usize,
) -> Vec<Vec<f32>> {
    // ...

    matrix
        .iter_mut()
        .flat_map(|row| row.iter_mut())
        .for_each(|value| *value /= max);

    matrix
}

Putting all together

Finally, implement a new method on the struct using the helper functions:

impl CoOccurrence {
    pub fn new(documents: Documents, words: Words, window_size: WindowSize) -> Self {
        // Create the word indexes
        let words_indexes = create_words_indexes(words);
        let length = words.len();

        Self {
            // Create the matrix
            matrix: get_matrix(documents, &words_indexes, length, window_size),
            words: words.to_vec(),
            words_indexes,
        }
    }
}

And create methods to access the private fields of the struct:

impl CoOccurrence {
    // ...

    /// Get the numeric label of a word.
    pub fn get_label(&self, word: &str) -> Option<usize> {
        self.words_indexes.get(word).map(|w| w.to_owned())
    }

    /// Get the word of a numeric label.
    pub fn get_word(&self, label: usize) -> Option<String> {
        self.words.get(label).map(|w| w.to_owned())
    }

    /// Get the matrix of the co-occurrence.
    pub fn get_matrix(&self) -> &Vec<Vec<f32>> {
        &self.matrix
    }

    /// Get the labels of the co-occurrence.
    pub fn get_labels(&self) -> &HashMap<String, usize> {
        &self.words_indexes
    }

    /// Get all relations of a given word.
    pub fn get_relations(&self, word: &str) -> Option<Vec<(String, f32)>> {
        let label = match self.get_label(word) {
            Some(l) => l,
            None => return None,
        };

         Some(
            self.matrix[label]
                .iter()
                .enumerate()
                .filter_map(|(i, &v)| {
                    if v > 0.0 {
                        if let Some(w) = self.get_word(i) {
                            return Some((w, v));
                        }
                    }

                    None
                })
                .collect::<Vec<(String, f32)>>(),
        )
    }

    /// Get the row of a given word.
    pub fn get_matrix_row(&self, word: &str) -> Option<Vec<f32>> {
        let label = match self.get_label(word) {
            Some(l) => l,
            None => return None,
        };
        Some(self.matrix[label].to_owned())
    }

    /// Get the relation between two words.
    pub fn get_relation(&self, word1: &str, word2: &str) -> Option<f32> {
        let label1 = match self.get_label(word1) {
            Some(l) => l,
            None => return None,
        };
        let label2 = match self.get_label(word2) {
            Some(l) => l,
            None => return None,
        };
        Some(self.matrix[label1][label2])
    }
}

Conclusion

In this article, we've successfully created a basic implementation of the co-occurrence matrix algorithm, a powerful tool especially useful for analyzing the relationship between keywords and ranking words within documents. This algorithm plays a crucial role in developing preference algorithms, offering insights into how words coalesce to convey meaning and context.

You can access all the code discussed in this article through this repository. For integration with existing projects consider using keyword_extraction crate available on crates.io.

About the Author

Hey there! I am Afonso Barracha, a back-end developer with a soft spot for GraphQL. If you enjoyed reading this article, why not show some love by buying me a coffee?

Do not miss out on any of my latest articles – follow me here on dev and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

Rust GraphQL APIs for NodeJS Developers: Introduction

Afonso Barracha — Thu, 08 Feb 2024 10:13:20 +0000

Series Intro

Although I'm a fan of NodeJS, it faces a significant limitation: lack of multi-threading support. Furthermore, TypeScript/JavaScript can be slow and resource-intensive when executing processor-heavy operations.

In my usual NodeJS tech stack, which includes GraphQL, NestJS, SQL (predominantly PostgreSQL with MikroORM), I encountered these limitations. To overcome them, I've developed a new stack utilizing Rust, which still offers some ease of development:

Async-GraphQL: A server-side GraphQL library implemented in Rust. It complies fully with the GraphQL specification and most of its extensions, providing type safety and high performance.
SQL with SeaORM: SQL, or Structured Query Language, is the foundation for popular databases like PostgreSQL. SeaORM is a relational ORM designed to facilitate the building of web services in Rust.
Actix-Web: A powerful, pragmatic, and extremely fast web framework for Rust.
Tokio: An asynchronous runtime for the Rust programming language, enabling efficient and scalable non-blocking I/O operations.

Introduction

In this article, I will explore and compare the NodeJS frameworks I typically use against my experiences with Rust, providing an overview of how to utilize them effectively.

Why Use Rust?

For applications requiring intensive processor tasks such as Cryptography Algorithms, Keyword Extraction Algorithms, or PDF generation, Rust emerges as the ideal language for server development. Its performance and efficiency in handling such tasks are unparalleled.

NestJS vs. Actix-Web

For general I/O-heavy applications, I prefer NestJS, a progressive Node.js framework designed for building efficient, reliable, and scalable server-side applications.

However, it's important to note the limitation of Node.JS: its single-threaded nature. For processor-intensive tasks, this can lead to blocking the thread being used by a user, alongside non-scalable resource usage.

Transitioning to Actix-Web

Actix-web, while not as feature-rich as NestJS, resembles lighter-weight frameworks such as Fastify or Express. This means much of the folder structure and architectural decisions are left to the developer. I recommend adopting a Model-Service-Resolver (MSR) structure for your application, consisting of:

Model: Represents the Data Transfer Objects (DTOs) and GraphQL Objects that expose API data.
Service: Manages data processing for specific models, encapsulating the business logic.
Resolver: Handles GraphQL queries and mutations, interacting with specific models.
Providers: External services utilized by the application, such as PostgreSQL or Redis.

Model

For our models, we utilize SeaORM, which necessitates a Workspace structure. Begin by creating a library named entities:

$ cargo new entities --lib

Then, include it in your main Cargo.toml file:

# ...

[workspace]
members = [".", "entities"]

In the entities library Cargo.toml, add the necessary dependencies:

[dependencies]
serde = { version = "1.0", features = ["derive"] }
chrono = "0.4"
sea-orm = { version = "0.12", features = [
    "sqlx-postgres",
    "runtime-actix-native-tls",
] }

Creating an entity

Follow the SeaORM documentation to create an entity. For example, to define a User entity, create a user.rs file and reference it in your lib.rs:

use chrono;
use sea_orm::QueryOrder;
use sea_orm::{entity::prelude::*, ActiveValue, Condition};

#[derive(Clone, Debug, PartialEq, DeriveEntityModel)]
#[sea_orm(table_name = "users")]
pub struct Model {
    #[sea_orm(primary_key)]
    pub id: i32,
    pub name: String,
    #[graphql(skip)]
    pub date_of_birth: String,
    pub created_at: DateTime,
    pub updated_at: DateTime,
}

// Define a relationship enum if needed (empty for this example)
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {
}

// Implement `ActiveModelBehavior` for auto-updating timestamps
#[async_trait::async_trait]
impl ActiveModelBehavior for ActiveModel {
    async fn before_save<C: ConnectionTrait>(mut self, _: &C, insert: bool) -> Result<Self, DbErr> {
        let current_time = Utc::now().naive_utc();
        self.updated_at = ActiveValue::Set(current_time);
        if insert {
            self.created_at = ActiveValue::Set(current_time);
        }
        Ok(self)
    }
}

This approach is slightly more complex than using MikroORM for entity definition, which would typically look like this in TypeScript:

import { Entity, PrimaryKey, Property } from '@mikro-orm/core';
import { IsString, Length } from 'class-validator';

@Entity({ tableName: 'users' })
export class UserEntity implements IUser {
  @PrimaryKey()
  public id: number;

  @Property({ columnType: 'varchar', length: 100 })
  @IsString()
  @Length(3, 100)
  @Matches(NAME_REGEX, {
    message: 'Name must not have special characters',
  })
  public name: string;

  @Property({ columnType: 'varchar', length: 10 })
  @IsString()
  @Length(10, 10)
  public dateOfBith: string;

  @Property({ onCreate: () => new Date() })
  public createdAt: Date = new Date();

  @Property({ onUpdate: () => new Date() })
  public updatedAt: Date = new Date();
}

Migrations

For migrations, both schema-first and entity-first approaches are viable. Here, we'll focus on entity-first. Begin by installing necessary CLI tools and setting up the migration folder:

$ cargo install sqlx-cli
$ cargo install sea-orm-cli
$ sea-orm-cli migrate init -d migrations

Include the new migrations directory in your workspace in the Cargo.toml file, and add the necessary dependencies:

# ...

[workspace]
members = [".", "entities", "migrations"]

[dependencies]
# ...
anyhow = "1"
actix-web = "4"
async-graphql-actix-web = "7"
async-graphql = { version = "7", features = ["default", "dataloader"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
sea-orm = { version = "0.12", features = [
    "sqlx-postgres",
    "runtime-actix-native-tls",
] }

Then, proceed to create and modify migration files as required, ensuring they follow the structured approach for creating or dropping tables based on your entities:

use sea_orm_migration::{
    prelude::*,
    sea_orm::{DbBackend, Schema},
};

use entities::user::{Column, Entity};

#[derive(DeriveMigrationName)]
pub struct Migration;

#[async_trait::async_trait]
impl MigrationTrait for Migration {
    async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
        let schema = Schema::new(DbBackend::Postgres);
        manager
            .create_table(
                schema
                    .create_table_from_entity(Entity)
                    .if_not_exists()
                    .to_owned(),
            )
            .await
    }

    async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
        manager
            .drop_table(Table::drop().table(Entity).to_owned())
            .await
    }
}

DTOs (Data Transfer Objects)

In async-graphql, DTOs are essential for constructing objects based on entities. Implement the From trait for entity-to-DTO conversions, and define complex GraphQL object fields and resolvers as needed for enhanced API functionality.

use async_graphql::dataloader::DataLoader;
use async_graphql::{ComplexObject, Context, Error, Result, SimpleObject};
use chrono::{NaiveDate, Utc};

use entities::user::Model;

#[derive(SimpleObject, Debug, Clone)]
#[graphql(complex)]
pub struct User {
    pub id: i32,
    pub name: String,
    pub username: String,
    #[graphql(skip)]
    pub date_of_birth: String,
    pub created_at: i64,
    pub updated_at: i64,
}

impl From<Model> for User {
    fn from(value: Model) -> Self {
        Self {
            id: value.id,
            name: value.name,
            email: value.email,
            username: value.username,
            date_of_birth: value.date_of_birth.to_string(),
            created_at: value.created_at.timestamp(),
            updated_at: value.updated_at.timestamp(),
        }
    }
}

#[ComplexObject]
impl User {
    pub async fn age(&self) -> Result<u32> {
        let date_of_birth = NaiveDate::parse_from_str(&self.date_of_birth, "%Y-%m-%d")
            .map_err(|_| Error::from("Invalid date of birth"))?;

        if let Some(age) = Utc::now().date_naive().years_since(date_of_birth) {
            Ok(age)
        } else {
            Err(Error::from("Invalid date of birth"))
        }
    }
}

Providers

In your application, providers act as the bridge between your application and external resources. Typically, you might find yourself needing providers for the following:

Database: Manages connections to your database system.
Redis Cache: Handles caching mechanisms to improve response times and reduce database load.
Object Storage Service: Facilitates interactions with object storage solutions for managing binary data, such as images or documents.

Creating a Database Provider

The first step in setting up your providers is to create a dedicated providers folder. Within this folder, you'll manage the various external resource providers your application uses. For the database provider, begin by creating a database.rs file:

use std::env;

use anyhow::Result;
use sea_orm::DatabaseConnection;

#[derive(Clone, Debug)]
pub struct Database {
    connection: DatabaseConnection,
}

impl Database {
    pub async fn new() -> Result<Self> {
        let database_url =
            env::var("DATABASE_URL").expect("Missing the DATABASE_URL environment variable.");
        let connection = sea_orm::Database::connect(&database_url).await?;

        Ok(Self { connection })
    }

    pub fn get_connection(&self) -> &DatabaseConnection {
        &self.connection
    }
}

This example demonstrates how to define a DatabaseProvider struct that encapsulates the functionality to establish a connection to your database using an environment variable (DATABASE_URL). The new async function initializes a new DatabaseProvider instance by connecting to the database, and the get_connection method provides a reference to the established DatabaseConnection.

Next Steps

Following the database provider setup, you would similarly create providers for Redis Cache and Object Storage Service. The implementation details would vary based on the specific libraries and services you choose to use. Remember, the goal of these providers is to centralize and abstract the logic for interacting with external resources, making your application's main logic cleaner and more maintainable.

Service

Before we dive into creating services, it's essential to establish a robust error handling mechanism. Unlike languages that use try/catch blocks, Rust utilizes Result types, which can be gracefully handled using the ? operator. This approach necessitates the creation of custom error handlers to manage different types of errors effectively.

Error Handling

Setting Up Tracing

First, ensure your application can trace errors by adding the necessary dependencies to your Cargo.toml file:

# ...
tracing = "0.1"
tracing-opentelemetry = "0.22"
tracing-actix-web = "0.7"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tracing-bunyan-formatter = "0.3"
tracing-log = "0.2"
derive_more = "0.99"

Then, initialize a telemetry system in a startup/telemetry.rs module:

use tracing::{subscriber::set_global_default, Subscriber};
use tracing_bunyan_formatter::{BunyanFormattingLayer, JsonStorageLayer};
use tracing_log::LogTracer;
use tracing_subscriber::{layer::SubscriberExt, EnvFilter, Registry};

pub struct Telemetry;

impl Telemetry {
    pub fn get_subscriber(name: &str, env_filter: &str) -> impl Subscriber + Send + Sync {
        let env_filter = EnvFilter::try_from_default_env().unwrap_or(EnvFilter::new(env_filter));
        let formatting_layer = BunyanFormattingLayer::new(name.into(), std::io::stdout);
        Registry::default()
            .with(env_filter)
            .with(JsonStorageLayer)
            .with(formatting_layer)
    }

    pub fn init_subscriber(subscriber: impl Subscriber + Send + Sync) {
        LogTracer::init().expect("Failed to set logger");
        set_global_default(subscriber).expect("Failed to set subscriber");
    }
}

Custom Error Handling

Create a common/error_handling.rs module to define a structure for internal errors and an enumeration for categorizing different types of service errors:

use derive_more::Display;

#[derive(Debug, Display)]
pub struct InternalCause(String);

impl InternalCause {
    pub fn new(cause: &str) -> Self {
        Self(cause.to_string())
    }
}

#[derive(Debug, Display)]
pub enum ServiceError {
    InternalServerError(String),
    BadRequest(String),
    Unauthorized(String),
    NotFound(String),
    Forbidden(String),
    Conflict(String),
}

Implement methods for generating specific error types and integrating with tracing for logging:

pub const INTERNAL_SERVER_ERROR: &'static str = "Internal Server Error";
pub const INTERNAL_SERVER_ERROR_STATUS_CODE: u16 = 500;
pub const BAD_REQUEST: &'static str = "Bad Request";
pub const BAD_REQUEST_STATUS_CODE: u16 = 400;
pub const UNAUTHORIZED: &'static str = "Unauthorized";
pub const UNAUTHORIZED_STATUS_CODE: u16 = 401;
pub const NOT_FOUND: &'static str = "Not Found";
pub const NOT_FOUND_STATUS_CODE: u16 = 404;
pub const FORBIDDEN: &'static str = "Forbidden";
pub const FORBIDDEN_STATUS_CODE: u16 = 403;
pub const CONFLICT: &'static str = "Conflict";
pub const CONFLICT_STATUS_CODE: u16 = 409;
pub const SOMETHING_WENT_WRONG: &'static str = "Something went wrong";

impl ServiceError {
    pub fn to_str_name(&self) -> &'static str {
        match self {
            ServiceError::InternalServerError(_) => INTERNAL_SERVER_ERROR,
            ServiceError::BadRequest(_) => BAD_REQUEST,
            ServiceError::Unauthorized(_) => UNAUTHORIZED,
            ServiceError::NotFound(_) => NOT_FOUND,
            ServiceError::Forbidden(_) => FORBIDDEN,
            ServiceError::Conflict(_) => CONFLICT,
        }
    }

    pub fn get_status_code(&self) -> u16 {
        match self {
            ServiceError::InternalServerError(_) => INTERNAL_SERVER_ERROR_STATUS_CODE,
            ServiceError::BadRequest(_) => BAD_REQUEST_STATUS_CODE,
            ServiceError::Unauthorized(_) => UNAUTHORIZED_STATUS_CODE,
            ServiceError::NotFound(_) => NOT_FOUND_STATUS_CODE,
            ServiceError::Forbidden(_) => FORBIDDEN_STATUS_CODE,
            ServiceError::Conflict(_) => CONFLICT_STATUS_CODE,
        }
    }

    pub fn internal_server_error<T: std::fmt::Display + std::fmt::Debug>(
        message: &str,
        cause: Option<T>,
    ) -> Self {
        let error = Self::InternalServerError(message.to_string());

        if let Some(cause) = cause {
            tracing::error!(INTERNAL_SERVER_ERROR, %message, %cause);
        } else {
            tracing::error!(INTERNAL_SERVER_ERROR, %message);
        }

        error
    }

    pub fn bad_request<T: std::fmt::Display + std::fmt::Debug>(
        message: &str,
        cause: Option<T>,
    ) -> Self {
        let error = Self::BadRequest(message.to_string());

        if let Some(cause) = cause {
            tracing::error!(BAD_REQUEST, %message, %cause);
        } else {
            tracing::error!(BAD_REQUEST, %message);
        }

        error
    }

    pub fn unauthorized<T: std::fmt::Display + std::fmt::Debug>(
        message: &str,
        cause: Option<T>,
    ) -> Self {
        let error = Self::Unauthorized(message.to_string());

        if let Some(cause) = cause {
            tracing::error!(UNAUTHORIZED, %message, %cause);
        } else {
            tracing::error!(UNAUTHORIZED, %message);
        }

        error
    }

    pub fn not_found<T: std::fmt::Display + std::fmt::Debug>(
        message: &str,
        cause: Option<T>,
    ) -> Self {
        let error = Self::NotFound(message.to_string());

        if let Some(cause) = cause {
            tracing::error!(NOT_FOUND, %message, %cause);
        } else {
            tracing::error!(NOT_FOUND, %message);
        }

        error
    }

    pub fn forbidden<T: std::fmt::Display + std::fmt::Debug>(
        message: &str,
        cause: Option<T>,
    ) -> Self {
        let error = Self::Forbidden(message.to_string());

        if let Some(cause) = cause {
            tracing::error!(FORBIDDEN, %message, %cause);
        } else {
            tracing::error!(FORBIDDEN, %message);
        }

        error
    }

    pub fn conflict<T: std::fmt::Display + std::fmt::Debug>(
        message: &str,
        cause: Option<T>,
    ) -> Self {
        let error = Self::Conflict(message.to_string());

        if let Some(cause) = cause {
            tracing::error!(CONFLICT, %message, %cause);
        } else {
            tracing::error!(CONFLICT, %message);
        }

        error
    }
}

Finally, extend the error handling to integrate with frameworks like actix_web and async_graphql by implementing the ResponseError and converting between ServiceError and GraphQLError:

use actix_web::{error, http::StatusCode, HttpResponse};
use async_graphql::{Error, ErrorExtensions};
// ...

impl error::ResponseError for ServiceError {
    fn status_code(&self) -> StatusCode {
        match *self {
            ServiceError::InternalServerError(_) => StatusCode::INTERNAL_SERVER_ERROR,
            ServiceError::BadRequest(_) => StatusCode::BAD_REQUEST,
            ServiceError::Unauthorized(_) => StatusCode::UNAUTHORIZED,
            ServiceError::NotFound(_) => StatusCode::NOT_FOUND,
            ServiceError::Forbidden(_) => StatusCode::FORBIDDEN,
            ServiceError::Conflict(_) => StatusCode::CONFLICT,
        }
    }

    fn error_response(&self) -> HttpResponse {
        match *self {
            ServiceError::InternalServerError(ref message) => {
                HttpResponse::InternalServerError().json(message)
            }
            ServiceError::BadRequest(ref message) => HttpResponse::BadRequest().json(message),
            ServiceError::Unauthorized(ref message) => HttpResponse::Unauthorized().json(message),
            ServiceError::NotFound(ref message) => HttpResponse::NotFound().json(message),
            ServiceError::Forbidden(ref message) => HttpResponse::Forbidden().json(message),
            ServiceError::Conflict(ref message) => HttpResponse::Conflict().json(message),
        }
    }
}

// Start by creating an identitical enum so there are no conflicts
#[derive(Debug)]
pub enum GraphQLError {
    InternalServerError(String),
    BadRequest(String),
    Unauthorized(String),
    NotFound(String),
    Forbidden(String),
    Conflict(String),
}

impl From<ServiceError> for GraphQLError {
    fn from(error: ServiceError) -> Self {
        match error {
            ServiceError::InternalServerError(message) => {
                GraphQLError::InternalServerError(message)
            }
            ServiceError::BadRequest(message) => GraphQLError::BadRequest(message),
            ServiceError::Unauthorized(message) => GraphQLError::Unauthorized(message),
            ServiceError::NotFound(message) => GraphQLError::NotFound(message),
            ServiceError::Forbidden(message) => GraphQLError::Forbidden(message),
            ServiceError::Conflict(message) => GraphQLError::Conflict(message),
        }
    }
}

// Create the Into trait for the new enum
impl Into<Error> for GraphQLError {
    fn into(self) -> Error {
        match self {
            GraphQLError::InternalServerError(message) => {
                Error::new(message).extend_with(|_, e| {
                    e.set("type", "Internal Server Error");
                    e.set("code", "500");
                })
            }
            GraphQLError::BadRequest(message) => Error::new(message).extend_with(|_, e| {
                e.set("type", "Bad Request");
                e.set("code", "400");
            }),
            GraphQLError::Unauthorized(message) => Error::new(message).extend_with(|_, e| {
                e.set("type", "Unauthorized");
                e.set("code", "401");
            }),
            GraphQLError::NotFound(message) => Error::new(message).extend_with(|_, e| {
                e.set("type", "Not Found");
                e.set("code", "404");
            }),
            GraphQLError::Forbidden(message) => Error::new(message).extend_with(|_, e| {
                e.set("type", "Forbidden");
                e.set("code", "403");
            }),
            GraphQLError::Conflict(message) => Error::new(message).extend_with(|_, e| {
                e.set("type", "Conflict");
                e.set("code", "409");
            }),
        }
    }
}

Creating a basic User Service

Creating a user

Define an input structure for creating a user within the dtos/inputs directory:

use async_graphql::InputObject;

#[derive(InputObject, Debug)]
pub struct CreateUser {
    pub id: i32,
    pub name: String,
    pub username: String,
    pub date_of_birth: String,
}

Implement the service method to create a user, leveraging your database provider and handling potential errors:

use entities::{
    user::{ActiveModel, Column, Entity, Model},
};
use sea_orm::{ActiveModelTrait, PaginatorTrait, Set};

use create::dtos::inputs::CreateUser;
use crate::providers::Database;

pub fn create_user(db: &Database, input: CreateUser) -> Result<Model, ServiceError> {
    let username = input.to_lowercase();
    tracing::info_span!("users_service::create_user", %username);

    let count = Entity::find()
          .filter(Column::Username.eq(&username))
          .count(db.get_connection())
          .await?;

    if count > 0 {
        return Err(ServiceError::conflict::<Error>("User already exists", None));
    }

    let user = ActiveModel {
        name: Set(input.name),
        date_of_birth: Set(input.date_of_birth),
        username: Set(username)
    }
    .insert(db.get_connection())
    .await?;


    let user_id = user.id;
    tracing::info_span!("User created", %user_id);
    Ok(user)
}

Finding a User by ID

Implement a method to find a user by their ID, which demonstrates handling potential not found errors:

// ...

pub async fn find_one_by_id(db: &Database, id: i32) -> Result<Model, ServiceError> {
    tracing::info_span!("users_service::find_one_by_id", %id);
    let user = Entity::find_by_id(id).one(db.get_connection()).await?;
    match user {
        Some(value) => {
            tracing::info!("User found");
            Ok(value)
        }
        None => Err(ServiceError::not_found::<Error>(USER_NOT_FOUND, None)),
    }
}

Through these implementations, your service layer will efficiently handle business logic, interact with the database, and manage errors. These foundations ensure that your application is robust, maintainable, and scalable.

Resolvers

In a GraphQL server, resolvers are crucial as they determine how each query or mutation translates into an operation that fetches or manipulates data. This section outlines how to implement resolvers that interact with our previously defined services.

Structuring Queries and Mutations

For organization, async-graphql dictates that our GraphQL schema has to have distinct structs for queries and mutations. This style helps in maintaining clear boundaries between data retrieval (queries) and data manipulation (mutations) operations.

use async_graphql::{Context, Object, Result};

use crate:dtos::inputs::CreateUser;
use crate::providers::Database;
use crate::services::users_service;

#[derive(Default)]
pub struct UsersQuery;

#[derive(Default)]
pub struct UsersMutation;

Implementing Resolver Methods

Within these structs, we define methods corresponding to each query and mutation defined in our GraphQL schema. These methods utilize the services layer to interact with the database and perform the necessary business logic.

Query Resolvers

For fetching user data by ID, we implement a method in the UsersQuery struct:

// ...

#[Object]
impl UsersQuery {
    async fn user_by_id(&self, ctx: &Context<'_>, id: i32) -> Result<User> {
       Ok(
           users_service::find_one_by_id(ctx.data::<Database>()?, id)
                .await?
                .into(),
        )
    }
}

This resolver accesses the Database provider from the context, passes it to the users_service::find_one_by_id method, and maps the service result into the GraphQL User object.

Mutation Resolvers

For creating a new user, we define a method in the UsersMutation struct:

// ...

#[Object]
impl UsersMutation {
    async fn creater_user(&self, ctx: &Context<'_>, input: CreateUser) -> Result<User> {
        Ok(
            users_service::create_user(ctx.data::<Database>()?, input)
                .await?
                .into(),
        )
    }
}

Similar to the query resolver, this mutation accesses the database through the context and utilizes the users_service::create_user method to add a new user. The result is then mapped to the GraphQL User object.

Schema Configuration

We define the schema by merging query and mutation roots. This setup allows your GraphQL server to understand how to process incoming queries and mutations.

Create a file called startup/schema_builder.rs with the query root and mutation root:

use async_graphql::{EmptySubscription, MergedObject, Schema};

use crate::providers::Database;
use crate::resolvers::users_resolver;


#[derive(MergedObject, Default)]
pub struct MutationRoot(users_resolver::UsersMutation);

#[derive(MergedObject, Default)]
pub struct QueryRoot(users_resolver::UsersQuery);

pub fn build_schema(database: &Database) -> Schema<QueryRoot, MutationRoot, EmptySubscription> {
    Schema::build(
        QueryRoot::default(),
        MutationRoot::default(),
        EmptySubscription,
    )
    .data(database.to_owned())
    .finish()
}

GraphQL Configuration

Next, we configure Actix Web to serve the GraphQL API and Playground, facilitating both API interaction and a user-friendly interface for testing queries and mutations.

Create the functions for the GraphQL API POST (the main API) and GET (the playground) routes:

use actix_web::{web::Data, HttpRequest, HttpResponse, Result};
use async_graphql::{
    http::{playground_source, GraphQLPlaygroundConfig},
    // ...
};
use async_graphql_actix_web::{GraphQLRequest, GraphQLResponse};

// ...

pub async fn graphql_request(
    schema: Data<Schema<QueryRoot, MutationRoot, EmptySubscription>>,
    req: HttpRequest,
    gql_req: GraphQLRequest,
) -> GraphQLResponse {
    schema
        .execute(gql_req.into_inner())
        .await
        .into()
}

pub async fn graphql_playground() -> Result<HttpResponse> {
    let source = playground_source(GraphQLPlaygroundConfig::new("/api/graphql"));
    Ok(HttpResponse::Ok()
        .content_type("text/html; charset=utf-8")
        .body(source))
}

Application Initialization

For the main application setup, we focus on integrating the schema with Actix Web and configuring the server to listen for incoming requests.

Create a file called startup/app.rs:

use std::{io, net::TcpListener};

use actix_web::guard;
use actix_web::{dev::Server, web, App, HttpServer};
use anyhow::Error;
use tracing_actix_web::TracingLogger;

use crate::providers::Database

use super::schema_builder::{build_schema, graphql_playground, graphql_request};


pub struct ActixApp {
    port: u16,
    server: Server,
}


impl ActixApp {
    pub async fn new() -> Result<Self, Error> {
        if let Err(e) = dotenvy::dotenv() {
            tracing::warn!("Failed to load .env file: {}", e);
            tracing::warn!("Using default environment variables");
        }

        let host = env::var("HOST").unwrap_or_else(|_| "127.0.0.1".to_string());
        let port = env::var("PORT")
            .unwrap_or_else(|_| "8080".to_string())
            .parse::<u16>()
            .unwrap_or(8080);
        let listener = TcpListener::bind(format!("{}:{}", &host, &port))?;
        let port = listener.local_addr().unwrap().port();
        let server = HttpServer::new(move || {
            App::new()
                .wrap(TracingLogger::default())
                .configure(Self::build_app_config(Environment::new(), port, &db))
        })
        .listen(listener)?
        .run();
        tracing::info!("Server running on port {}", port);
        Ok(Self { port, server })
    }

    pub fn port(&self) -> u16 {
        self.port
    }

    pub async fn start_server(self) -> Result<(), io::Error> {
        self.server.await
    }

    pub fn build_app_config(
        port: u16,
        db: &Database,
    ) -> impl Fn(&mut web::ServiceConfig) {
        let db = db.clone();
        move |cfg: &mut web::ServiceConfig| {
            cfg.app_data(web::Data::new(build_schema(
                &db,
                &jwt,
                ObjectStorage::new(&environment),
            )))
            .service(
                web::resource("/api/graphql")
                    .guard(guard::Post())
                    .to(graphql_request),
            )
            .service(
                web::resource("/api/graphql")
                    .guard(guard::Get())
                    .to(graphql_playground),
            );
        }
    }
}

Running the API

To get the API up and running, we'll utilize the tokio::main macro for our main function, facilitating an asynchronous runtime necessary for our Actix web server. This setup involves initializing telemetry for application insights and starting the Actix application.

Here's how you can do it:

use std::fmt::{Debug, Display};
use tokio::task::JoinError;
use your_project_name::startup::{ActixApp, Telemetry};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize telemetry for structured logging.
    let subscriber = Telemetry::get_subscriber("your_project_name", "info"); // Customize the application name and log level as needed.
    Telemetry::init_subscriber(subscriber);

    // Create and start the Actix application.
    let application = ActixApp::new().await?;
    let application_task = tokio::spawn(application.start());

    // Monitor the application's exit status.
    tokio::select! {
        outcome = application_task => report_exit("API", outcome),
    };

    Ok(())
}

// Helper function to log the outcome of the application task.
fn report_exit(task_name: &str, outcome: Result<Result<(), impl Debug + Display>, JoinError>) {
    match outcome {
        Ok(Ok(())) => {
            tracing::info!("{} has exited", task_name)
        }
        Ok(Err(e)) => {
            tracing::error!(
                error.cause_chain = ?e,
                error.message = %e,
                "{} failed",
                task_name
            )
        }
        Err(e) => {
            tracing::error!(
                error.cause_chain = ?e,
                error.message = %e,
                "{}' task failed to complete",
                task_name
            )
        }
    }
}

Conclusion

Throughout this article, we embarked on a comparative journey between the NestJS framework and its Rust counterpart, exploring not just the theoretical differences but also delving into the practical aspects of using them in real-world applications. From setting up the project structure and handling data models to implementing services, resolvers, and finally running the API, we've covered a comprehensive guide to understanding the strengths and nuances of developing server-side applications in Rust as an alternative to NodeJS with NestJS.

As we conclude, it's evident that the choice between NestJS and Rust depends on specific project requirements, performance considerations, and developer proficiency. While NestJS offers a quick and efficient way to build server-side applications with JavaScript/TypeScript, Rust provides an avenue for achieving unparalleled performance and reliability, albeit with a steeper learning curve.

About the Author

Hey there! I am Afonso Barracha, a back-end developer with a soft spot for GraphQL. If you enjoyed reading this article, why not show some love by buying me a coffee?

Do not miss out on any of my latest articles – follow me here on dev and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

NestJS and Mercurius full GraphQL Back-end template

Afonso Barracha — Sat, 20 May 2023 14:13:47 +0000

Intro

I am excited to share my GraphQL Back-end template that is not only easy to develop but also cost-effective to host, built with NestJS and Mercurius.

Where can you find it?

You can find this template on my GitHub repository.

Using it is super simple, just click on the Use this template to begin. Afterward, make sure to modify the CODEOWNERS on the .github folder to reflect your GitHub username. You can then remove or alter the PULL_REQUEST_TEMPLATE.md. Lastly, in the main folder, you will most likely want to update the README.md and CONTRIBUTING.md files.

What about the License and Header?

The template is licensed under the Mozilla Public License 2.0, which means that only modifications to the existing files need to be disclosed. Any additional modules you add to the template remain your private property.

As for the header, it is conveniently placed for easy modification to suit your project in the .eslintrc.js file. After updating it, to change the header on all files, all you need to do is run yarn lint.

How is hosting made easy?

The template run on the Node.JS cluster module, utilizing a round-robin model for load balancing, making it easily scalable on a single Virtual Private Cloud (VPC). All you need to do is adjust the WORKOUTS_COUNT environment variable. By default, the template uses all available threads to run the app in a production environment, if this variable is not specified.

This structure enables affordable hosting on any Ubuntu/Debian-based Linux VPC using open-source Platform as a Service (PaaS) tools like dokku or caprover. You can scale your VPC vertically whenever your app needs to grow.

Conclusion

My motivation behind making this template publicly available was to support solo developers in bringing their ideas to life with ease, either for production projects or small hackathons. This template handles the more tedious and repetitive tasks, so you can focus on what truly matters: developing your project.

About the Author

Do not miss out on any of my latest articles – follow me here on dev, on twitter and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

NestJS Authentication with OAuth2.0: Adding External Providers

Afonso Barracha — Fri, 14 Apr 2023 22:11:24 +0000

Series Intro

This series will cover the full implementation of OAuth2.0 Authentication in NestJS for the following types of APIs:

Express REST API;
Fastify REST API;
Apollo GraphQL API.

And it is divided in 5 parts:

Configuration and operations;
Express Local OAuth REST API;
Fastify Local OAuth REST API;
Apollo Local OAuth GraphQL API;
Adding External OAuth Providers to our API;

Lets start the fifth and last part of this series.

Tutorial Intro

On this tutorial we will add to the Fastify API from the third part the capability of signing in with external OAuth 2.0 providers such as Microsoft, Google, Facebook and GitHub.

This tutorial will be slightly more complex than the previous ones, as on the one hand, we need an implementation that works for both adapters (Fastify and Express), and on the other hand it should apply to the maximum number of external providers.

TLDR: if you do not have 45 minutes to read the article, the source code can be found on this repo. While you at it if you liked what you read consider buying me a coffee.

The Flow

The flow is a bit complex and requires access to the front-end, as we will need to be able to redirect back to it:

Note: Apple's flow is a bit different and will not be covered in this article.

Options

There are several ways to implement this flow in NestJS, for example this are my recommended 3 ways:

Use adapter specific packages:
- Passport for Express;
- Fastify-OAuth2 for Fastify;
Implement an NestJS External OAuth Provider, using dynamic modules;
Implementing External OAuth Routes under a feature flags.

My preferred approach is the 3rd one, and that is the one we will implement on this tutorial.

Approach

We will:

Allow for multiple types of authentication for each user;
Only require confirmation when the user was registed through Local OAuth;
Users can create a password for Local OAuth even if they registered with an external provider.

Since the setup is quite different, we will separate the Local OAuth and External OAuth into distinct modules.

This tutorial only covers the code and API logic, it will not show you how to register your app on Microsoft, Google, Facebook or GitHub. I will assume that you already know how to do that, if you do not, you can find how in each providers' official documentation.

Set up

Start by upgrading the library to the latest npm package versions:

$ yarn upgrade-interactive

NOTE: if you update Mikro-ORM to the latest version you will need to update the imports on the common.service.ts to the EntityRepository of the @mikro-orm/postgresql package.

To facilitate our implementation we will use the simple-oauth2 library, so start by installing it:

$ yarn add simple-oauth2
$ yarn add -D @types/simple-oauth2

And for API calls we will use Axios, nest has its own Axios integration, so install them as well:

$ yarn add @nestjs/axios axios

Changes

There are several changes that we need to make on our previous API implementation.

User changes

Entities

OAuth Provider Entity

First we need an enum with the possible providers, for this example we will allow 5 providers. Create the oauth-providers.enum.ts on the enums directory:

export enum OAuthProvidersEnum {
  LOCAL = 'local',
  MICROSOFT = 'microsoft',
  GOOGLE = 'google',
  FACEBOOK = 'facebook',
  GITHUB = 'github',
}

Since we will allow multiple providers we need to create a OAuthProviderEntity to keep track of the user's active providers. For simplicity we will not allow the user to manually to disconnect from the providers.

Create the oauth-provider.entity.ts:

import {
  Entity,
  Enum,
  ManyToOne,
  PrimaryKeyType,
  Property,
  Unique,
} from '@mikro-orm/core';
import { IsEnum } from 'class-validator';
import { OAuthProvidersEnum } from '../enums/oauth-providers.enum';
import { IOAuthProvider } from '../interfaces/oauth-provider.interface';
import { UserEntity } from './user.entity';

@Entity({ tableName: 'oauth_providers' })
@Unique({ properties: ['provider', 'user'] })
export class OAuthProviderEntity implements IOAuthProvider {
  @Enum({
    items: () => OAuthProvidersEnum,
    primary: true,
    columnType: 'varchar',
    length: 9,
  })
  @IsEnum(OAuthProvidersEnum)
  public provider: OAuthProvidersEnum;

  @ManyToOne({
    entity: () => UserEntity,
    primary: true,
    onDelete: 'cascade',
  })
  public user: UserEntity;

  @Property({ onCreate: () => new Date() })
  public createdAt: Date = new Date();

  @Property({ onUpdate: () => new Date() })
  public updatedAt: Date = new Date();

  [PrimaryKeyType]?: [OAuthProvidersEnum, number];
}

And import it on the UsersModule:

// ...
import { OAuthProviderEntity } from './entities/oauth-provider.entity';
// ...

@Module({
  imports: [MikroOrmModule.forFeature([UserEntity, OAuthProviderEntity])],
  // ...
})
export class UsersModule {}

Finally inject it into the UserService:

//...  
import { OAuthProviderEntity } from './entities/oauth-provider.entity';
// ...

@Injectable()
export class UsersService {
  constructor(
    // ...
    @InjectRepository(OAuthProviderEntity)
    private readonly oauthProvidersRepository: EntityRepository<OAuthProviderEntity>,
    // ...
  ) {}

  // ...
}

User Entity

For the user entity we need to make some changes. Now the Password is not mandatory, so change the BCRYPT_HASH regex into BCRYPT_HASH_OR_USET:

export const BCRYPT_HASH_OR_UNSET =
  /(UNSET|(\$2[abxy]?\$\d{1,2}\$[A-Za-z\d\./]{53}))/;

And add the one-to-many relation for the OAuthProviderEntity:

import {
  Collection,
  // ...
  OneToMany,
  // ...
} from '@mikro-orm/core';
// ...
import { OAuthProviderEntity } from './oauth-provider.entity';

@Entity({ tableName: 'users' })
export class UserEntity implements IUser {
  // ...

  @OneToMany(() => OAuthProviderEntity, (oauth) => oauth.user)
  public oauthProviders = new Collection<OAuthProviderEntity, UserEntity>(this);
}

Because of the external providers the users now can be confirmed automatically, so update the CredentialsEmbeddable:

import { Embeddable, Property } from '@mikro-orm/core';
import dayjs from 'dayjs';
import { ICredentials } from '../interfaces/credentials.interface';

@Embeddable()
export class CredentialsEmbeddable implements ICredentials {
  @Property({ default: 0 })
  public version: number = 0;

  @Property({ default: '' })
  public lastPassword: string = '';

  @Property({ default: dayjs().unix() })
  public passwordUpdatedAt: number = dayjs().unix();

  @Property({ default: dayjs().unix() })
  public updatedAt: number = dayjs().unix();

  constructor(isConfirmed = false) {
    this.version = isConfirmed ? 1 : 0;
  }

  public updatePassword(password: string): void {
    this.version++;
    this.lastPassword = password;
    const now = dayjs().unix();
    this.passwordUpdatedAt = now;
    this.updatedAt = now;
  }

  public updateVersion(): void {
    this.version++;
    this.updatedAt = dayjs().unix();
  }
}

Service

When we create an user, we need to create an OAuthProvider as well, therefore create the createOAuthProvider private method:

// ...
import { OAuthProvidersEnum } from './enums/oauth-providers.enum';

@Injectable()
export class UsersService {
  // ...

  private async createOAuthProvider(
    provider: OAuthProvidersEnum,
    userId: number,
  ): Promise<OAuthProviderEntity> {
    const oauthProvider = this.oauthProvidersRepository.create({
      provider,
      user: userId,
    });
    await this.commonService.saveEntity(
      this.oauthProvidersRepository,
      oauthProvider,
      true,
    );
    return oauthProvider;
  }
}

And update the create public method, adding a new provider argument, making the password an optional argument, and confirming the user if the provider is not local:

// ...

@Injectable()
export class UsersService {
  // ...

  public async create(
    provider: OAuthProvidersEnum,
    email: string,
    name: string,
    password?: string,
  ): Promise<UserEntity> {
    const isConfirmed = provider !== OAuthProvidersEnum.LOCAL;
    const formattedEmail = email.toLowerCase();
    await this.checkEmailUniqueness(formattedEmail);
    const formattedName = this.commonService.formatName(name);
    const user = this.usersRepository.create({
      email: formattedEmail,
      name: formattedName,
      username: await this.generateUsername(formattedName),
      password: isUndefined(password) ? 'UNSET' : await hash(password, 10),
      confirmed: isConfirmed,
      credentials: new CredentialsEmbeddable(isConfirmed),
    });
    await this.commonService.saveEntity(this.usersRepository, user, true);
    await this.createOAuthProvider(provider, user.id);
    return user;
  }
}

To be able to accept external OAuth we need to be able to get or create a new user, so create the findOrCreate public method:

// ...

@Injectable()
export class UsersService {
  // ...

  public async findOrCreate(
    provider: OAuthProvidersEnum,
    email: string,
    name: string,
  ): Promise<UserEntity> {
    const formattedEmail = email.toLowerCase();
    const user = await this.usersRepository.findOne(
      {
        email: formattedEmail,
      },
      {
        populate: ['oauthProviders'],
      },
    );

    if (isUndefined(user) || isNull(user)) {
      return this.create(provider, email, name);
    }
    if (
      isUndefined(
        user.oauthProviders.getItems().find((p) => p.provider === provider),
      )
    ) {
      await this.createOAuthProvider(provider, user.id);
    }

    return user;
  }

  // ...
}

Both Update and Reset Password have to take into account external providers, hence we need to refactor them a bit, start by extracting the password change logic into its own method:

// ...

@Injectable()
export class UsersService {
  // ...

  private async changePassword(
    user: UserEntity,
    password: string,
  ): Promise<UserEntity> {
    user.credentials.updatePassword(user.password);
    user.password = await hash(password, 10);
    await this.commonService.saveEntity(this.usersRepository, user);
    return user;
  }

  // ...
}

Next, update the updatePassword and resetPassword public methods:

// ...

@Injectable()
export class UsersService {
  // ...

  public async updatePassword(
    userId: number,
    newPassword: string,
    password?: string,
  ): Promise<UserEntity> {
    const user = await this.findOneById(userId);

    if (user.password === 'UNSET') {
      await this.createOAuthProvider(OAuthProvidersEnum.LOCAL, user.id);
    } else {
      if (isUndefined(password) || isNull(password)) {
        throw new BadRequestException('Password is required');
      }
      if (!(await compare(password, user.password))) {
        throw new BadRequestException('Wrong password');
      }
      if (await compare(newPassword, user.password)) {
        throw new BadRequestException('New password must be different');
      }
    }

    return await this.changePassword(user, newPassword);
  }

  public async resetPassword(
    userId: number,
    version: number,
    password: string,
  ): Promise<UserEntity> {
    const user = await this.findOneByCredentials(userId, version);
    return await this.changePassword(user, password);
  }

  // ...
}

As a last update, just add a findOAuthProviders so users can know how many providers they have active:

// ...

@Injectable()
export class UsersService {
  // ...

  public async findOAuthProviders(
    userId: number,
  ): Promise<OAuthProviderEntity[]> {
    return await this.oauthProvidersRepository.find(
      {
        user: userId,
      },
      { orderBy: { provider: QueryOrder.ASC } },
    );
  }
  // ...
}

Jwt Changes

There is just one minor change, as we need a private method from the AuthService to be shared between it and the future Oauth2Service for external providers.

Service

Move the AuthService's generateAuthTokens private method to the JwtService so we can share its logic with the Oauth2Service:

// ...

@Injectable()
export class JwtService {
  // ...

  public async generateAuthTokens(
    user: IUser,
    domain?: string,
    tokenId?: string,
  ): Promise<[string, string]> {
    return Promise.all([
      this.generateToken(user, TokenTypeEnum.ACCESS, domain, tokenId),
      this.generateToken(user, TokenTypeEnum.REFRESH, domain, tokenId),
    ]);
  }
}

Auth Changes

Module

Move the ThrottlerModule import into the AppModule:

// ...
import { ThrottlerModule } from '@nestjs/throttler';
// ...

@Module({
  imports: [
    // ...
    ThrottlerModule.forRootAsync({
      imports: [ConfigModule],
      useClass: ThrottlerConfig,
    }),
    // ...
  ],
  // ...
})
export class AppModule {}

Service

Now that we have a public generateAuthTokens remove the private version, and use the public version from the JwtService:

// ...

@Injectable()
export class AuthService {
  // ...

  public async confirmEmail(
    dto: ConfirmEmailDto,
    domain?: string,
  ): Promise<IAuthResult> {
    // ...

    const [accessToken, refreshToken] =
      await this.jwtService.generateAuthTokens(user, domain);
    return { user, accessToken, refreshToken };
  }

  public async signIn(dto: SignInDto, domain?: string): Promise<IAuthResult> {
    // ...

    const [accessToken, refreshToken] =
      await this.jwtService.generateAuthTokens(user, domain);
    return { user, accessToken, refreshToken };
  }

  public async refreshTokenAccess(
    refreshToken: string,
    domain?: string,
  ): Promise<IAuthResult> {
    // ...

    const [accessToken, newRefreshToken] =
      await this.jwtService.generateAuthTokens(user, domain, tokenId);
    return { user, accessToken, refreshToken: newRefreshToken };
  }

  // ...

  public async updatePassword(
    userId: number,
    dto: ChangePasswordDto,
    domain?: string,
  ): Promise<IAuthResult> {
    // ...

    const [accessToken, refreshToken] =
      await this.jwtService.generateAuthTokens(user, domain);
    return { user, accessToken, refreshToken };
  }

  // ...
}

And the order of the inputs of the UsersService's updatePassword method change:

// ...

@Injectable()
export class AuthService {
  // ...

  public async updatePassword(
    userId: number,
    dto: ChangePasswordDto,
    domain?: string,
  ): Promise<IAuthResult> {
    // ...
    const user = await this.usersService.updatePassword(
      userId,
      password1,
      password,
    );
    // ...
  }

  // ...
}

On a last note, update the signUp method to pass the OAuthProviderEnum.LOCAL:

// ...
import { OAuthProvidersEnum } from '../users/enums/oauth-providers.enum';
// ...

@Injectable()
export class AuthService {
  // ...

  public async signUp(dto: SignUpDto, domain?: string): Promise<IMessage> {
    // ...
    const user = await this.usersService.create(
      OAuthProvidersEnum.LOCAL,
      email,
      name,
      password1,
    );
    // ...
  }

  // ...
}

OAuth2 Resource

To set up our external providers we will use a new REST resource named oauth2:

$ nest g res oauth2

OAuth Class

Before starting working on the service and controller we need to create an external provider wrapper. It is just a class that encapsulates the OAuth provider logic, so create a classes directory with an oauth.class.ts file inside:

export class OAuthClass {}

Before implementing it we need to understand what we need from it. The OAuthClass will consist on three parts:

Provider: the provider URLs and Client options, we abstract this with the AuthorizationCode from the simple-oauth2 package;
Authorization URL: the URL with the parameters that we temporarily redirect our user to sign-in/up;
Data URL: the URL we use to get the user data when we get the external provider's access token.

Types

Create a new interfaces directory with the following interfaces:

IAuthParams: the parameters necessary for generating the Authorization URL, normally the redirect_uri (also known as the callback URL) and scope.
```
export interface IAuthParams {
  readonly redirect_uri: string;
  readonly scope: string | string[];
}
```

ICallbackQuery: the query parameters that you receive on the (redirect_uri) callback URL.

export interface ICallbackQuery {
  readonly code: string;
  readonly state: string;
}

IProvider: the provider URLs (hosts) and relative paths.

export interface IProvider {
  readonly tokenHost: string;
  readonly tokenPath: string;
  readonly authorizeHost: string;
  readonly authorizePath: string;
  readonly refreshPath?: string;
  readonly revokePath?: string;
}

User Responses: this one is not a single interface but many interfaces, one for each provider.

export interface IMicrosoftUser {
  readonly businessPhones: string[];
  readonly displayName: string;
  readonly givenName: string;
  readonly jobTitle: string;
  readonly mail: string;
  readonly mobilePhone: string;
  readonly officeLocation: string;
  readonly preferredLanguage: string;
  readonly surname: string;
  readonly userPrincipalName: string;
  readonly id: string;
}

export interface IGoogleUser {
  readonly sub: string;
  readonly name: string;
  readonly given_name: string;
  readonly family_name: string;
  readonly picture: string;
  readonly email: string;
  readonly email_verified: boolean;
  readonly locale: string;
  readonly hd: string;
}

export interface IFacebookUser {
  readonly id: string;
  readonly name: string;
  readonly email: string;
}

interface IGitHubPlan {
  readonly name: string;
  readonly space: number;
  readonly private_repos: number;
  readonly collaborators: number;
}

export interface IGitHubUser {
  readonly login: string;
  readonly id: number;
  readonly node_id: string;
  readonly avatar_url: string;
  readonly gravatar_id: string;
  readonly url: string;
  readonly html_url: string;
  readonly followers_url: string;
  readonly following_url: string;
  readonly gists_url: string;
  readonly starred_url: string;
  readonly subscriptions_url: string;
  readonly organizations_url: string;
  readonly repos_url: string;
  readonly events_url: string;
  readonly received_events_url: string;
  readonly type: string;
  readonly site_admin: boolean;
  readonly name: string;
  readonly company: string;
  readonly blog: string;
  readonly location: string;
  readonly email: string;
  readonly hireable: boolean;
  readonly bio: string;
  readonly twitter_username: string;
  readonly public_repos: number;
  readonly public_gists: number;
  readonly followers: number;
  readonly following: number;
  readonly created_at: string;
  readonly updated_at: string;
  readonly private_gists: number;
  readonly total_private_repos: number;
  readonly owned_private_repos: number;
  readonly disk_usage: number;
  readonly collaborators: number;
  readonly two_factor_authentication: boolean;
  readonly plan: IGitHubPlan;
}

Private Static Params

Back on the class, start adding the providers as static parameters, you can find most of them in the fastify-oauth2 repository.

First the providers:

import { IAuthParams } from '../interfaces/auth-params.interface';
import { IProvider } from '../interfaces/provider.interface';

export class OAuthClass {
  private static readonly [OAuthProvidersEnum.MICROSOFT]: IProvider = {
    authorizeHost: 'https://login.microsoftonline.com',
    authorizePath: '/common/oauth2/v2.0/authorize',
    tokenHost: 'https://login.microsoftonline.com',
    tokenPath: '/common/oauth2/v2.0/token',
  };
  private static readonly [OAuthProvidersEnum.GOOGLE]: IProvider = {
    authorizeHost: 'https://accounts.google.com',
    authorizePath: '/o/oauth2/v2/auth',
    tokenHost: 'https://www.googleapis.com',
    tokenPath: '/oauth2/v4/token',
  };
  private static readonly [OAuthProvidersEnum.FACEBOOK]: IProvider = {
    authorizeHost: 'https://facebook.com',
    authorizePath: '/v9.0/dialog/oauth',
    tokenHost: 'https://graph.facebook.com',
    tokenPath: '/v9.0/oauth/access_token',
  };
  private static readonly [OAuthProvidersEnum.GITHUB]: IProvider = {
    authorizeHost: 'https://github.com',
    authorizePath: '/login/oauth/authorize',
    tokenHost: 'https://github.com',
    tokenPath: '/login/oauth/access_token',
  };
}

And then the user data urls:

// ...

export class OAuthClass {
  // ...
  private static userDataUrls: Record<OAuthProvidersEnum, string> = {
    [OAuthProvidersEnum.GOOGLE]:
      'https://www.googleapis.com/oauth2/v3/userinfo',
    [OAuthProvidersEnum.MICROSOFT]: 'https://graph.microsoft.com/v1.0/me',
    [OAuthProvidersEnum.FACEBOOK]:
      'https://graph.facebook.com/v16.0/me?fields=email,name',
    [OAuthProvidersEnum.GITHUB]: 'https://api.github.com/user',
    [OAuthProvidersEnum.LOCAL]: '',
  };
}

We need a static method to generate the Authorization Parameters. It will take the provider and the API url as arguments, and return the params:

import { randomBytes } from 'crypto';
import { OAuthProvidersEnum } from '../../users/enums/oauth-providers.enum';
// ...

export class OAuthClass {
  // ...

  private static genAuthorization(
    provider: OAuthProvidersEnum,
    url: string,
  ): IAuthParams {
    // generates the callback url given the provider
    const redirect_uri = `${url}/api/auth/ext/${provider}/callback`;

    switch (provider) {
      case OAuthProvidersEnum.GOOGLE:
        return {
          redirect_uri,
          scope: [
            'https://www.googleapis.com/auth/userinfo.email',
            'https://www.googleapis.com/auth/userinfo.profile',
          ],
        };
      case OAuthProvidersEnum.MICROSOFT:
        return {
          redirect_uri,
          scope: ['openid', 'profile', 'email'],
        };
      case OAuthProvidersEnum.FACEBOOK:
        return {
          redirect_uri,
          scope: ['email', 'public_profile'],
        };
      case OAuthProvidersEnum.GITHUB:
        return {
          redirect_uri,
          scope: ['user:email', 'read:user'],
        };
    }
  }
}

Note: I made the scopes static for consistent, but they could have been passed as a parameter, this depends on your preference.

Private Non-static Params

This class will have three main non-static parameters:

Code: a class to get the access token provided by the simple-oauth2 package;
Authorization: the authorization params for the redirect URI;
User Data URL: the URL to fetch the user data after getting the external provider's access token.

// ...
import { AuthorizationCode } from 'simple-oauth2';
import { IAuthParams } from '../interfaces/auth-params.interface';
import { IClient } from '../interfaces/client.interface';
// ...

export class OAuthClass {
   // ...

  private readonly code: AuthorizationCode;
  private readonly authorization: IAuthParams;
  private readonly userDataUrl: string;

  constructor(
    private readonly provider: OAuthProvidersEnum,
    private readonly client: IClient,
    private readonly url: string,
  ) {
    if (provider === OAuthProvidersEnum.LOCAL) {
      throw new Error('Invalid provider');
    }

    this.code = new AuthorizationCode({
      client,
      auth: OAuthClass[provider],
    });
    this.authorization = OAuthClass.genAuthorization(provider, url);
    this.userDataUrl = OAuthClass.userDataUrls[provider];
  }
}

Get Methods

Get methods are like parameters, but provided by methods with no arguments, this class will have 2:

Get Data URL:

// ...

export class OAuthClass {
   // ...

  public get dataUrl(): string {
    return this.userDataUrl;
  }
}

Get Authorization URL (this is provided by the AuthorizationCode from simple-oauth2) with a random state:

// ...

export class OAuthClass {
   // ...

  public get authorizationUrl(): [string, string] {
    const state = randomBytes(16).toString('hex');
    return [this.code.authorizeURL({ ...this.authorization, state }), state];
  }
}

Normal Methods

We need a method to get the access token from the external provided. Most of the underlying logic is already provided by the AuthorizationCode class from the simple-oauth2 package.

// ...

export class OAuthClass {
   // ...

  public async getToken(code: string): Promise<string> {
    const result = await this.code.getToken({
      code,
      redirect_uri: this.authorization.redirect_uri,
      scope: this.authorization.scope,
    });
    return result.token.access_token as string;
  }
}

Config

Create an interface with all our external providers, we will call it oauth2.interface.ts:

import { IClient } from '../../oauth2/interfaces/client.interface';

export interface IOAuth2 {
  readonly microsoft: IClient | null;
  readonly google: IClient | null;
  readonly facebook: IClient | null;
  readonly github: IClient | null;
}

Add it and the url to the config.interface.ts:

import { IOAuth2 } from './oauth2.interface';

export interface IConfig {
  // ...
  readonly url: string;
  // ...
  readonly oauth2: IOAuth2;
}

Update the config schema with the new ENV variables:

import Joi from 'joi';

export const validationSchema = Joi.object({
  // ...
  URL: Joi.string().uri().required(),
  // ...
  MICROSOFT_CLIENT_ID: Joi.string().optional(),
  MICROSOFT_CLIENT_SECRET: Joi.string().optional(),
  GOOGLE_CLIENT_ID: Joi.string().optional(),
  GOOGLE_CLIENT_SECRET: Joi.string().optional(),
  FACEBOOK_CLIENT_ID: Joi.string().optional(),
  FACEBOOK_CLIENT_SECRET: Joi.string().optional(),
  GITHUB_CLIENT_ID: Joi.string().optional(),
  GITHUB_CLIENT_SECRET: Joi.string().optional(),
});

And finally add them to the config function:

// ...

export function config(): IConfig {
  // ...

  return {
    // ...
    url: process.env.URL,
    // ...
    oauth2: {
      microsoft:
        isUndefined(process.env.MICROSOFT_CLIENT_ID) ||
        isUndefined(process.env.MICROSOFT_CLIENT_SECRET)
          ? null
          : {
              id: process.env.MICROSOFT_CLIENT_ID,
              secret: process.env.MICROSOFT_CLIENT_SECRET,
            },
      google:
        isUndefined(process.env.GOOGLE_CLIENT_ID) ||
        isUndefined(process.env.GOOGLE_CLIENT_SECRET)
          ? null
          : {
              id: process.env.GOOGLE_CLIENT_ID,
              secret: process.env.GOOGLE_CLIENT_SECRET,
            },
      facebook:
        isUndefined(process.env.FACEBOOK_CLIENT_ID) ||
        isUndefined(process.env.FACEBOOK_CLIENT_SECRET)
          ? null
          : {
              id: process.env.FACEBOOK_CLIENT_ID,
              secret: process.env.FACEBOOK_CLIENT_SECRET,
            },
      github:
        isUndefined(process.env.GITHUB_CLIENT_ID) ||
        isUndefined(process.env.GITHUB_CLIENT_SECRET)
          ? null
          : {
              id: process.env.GITHUB_CLIENT_ID,
              secret: process.env.GITHUB_CLIENT_SECRET,
            },
    },
  };
}

Module

Start by importing the necessary dependencies, namely the UsersModule, JwtModule and HttpModule, and export the Oauth2Service:

import { HttpModule } from '@nestjs/axios';
import { Module } from '@nestjs/common';
import { JwtModule } from '../jwt/jwt.module';
import { UsersModule } from '../users/users.module';
import { Oauth2Controller } from './oauth2.controller';
import { Oauth2Service } from './oauth2.service';

@Module({
  imports: [
    UsersModule,
    JwtModule,
    HttpModule.register({
      timeout: 5000,
      maxRedirects: 5,
    }),
  ],
  controllers: [Oauth2Controller],
  providers: [Oauth2Service],
})
export class Oauth2Module {}

Service

Inject all the necessary services:

import { Cache } from 'cache-manager';
import { HttpService } from '@nestjs/axios';
import { CACHE_MANAGER } from '@nestjs/cache-manager';
import { Inject, Injectable } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { CommonService } from '../common/common.service';
import { JwtService } from '../jwt/jwt.service';
import { UsersService } from '../users/users.service';

@Injectable()
export class Oauth2Service {
  constructor(
    @Inject(CACHE_MANAGER)
    private readonly cacheManager: Cache,
    private readonly usersService: UsersService,
    private readonly jwtService: JwtService,
    private readonly configService: ConfigService,
    private readonly httpService: HttpService,
    private readonly commonService: CommonService,
  ) {}
}

And the private parameters for each external provider:

// ...
import { OAuthProvidersEnum } from '../users/enums/oauth-providers.enum';
import { OAuthClass } from './classes/oauth.class';

@Injectable()
export class Oauth2Service {
  private readonly [OAuthProvidersEnum.MICROSOFT]: OAuthClass | null;
  private readonly [OAuthProvidersEnum.GOOGLE]: OAuthClass | null;
  private readonly [OAuthProvidersEnum.FACEBOOK]: OAuthClass | null;
  private readonly [OAuthProvidersEnum.GITHUB]: OAuthClass | null;

  // ...
}

We will use a private static method to create these parameters:

// ...
import { isNull } from '../common/utils/validation.util';

@Injectable()
export class Oauth2Service {
  // ...

  constructor(
    // ...
  ) {
    const url = configService.get<string>('url');
    this[OAuthProvidersEnum.MICROSOFT] = Oauth2Service.setOAuthClass(
      OAuthProvidersEnum.MICROSOFT,
      configService,
      url,
    );
    this[OAuthProvidersEnum.GOOGLE] = Oauth2Service.setOAuthClass(
      OAuthProvidersEnum.GOOGLE,
      configService,
      url,
    );
    this[OAuthProvidersEnum.FACEBOOK] = Oauth2Service.setOAuthClass(
      OAuthProvidersEnum.FACEBOOK,
      configService,
      url,
    );
    this[OAuthProvidersEnum.GITHUB] = Oauth2Service.setOAuthClass(
      OAuthProvidersEnum.GITHUB,
      configService,
      url,
    );
  }

  private static setOAuthClass(
    provider: OAuthProvidersEnum,
    configService: ConfigService,
    url: string,
  ): OAuthClass | null {
    const client = configService.get<IClient | null>(
      `oauth2.${provider.toLowerCase()}`,
    );

    if (isNull(client)) {
      return null;
    }

    return new OAuthClass(provider, client, url);
  }
}

And, since the provider can be null, we need to check everytime we call it. Thus, create a wrapper method to get the provider:

// ...
import {
  // ...
  NotFoundException,
} from '@nestjs/common';
// ...

@Injectable()
export class Oauth2Service {
  // ...

  private getOAuth(provider: OAuthProvidersEnum): OAuthClass {
    const oauth = this[provider];

    if (isNull(oauth)) {
      throw new NotFoundException('Page not found');
    }

    return oauth;
  }
}

Core Logic

The core logic is comprised of 4 methods:

Get Authorization URL: to redirect the user to the provider.

// ...

@Injectable()
export class Oauth2Service {
  // ...

  public async getAuthorizationUrl(
    provider: OAuthProvidersEnum,
  ): Promise<string> {
    const [url, state] = this.getOAuth(provider).authorizationUrl;
    // Cache state for 2 minutes
    await this.commonService.throwInternalError(
      this.cacheManager.set(this.getOAuthStateKey(state), provider, 120 * 1000),
    );
    return url;
  }

  private getOAuthStateKey(state: string): string {
    return `oauth_state:${state}`;
  }

  // ...
}

Get Access Token: get the provider's access token given the callback code and state.

// ...

@Injectable()
export class Oauth2Service {
  // ...

  private async getAccessToken(
    provider: OAuthProvidersEnum,
    code: string,
    state: string,
  ): Promise<string> {
    const oauth = this.getOAuth(provider);
    const stateProvider = await this.commonService.throwInternalError(
      this.cacheManager.get<OAuthProvidersEnum>(this.getOAuthStateKey(state)),
);

    if (!stateProvider || provider !== stateProvider) {
      throw new UnauthorizedException('Corrupted state');
    }

    return await this.commonService.throwInternalError(oauth.getToken(code));
  }
}

Get User Data: this is dependent on the previous method, and uses its access token to get the user data from the provider.

// ...
import { AxiosError } from 'axios';
import { catchError, firstValueFrom } from 'rxjs';

@Injectable()
export class Oauth2Service {
  // ...

  public async getUserData<T extends Record<string, any>>(
    provider: OAuthProvidersEnum,
    cbQuery: ICallbackQuery,
  ): Promise<T> {
    const { code, state } = cbQuery;
    const accessToken = await this.getAccessToken(provider, code, state);
    const userData = await firstValueFrom(
      this.httpService
        .get<T>(this.getOAuth(provider).dataUrl, {
          headers: {
            'Content-Type': 'application/json',
            Authorization: `Bearer ${accessToken}`,
          },
        })
        .pipe(
          catchError((error: AxiosError) => {
            throw new UnauthorizedException(error.response.data);
          }),
        ),
    );
    return userData.data;
  }

  // ...
}

Login: finds or creates a new user and generates the Auth tokens.

// ...

@Injectable()
export class Oauth2Service {
  // ...

  public async login(
    provider: OAuthProvidersEnum,
    email: string,
    name: string,
  ): Promise<[string, string]> {
    const user = await this.usersService.findOrCreate(provider, email, name);
    return this.jwtService.generateAuthTokens(user);
  }

  // ...
}

Controller

Before start writing the controllers logic we need to understand the concept of a feature flag, if you ask ChatGPT, it will answer as follows:

A Feature Flag is a software development technique that allows developers to turn specific functionality or features of their application on or off, without deploying new code. It is also known as a feature toggle or feature switch.

A great article on how to implement them in NestJS can be found on the wanago's blog.

OAuth Flag Guard

To use feature flags we will use a mixin guard that checks if provider is null or not:

import {
  CanActivate,
  ExecutionContext,
  Injectable,
  mixin,
  NotFoundException,
  Type,
} from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { FastifyRequest } from 'fastify';
import { isNull } from '../../common/utils/validation.util';
import { OAuthProvidersEnum } from '../../users/enums/oauth-providers.enum';
import { IClient } from '../interfaces/client.interface';

export const OAuthFlagGuard = (
  provider: OAuthProvidersEnum,
): Type<CanActivate> => {
  @Injectable()
  class OAuthFlagGuardClass implements CanActivate {
    constructor(private readonly configService: ConfigService) {}

    public canActivate(context: ExecutionContext): boolean {
      const client = this.configService.get<IClient | null>(
        `oauth2.${provider}`,
      );

      if (isNull(client)) {
        const request = context.switchToHttp().getRequest<FastifyRequest>();
        throw new NotFoundException(`Cannot ${request.method} ${request.url}`);
      }

      return true;
    }
  }

  return mixin(OAuthFlagGuardClass);
};

DTOs

There is only necessity for one DTO, the CallbackQueryDto:

import { IsString } from 'class-validator';
import { ICallbackQuery } from '../interfaces/callback-query.interface';

export abstract class CallbackQueryDto implements ICallbackQuery {
  @IsString()
  public code: string;

  @IsString()
  public state: string;
}

Controller Set-up

Add the same private parameters as the ones in the AuthController, plus an url parameter for the front-end url. Futhermore, inject the Oauth2Service and the ConfigService.

import {
  Controller,
  // ...
  UseGuards,
} from '@nestjs/common';
import { FastifyThrottlerGuard } from '../auth/guards/fastify-throttler.guard';
import { ConfigService } from '@nestjs/config';
import { Oauth2Service } from './oauth2.service';

@ApiTags('Oauth2')
@Controller('api/auth/ext')
@UseGuards(FastifyThrottlerGuard)
export class Oauth2Controller {
  private readonly url: string;
  private readonly cookiePath = '/api/auth';
  private readonly cookieName: string;
  private readonly refreshTime: number;
  private readonly testing: boolean;

  constructor(
    private readonly oauth2Service: Oauth2Service,
    private readonly configService: ConfigService,
  ) {
    this.url = `https://${this.configService.get<string>('domain')}`;
    this.cookieName = this.configService.get<string>('REFRESH_COOKIE');
    this.refreshTime = this.configService.get<number>('jwt.refresh.time');
    this.testing = this.configService.get<boolean>('testing');
  }
}

Controller main logic

The logic for all the routes will be the same, we will:

Temporary redirect the user to the external provider URL:

import {
  // ...
  HttpStatus,
} from '@nestjs/common';
// ...
import { FastifyReply } from 'fastify';

@ApiTags('Oauth2')
@Controller('api/auth/ext')
@UseGuards(FastifyThrottlerGuard)
export class Oauth2Controller {
  // ...

  private async startRedirect(
    res: FastifyReply,
    provider: OAuthProvidersEnum,
  ): Promise<FastifyReply> {
    return res
      .status(HttpStatus.TEMPORARY_REDIRECT)
      .redirect(await this.oauth2Service.getAuthorizationUrl(provider));
  }
}

Recieve the response on our callback URL, and permanent redirect to the front-end with an access and refresh token pair:

// ...

@ApiTags('Oauth2')
@Controller('api/auth/ext')
@UseGuards(FastifyThrottlerGuard)
export class Oauth2Controller {
  // ...

  private async loginAndRedirect(
    res: FastifyReply,
    provider: OAuthProvidersEnum,
    email: string,
    name: string,
  ): Promise<FastifyReply> {
    const [accessToken, refreshToken] = await this.oauth2Service.login(
      provider,
      email,
      name,
    );
    return res
      .cookie(this.cookieName, refreshToken, {
        secure: !this.testing,
        httpOnly: true,
        signed: true,
        path: this.cookiePath,
        expires: new Date(Date.now() + this.refreshTime * 1000),
  })
      .status(HttpStatus.PERMANENT_REDIRECT)
      .redirect(`${this.url}/?access_token=${accessToken}`);
  }
}

Controller routes

Create a redirect route for each provider:

import {
  // ...
  Query,
  Res,
  // ...
} from '@nestjs/common';
// ...
import { ApiNotFoundResponse, ApiResponse, ApiTags } from '@nestjs/swagger';
import { Public } from '../auth/decorators/public.decorator';
import { CallbackQueryDto } from './dtos/callback-query.dto';
import { OAuthFlagGuard } from './guards/oauth-flag.guard';
// ...

@ApiTags('Oauth2')
@Controller('api/auth/ext')
@UseGuards(FastifyThrottlerGuard)
export class Oauth2Controller {
  // ...

  @Public()
  @UseGuards(OAuthFlagGuard(OAuthProvidersEnum.MICROSOFT))
  @Get('microsoft')
  @ApiResponse({
    description: 'Redirects to Microsoft OAuth2 login page',
    status: HttpStatus.TEMPORARY_REDIRECT,
  })
  @ApiNotFoundResponse({
    description: 'OAuth2 is not enabled for Microsoft',
  })
  public async microsoft(@Res() res: FastifyReply): Promise<FastifyReply> {
    return this.startRedirect(res, OAuthProvidersEnum.MICROSOFT);
  }

  @Public()
  @UseGuards(OAuthFlagGuard(OAuthProvidersEnum.GOOGLE))
  @Get('google')
  @ApiResponse({
    description: 'Redirects to Google OAuth2 login page',
    status: HttpStatus.TEMPORARY_REDIRECT,
  })
  @ApiNotFoundResponse({
    description: 'OAuth2 is not enabled for Google',
  })
  public async google(@Res() res: FastifyReply): Promise<FastifyReply> {
    return this.startRedirect(res, OAuthProvidersEnum.GOOGLE);
  }

  @Public()
  @UseGuards(OAuthFlagGuard(OAuthProvidersEnum.FACEBOOK))
  @Get('facebook')
  @ApiResponse({
    description: 'Redirects to Facebook OAuth2 login page',
    status: HttpStatus.TEMPORARY_REDIRECT,
  })
  @ApiNotFoundResponse({
    description: 'OAuth2 is not enabled for Facebook',
  })
  public async facebook(@Res() res: FastifyReply): Promise<FastifyReply> {
    return this.startRedirect(res, OAuthProvidersEnum.FACEBOOK);
  }

  @Public()
  @UseGuards(OAuthFlagGuard(OAuthProvidersEnum.GITHUB))
  @Get('github')
  @ApiResponse({
    description: 'Redirects to GitHub OAuth2 login page',
    status: HttpStatus.TEMPORARY_REDIRECT,
  })
  @ApiNotFoundResponse({
    description: 'OAuth2 is not enabled for GitHub',
  })
  public async github(@Res() res: FastifyReply): Promise<FastifyReply> {
    return this.startRedirect(res, OAuthProvidersEnum.GITHUB);
  }

  // ...
}

And a callback route for each provider:

// ...
import {
  IFacebookUser,
  IGitHubUser,
  IGoogleUser,
  IMicrosoftUser,
} from './interfaces/user-response.interface';

@ApiTags('Oauth2')
@Controller('api/auth/ext')
@UseGuards(FastifyThrottlerGuard)
export class Oauth2Controller {
  // ...

  @Public()
  @UseGuards(OAuthFlagGuard(OAuthProvidersEnum.MICROSOFT))
  @Get('microsoft/callback')
  @ApiResponse({
    description: 'Redirects to the frontend with the JWT token',
    status: HttpStatus.PERMANENT_REDIRECT,
  })
  @ApiNotFoundResponse({
    description: 'OAuth2 is not enabled for Microsoft',
  })
  public async microsoftCallback(
    @Query() cbQuery: CallbackQueryDto,
    @Res() res: FastifyReply,
  ): Promise<FastifyReply> {
    const provider = OAuthProvidersEnum.MICROSOFT;
    const { displayName, mail } =
      await this.oauth2Service.getUserData<IMicrosoftUser>(provider, cbQuery);
    return this.loginAndRedirect(res, provider, mail, displayName);
  }

  @Public()
  @UseGuards(OAuthFlagGuard(OAuthProvidersEnum.GOOGLE))
  @Get('google/callback')
  @ApiResponse({
    description: 'Redirects to the frontend with the JWT token',
    status: HttpStatus.PERMANENT_REDIRECT,
  })
  @ApiNotFoundResponse({
    description: 'OAuth2 is not enabled for Google',
  })
  public async googleCallback(
    @Query() cbQuery: CallbackQueryDto,
    @Res() res: FastifyReply,
  ): Promise<FastifyReply> {
    const provider = OAuthProvidersEnum.GOOGLE;
    const { name, email } = await this.oauth2Service.getUserData<IGoogleUser>(
      provider,
      cbQuery,
    );
    return this.loginAndRedirect(res, provider, email, name);
  }

  @Public()
  @UseGuards(OAuthFlagGuard(OAuthProvidersEnum.FACEBOOK))
  @Get('facebook/callback')
  @ApiResponse({
    description: 'Redirects to the frontend with the JWT token',
    status: HttpStatus.PERMANENT_REDIRECT,
  })
  @ApiNotFoundResponse({
    description: 'OAuth2 is not enabled for Facebook',
  })
  public async facebookCallback(
    @Query() cbQuery: CallbackQueryDto,
    @Res() res: FastifyReply,
  ): Promise<FastifyReply> {
    const provider = OAuthProvidersEnum.FACEBOOK;
    const { name, email } = await this.oauth2Service.getUserData<IFacebookUser>(
      provider,
      cbQuery,
    );
    return this.loginAndRedirect(res, provider, email, name);
  }

  @Public()
  @UseGuards(OAuthFlagGuard(OAuthProvidersEnum.GITHUB))
  @Get('github/callback')
  @ApiResponse({
    description: 'Redirects to the frontend with the JWT token',
    status: HttpStatus.PERMANENT_REDIRECT,
  })
  @ApiNotFoundResponse({
    description: 'OAuth2 is not enabled for GitHub',
  })
  public async githubCallback(
    @Query() cbQuery: CallbackQueryDto,
    @Res() res: FastifyReply,
  ): Promise<FastifyReply> {
    const provider = OAuthProvidersEnum.GITHUB;
    const { name, email } = await this.oauth2Service.getUserData<IGitHubUser>(
      provider,
      cbQuery,
    );
    return this.loginAndRedirect(res, provider, email, name);
  }

  // ...
}

Conclusion

This is the end of my OAuth 2.0 series with NestJS, with this series you have learnt how to create a production grade OAuth 2.0 microservice using NodeJS, with both local and external OAuth.

The full source code can be found on this repo.

About the Author

Do not miss out on any of my latest articles – follow me here on dev, on twitter and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

What is your opinion on the new "Rust Trademark Policy"?

Afonso Barracha — Mon, 10 Apr 2023 18:45:25 +0000

Firstly sorry for the formality of the post but today, I took the time to examine the new Rust Trademark Policy, which can be accessed through the Rust Trademark Policy Comment Form.

If you are a Rustacean, or are on the process of turning into one, please review it. The more we make our voice heard the easier they will amend this policy.

The feedback that I left was:

As a content creator, particularly a blog writer, I have some concerns regarding the implications of this policy on my hobby. These concerns include:

The requirement to include a disclaimer alongside the Rust Logo on tutorials and blog posts, clarifying that they are not endorsed by the Rust Foundation. This stipulation may be perceived as unnecessary since most readers are aware that online tutorials are typically created by community members rather than the foundation itself.

The inability to use the Rust Logo as a selling point for Software as a Service (SaaS) products. For instance, if I were to develop a project using the FAST stack (Flutter, Actix, SeaORM, and Tokio Runtime) and make it publicly available, I would not be allowed to display the Rust Logo on the landing page.

The policy dictates that the Rust Logo must be smaller than the hosting website's logo, which could result in a deviation from modern design principles. Consequently, if you have an educational platform with a dedicated section for Rust, the Rust Logo must be smaller than the logo displayed on the navigation bar.

The policy prohibits the inclusion of the word "Rust" in package, repository, library, and crate names. While I understand the need to differentiate between official and unofficial packages, enforcing this rule through a trademark may not be the most effective approach. A tacit agreement within the community could potentially suffice.

While I may not possess a strong artistic inclination, the restriction on modifying the logo beyond scaling, including colour changes, could potentially impede the creative freedom of other content creators.

In conclusion, while the Rust Trademark Policy aims to protect the Rust brand, it is essential to consider its impact on content creators, who play a vital role in promoting and expanding the Rust ecosystem. Balancing the needs of the Rust Foundation and the creator community is crucial for fostering a healthy and collaborative environment.

These were my reservations, what are yours?

Rust Keyword Extraction: Creating the TF-IDF algorithm from scratch (and a Tokenizer)

Afonso Barracha — Sun, 02 Apr 2023 14:03:38 +0000

Disclosure: The material in this Tutorial has not been reviewed, endorsed, or approved of by the Rust Foundation. For more information on the Rust Foundation Trademark Policy, click here.

Series Intro

I tried unsupervised NLP models and keyword extraction, but even those were too slow for a simple VPS. That is when I realised Rust and Actix-Web could be the answer.

Therefore, let's explore Rust for NLP together and dive into the awesome world of text mining and keyword extraction.

What will I cover in this series

I will mainly cover 3 algorithms:

TF-IDF: Term Frequency-Inverse Document Frequency algorithm converts raw text into insightful numerical data.
Co-occurrence Matrix: the Co-occurrence Matrix covers underlying patterns and associations in a language by examining word pairs and building semantic networks in textual data.
RAKE: Rapid Automatic Keyword Extraction Algorithm identifies crucial words and phrases to enhance text summarization, topic modeling, and information retrieval in any text-based project.

Intro

Before implementing the TF-IDF algorithm we have to know what it is, and what it does. Term Frequency-Inverse Document Frequency algorithm uses a weighting system that weights the importance of each word in a group of documents based on two components:

Term Frequency (TF): this element calculates how often a term appears in a document. It is calculated by dividing the word count by the count of all words.
Inverse Document Frequency (IDF): calculates the importance of a word relative to the corpus of the document. It is calculated as the logarithm of the total number of documents in the corpus divided by the number of documents in which the word appears.

Steps to implement TF-IDF

Note: if you do not have time to read this entire article, you can find all the code in this repository. While at it consider buying me a coffee if you liked what you saw.

Here are the steps to implement TF-IDF:

Tokenize the documents by converting them into a vector of words, and then remove punctuation and stop-words. Stop-words are common, language-specific words that do not carry much meaning and can typically be ignored.
Calculate the Term Frequency (TF) for each word.
Calculate the Inverse Document Frequency (IDF) for each word.
Multiply the TF by IDF to get the Term Frequency-Inverse Document Frequency (TF-IDF) for each word.
L2 (Pythagoras Theorem type of vector distance) normalize the base TF-IDF calculation

Implementation

Let start by creating a new RUST project with Cargo:

$ cargo new keyword_extraction --lib

Since we will have several algorithms we can take advantage of the cargo workspaces functionality.

Text process

Tokenizer initialisation

Create a library called tokenizer:

$ cargo new tokenizer --lib

And add it to the main Cargo.toml file:

[package]
name = "keyword_extraction"
version = "0.1.0"
edition = "2021"
license = "GPL-3.0-or-later"
authors = ["Afonso Barracha <barracha.afonso@gmail.com>"]
publish = false

[lib]
name = "keyword_extraction"
path = "src/lib.rs"

[workspace]
members = [
    ".",
    "tokenizer"
]

[dependencies]
tokenizer = { path = "./tokenizer" }

And update the tokenizer's Cargo.toml:

[package]
name = "tokenizer"
version = "0.1.0"
edition = "2021"
publish = false
authors = ["Afonso Barracha <barracha.afonso@gmail.com>"]
description = "A simple tokenizer for the Rust programming language"
license = "GPL-3.0-or-later"

[lib]
name = "tokenizer"
path = "src/lib.rs"

To process the document we will use a common used library for string processing unicode_segmentation, as the description of the library states:

This crate provides Grapheme Cluster, Word and Sentence boundaries according to Unicode Standard Annex #29 rules.

So we will use it to break (tokenize) the block of text (article, job offer, etc.) into sentences, paragraphs and words.

Start by adding this and the regex library to the Cargo.toml file:

# ...

[dependencies]
unicode-segmentation = "^1"
regex = "^1"

On the top of the lib.rs file import UnicodeSegmentation trait from the this crate, the Regex struct from the regex crate, and the HashSet from the standard library:

use std::collections::HashSet;

use regex::Regex;
use unicode_segmentation::UnicodeSegmentation;

On the lib.rs file add a new struct that contains the text, the stop-words and punctuation used in the text:

pub struct Tokenizer {
    text: String,
    stopwords: HashSet<String>,
    punctuation: HashSet<String>,
}

On the struct implement a new method for its initialisation:

impl Tokenizer {
    pub fn new(text: &str, stopwords: Vec<String>, punctuation: Option<Vec<String>>) -> Self {
        Self {
            text: text.to_owned(),
            stopwords: stopwords
                .iter()
                .map(|s| s.to_owned())
                .collect::<HashSet<String>>(),
            punctuation: punctuation
                .unwrap_or(
                    vec![
                        "!", "\"", "#", "$", "%", "&", "'", "(", ")", "*", "+", ",", ";", ".", "/",
                        ":", ",", "<", "=", ">", "?", "@", "[", "\\", "]", "^", "_", "`", "{", "|",
                        "}", "~", "-",
                    ]
                    .iter()
                    .map(|s| s.to_string())
                    .collect::<Vec<String>>(),
                )
                .iter()
                .map(|s| s.to_string())
                .collect::<HashSet<String>>(),
        }
    }
}

Punctuation for most languages are the same so we make it optional and use the vec! macro to initialise the most common characters.

Using the iterator pattern we turn the vectors of stop-words and punctuation into hash sets for easy and efficient lookup.

Tokenizer Word split

The UnicodeSegmentation trait gives strings a very cool method, namely split_word_bounds() which breaks our block of text into a Vec<String> where words are separated from most punctuation characters, however some characters like "'s", "." and "," still need to be filtered so create a function to create a regex to filter those characters:

// ...

fn get_special_char_regex() -> Regex {
    Regex::new(r"('s|,|\.)").unwrap()
}

Finally to process and filter those words we will use the iterator trait filter_map method,

Create a helper function that returns the Option<String> monad:

fn process_word(
    w: &str,
    special_char_regex: &Regex,
    stopwords: &HashSet<String>,
    punctuation: &HashSet<String>,
) -> Option<String> {
    let word = special_char_regex.replace_all(w.trim(), "").to_lowercase();

    if word.is_empty()
        || (word.graphemes(true).count() == 1) && punctuation.contains(&word)
        || stopwords.contains(&word)
    {
        return None;
    }

    Some(word)
}

We do 4 actions on that function, we:

Process the word by removing the special characters and turning it into lowercase:
```
let word = special_char_regex.replace_all(w.trim(), "").to_lowercase();
```
We remove filter the word out if it is an empty string:
```
if word.is_empty() {
    return None;
}
```
If the string is one character long we check if it is a punctuation character (that was split by the split_word_bounds() method):
```
if (word.graphemes(true).count() == 1) && punctuation.contains(&word) {
    return None;
}
```
Note: the graphemes method is provided by the UnicodeSegmentation trait as well

We check if the word is a stop word, and filter it if it is:

if stopwords.contains(&word) {
    return None;
}

Now we can use the function inside an iterator on methods of our Tokenizer trait, like the one to split into words:

impl Tokenizer {
    // ...

    pub fn split_into_words(&self) -> Vec<String> {
        self.text
            .split_word_bounds()
            .filter_map(|w| {
                process_word(
                    w,
                    &get_special_char_regex(),
                    &self.stopwords,
                    &self.punctuation,
                )
            })
            .collect::<Vec<String>>()
    }
}

The method is pretty self explanatory, we split_word_bounds to get an iterator of words, we filter_map and use the process_word function to process the words, and we collect everything into a Vec<String>.

Tokenizer Sentence Split

To split our we code into documents we will need to split the text block into sentences, again UnicodeSegmentation has a good method for that unicode_sentences(). But before that we need to remove new lines ("\n" and "\r") and tabs ("\t"), so create a regex for that:

fn get_sentence_space_regex() -> Regex {
    Regex::new(r"^([\.!?])[\n\t\r]").unwrap()
}

And use it to filter them, like the words, we still need to remove the stop words:

impl Tokenizer {
    // ...

    pub fn split_into_sentences(&self) -> Vec<String> {
        let special_char_regex = get_special_char_regex();
        get_sentence_space_regex()
            .replace_all(&self.text, ".")
            .unicode_sentences()
            .map(|s| {
                s.split_word_bounds()
                    .filter_map(|w| {
                        process_word(w, &special_char_regex, &self.stopwords, &self.punctuation)
                    })
                    .collect::<Vec<String>>()
                    .join(" ")
            })
            .collect::<Vec<String>>()
    }
}

The code is pretty much the same as the previous section, but now we have to split it in sentences, and for each sentence we need to process the words.

Tokenizer Paragraph Split

To split our code into paragraphs we just need to divide our code with the delimiter of new lines, so create an Regex for new lines:

fn get_newline_regex() -> Regex {
    Regex::new(r"(\r|\n|\r\n)").unwrap()
}

And apply it to the text:

impl Tokenizer {
    // ...

    pub fn split_into_paragraphs(&self) -> Vec<String> {
        get_newline_regex()
            .split(&self.text)
            .filter_map(|s| {
                if s.trim().is_empty() {
                    return None;
                }

                Some(
                    s.unicode_sentences()
                        .map(|s| {
                            s.split_word_bounds()
                                .filter_map(|w| {
                                    process_word(
                                        w,
                                        &get_special_char_regex(),
                                        &self.stopwords,
                                        &self.punctuation,
                                    )
                                })
                                .collect::<Vec<String>>()
                                .join(" ")
                        })
                        .collect::<Vec<String>>()
                        .join(" "),
                )
            })
            .collect::<Vec<String>>()
    }
}

Note that now we need three iterators, one for the paragraphs, one for the sentences, and one for the words of the sentences.

Calculate TF-IDF

At the end of the day TF-IDF is just a fancy HashMap that has the words as keys, and a 32 bits float (f32) score as values.

Start by generating a tf_idf library:

$ cargo new tf_idf --lib

And import the necessary libraries, including the Tokenizer library:

use std::cmp::Ordering;
use std::collections::{HashMap, HashSet};

use tokenizer::Tokenizer;

Create a tuple struct TfIdf that accepts a single HashMap params:

pub struct TfIdf(HashMap<String, f64>);

Calculate TF

To calculate the TF we have to create two functions.

This function takes the documents and returns a word hash map with the number of occurence of each word in all documents:

fn generate_word_hashmap(documents: &Vec<String>) -> HashMap<String, f32> {
    documents
        .iter()
        .flat_map(|document| document.split_whitespace().map(|word| word.to_string()))
        .fold(HashMap::new(), |mut acc, word| {
            let count = acc.entry(word).or_insert(0.0);
            *count += 1.0;
            acc
        })
}

While this one takes a mutable hash map and returns a word hash map with its TF score:

fn calculate_tf(tf: HashMap<String, f32>) -> HashMap<String, f32> {
    let total_words = tf.values().sum::<f32>();

    tf.iter()
        .map(|(word, count)| (word.to_string(), count / total_words))
        .collect::<HashMap<String, f32>>()
}

Calculate IDF

You need to two functions to calculate the IDF score as well:

One takes the documents, and returns on how many documents each word appeared:

fn generate_unique_word_hashmap(documents: &Vec<String>) -> HashMap<String, f32> {
    documents
        .iter()
        .map(|document| {
            document
                .split_whitespace()
                .map(|s| s.to_owned())
                .collect::<HashSet<String>>()
        })
        .flat_map(|unique_words| unique_words.into_iter())
        .fold(HashMap::new(), |mut acc, word| {
            let count = acc.entry(word).or_insert(0.0);
            *count += 1.0;
            acc
        })
}

The other takes the documents vector length, a hash map with the unique words and returns a hash map of IDF scores:

fn calculate_idf(docs_len: f32, word_hashmap: HashMap<String, f32>) -> HashMap<String, f32> {
    let one = 1.0_f32;

    word_hashmap
        .iter()
        .map(|(word, count)| {
            let documents_with_term = (docs_len + one) / (count + one);
            (word.to_string(), documents_with_term.ln() + one)
        })
        .collect::<HashMap<String, f32>>()
}

Calculate the TF-IDF

The TF-IDF is just a hash map of the product of the previous two hash maps:

fn calculate_tf_idf(tf: HashMap<String, f32>, idf: HashMap<String, f32>) -> HashMap<String, f32> {
    tf.iter()
        .map(|(word, count)| (word.to_string(), count * idf.get(word).unwrap_or(&0.0_f32)))
        .collect::<HashMap<String, f32>>()
}

L2 Normalization

L2 norm, or Euclidean norm, is the distance of two vectors coordinates within a vector space. It is basically the same as the Pythagoras Theorem but for all the values in the vector:

l2 = sqrt(x*x + y*y + ...)

So the function that calculates this is something like:

fn l2_normalize(tf_id: HashMap<String, f32>) -> HashMap<String, f32> {
    let l2_norm = tf_id
        .values()
        .map(|value| value * value)
        .sum::<f32>()
        .sqrt();

    tf_id
        .iter()
        .map(|(key, value)| (key.clone(), value / l2_norm))
        .collect::<HashMap<String, f32>>()
}

Putting it all together

To initialise the TfIdf struct implement a new method:

impl TfIdf {
    pub fn new(documents: &Vec<String>) -> Self {
        Self(l2_normalize(calculate_tf_idf(
            calculate_tf(generate_word_hashmap(documents)),
            calculate_idf(
                documents.len() as f32,
                generate_unique_word_hashmap(documents),
            ),
        )))
    }
}

Adding the Final Touches

As you can see on our struct the only param is private, hence we need methods to access its data, one for checking the score for a single word, and one to get the top N words.

The get_score method, is just an wrapper on the hash map:

impl TfIdf {
    // ...

    pub fn get_score(&self, word: &str) -> f32 {
        *self.0.get(word).unwrap_or(&0.0)
    }
}

While the get_n_best gets the top n words:

impl TfIdf {
    // ...

    pub fn get_n_best(&self, n: usize) -> Vec<(String, f32)> {
        let mut sorted_tfidf = self.0.iter().collect::<Vec<(&String, &f32)>>();
        sorted_tfidf.sort_by(|a, b| {
            let order = b.1.partial_cmp(&a.1).unwrap_or(Ordering::Equal);

            if order == Ordering::Equal {
                return a.0.cmp(&b.0);
            }

            order
        });
        sorted_tfidf
            .iter()
            .take(n)
            .map(|(word, score)| (word.to_string(), **score))
            .collect::<Vec<(String, f32)>>()
    }
}

Conclusion

In this article we learn how to implement one of the most common algorithms in NLP keyword extraction, the TF-IDF algorithm. This article is particularly useful for developing text based Data Analysis services like article, job offers, etc, analysers.

All the code from this article can be found here.

About the Author

Do not miss out on any of my latest articles – follow me here on dev and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

NestJS Authentication with OAuth2.0: Apollo Local OAuth GraphQL API

Afonso Barracha — Sun, 26 Mar 2023 12:26:27 +0000

Hey there, dear readers! I apologise for the delayed Article. I had been traveling abroad for almost a month, and upon returning, this series slipped my conscious. I am sorry for any frustration that this may have caused.

We are nearing the end of this series, with only two more articles to go (including this one). I am committed to delivering them before mid-April, so you will not have to wait too long. But hold on, the excitement does not end there! I have got big plans for the future of my blog.

In fact, I am thrilled to announce that I'll be introducing two new series:

Rust Algorithm Development: In this series, I will be guiding you through implementing some basic NLP algorithms from scratch. Specifically, TF-IDF, Co-occurrence Matrix and RAKE. I know it might seem a bit random, but since machine learning is becoming more prevalent in our lives, I thought it would be fun to dip my toes into the topic since I used to be a Data Analyst (technically an Econometrician)
Advance GraphQL with Mercurius and NestJS: Throughout this series, I will be your guide as we develop a Discord clone. We will start by creating a monolithic back-end and then divide it into microservices to make it more scalable and future proof.

I hope you all enjoy the remainder of the series, and thank you for your understanding!

Series Intro

This series will cover the full implementation of OAuth2.0 Authentication in NestJS for the following types of APIs:

Express REST API;
Fastify REST API;
Apollo GraphQL API.

And it is divided in 5 parts:

Configuration and operations;
Express Local OAuth REST API;
Fastify Local OAuth REST API;
Apollo Local OAuth GraphQL API;
Adding External OAuth Providers to our API;

Lets start the fourth part of this series.

Tutorial Intro

On this tutorial we will look at the previous Express REST API tutorial, and transform it to an Apollo GraphQL API.

DISCLAIMER: Although it is possible to have an OAuth System in GraphQL, it is recommended to use REST architecture instead. This article is for educational purposes only. Please do not use this code in Production. To discourage the use of this code in production I removed all unit tests from it

NOTE: if you do not have time to read the entire article the repository can be found here. Do not use this code in production, and please remember that there is nothing wrong with having a Hybrid API, with REST for Authentication and GraphQL for the rest of the API, that is what is recommended. While you at it consider buying me a coffee.

Set up

Disclaimers aside, start by installing the required libraries for GraphQL development:

$ yarn add @nestjs/graphql @nestjs/apollo @apollo/server graphql @apollo/federation @apollo/subgraph

On the config directory create a new file for the GraphQL configuration:

import { ApolloDriver, ApolloDriverConfig } from '@nestjs/apollo';
import { ConfigService } from '@nestjs/config';
import { GqlOptionsFactory } from '@nestjs/graphql';
import { IContext } from './interfaces/context.interface';

export class GraphQLConfig implements GqlOptionsFactory {
  private readonly testing: boolean;

  constructor(private readonly configService: ConfigService) {
    this.testing = this.configService.get('testing');
  }

  public createGqlOptions(): ApolloDriverConfig {
    return {
      driver: ApolloDriver,
      context: ({ req, res }): IContext => ({
        req,
        res,
      }),
      path: '/api/graphql',
      autoSchemaFile: './schema.gql',
      sortSchema: true,
      playground: this.testing,
      introspection: true,
    };
  }
}

And register the GraphQLModule with it in the AppModule:

// ...
import { GraphQLConfig } from './config/graphql.config';
// ...

@Module({
  imports: [
    // ...
    GraphQLModule.forRootAsync({
      imports: [ConfigModule],
      driver: ApolloDriver,
      useClass: GraphQLConfig,
    }),
    // ...
  ],
  // ...
})
export class AppModule {}

Differences overview

If you are reading this article, I will assume that you already have experience developing GraphQL APIs using NestJS, but to summarise there are a few differences from REST to GraphQL:

We use Resolvers and not Controllers to write Mutations (equivalent to POST, PATCH & Delete) and Queries (GET equivalent);
Request and Response are not passed directly to the Resolver, but through the Context parameter.

Having this in mind, going from REST to GraphQL is not that hard.

Resolvers

In general before developing our resolvers we need to create:

Object Types:
- How our entities maps to the GraphQL schema;
- Impl: add @ObjectType() decorator on the entity.
Args Types:
- The arguments that go into our Queries (like array fields in Python *args);
- Impl: add @ArgsType() decorator on DTOs.
Input Types:
- This are Input Objects where a single argument will accept an object (like named parameters in Python **kwargs);
- Impl: add @InputType() decorator on DTOs.
Fields:
- For any field that should be mapped on any kind of object or DTO;
- Impl: add @Field decorator on each parameter.

As for standards and rules we will use the Apollo recommendations and be explicit with our queries and mutations.

Updates

Common

On the common service start by creating a new entities folder with a gql folder inside and add a MessageType there, based on the MessageMapper:

message.type.ts:

import { Field, ObjectType } from '@nestjs/graphql';
import { ApiProperty } from '@nestjs/swagger';
import { v4 } from 'uuid';
import { IMessage } from '../../interfaces/message.interface';

@ObjectType('Message')
export class MessageType implements IMessage {
  @Field(() => String)
  public id: string;

  @Field(() => String)
  @ApiProperty({
    description: 'Message',
    example: 'Hello World',
    type: String,
  })
  public message: string;

  constructor(message: string) {
    this.id = v4();
    this.message = message;
  }
}

We need a new DTO as well, the IdDto, start by creating a dtos folder and add the id.dto.ts there:

import { ArgsType, Field, Int } from '@nestjs/graphql';
import { IsInt, Min } from 'class-validator';

@ArgsType()
export abstract class IdDto {
  @Field(() => Int)
  @IsInt()
  @Min(1)
  public id: number;
}

Note that if your service should allow to filter users consider reading my article on cursor pagination. Shameless plug I know

Lastly since we are creating a micro-service we need to Federate our schema, so create a IFederatedInstance interface on the interfaces folder:

export interface IFederatedInstance<T extends string> {
  readonly __typename: T;
  readonly id: number;
}

Read more about apollo federation on the apollo docs.

Users

Entities

Start by adding the @ObjectType decorator to the UserEntity, with the @Field decorator for every field you want to be queryable:

import { Embedded, Entity, PrimaryKey, Property } from '@mikro-orm/core';
import { Directive, Field, Int, ObjectType } from '@nestjs/graphql';
import { IsBoolean, IsEmail, IsString, Length, Matches } from 'class-validator';
import {
  BCRYPT_HASH,
  NAME_REGEX,
  SLUG_REGEX,
} from '../../common/consts/regex.const';
import { CredentialsEmbeddable } from '../embeddables/credentials.embeddable';
import { IUser } from '../interfaces/user.interface';
import { privateMiddleware } from '../middleware/private.middleware';

@ObjectType('User')
@Directive('@key(fields: "id")')
@Entity({ tableName: 'users' })
export class UserEntity implements IUser {
  @Field(() => Int)
  @PrimaryKey()
  public id: number;

  @Field(() => String)
  @Property({ columnType: 'varchar', length: 100 })
  @IsString()
  @Length(3, 100)
  @Matches(NAME_REGEX, {
    message: 'Name must not have special characters',
  })
  public name: string;

  @Field(() => String)
  @Property({ columnType: 'varchar', length: 106 })
  @IsString()
  @Length(3, 106)
  @Matches(SLUG_REGEX, {
    message: 'Username must be a valid slug',
  })
  public username: string;

  @Field(() => String, { nullable: true, middleware: [privateMiddleware] })
  @Property({ columnType: 'varchar', length: 255 })
  @IsString()
  @IsEmail()
  @Length(5, 255)
  public email: string;

  @Property({ columnType: 'varchar', length: 60 })
  @IsString()
  @Length(59, 60)
  @Matches(BCRYPT_HASH)
  public password: string;

  @Property({ columnType: 'boolean', default: false })
  @IsBoolean()
  public confirmed: true | false = false;

  @Embedded(() => CredentialsEmbeddable)
  public credentials: CredentialsEmbeddable = new CredentialsEmbeddable();

  @Property({ onCreate: () => new Date() })
  public createdAt: Date = new Date();

  @Property({ onUpdate: () => new Date() })
  public updatedAt: Date = new Date();
}

In order for only the current logged user to have access to his own email we need to create a privateMiddleware, add it to new a middleware directory:

import { FieldMiddleware, MiddlewareContext } from '@nestjs/graphql';
import { isNull, isUndefined } from '../../common/utils/validation.util';
import { IContext } from '../../config/interfaces/context.interface';
import { IUser } from '../interfaces/user.interface';

export const privateMiddleware: FieldMiddleware = async (
  ctx: MiddlewareContext<IUser, IContext, unknown>,
  next,
) => {
  const user = ctx.context.req.user;

  if (isUndefined(user) || isNull(user) || ctx.source.id !== user) {
    return null;
  }

  return next();
};

Users Service

Before defining the new users.resolver.ts file we need to do some changes to UsersService based on being explicit with our mutation, hence we need to divide the update method into each user's parameter:

updateName:

@Injectable()
export class UsersService {
  // ...

  public async updateName(userId: number, name: string): Promise<UserEntity> {
    const user = await this.findOneById(userId);
    const formattedName = this.commonService.formatName(name);

    if (user.name === formattedName) {
      throw new BadRequestException('Name must be different');
    }

    user.name = formattedName;
    await this.commonService.saveEntity(this.usersRepository, user);
    return user;
  }

  // ...
}

updateUsername:

@Injectable()
export class UsersService {
  // ...

  public async updateUsername(
    userId: number,
    username: string,
  ): Promise<UserEntity> {
    const user = await this.findOneById(userId);
    const formattedUsername = username.toLowerCase();

    if (user.username === formattedUsername) {
      throw new BadRequestException('Username should be different');
    }

    await this.checkUsernameUniqueness(formattedUsername);
    user.username = formattedUsername;
    await this.commonService.saveEntity(this.usersRepository, user);
    return user;
  }

  // ...
}

DTOs

On the DTOs we created in the previous tutorials, and new DTOs you will need to add the @ArgsType() decorator. So the old decorators would be something like this:

PasswordDto on password.dto.ts:

import { ArgsType, Field } from '@nestjs/graphql';
import { ApiProperty } from '@nestjs/swagger';
import { IsString, MinLength } from 'class-validator';

@ArgsType()
export abstract class PasswordDto {
  @Field(() => String)
  @ApiProperty({
    description: 'The password of the user',
    minLength: 1,
    type: String,
  })
  @IsString()
  @MinLength(1)
  public password: string;
}

ChangeEmailDto on change-email.dto.ts;

import { ArgsType, Field } from '@nestjs/graphql';
import { ApiProperty } from '@nestjs/swagger';
import { IsEmail, IsString, Length } from 'class-validator';
import { PasswordDto } from './password.dto';

@ArgsType()
export abstract class ChangeEmailDto extends PasswordDto {
  @Field(() => String)
  @ApiProperty({
    description: 'The email of the user',
    example: 'someone@gmail.com',
    minLength: 5,
    maxLength: 255,
    type: String,
  })
  @IsString()
  @IsEmail()
  @Length(5, 255)
  public email: string;
}

As for new create a NameDto and UsernameDto:

name.dto.ts:

import { ArgsType, Field } from '@nestjs/graphql';
import { IsString, Length, Matches } from 'class-validator';
import { NAME_REGEX } from '../../common/consts/regex.const';

@ArgsType()
export abstract class NameDto {
  @Field(() => String)
  @IsString()
  @Length(3, 100)
  @Matches(NAME_REGEX, {
    message: 'Name must not have special characters',
  })
  public name: string;
}

username.dto.ts:

import { ArgsType, Field } from '@nestjs/graphql';
import { IsString, Length, Matches } from 'class-validator';
import { SLUG_REGEX } from '../../common/consts/regex.const';

@ArgsType()
export abstract class UsernameDto {
  @Field(() => String)
  @IsString()
  @Length(3, 106)
  @Matches(SLUG_REGEX, {
    message: 'Username must be a valid slug',
  })
  public username: string;
}

Resolver

To create the users.resolver.ts run the following command:

$ nest g r users

Now add a return type for our resolver and import the user service:

import { Resolver } from '@nestjs/graphql';
import { UserEntity } from './entities/user.entity';
import { UsersService } from './users.service';

@Resolver(() => UserEntity)
export class UsersResolver {
  constructor(private readonly usersService: UsersService) {}
}

On the resolver we need to add two queries and four mutations.

Queries:

User by ID:

import {
  Args,
  Query,
  // ...
} from '@nestjs/graphql';
import { Public } from '../auth/decorators/public.decorator';
import { UserEntity } from './entities/user.entity';
// ...

@Resolver(() => UserEntity)
export class UsersResolver {
  // ...

  @Public()
  @Query(() => UserEntity)
  public async userById(@Args() idDto: IdDto): Promise<UserEntity> {
    return await this.usersService.findOneById(idDto.id);
  }

}

User by username:

import {
  Args,
  Query,
  // ...
} from '@nestjs/graphql';
import { Public } from '../auth/decorators/public.decorator';
import { UserEntity } from './entities/user.entity';
// ...

@Resolver(() => UserEntity)
export class UsersResolver {
  // ...

  @Public()
  @Query(() => UserEntity)
  public async userByUsername(
    @Args() usernameDto: UsernameDto,
  ): Promise<UserEntity> {
    return await this.usersService.findOneByUsername(usernameDto.username);
  }

}

Mutations

For mutations is a bit more complicated, specially because of delete. First we need to change the cookiePath to the GraphQL path '/api/graphl' so the cookie is blacklisted when the user deletes his or her account.

import { ConfigService } from '@nestjs/config';
// ...

@Resolver(() => UserEntity)
export class UsersResolver {
  private readonly cookiePath = '/api/graphql';
  private readonly cookieName: string;

  constructor(
    private readonly usersService: UsersService,
    private readonly configService: ConfigService,
  ) {
    this.cookieName = this.configService.get<string>('REFRESH_COOKIE');
  }

  // ...
}

Finally, start adding the delete mutation:

// ...
import {
  // ...
  Context,
  Mutation,
  // ...
} from '@nestjs/graphql';
import { MessageType } from '../common/entities/gql/message.type';
import { PasswordDto } from './dtos/password.dto';
// ...

@Resolver(() => UserEntity)
export class UsersResolver {
  // ...

  @Mutation(() => MessageType)
  public async deleteUser(
    @Context('res') res: Response,
    @CurrentUser() id: number,
    @Args() passwordDto: PasswordDto,
  ): Promise<MessageType> {
    await this.usersService.delete(id, passwordDto);
    res.clearCookie(this.cookieName, { path: this.cookiePath });
    return new MessageType('User deleted successfully');
  }
}

As you can see we use the context to pass the Response object into our mutation.

Next, create an update mutation for each editable parameter:

Update Email:

// ...
import { ChangeEmailDto } from './dtos/change-email.dto';
// ...

@Resolver(() => UserEntity)
export class UsersResolver {
  // ...

  @Mutation(() => UserEntity)
  public async updateUserEmail(
    @CurrentUser() id: number,
    @Args() changeEmailDto: ChangeEmailDto,
  ): Promise<UserEntity> {
    return await this.usersService.updateEmail(id, changeEmailDto);
  }
}

Update Name:

// ...
import { NameDto } from './dtos/name.dto';
// ...

@Resolver(() => UserEntity)
export class UsersResolver {
  // ...

  @Mutation(() => UserEntity)
  public async updateUserName(
    @CurrentUser() id: number,
    @Args() nameDto: NameDto,
  ): Promise<UserEntity> {
    return await this.usersService.updateName(id, nameDto.name);
  }
}

Update Username:

// ...
import { UsernameDto } from './dtos/username.dto';
// ...

@Resolver(() => UserEntity)
export class UsersResolver {
  // ...

  @Mutation(() => UserEntity)
  public async updateUserUsername(
    @CurrentUser() id: number,
    @Args() usernameDto: UsernameDto,
  ): Promise<UserEntity> {
    return await this.usersService.updateUsername(id, usernameDto.username);
  }
}

Finally since we are using micro-services, we need to add a @ResolveReference decorator so other sub-graphs can have access to the UserEntity:

// ...
import {
  // ...
  ResolveReference,
} from '@nestjs/graphql';
import { IFederatedInstance } from '../common/interfaces/federated-instance.interface';

@Resolver(() => UserEntity)
export class UsersResolver {
  // ...

  @ResolveReference()
  public async resolveReference(
    reference: IFederatedInstance<'User'>,
  ): Promise<UserEntity> {
    return this.usersService.findOneById(reference.id);
  }
}

Auth

Entities

Create a gql directory and add an auth.type.ts type:

import { Field, ObjectType } from '@nestjs/graphql';
import { UserEntity } from '../../../users/entities/user.entity';
import { IUser } from '../../../users/interfaces/user.interface';

@ObjectType('Auth')
export abstract class AuthType {
  @Field(() => UserEntity)
  public user: IUser;

  @Field(() => String)
  public accessToken: string;
}

DTOs

Start by updating the:

Email DTO:

import { ArgsType, Field } from '@nestjs/graphql';
import { ApiProperty } from '@nestjs/swagger';
import { IsEmail, IsString, Length } from 'class-validator';

@ArgsType()
export abstract class EmailDto {
  @Field(() => String)
  @ApiProperty({
    description: 'The email of the user',
    example: 'someone@gmail.com',
    minLength: 5,
    maxLength: 255,
    type: String,
  })
  @IsString()
  @IsEmail()
  @Length(5, 255)
  public email: string;
}

Passwords DTO:

import { ArgsType, Field, InputType } from '@nestjs/graphql';
import { ApiProperty } from '@nestjs/swagger';
import { IsString, Length, Matches, MinLength } from 'class-validator';
import { PASSWORD_REGEX } from '../../common/consts/regex.const';

@InputType({ isAbstract: true })
@ArgsType()
export abstract class PasswordsDto {
  @Field(() => String)
  @ApiProperty({
    description: 'New password',
    minLength: 8,
    maxLength: 35,
    type: String,
  })
  @IsString()
  @Length(8, 35)
  @Matches(PASSWORD_REGEX, {
    message:
      'Password requires a lowercase letter, an uppercase letter, and a number or symbol',
  })
  public password1!: string;

  @Field(() => String)
  @ApiProperty({
    description: 'Password confirmation',
    minLength: 1,
    type: String,
  })
  @IsString()
  @MinLength(1)
  public password2!: string;
}

Since we are going to use the AuthResolver to create new users and do updates that require complex inputs (e.g. updating the Password). We need to create an inputs folder and add the following inputs:

import { Field, InputType } from '@nestjs/graphql';
import { IsString, Length, MinLength } from 'class-validator';

@InputType('SignInInput')
export abstract class SignInInput {
  @Field(() => String)
  @IsString()
  @Length(3, 255)
  public emailOrUsername!: string;

  @Field(() => String)
  @IsString()
  @MinLength(1)
  public password!: string;
}

import { Field, InputType } from '@nestjs/graphql';
import { IsEmail, IsString, Length, Matches } from 'class-validator';
import { NAME_REGEX } from '../../common/consts/regex.const';
import { PasswordsDto } from '../dtos/passwords.dto';

@InputType('SignUpInput')
export abstract class SignUpInput extends PasswordsDto {
  @Field(() => String)
  @IsString()
  @Length(3, 100, {
    message: 'Name has to be between 3 and 100 characters.',
  })
  @Matches(NAME_REGEX, {
    message: 'Name can not contain special characters.',
  })
  public name!: string;

  @Field(() => String)
  @IsString()
  @IsEmail()
  @Length(5, 255)
  public email!: string;
}

Reset Password Input:

import { Field, InputType } from '@nestjs/graphql';
import { IsJWT, IsString } from 'class-validator';
import { PasswordsDto } from '../dtos/passwords.dto';

@InputType('ResetPasswordInput')
export abstract class ResetPasswordInput extends PasswordsDto {
  @Field(() => String)
  @IsString()
  @IsJWT()
  public resetToken!: string;
}

Update Password Input:

import { Field, InputType } from '@nestjs/graphql';
import { IsString, MinLength } from 'class-validator';
import { PasswordsDto } from '../dtos/passwords.dto';

@InputType('UpdatePasswordInput')
export abstract class UpdatePasswordInput extends PasswordsDto {
  @Field(() => String)
  @IsString()
  @MinLength(1)
  public password!: string;
}

Decorators

The decorators need to be updated to accept GraphQL requests:

Current User Decorator:

import { createParamDecorator, ExecutionContext } from '@nestjs/common';
import { GqlExecutionContext } from '@nestjs/graphql';
import { Request } from 'express-serve-static-core';
import { IContext } from '../../config/interfaces/context.interface';

export const CurrentUser = createParamDecorator(
  (_, context: ExecutionContext): number | undefined => {
    if (context.getType() === 'http') {
      return context.switchToHttp().getRequest<Request>()?.user;
    }

    return GqlExecutionContext.create(context).getContext<IContext>().req?.user;
  },
);

Origin Decorator:

import { createParamDecorator, ExecutionContext } from '@nestjs/common';
import { GqlExecutionContext } from '@nestjs/graphql';
import { Request } from 'express-serve-static-core';
import { IContext } from '../../config/interfaces/context.interface';

export const Origin = createParamDecorator(
  (_, context: ExecutionContext): string | undefined => {
    if (context.getType() === 'http') {
      return context.switchToHttp().getRequest<Request>().headers?.origin;
    }

    return GqlExecutionContext.create(context).getContext<IContext>().req
  .headers?.origin;
  },
);

You can determine if you are using GraphQL or HTTP simply by using the context.getType() method.

Guards

The Auth Guard right now only accepts REST requests, so lets change that:

import {
  CanActivate,
  ExecutionContext,
  Injectable,
  UnauthorizedException,
} from '@nestjs/common';
import { Reflector } from '@nestjs/core';
import { isJWT } from 'class-validator';
import { Request } from 'express-serve-static-core';
import { isNull, isUndefined } from '../../common/utils/validation.util';
import { TokenTypeEnum } from '../../jwt/enums/token-type.enum';
import { JwtService } from '../../jwt/jwt.service';
import { IS_PUBLIC_KEY } from '../decorators/public.decorator';

@Injectable()
export class AuthGuard implements CanActivate {
  constructor(
    private readonly reflector: Reflector,
    private readonly jwtService: JwtService,
  ) {}

  public async canActivate(context: ExecutionContext): Promise<boolean> {
    const isPublic = this.reflector.getAllAndOverride<boolean>(IS_PUBLIC_KEY, [
      context.getHandler(),
      context.getClass(),
    ]);
    const activate = await this.setHttpHeader(
      context.switchToHttp().getRequest(),
      isPublic,
    );

    if (!activate) {
      throw new UnauthorizedException();
    }

    return activate;
  }

  /**
   * Sets HTTP Header
   *
   * Checks if the header has a valid Bearer token, validates it and sets the User ID as the user.
   */
  private async setHttpHeader(
    req: Request,
    isPublic: boolean,
  ): Promise<boolean> {
    const auth = req.headers?.authorization;

    if (isUndefined(auth) || isNull(auth) || auth.length === 0) {
      return isPublic;
    }

    const authArr = auth.split(' ');
    const bearer = authArr[0];
    const token = authArr[1];

    if (isUndefined(bearer) || isNull(bearer) || bearer !== 'Bearer') {
      return isPublic;
    }
    if (isUndefined(token) || isNull(token) || !isJWT(token)) {
      return isPublic;
    }

    try {
      const { id } = await this.jwtService.verifyToken(
        token,
        TokenTypeEnum.ACCESS,
      );
      req.user = id;
      return true;
    } catch (_) {
      return isPublic;
    }
  }
}

Resolver

The AuthResolver and AuthController are not that different, they do exactly the same thing, just change the routes to Queries and Mutations.

Start by creating the resolver:

$ nest g r auth

And copy the AuthController constructor to the resolver:

import { ConfigService } from '@nestjs/config';
import { Resolver } from '@nestjs/graphql';
import { UsersService } from '../users/users.service';
import { AuthService } from './auth.service';

@Resolver(() => AuthType)
export class AuthResolver {
  private readonly cookiePath = '/api/graphql';
  private readonly cookieName: string;
  private readonly refreshTime: number;
  private readonly testing: boolean;

  constructor(
    private readonly authService: AuthService,
    private readonly usersService: UsersService,
    private readonly configService: ConfigService,
  ) {
    this.cookieName = this.configService.get<string>('REFRESH_COOKIE');
    this.refreshTime = this.configService.get<number>('jwt.refresh.time');
    this.testing = this.configService.get<boolean>('testing');
  }
}

As you can see now our cookie path is "/api/graphql", so create a new saveRefreshCookie:

// ...
import { Response } from 'express-serve-static-core';

@Resolver(() => AuthType)
export class AuthResolver {
  // ...

  private saveRefreshCookie(res: Response, refreshToken: string): void {
    res.cookie(this.cookieName, refreshToken, {
      secure: !this.testing,
      httpOnly: true,
      signed: true,
      path: this.cookiePath,
      expires: new Date(Date.now() + this.refreshTime * 1000),
    });
  }
}

And copy the private refreshTokenFromReq method as well:

import { UnauthorizedException } from '@nestjs/common';
// ...
import { Request, Response } from 'express-serve-static-core';

@Resolver(() => AuthType)
export class AuthResolver {
  // ...

  private refreshTokenFromReq(req: Request): string {
    const token: string | undefined = req.signedCookies[this.cookieName];

    if (isUndefined(token)) {
      throw new UnauthorizedException();
    }

    return token;
  }
}

Taking in mind that all the routes are the same but now we use Mutations and Queries, hence we get the following mutations:

import { UnauthorizedException } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { Args, Context, Mutation, Resolver } from '@nestjs/graphql';
import { Request, Response } from 'express-serve-static-core';
import { MessageType } from '../common/entities/gql/message.type';
import { IMessage } from '../common/interfaces/message.interface';
import { isUndefined } from '../common/utils/validation.util';
import { UsersService } from '../users/users.service';
import { AuthService } from './auth.service';
import { CurrentUser } from './decorators/current-user.decorator';
import { Origin } from './decorators/origin.decorator';
import { Public } from './decorators/public.decorator';
import { ConfirmEmailDto } from './dtos/confirm-email.dto';
import { EmailDto } from './dtos/email.dto';
import { AuthType } from './entities/gql/auth.type';
import { ResetPasswordInput } from './inputs/reset-password.input';
import { SignInInput } from './inputs/sign-in.input';
import { SignUpInput } from './inputs/sign-up.input';
import { UpdatePasswordInput } from './inputs/update-password.input';

@Resolver(() => AuthType)
export class AuthResolver {
  // ...

  @Public()
  @Mutation(() => MessageType)
  public async signUp(
    @Origin() origin: string | undefined,
    @Args('input') signUpInput: SignUpInput,
  ): Promise<IMessage> {
    return await this.authService.signUp(signUpInput, origin);
  }

  @Public()
  @Mutation(() => AuthType)
  public async signIn(
    @Context('res') res: Response,
    @Origin() origin: string | undefined,
    @Args('input') signInInput: SignInInput,
  ): Promise<AuthType> {
    const { refreshToken, ...authType } = await this.authService.signIn(
      signInInput,
      origin,
    );
    this.saveRefreshCookie(res, refreshToken);
    return authType;
  }

  @Public()
  @Mutation(() => AuthType)
  public async refreshAccess(
    @Context('req') req: Request,
    @Context('res') res: Response,
  ): Promise<AuthType> {
    const token = this.refreshTokenFromReq(req);
    const { refreshToken, ...authType } =
      await this.authService.refreshTokenAccess(token, req.headers.origin);
    this.saveRefreshCookie(res, refreshToken);
    return authType;
  }

  @Mutation(() => MessageType)
  public async logout(
    @Context('req') req: Request,
    @Context('res') res: Response,
  ): Promise<IMessage> {
    const token = this.refreshTokenFromReq(req);
    res.clearCookie(this.cookieName);
    return this.authService.logout(token);
  }

  @Public()
  @Mutation(() => AuthType)
  public async confirmEmail(
    @Origin() origin: string | undefined,
    @Args() confirmEmailDto: ConfirmEmailDto,
    @Context('res') res: Response,
  ) {
    const { refreshToken, ...authType } = await this.authService.confirmEmail(
      confirmEmailDto,
      origin,
    );
    this.saveRefreshCookie(res, refreshToken);
    return authType;
  }

  @Public()
  @Mutation(() => MessageType)
  public async forgotPassword(
    @Origin() origin: string | undefined,
    @Args() emailDto: EmailDto,
  ): Promise<IMessage> {
    return await this.authService.resetPasswordEmail(emailDto, origin);
  }

  @Public()
  @Mutation(() => MessageType)
  public async resetPassword(
    @Args('input') input: ResetPasswordInput,
  ): Promise<IMessage> {
    return await this.authService.resetPassword(input);
  }

  @Mutation(() => AuthType)
  public async updatePassword(
    @CurrentUser() id: number,
    @Origin() origin: string | undefined,
    @Args('input') input: UpdatePasswordInput,
    @Context('res') res: Response,
  ): Promise<AuthType> {
    const { refreshToken, ...authType } = await this.authService.updatePassword(
      id,
      input,
      origin,
    );
    this.saveRefreshCookie(res, refreshToken);
    return authType;
  }

  //...
}

And the following me query:

// ...
import { Query } from '@nestjs/graphql';
// ...
import { UserEntity } from '../users/entities/user.entity';
// ...

@Resolver(() => AuthType)
export class AuthResolver {
  // ...

  @Query(() => UserEntity)
  public async me(@CurrentUser() id: number): Promise<UserEntity> {
    return await this.usersService.findOneById(id);
  }

  // ...
}

Conclusion

With this you are now able to implement (although again, you should not) a Local OAuth API in GraphQL.

The repository for the code can be found here.

About the Author

Do not miss out on any of my latest articles – follow me here on dev and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

NestJS Authentication with OAuth2.0: Fastify Local OAuth REST API

Afonso Barracha — Fri, 27 Jan 2023 14:29:59 +0000

Series Intro

This series will cover the full implementation of OAuth2.0 Authentication in NestJS for the following types of APIs:

Express REST API;
Fastify REST API;
Apollo GraphQL API.

And it is divided in 5 parts:

Configuration and operations;
Express Local OAuth REST API;
Fastify Local OAuth REST API;
Apollo Local OAuth GraphQL API;
Adding External OAuth Providers to our API;

Lets start the third part of this series.

Tutorial Intro

On this tutorial we will change the adapter from our previous REST API, from Express to Fastify.

TLDR: if you do not have 25 minutes to read the article, the code can be found on this repo

Set Up

Start by removing express and its dependencies:

$ yarn remove @types/express @types/express-serve-static-core @nestjs/platform-express cookie-parser helmet @types/cookie-parser

And install the fastify ones:

$ yarn add @nestjs/platform-fastify fastify @fastify/cookie @fastify/cors @fastify/csrf-protection @fastify/helmet

Auth Module

Guards

We need to remove express.d.ts and add fastify.d.ts:

import { FastifyRequest as Request } from 'fastify';

declare module 'fastify' {
  interface FastifyRequest extends Request {
    user?: number;
  }
}

Auth Guard

Only one of the types change, from Request to FastifyRequest:

import {
  CanActivate,
  ExecutionContext,
  Injectable,
  UnauthorizedException,
} from '@nestjs/common';
import { Reflector } from '@nestjs/core';
import { isJWT } from 'class-validator';
import { FastifyRequest } from 'fastify';
import { isNull, isUndefined } from '../../common/utils/validation.util';
import { TokenTypeEnum } from '../../jwt/enums/token-type.enum';
import { JwtService } from '../../jwt/jwt.service';
import { IS_PUBLIC_KEY } from '../decorators/public.decorator';

@Injectable()
export class AuthGuard implements CanActivate {
  constructor(
    private readonly reflector: Reflector,
    private readonly jwtService: JwtService,
  ) {}

  public async canActivate(context: ExecutionContext): Promise<boolean> {
    const isPublic = this.reflector.getAllAndOverride<boolean>(IS_PUBLIC_KEY, [
      context.getHandler(),
      context.getClass(),
    ]);
    const activate = await this.setHttpHeader(
      context.switchToHttp().getRequest<FastifyRequest>(),
      isPublic,
    );

    if (!activate) {
      throw new UnauthorizedException();
    }

    return activate;
  }

  /**
   * Sets HTTP Header
   *
   * Checks if the header has a valid Bearer token, validates it and sets the User ID as the user.
   */
  private async setHttpHeader(
    req: FastifyRequest,
    isPublic: boolean,
  ): Promise<boolean> {
    const auth = req.headers?.authorization;

    if (isUndefined(auth) || isNull(auth) || auth.length === 0) {
      return isPublic;
    }

    const authArr = auth.split(' ');
    const bearer = authArr[0];
    const token = authArr[1];

    if (isUndefined(bearer) || isNull(bearer) || bearer !== 'Bearer') {
      return isPublic;
    }
    if (isUndefined(token) || isNull(token) || !isJWT(token)) {
      return isPublic;
    }

    try {
      const { id } = await this.jwtService.verifyToken(
        token,
        TokenTypeEnum.ACCESS,
      );
      req.user = id;
      return true;
    } catch (_) {
      return isPublic;
    }
  }
}

(Optional) Throttler Guard

We can no longer use the NestJS default ThrottlerGuard as that one is implemented for express, so create a custom one:

// fastify-throttler.guard.ts

import { ExecutionContext, Injectable } from '@nestjs/common';
import { ThrottlerGuard } from '@nestjs/throttler';
import { FastifyReply, FastifyRequest } from 'fastify';

@Injectable()
export class FastifyThrottlerGuard extends ThrottlerGuard {
  public getRequestResponse(context: ExecutionContext) {
    const http = context.switchToHttp();
    return {
      req: http.getRequest<FastifyRequest>(),
      res: http.getResponse<FastifyReply>(),
    };
  }
}

And add it to the auth controller:

// ...
import { FastifyThrottlerGuard } from './guards/fastify-throttler.guard';
// ...

@ApiTags('Auth')
@Controller('api/auth')
@UseGuards(FastifyThrottlerGuard)
export class AuthController {
  // ...
}

Controller

Private Methods

There is a slight change on the refresTokenFromReq method:

import {
  // ...
  Controller,
  // ...
  UseGuards,
} from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import {
  // ...
  ApiTags,
  // ...
} from '@nestjs/swagger';
import { FastifyReply, FastifyRequest } from 'fastify';
// ...
import { isNull, isUndefined } from '../common/utils/validation.util';
// ...

@ApiTags('Auth')
@Controller('api/auth')
@UseGuards(FastifyThrottlerGuard)
export class AuthController {
  // ...

  private refreshTokenFromReq(req: FastifyRequest): string {
    const token: string | undefined = req.cookies[this.cookieName];

    if (isUndefined(token) || isNull(token)) {
      throw new UnauthorizedException();
    }

    const { valid, value } = req.unsignCookie(token);

    if (!valid) {
      throw new UnauthorizedException();
    }

    return value;
  }

  // ...
}

And since fastify does not have the json method we need to set the Content-Type header to application/json:

// ...

@ApiTags('Auth')
@Controller('api/auth')
@UseGuards(FastifyThrottlerGuard)
export class AuthController {
  // ...

  private saveRefreshCookie(
    res: FastifyReply,
    refreshToken: string,
  ): FastifyReply {
    return res
      .cookie(this.cookieName, refreshToken, {
        secure: !this.testing,
        httpOnly: true,
        signed: true,
        path: this.cookiePath,
        expires: new Date(Date.now() + this.refreshTime * 1000),
      })
      .header('Content-Type', 'application/json');
  }
}

Enpoints

There aren't that many changes that we need to implement on our previous express API, we just need to change from:

Request to FastifyRequest on the Req decorator;
Response to FastifyReply on the Res decorator;
json to send method.

// ...

@ApiTags('Auth')
@Controller('api/auth')
@UseGuards(FastifyThrottlerGuard)
export class AuthController {
  // ...

  @Public()
  @Post('/sign-up')
  @ApiCreatedResponse({
    type: MessageMapper,
    description: 'The user has been created and is waiting confirmation',
  })
  @ApiConflictResponse({
    description: 'Email already in use',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body',
  })
  public async signUp(
    @Origin() origin: string | undefined,
    @Body() signUpDto: SignUpDto,
  ): Promise<IMessage> {
    return await this.authService.signUp(signUpDto, origin);
  }

  @Public()
  @Post('/sign-in')
  @ApiOkResponse({
    type: AuthResponseMapper,
    description: 'Logs in the user and returns the access token',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body',
  })
  @ApiUnauthorizedResponse({
    description: 'Invalid credentials or User is not confirmed',
  })
  public async signIn(
    @Res() res: FastifyReply,
    @Origin() origin: string | undefined,
    @Body() singInDto: SignInDto,
  ): Promise<void> {
    const result = await this.authService.signIn(singInDto, origin);
    this.saveRefreshCookie(res, result.refreshToken)
      .status(HttpStatus.OK)
      .send(AuthResponseMapper.map(result));
  }

  @Public()
  @Post('/refresh-access')
  @ApiOkResponse({
    type: AuthResponseMapper,
    description: 'Refreshes and returns the access token',
  })
  @ApiUnauthorizedResponse({
    description: 'Invalid token',
  })
  @ApiBadRequestResponse({
    description:
      'Something is invalid on the request body, or Token is invalid or expired',
  })
  public async refreshAccess(
    @Req() req: FastifyRequest,
    @Res() res: FastifyReply,
  ): Promise<void> {
    const token = this.refreshTokenFromReq(req);
    const result = await this.authService.refreshTokenAccess(
      token,
      req.headers.origin,
    );
    this.saveRefreshCookie(res, result.refreshToken)
      .status(HttpStatus.OK)
      .send(AuthResponseMapper.map(result));
  }

  @Post('/logout')
  @ApiOkResponse({
    type: MessageMapper,
    description: 'The user is logged out',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body',
  })
  @ApiUnauthorizedResponse({
    description: 'Invalid token',
  })
  public async logout(
    @Req() req: FastifyRequest,
    @Res() res: FastifyReply,
  ): Promise<void> {
    const token = this.refreshTokenFromReq(req);
    const message = await this.authService.logout(token);
    res
      .clearCookie(this.cookieName, { path: this.cookiePath })
      .header('Content-Type', 'application/json')
      .status(HttpStatus.OK)
      .send(message);
  }

  @Public()
  @Post('/confirm-email')
  @ApiOkResponse({
    type: AuthResponseMapper,
    description: 'Confirms the user email and returns the access token',
  })
  @ApiUnauthorizedResponse({
    description: 'Invalid token',
  })
  @ApiBadRequestResponse({
    description:
      'Something is invalid on the request body, or Token is invalid or expired',
  })
  public async confirmEmail(
    @Origin() origin: string | undefined,
    @Body() confirmEmailDto: ConfirmEmailDto,
    @Res() res: FastifyReply,
  ): Promise<void> {
    const result = await this.authService.confirmEmail(confirmEmailDto);
    this.saveRefreshCookie(res, result.refreshToken)
      .status(HttpStatus.OK)
      .send(AuthResponseMapper.map(result));
  }

  @Public()
  @Post('/forgot-password')
  @HttpCode(HttpStatus.OK)
  @ApiOkResponse({
    type: MessageMapper,
    description:
      'An email has been sent to the user with the reset password link',
  })
  public async forgotPassword(
    @Origin() origin: string | undefined,
    @Body() emailDto: EmailDto,
  ): Promise<IMessage> {
    return this.authService.resetPasswordEmail(emailDto, origin);
  }

  @Public()
  @Post('/reset-password')
  @HttpCode(HttpStatus.OK)
  @ApiOkResponse({
    type: MessageMapper,
    description: 'The password has been reset',
  })
  @ApiBadRequestResponse({
    description:
      'Something is invalid on the request body, or Token is invalid or expired',
  })
  public async resetPassword(
    @Body() resetPasswordDto: ResetPasswordDto,
  ): Promise<IMessage> {
    return this.authService.resetPassword(resetPasswordDto);
  }

  @Patch('/update-password')
  @ApiOkResponse({
    type: AuthResponseMapper,
    description: 'The password has been updated',
  })
  @ApiUnauthorizedResponse({
    description: 'The user is not logged in.',
  })
  public async updatePassword(
    @CurrentUser() userId: number,
    @Origin() origin: string | undefined,
    @Body() changePasswordDto: ChangePasswordDto,
    @Res() res: FastifyReply,
  ): Promise<void> {
    const result = await this.authService.updatePassword(
      userId,
      changePasswordDto,
      origin,
    );
    this.saveRefreshCookie(res, result.refreshToken)
      .status(HttpStatus.OK)
      .send(AuthResponseMapper.map(result));
  }

  @Get('/me')
  @ApiOkResponse({
    type: AuthResponseUserMapper,
    description: 'The user is found and returned.',
  })
  @ApiUnauthorizedResponse({
    description: 'The user is not logged in.',
  })
  public async getMe(@CurrentUser() id: number): Promise<IAuthResponseUser> {
    const user = await this.usersService.findOneById(id);
    return AuthResponseUserMapper.map(user);
  }

  // ...
}

User Module

Controller

There is just one change on the return type of the Res decorator (from Response to FastifyReply) on the delete endpoint:

// ...
import { FastifyReply } from 'fastify';
// ...

@ApiTags('Users')
@Controller('api/users')
export class UsersController {
  // ...

  @Delete()
  @ApiNoContentResponse({
    description: 'The user is deleted.',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body, or wrong password.',
  })
  @ApiUnauthorizedResponse({
    description: 'The user is not logged in.',
  })
  public async deleteUser(
    @CurrentUser() id: number,
    @Body() dto: PasswordDto,
    @Res() res: FastifyReply,
  ): Promise<void> {
    await this.usersService.delete(id, dto);
    res
      .clearCookie(this.cookieName, { path: this.cookiePath })
      .status(HttpStatus.NO_CONTENT)
      .send();
  }
}

Main

Finally change the main file to a NestFastifyApplication, and register all the plugins we installed earlier:

import fastifyCookie from '@fastify/cookie';
import fastifyCors from '@fastify/cors';
import fastifyCsrfProtection from '@fastify/csrf-protection';
import fastifyHelmet from '@fastify/helmet';
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import {
  FastifyAdapter,
  NestFastifyApplication,
} from '@nestjs/platform-fastify';
import { DocumentBuilder, SwaggerModule } from '@nestjs/swagger';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create<NestFastifyApplication>(
    AppModule,
    new FastifyAdapter(),
  );
  const configService = app.get(ConfigService);
  app.register(fastifyCookie, {
    secret: configService.get<string>('COOKIE_SECRET'),
  });
  app.register(fastifyHelmet);
  app.register(fastifyCsrfProtection, { cookieOpts: { signed: true } });
  app.register(fastifyCors, {
    credentials: true,
    origin: `https://${configService.get<string>('domain')}`,
  });
  app.useGlobalPipes(
    new ValidationPipe({
      transform: true,
    }),
  );

  const swaggerConfig = new DocumentBuilder()
    .setTitle('NestJS Authentication API')
    .setDescription('An OAuth2.0 authentication API made with NestJS')
    .setVersion('0.0.1')
    .addBearerAuth()
    .addTag('Authentication API')
    .build();
  const document = SwaggerModule.createDocument(app, swaggerConfig);
  SwaggerModule.setup('api/docs', app, document);

  await app.listen(
    configService.get<number>('port'),
    configService.get<boolean>('testing') ? '127.0.0.1' : '0.0.0.0',
  );
}

bootstrap();

Conclusion

I hope that I have shown just how easy it is to change an API from Express to Fastify with NestJS.

The github for this tutorial can be found here.

About the Author

Do not miss out on any of my latest articles – follow me here on dev and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!

NestJS Authentication with OAuth2.0: Express Local OAuth REST API

Afonso Barracha — Fri, 20 Jan 2023 12:55:08 +0000

Series Intro

This series will cover the full implementation of OAuth2.0 Authentication in NestJS for the following types of APIs:

Express REST API;
Fastify REST API;
Apollo GraphQL API.

And it is divided in 5 parts:

Configuration and operations;
Express Local OAuth REST API;
Fastify Local OAuth REST API;
Apollo Local OAuth GraphQL API;
Adding External OAuth Providers to our API;

Lets start the second part of this series.

Tutorial Intro

On this tutorial we will continue building on the previous article, by creating an Express REST API.

TLDR: if you do not have 45 minutes to read the article, the code can be found on this repo

Set up

Start by creating controllers for both the auth and users:

$ nest g co auth
$ nest g co users

Auth Module

Before implementing the endpoints we need both an auth guard, and some custom decorators:

Public: to make an endpoint public;
Origin: to get the origin from cross-origin requests:
CurrentUser: to get the current logged user id;

Decorators

Before creating decorators, since we will have a user field on our express Request interface install express-serve-static-core:

$ yarn add -D @types/express-serve-static-core

And extend express on a types folder:

// express.d.ts

import { Request as ExpressRequest } from 'express';

declare module 'express-serve-static-core' {
  interface Request extends ExpressRequest {
    user?: number;
  }
}

To be able to create a global auth guard we need to create the Public decorator first, this decorator can be found here in the docs, on the decorators folder add:

// public.decorator.ts

import { SetMetadata } from '@nestjs/common';

export const IS_PUBLIC_KEY = 'isPublic';
export const Public = () => SetMetadata(IS_PUBLIC_KEY, true);

To validate that our tokens come from the correct origin we need to get the domain from the the front-end:

// origin.decorator.ts

import { createParamDecorator, ExecutionContext } from '@nestjs/common';
import { Request } from 'express-serve-static-core';

export const Origin = createParamDecorator(
  (_, context: ExecutionContext): string | undefined => {
    return context.switchToHttp().getRequest<Request>().headers?.origin;
  },
);

Finally we need to be able to get the current logged user ID:

// current-user.decorator.ts

import { createParamDecorator, ExecutionContext } from '@nestjs/common';
import { Request } from 'express-serve-static-core';

export const CurrentUser = createParamDecorator(
  (_, context: ExecutionContext): number | undefined => {
    return context.switchToHttp().getRequest<Request>()?.user;
  },
);

Guards

Auth Guard

We need a global AuthGuard to protect our endpoints, on a guards folder, we can create a CanActivate guard:

import {
  CanActivate,
  ExecutionContext,
  Injectable,
  UnauthorizedException,
} from '@nestjs/common';
import { Reflector } from '@nestjs/core';
import { isJWT } from 'class-validator';
import { Request } from 'express';
import { isNull, isUndefined } from '../../common/utils/validation.util';
import { TokenTypeEnum } from '../../jwt/enums/token-type.enum';
import { JwtService } from '../../jwt/jwt.service';
import { IS_PUBLIC_KEY } from '../decorators/public.decorator';

@Injectable()
export class AuthGuard implements CanActivate {
  constructor(
    private readonly reflector: Reflector,
    private readonly jwtService: JwtService,
  ) {}

  public async canActivate(context: ExecutionContext): Promise<boolean> {
    const isPublic = this.reflector.getAllAndOverride<boolean>(IS_PUBLIC_KEY, [
      context.getHandler(),
      context.getClass(),
    ]);
    const activate = await this.setHttpHeader(
      context.switchToHttp().getRequest(),
      isPublic,
    );

    if (!activate) {
      throw new UnauthorizedException();
    }

    return activate;
  }

  /**
   * Sets HTTP Header
   *
   * Checks if the header has a valid Bearer token, validates it and sets the User ID as the user.
   */
  private async setHttpHeader(
    req: Request,
    isPublic: boolean,
  ): Promise<boolean> {
    const auth = req.headers?.authorization;

    if (isUndefined(auth) || isNull(auth) || auth.length === 0) {
      return isPublic;
    }

    const authArr = auth.split(' ');
    const bearer = authArr[0];
    const token = authArr[1];

    if (isUndefined(bearer) || isNull(bearer) || bearer !== 'Bearer') {
      return isPublic;
    }
    if (isUndefined(token) || isNull(token) || !isJWT(token)) {
      return isPublic;
    }

    try {
      const { id } = await this.jwtService.verifyToken(
        token,
        TokenTypeEnum.ACCESS,
      );
      req.user = id;
      return true;
    } catch (_) {
      return isPublic;
    }
  }
}

Finally add the guard to the AppModule:

// ...
import { CacheModule, Module } from '@nestjs/common';
// ...
import { APP_GUARD } from '@nestjs/core';
import { AuthGuard } from './auth/guards/auth.guard';
// ...

@Module({
  // ...
  providers: [
    {
      provide: APP_GUARD,
      useClass: AuthGuard,
    },
  ],
})
export class AppModule {}

(Optional) Throttler Guard

To protect our auth endpoints against brute force attacks we should add rate limiting to our API, NestJS has a ThrottlerModule, start by installing it (and optionally the Redis storage for it):

$ yarn add @nestjs/throttler nestjs-throttler-storage-redis

Add the throttler options to .env:

# ...
# Throttler config
THROTTLE_TTL=60
THROTTLE_LIMIT=20

To the schema:

import Joi from 'joi';

export const validationSchema = Joi.object({
  // ...
  THROTTLE_TTL: Joi.number().required(),
  THROTTLE_LIMIT: Joi.number().required(),
});

To the config.interface.ts file:

// ...
import { ThrottlerModuleOptions } from '@nestjs/throttler';
// ...

export interface IConfig {
  // ...
  throttler: ThrottlerModuleOptions;
  // ...
}

And on the index.ts of the config folder:

// ...
import { IConfig } from './interfaces/config.interface';
// ...

export function config(): IConfig {
  // ...

  return {
    // ...
    throttler: {
      ttl: parseInt(process.env.THROTTLE_TTL, 10),
      limit: parseInt(process.env.THROTTLE_LIMIT, 10),
    },
    // ...
  };
}

Now add a config class for the throttler:

// throttler.config.ts

import { Injectable } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import {
  ThrottlerModuleOptions,
  ThrottlerOptionsFactory,
} from '@nestjs/throttler';
import { RedisOptions } from 'ioredis';
import { ThrottlerStorageRedisService } from 'nestjs-throttler-storage-redis';

@Injectable()
export class ThrottlerConfig implements ThrottlerOptionsFactory {
  constructor(private readonly configService: ConfigService) {}

  createThrottlerOptions(): ThrottlerModuleOptions {
    return {
      ...this.configService.get<ThrottlerModuleOptions>('throttler'),
      storage: new ThrottlerStorageRedisService(
        this.configService.get<RedisOptions>('redis'),
      ),
    };
  }
}

import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { ThrottlerModule } from '@nestjs/throttler';
import { ThrottlerConfig } from '../config/throttler.config';
// ...

@Module({
  imports: [
    // ...
    ThrottlerModule.forRootAsync({
      imports: [ConfigModule],
      useClass: ThrottlerConfig,
    }),
  ],
  // ...
})
export class AuthModule {}

Finally add the ThrottlerGuard to the auth controller:

import { Controller, UseGuards } from '@nestjs/common';
import { ThrottlerGuard } from '@nestjs/throttler';

@Controller('auth')
@UseGuards(ThrottlerGuard)
export class AuthController {}

Controller

Mappers

Some data is private and should never be sent by the response of our API, such as the user password and the refresh token. Therefore, we need to create mapper functions that reflect what we want to send as a response of our API.

Interfaces

There are two interfaces that we need, one with the user data that we want to send and one for the Auth Result.

Auth Response User

// auth-response-user.interface.ts

export interface IAuthResponseUser {
  id: number;
  name: string;
  username: string;
  email: string;
  createdAt: string;
  updatedAt: string;
}

Auth Response

// auth-response.interface.ts

import { IAuthResponseUser } from './auth-response-user.interface';

export interface IAuthResponse {
  user: IAuthResponseUser;
  accessToken: string;
}

Classes

Now for the mapper we can either add a static method to our controllers or create separate classes with a map static method. I recommend the latter as it is easier to add OpenAPI specifications latter on.

Create a new folder called mappers and add the following files:

auth-response-user.mapper.ts:

import { IUser } from '../../users/interfaces/user.interface';
import { IAuthResponseUser } from '../interfaces/auth-response-user.interface';

export class AuthResponseUserMapper implements IAuthResponseUser {
  public id: number;
  public name: string;
  public username: string;
  public email: string;
  public createdAt: string;
  public updatedAt: string;

  constructor(values: IAuthResponseUser) {
    Object.assign(this, values);
  }

  public static map(user: IUser): AuthResponseUserMapper {
    return new AuthResponseUserMapper({
      id: user.id,
      name: user.name,
      username: user.username,
      email: user.email,
      createdAt: user.createdAt.toISOString(),
      updatedAt: user.updatedAt.toISOString(),
    });
  }
}

auth-response.mapper.ts:

import { IAuthResponse } from '../interfaces/auth-response.interface';
import { IAuthResult } from '../interfaces/auth-result.interface';
import { AuthResponseUserMapper } from './auth-response-user.mapper';

export class AuthResponseMapper implements IAuthResponse {
  public user: AuthResponseUserMapper;
  public accessToken: string;

  constructor(values: IAuthResponse) {
    Object.assign(this, values);
  }

  public static map(result: IAuthResult): AuthResponseMapper {
    return new AuthResponseMapper({
      user: AuthResponseUserMapper.map(result.user),
      accessToken: result.accessToken,
    });
  }
}

Methods

While most of the business logic resides on the service, we still need some logic to be added here on the controller.

Start by injecting the AuthService and UserService, and by getting the COOKIE_NAME and JWT_REFRESH_TIME from the ConfigService:

import { Controller, UseGuards } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { UsersService } from '../users/users.service';
import { AuthService } from './auth.service';
// ...

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  private readonly cookiePath = '/api/auth';
  private readonly cookieName: string;
  private readonly refreshTime: number;
  private readonly testing: boolean;

  constructor(
    private readonly authService: AuthService,
    private readonly usersService: UsersService,
    private readonly configService: ConfigService,
  ) {
    this.cookieName = this.configService.get<string>('REFRESH_COOKIE');
    this.refreshTime = this.configService.get<number>('jwt.refresh.time');
    this.testing = this.configService.get<string>('NODE_ENV') !== 'production';
  }
}

And add two methods:

Getting refresh token out of the request:

// ...

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  private refreshTokenFromReq(req: Request): string {
    const token: string | undefined = req.signedCookies[this.cookieName];

    if (isUndefined(token)) {
      throw new UnauthorizedException();
    }

    return token;
  }
}

Saving the refresh token in a cookie:

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  private saveRefreshCookie(res: Response, refreshToken: string): Response {
    return res.cookie(this.cookieName, refreshToken, {
      secure: !this.testing,
      httpOnly: true,
      signed: true,
      path: this.cookiePath,
      expires: new Date(Date.now() + this.refreshTime * 1000),
    });
  }
}

Endpoints

The endpoints are the same as the public methods we have on our service.

Sign Up

import {
  Body,
  //...
  Post,
  // ...
} from '@nestjs/common';
// ...
import { Origin } from './decorators/origin.decorator';
import { Public } from './decorators/public.decorator';
// ...
import { SignUpDto } from './dtos/sign-up.dto';


@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  @Public()
  @Post('/sign-up')
  public async signUp(
    @Origin() origin: string | undefined,
    @Body() signUpDto: SignUpDto,
  ): Promise<IMessage> {
    return this.authService.signUp(signUpDto, origin);
  }

  // ...
}

Sign In

import {
  Body,
  HttpStatus,
  Res,
  // ...
} from '@nestjs/common';
import { SignInDto } from './dtos/sign-in.dto';
import { AuthResponseMapper } from './mappers/auth-response.mapper';

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  @Public()
  @Post('/sign-in')
  public async signIn(
    @Res() res: Response,
    @Origin() origin: string | undefined,
    @Body() singInDto: SignInDto,
  ): Promise<void> {
    const result = await this.authService.signIn(singInDto, origin);
    this.saveRefreshCookie(res, result.refreshToken)
      .status(HttpStatus.OK)
      .json(AuthResponseMapper.map(result));
  }

  // ...
}

Refresh Access

import {
  //...
  Req,
  // ...
} from '@nestjs/common';
// ...

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  @Public()
  @Post('/refresh-access')
  public async refreshAccess(
    @Req() req: Request,
    @Res() res: Response,
  ): Promise<void> {
    const token = this.refreshTokenFromReq(req);
    const result = await this.authService.refreshTokenAccess(
      token,
      req.headers.origin,
    );
    this.saveRefreshCookie(res, result.refreshToken)
      .status(HttpStatus.OK)
      .json(AuthResponseMapper.map(result));
  }

  // ...
}

Logout

import {
  //...
  HttpCode,
  // ...
} from '@nestjs/common';
// ...

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  @Post('/logout')
  @HttpCode(HttpStatus.OK)
  public async logout(
    @Req() req: Request,
    @Res() res: Response,
  ): Promise<void> {
    const token = this.refreshTokenFromReq(req);
    const message = await this.authService.logout(token);
    res
      .clearCookie(this.cookieName, { path: this.cookiePath })
      .status(HttpStatus.OK)
      .json(message);
  }

  // ...
}

Confirm Email

// ...

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  @Public()
  @Post('/confirm-email')
  public async confirmEmail(
    @Body() confirmEmailDto: ConfirmEmailDto,
    @Res() res: Response,
  ): Promise<void> {
    const result = await this.authService.confirmEmail(confirmEmailDto);
    this.saveRefreshCookie(res, result.refreshToken)
      .status(HttpStatus.OK)
      .json(AuthResponseMapper.map(result));
  }

  // ...
}

Forgot Password

// ...

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  @Post('/forgot-password')
  @HttpCode(HttpStatus.OK)
  public async forgotPassword(
    @Origin() origin: string | undefined,
    @Body() emailDto: EmailDto,
  ): Promise<IMessage> {
    return this.authService.resetPasswordEmail(emailDto, origin);
  }

  // ...
}

Reset Password

// ...

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  @Public()
  @Post('/reset-password')
  @HttpCode(HttpStatus.OK)
  public async resetPassword(
    @Body() resetPasswordDto: ResetPasswordDto,
  ): Promise<IMessage> {
    return this.authService.resetPassword(resetPasswordDto);
  }

  // ...
}

Update Password

import {
  // ...
  Patch,
  // ...
} from '@nestjs/common';
import { CurrentUser } from './decorators/current-user.decorator';
import { ChangePasswordDto } from './dtos/change-password.dto';
// ...

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  @Patch('/update-password')
  public async updatePassword(
    @CurrentUser() userId: number,
    @Origin() origin: string | undefined,
    @Body() changePasswordDto: ChangePasswordDto,
    @Res() res: Response,
  ): Promise<void> {
    const result = await this.authService.updatePassword(
      userId,
      changePasswordDto,
      origin,
    );
    this.saveRefreshCookie(res, result.refreshToken)
      .status(HttpStatus.OK)
      .json(AuthResponseMapper.map(result));
  }

  // ...
}

This one is necessary to get the current logged user

import {
  // ...
  Get,
  // ...
} from '@nestjs/common';
import { IAuthResponseUser } from '../auth/interfaces/auth-response-user.interface';
import { AuthResponseUserMapper } from '../auth/mappers/auth-response-user.mapper';
// ...

@Controller('api/auth')
@UseGuards(ThrottlerGuard)
export class AuthController {
  // ...

  @Get('/me')
  public async getMe(@CurrentUser() id: number): Promise<IAuthResponseUser> {
    const user = await this.usersService.findOneById(id);
    return AuthResponseUserMapper.map(user);
  }

  // ...
}

User Module

User Service

Before adding our controller we need to make some changes to the service:

Add a PasswordDto:

// password.dto.ts

import { IsString, MinLength } from 'class-validator';

export abstract class PasswordDto {
  @IsString()
  @MinLength(1)
  public password: string;
}

Update the ChangeEmailDto:

// change-email.dto.ts

import { IsEmail, IsString, Length } from 'class-validator';
import { PasswordDto } from './password.dto';

export abstract class ChangeEmailDto extends PasswordDto {
  @ApiProperty({
    description: 'The email of the user',
    example: 'someone@gmail.com',
    minLength: 5,
    maxLength: 255,
    type: String,
  })
  @IsString()
  @IsEmail()
  @Length(5, 255)
  public email: string;
}

Add password verification to delete method:

// ...
import {
  BadRequestException,
  // ...
} from '@nestjs/common';
import { compare, hash } from 'bcrypt';
// ...
import { PasswordDto } from './dtos/password.dto';
import { UserEntity } from './entities/user.entity';

@Injectable()
export class UsersService {
  // ...

  public async delete(userId: number, dto: PasswordDto): Promise<UserEntity> {
    const user = await this.findOneById(userId);

    if (!(await compare(dto.password, user.password))) {
      throw new BadRequestException('Wrong password');
    }

    await this.commonService.removeEntity(this.usersRepository, user);
    return user;
  }

  // ...
}

Add a method to find by id or username:

// ...

@Injectable()
export class UsersService {
// ...

  public async findOneByIdOrUsername(
    idOrUsername: string,
  ): Promise<UserEntity> {
    const parsedValue = parseInt(idOrUsername, 10);

    if (!isNaN(parsedValue) && parsedValue > 0 && isInt(parsedValue)) {
      return this.findOneById(parsedValue);
    }

    if (
      idOrUsername.length < 3 ||
      idOrUsername.length > 106 ||
      !SLUG_REGEX.test(idOrUsername)
    ) {
      throw new BadRequestException('Invalid username');
    }

    return this.findOneByUsername(idOrUsername);
  }

// ...
}

User Controller

Mappers

The user return by the user controller should not have the email, so create a new interface:

// response-user.interface.ts

export interface IResponseUser {
  id: number;
  name: string;
  username: string;
  createdAt: string;
  updatedAt: string;
}

And implement it:

// response-user.mapper.ts

import { IResponseUser } from '../interfaces/response-user.interface';
import { IUser } from '../interfaces/user.interface';

export class ResponseUserMapper implements IResponseUser {
  public id: number;
  public name: string;
  public username: string;
  public createdAt: string;
  public updatedAt: string;

  constructor(values: IResponseUser) {
    Object.assign(this, values);
  }

  public static map(user: IUser): ResponseUserMapper {
    return new ResponseUserMapper({
      id: user.id,
      name: user.name,
      username: user.username,
      createdAt: user.createdAt.toISOString(),
      updatedAt: user.updatedAt.toISOString(),
    });
  }
}

Endpoints

Start by adding the cookiePath and cookieName for the delete route:

import { Controller } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { UsersService } from './users.service';

@Controller('api/users')
export class UsersController {
  private cookiePath = '/api/auth';
  private cookieName: string;

  constructor(
    private readonly usersService: UsersService,
    private readonly configService: ConfigService,
  ) {
    this.cookieName = this.configService.get<string>('COOKIE_NAME');
  }
}

User controller is comprised mostly by user Read, Update and Delete operations.

Get User

Start by creating a dto for the params:

// get-user.params.ts

import { IsString, Length } from 'class-validator';

export abstract class GetUserParams {
  @IsString()
  @Length(1, 106)
  public idOrUsername: string;
}

And add it as the type of the Param decorator:

import {
  // ...
  Get,
  Param,
  // ...
} from '@nestjs/common';
// ...
import { GetUserParams } from './dtos/get-user.params';
// ...

@Controller('api/users')
export class UsersController {
  // ...

  @Public()
  @Get('/:idOrUsername')
  public async getUser(@Param() params: GetUserParams): Promise<IResponseUser> {
    const user = await this.usersService.findOneByIdOrUsername(idOrUsername);
    return ResponseUserMapper.map(user);
  }

  // ...
}

Update Email

Since it updtes the email we need to return AuthResponseUserMapper:

import {
  Body,
  // ...
  Patch,
} from '@nestjs/common';
// ...
import { ChangeEmailDto } from './dtos/change-email.dto';
// ...

@Controller('api/users')
export class UsersController {
  // ...

  @Patch('/email')
  public async updateEmail(
    @CurrentUser() id: number,
    @Body() dto: ChangeEmailDto,
  ): Promise<IAuthResponseUser> {
    const user = await this.usersService.updateEmail(id, dto);
    return AuthResponseUserMapper.map(user);
  }

  // ...
}

Update User

// ...
import { UpdateUserDto } from './dtos/update-user.dto';
// ...

@Controller('api/users')
export class UsersController {
  // ...

  @Patch()
  public async updateUser(
    @CurrentUser() id: number,
    @Body() dto: UpdateUserDto,
  ): Promise<IResponseUser> {
    const user = await this.usersService.update(id, dto);
    return ResponseUserMapper.map(user);
  }

  // ...
}

Delete User

import {
  // ...
  Delete,
  // ...,
  HttpStatus,
  // ...
} from '@nestjs/common';
import { PasswordDto } from './dtos/password.dto';
// ...

@Controller('api/users')
export class UsersController {
  // ...

  @Delete()
  @ApiNoContentResponse({
    description: 'The user is deleted.',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body, or wrong password.',
  })
  @ApiUnauthorizedResponse({
    description: 'The user is not logged in.',
  })
  public async deleteUser(
    @CurrentUser() id: number,
    @Body() dto: PasswordDto,
    @Res() res: Response,
  ): Promise<void> {
    await this.usersService.delete(id, dto);
    res
      .clearCookie(this.cookieName, { path: this.cookiePath })
      .status(HttpStatus.NO_CONTENT)
      .send();
  }

  // ...
}

Main file

Finally we need to set-up our main file, start by installing cookie-parser and helmet:

$ yarn add cookie-parser helmet
$ yarn add -D @types/cookie-parser

And add them, as well as a validation pipe to the main file:

import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import cookieParser from 'cookie-parser';
import helmet from 'helmet';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  const configService = app.get(ConfigService);
  app.use(cookieParser(configService.get<string>('COOKIE_SECRET')));
  app.use(helmet());
  app.enableCors({
    credentials: true,
    origin: `https://${configService.get<string>('domain')}`,
  });
  app.useGlobalPipes(
    new ValidationPipe({
      transform: true,
    }),
  );
  await app.listen(configService.get<number>('port'));
}

bootstrap();

(Optional) Swagger

How will our front-end team know how to use our API? This is where documentation is important, to create documentation automatic we can use NestJS Swagger, start by installing it:

$ yarn add @nestjs/swagger

And add descriptions to all DTOS, Mappers and endpoints, I will exemplify with the users, but is exactly the same process for the auth.

DTOs

Add ApiProperty decorators all dtos:

Username DTO

import { ApiProperty } from '@nestjs/swagger';
import { IsString, Length, Matches } from 'class-validator';
import { SLUG_REGEX } from '../../common/consts/regex.const';

export abstract class UsernameDto {
  @ApiProperty({
    description: 'The username of the user',
    minLength: 3,
    maxLength: 106,
    example: 'my-username',
    type: String,
  })
  @IsString()
  @Length(3, 106)
  @Matches(SLUG_REGEX, {
    message: 'Username must be a valid slugs',
  })
  public username: string;
}

Password DTO

import { ApiProperty } from '@nestjs/swagger';
import { IsString, MinLength } from 'class-validator';

export abstract class PasswordDto {
  @ApiProperty({
    description: 'The password of the user',
    minLength: 1,
    type: String,
  })
  @IsString()
  @MinLength(1)
  public password: string;
}

Change Email DTO

import { ApiProperty } from '@nestjs/swagger';
import { IsEmail, IsString, Length } from 'class-validator';
import { PasswordDto } from './password.dto';

export abstract class ChangeEmailDto extends PasswordDto {
  @ApiProperty({
    description: 'The email of the user',
    example: 'someone@gmail.com',
    minLength: 5,
    maxLength: 255,
    type: String,
  })
  @IsString()
  @IsEmail()
  @Length(5, 255)
  public email: string;
}

Get User Params*

import { ApiProperty } from '@nestjs/swagger';
import { IsString, Length } from 'class-validator';

export abstract class GetUserParams {
  @ApiProperty({
    description: 'The id or username of the user',
    type: String,
    example: "1 or 'username'",
  })
  @IsString()
  @Length(1, 106)
  public idOrUsername: string;
}

Mappers

Add ApiProperty decorators the ResponseUserMapper:

import { ApiProperty } from '@nestjs/swagger';
import { IResponseUser } from '../interfaces/response-user.interface';
import { IUser } from '../interfaces/user.interface';

export class ResponseUserMapper implements IResponseUser {
  @ApiProperty({
    description: 'User id',
    example: 123,
    minimum: 1,
    type: Number,
  })
  public id: number;

  @ApiProperty({
    description: 'User name',
    example: 'John Doe',
    minLength: 3,
    maxLength: 100,
    type: String,
  })
  public name: string;

  @ApiProperty({
    description: 'User username',
    example: 'john.doe1',
    minLength: 3,
    maxLength: 106,
    type: String,
  })
  public username: string;

  @ApiProperty({
    description: 'User creation date',
    example: '2021-01-01T00:00:00.000Z',
    type: String,
  })
  public createdAt: string;

  @ApiProperty({
    description: 'User last update date',
    example: '2021-01-01T00:00:00.000Z',
    type: String,
  })
  public updatedAt: string;

  constructor(values: IResponseUser) {
    Object.assign(this, values);
  }

  public static map(user: IUser): ResponseUserMapper {
    return new ResponseUserMapper({
      id: user.id,
      name: user.name,
      username: user.username,
      createdAt: user.createdAt.toISOString(),
      updatedAt: user.updatedAt.toISOString(),
    });
  }
}

Enpoints

NestJS swagger exports an api response type for each HTTP status code, add one for each response an endpoint can have, with a type of a Mapper for the 200's status codes:

// ...
import {
  ApiBadRequestResponse,
  ApiNoContentResponse,
  ApiNotFoundResponse,
  ApiOkResponse,
  ApiTags,
  ApiUnauthorizedResponse,
} from '@nestjs/swagger';
// ...

@ApiTags('Users')
@Controller('api/users')
export class UsersController {
  // ...

  @Public()
  @Get('/:idOrUsername')
  @ApiOkResponse({
    type: ResponseUserMapper,
    description: 'The user is found and returned.',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body',
  })
  @ApiNotFoundResponse({
    description: 'The user is not found.',
  })
  public async getUser(
    // ...
  ): Promise<IResponseUser> {
    // ...
  }

  @Patch('/email')
  @ApiOkResponse({
    type: AuthResponseUserMapper,
    description: 'The email is updated, and the user is returned.',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body, or wrong password.',
  })
  @ApiUnauthorizedResponse({
    description: 'The user is not logged in.',
  })
  public async updateEmail(
    // ...
  ): Promise<IAuthResponseUser> {
    // ...
  }

  @Patch()
  @ApiOkResponse({
    type: ResponseUserMapper,
    description: 'The username is updated.',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body.',
  })
  @ApiUnauthorizedResponse({
    description: 'The user is not logged in.',
  })
  public async updateUser(
    // ...
  ): Promise<IResponseUser> {
    // ...
  }

  @Delete()
  @ApiNoContentResponse({
    description: 'The user is deleted.',
  })
  @ApiBadRequestResponse({
    description: 'Something is invalid on the request body, or wrong password.',
  })
  @ApiUnauthorizedResponse({
    description: 'The user is not logged in.',
  })
  public async deleteUser(
    // ...
  ): Promise<void> {
    // ...
  }
}

Main file

Finally, add the document builder to your main file:

// ...
import { DocumentBuilder, SwaggerModule } from '@nestjs/swagger';
// ...

async function bootstrap() {
  // ...

  const swaggerConfig = new DocumentBuilder()
    .setTitle('NestJS Authentication API')
    .setDescription('A OAuth2.0 authentication API made with NestJS')
    .setVersion('0.0.1')
    .addBearerAuth()
    .addTag('Authentication API')
    .build();
  const document = SwaggerModule.createDocument(app, swaggerConfig);
  SwaggerModule.setup('api/docs', app, document);

  // ...
}

bootstrap();

Conclusion

With this you now have a full functioning Authentication Microsservice to use on your REST apps. The github for this tutorial can be found here.

About the Author

Do not miss out on any of my latest articles – follow me here on dev and LinkedIn to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!