Series Intro
Nowadays, due to performance constraints a lot of companies are moving away from NodeJS to Go for their network and API stacks. This series is for developers interest in making the jump from Node.js to Go.
On this series, I will teach the process of building an IDP (identity provider), a project I started to create a comunity driven self-hosted opensource version of clerk.
Series Requirements
This series requires some basic knowledge of Go, as it will teach you how to structure a Go API, and how to transfer Node.js to Go skills. Before starting this series, I recommend reading the following two books to gain familiarity with the Go programming language:
- Go Fundamentals: an introduction to the go programming language.
- Network Programming with Go: an introduction to Network Programming, from network nodes and TCP to HTTP fundamentals.
Or if you're in a hurry just take a tour of Go in the official website.
Topics to cover
This series will be divided in 6 parts:
- Project structure and Database design: I'll lay the foundation by setting up the project structure and designing the database schema for the IDP.
- Local and External Authentication with JWTs: I will touch on how I implemented local authentication using JSON Web Tokens (JWTs) Bearer tokens (RFC-6750) for our Accounts (tenants) including external auth (RFC-6749).
-
Production mailer: to send emails we will create our own queue using redis and the
"net/smtp"package from the standard library. -
Component testing with docker-compose: I'll touch on how to ensure the reliability of our API by writing endpoint tests using Docker Compose and Go's standard
net/httpandtestingpackages. - Apps and account mapping: add multiple apps support for each account.
- Deploying our API: We will deploy our App to a VPS (virtual private server) using coolify.
Intro
A well-organized project structure is a requirement for building maintainable and scalable APIs. In this article, we'll explore how to structure a Go API using a very common pattern in the Node.JS world, using the Model, Service, Controller (MSC) architecture.
We will leverage the following stack:
- Fiber: Go framework inspired in express.
- PostgreSQL with SQLC: SQLC is a Go code generator with type safety and compile-time checks for our SQL queries.
- Redis with fiber's Redis abstraction: a caching storage abstraction for fiber using Redis.
We will also design the database schema and create the necessary migrations to set up our data layer.
Packages structure
For the structure of our code we'll levarage the go-blueprint CLI tool made by the youtuber Melky.
To start the project install and initialize the project:
$ go install github.com/melkeydev/go-blueprint@latest
$ makedir devlogs && cd devlogs
$ go-blueprint create --name idp --framework fiber --driver postgres --git commit
This will lead to an initial file structure like:
C:/Users/user/IdeProjects/devlogs/idp:
├─── cmd
│ ├─── api
│ │ main.go
├─── internal
│ ├─── database
│ │ database.go
│ ├─── server
│ │ server.go
│ │ routes.go
│ │ routes_test.go
| ...
While the name of the module will be devlogs/idp, this is a bit far of what we want, which would be github.com/your-username/devlogs/idp. So update the module name to github.com/your-username/devlogs/idp and run:
$ go mod tidy
And move the files around for a MSC (Model, Service, Controller) architecture:
C:/Users/user/IdeProjects/devlogs/idp:
├─── cmd
│ ├─── api
│ │ main.go
├─── internal
│ ├─── config
│ │ config.go
│ │ encryption.go
│ │ logger.go
│ │ oauth.go
│ │ rate_limiter.go
│ │ tokens.go
│ │ ...
│ ├─── controllers
│ │ ├─── bodies
│ │ │ common.go
│ │ │ ...
│ │ ├─── params
│ │ │ common.go
│ │ │ ...
│ │ ├─── paths
│ │ │ common.go
│ │ │ ...
│ │ auth.go
│ │ controllers.go
│ │ ...
│ ├─── exceptions
│ │ controllers.go
│ │ services.go
│ ├─── providers
│ │ ├─── cache
│ │ │ cache.go
│ │ │ ...
│ │ ├─── database
│ │ │ database.go
│ │ │ ...
│ │ ├─── mailer
│ │ │ mailer.go
│ │ │ ...
│ │ ├─── oauth
│ │ │ oauth.go
│ │ │ ...
│ │ ├─── tokens
│ │ │ tokens.go
│ │ │ ...
│ │ ├─── encryption
│ │ │ encryption.go
│ │ │ ...
│ ├─── database
│ │ database.go
│ ├─── server
│ │ ├─── routes
│ │ │ routes.go
│ │ │ ...
│ │ logger.go
│ │ server.go
│ │ routes.go
│ ├─── services
│ │ ├─── dtos
│ │ │ common.go
│ │ │ ...
│ ├─── utils
│ │ ...
| ...
Most logic will live on the internal folder, where the main structure is as follows:
- Config: centralizes and loads all environment configurations.
-
Server: contains the Fiber instance initialization and endpoints routing
- /routes: specifies the routes for a given controller method.
- logger.go: builds the default configuration for the structure logger.
-
routes.go: has the main method to register all routes
RegisterFiberRoutes. -
server.go: builds a
FiberServerinstance with thefiber.Appinstance and routes router.
-
Controllers: handle incoming HTTP requests, process them using services, and return the appropriate HTTP responses.
- /bodies: specifies the controllers bodies.
- /params: specifies the controllers URL and Query parameters.
- /paths: where the routes path constants are defined so they can be easily shared.
-
Services: where most of our business logic and interactions with external providers lives.
- /dtos: where our data transfer objects are defined,
-
Providers: where we have our external providers such as Data stores, JWTs, etc.
- /cache: Redis storage interactions.
- /database: PostgreSQL interactions and model structs implementations.
- /mailer: connection to our mailing queue.
- /oauth: oauth interactions with external authentication providers.
- /tokens: jwt signing and verifying.
- /encryption: envelope encryption logic provider.
Configuration
As most APIs, the configuration will come from environment variables, we can load them with the os package from the standard library.
For local development we can load the environment variables from a .env file, by installing the following package:
$ go get github.com/joho/godotenv
This variables most of the time act as secrets, hence we will use a OOP style encapsulation with them where all members of configurations structs are private and immutable, while you can get their values with getters.
Our IDP will have main configurations groups apart from the base config:
- Logger: whether debug is active and the env to chose whether we want a text or JSON handler.
- Tokens: the private/public keys pairs and TTL for signing and verifying JWTs.
- OAuth: collection of client ids and secrets for each of the external authentication providers.
- Rate Limiter: specifies the max number of request withing a window that an IP can make.
- Encryption: list of KEK (Key encryption keys) provided to the environment.
Logger
On the internal/config directory create a logger.go file with the following struct and New function builder:
package config
type LoggerConfig struct {
isDebug bool
env string
serviceName string
}
func NewLoggerConfig(isDebug bool, env, serviceName string) LoggerConfig {
return LoggerConfig{
isDebug: isDebug,
env: env,
serviceName: serviceName,
}
}
func (l *LoggerConfig) IsDebug() bool {
return l.isDebug
}
func (l *LoggerConfig) Env() string {
return l.env
}
func (l *LoggerConfig) ServiceName() string {
return l.serviceName
}
NOTE: for most methods in Go it is recommended using pointer receivers. Pointers are just address pointing to the underlying memory of the struct, but for simplicity you can think of them as pass by reference instead of pass by value.
Tokens
JWT have 3 main parts that need to be provided by the environment:
- Public Key: the key that will be used to verify the token.
- Private Key: the key that will be used to sign the token.
- Time to live: the TTL in seconds of the token.
And the service will have 7 different JWTs with different key pairs:
- Access: for the access token.
- Account Credentials: for machine to machine access.
- Refresh: for the refresh token to refresh the access token on client to machine access.
- Confirmation: for the email confirmation token (JWT for confirmation can be overkill, however it saves on ram memory by not saving the hashed email in cache).
- Reset: for email resetting.
- OAuth: for a temporary authorization header for the token exchange.
- Two Factor: for a temporary authorization header for passing the two factor code.
Single Configuration
Create a tokens.go file on the /internal/config directory with the struct, new method and getters:
package config
type SingleJwtConfig struct {
publicKey string
privateKey string
ttlSec int64
}
func NewSingleJwtConfig(publicKey, privateKey string, ttlSec int64) SingleJwtConfig {
return SingleJwtConfig{
publicKey: publicKey,
privateKey: privateKey,
ttlSec: ttlSec,
}
}
func (s *SingleJwtConfig) PublicKey() string {
return s.publicKey
}
func (s *SingleJwtConfig) PrivateKey() string {
return s.privateKey
}
func (s *SingleJwtConfig) TtlSec() int64 {
return s.ttlSec
}
Tokens Configuration
Now do the same as previously but for each token type:
package config
// ...
type TokensConfig struct {
access SingleJwtConfig
accountCredentials SingleJwtConfig
refresh SingleJwtConfig
confirm SingleJwtConfig
reset SingleJwtConfig
oAuth SingleJwtConfig
twoFA SingleJwtConfig
}
func NewTokensConfig(
access SingleJwtConfig,
accountCredentials SingleJwtConfig,
refresh SingleJwtConfig,
confirm SingleJwtConfig,
oAuth SingleJwtConfig,
twoFA SingleJwtConfig,
) TokensConfig {
return TokensConfig{
access: access,
accountCredentials: accountCredentials,
refresh: refresh,
confirm: confirm,
oAuth: oAuth,
twoFA: twoFA,
}
}
func (t *TokensConfig) Access() SingleJwtConfig {
return t.access
}
func (t *TokensConfig) AccountCredentials() SingleJwtConfig {
return t.accountKeys
}
func (t *TokensConfig) Refresh() SingleJwtConfig {
return t.refresh
}
func (t *TokensConfig) Confirm() SingleJwtConfig {
return t.confirm
}
func (t *TokensConfig) Reset() SingleJwtConfig {
return t.reset
}
func (t *TokensConfig) OAuth() SingleJwtConfig {
return t.oAuth
}
func (t *TokensConfig) TwoFA() SingleJwtConfig {
return t.twoFA
}
OAuth
External authentication providers have two environment variables each:
- Client ID: the identifier of the app on the external IDP
- Client Secret: a secure secret to fetch the user info from the code exchange.
And we will add support for the 5 main western ones:
- GitHub
- Apple
- Microsoft
Single OAuth provider configuration
Create a oauth.go file on the /internal/config directory with the struct, new method and getters:
package config
type OAuthProviderConfig struct {
clientID string
clientSecret string
}
func NewOAuthProvider(clientID, clientSecret string) OAuthProviderConfig {
return OAuthProviderConfig{
clientID: clientID,
clientSecret: clientSecret,
}
}
func (o *OAuthProviderConfig) ClientID() string {
return o.clientID
}
func (o *OAuthProviderConfig) ClientSecret() string {
return o.clientSecret
}
OAuth providers configuration
Create a struct, new method and getter for each provider:
package config
// ...
type OAuthProvidersConfig struct {
gitHub OAuthProviderConfig
google OAuthProviderConfig
facebook OAuthProviderConfig
apple OAuthProviderConfig
microsoft OAuthProviderConfig
}
func NewOAuthProviders(gitHub, google, facebook, apple, microsoft OAuthProviderConfig) OAuthProvidersConfig {
return OAuthProvidersConfig{
gitHub: gitHub,
google: google,
facebook: facebook,
apple: apple,
microsoft: microsoft,
}
}
func (o *OAuthProvidersConfig) GitHub() OAuthProviderConfig {
return o.gitHub
}
func (o *OAuthProvidersConfig) Google() OAuthProviderConfig {
return o.google
}
func (o *OAuthProvidersConfig) Facebook() OAuthProviderConfig {
return o.facebook
}
func (o *OAuthProvidersConfig) Apple() OAuthProviderConfig {
return o.apple
}
func (o *OAuthProvidersConfig) Microsoft() OAuthProviderConfig {
return o.microsoft
}
Rate Limiter
The rate limiter has only two options:
- Max: the number of maximum number of requests per window size.
- Expiration Seconds: the window size in seconds.
Just create the rate_limiter.go file on the /internal/config directory as follows:
package config
type RateLimiterConfig struct {
max int64
expSec int64
}
func NewRateLimiterConfig(max, expSec int64) RateLimiterConfig {
return RateLimiterConfig{
max: max,
expSec: expSec,
}
}
func (r *RateLimiterConfig) Max() int64 {
return r.max
}
func (r *RateLimiterConfig) ExpSec() int64 {
return r.expSec
}
Encryption
Finaly to configure the KEKs for each encryption space:
- Accounts: encryption of secrets for TOTP two factor auth.
- Apps: encryption for the APPs private keys.
- Users: encryption of secrets for TOTP two factor auth.
As well as old keys for easy keys rotation.
Add them to encryption.go file:
package config
import "encoding/json"
type EncryptionConfig struct {
accountSecret string
appSecret string
userSecret string
oldSecrets []string
}
func NewEncryptionConfig(accountSecret, appSecret, userSecret, oldSecrets string) EncryptionConfig {
var secretSlice []string
if err := json.Unmarshal([]byte(oldSecrets), &secretSlice); err != nil {
panic(err)
}
return EncryptionConfig{
accountSecret: accountSecret,
appSecret: appSecret,
userSecret: userSecret,
oldSecrets: secretSlice,
}
}
func (e *EncryptionConfig) AccountSecret() string {
return e.accountSecret
}
func (e *EncryptionConfig) AppSecret() string {
return e.appSecret
}
func (e *EncryptionConfig) UserSecret() string {
return e.userSecret
}
func (e *EncryptionConfig) OldSecrets() []string {
return e.oldSecrets
}
Base config
On the base config you just need to add the environment variables in an array, as well as getters for each config parameter of the struct:
package config
import (
"log/slog"
"os"
"strconv"
"strings"
"github.com/google/uuid"
"github.com/joho/godotenv"
)
type Config struct {
port int64
maxProcs int64
databaseURL string
redisURL string
frontendDomain string
backendDomain string
cookieSecret string
cookieName string
emailPubChannel string
encryptionSecret string
serviceID uuid.UUID
loggerConfig LoggerConfig
tokensConfig TokensConfig
oAuthProvidersConfig OAuthProvidersConfig
rateLimiterConfig RateLimiterConfig
encryptionConfig EncryptionConfig
}
func (c *Config) Port() int64 {
return c.port
}
func (c *Config) MaxProcs() int64 {
return c.maxProcs
}
func (c *Config) DatabaseURL() string {
return c.databaseURL
}
func (c *Config) RedisURL() string {
return c.redisURL
}
func (c *Config) FrontendDomain() string {
return c.frontendDomain
}
func (c *Config) BackendDomain() string {
return c.backendDomain
}
func (c *Config) CookieSecret() string {
return c.cookieSecret
}
func (c *Config) CookieName() string {
return c.cookieName
}
func (c *Config) EmailPubChannel() string {
return c.emailPubChannel
}
func (c *Config) EncryptionSecret() string {
return c.encryptionSecret
}
func (c *Config) ServiceID() uuid.UUID {
return c.serviceID
}
func (c *Config) LoggerConfig() LoggerConfig {
return c.loggerConfig
}
func (c *Config) TokensConfig() TokensConfig {
return c.tokensConfig
}
func (c *Config) OAuthProvidersConfig() OAuthProvidersConfig {
return c.oAuthProvidersConfig
}
func (c *Config) RateLimiterConfig() RateLimiterConfig {
return c.rateLimiterConfig
}
func (c *Config) EncryptionConfig() EncryptionConfig {
return c.encryptionConfig
}
var variables = [40]string{
"PORT",
"ENV",
"DEBUG",
"SERVICE_NAME",
"SERVICE_ID",
"MAX_PROCS",
"DATABASE_URL",
"REDIS_URL",
"FRONTEND_DOMAIN",
"BACKEND_DOMAIN",
"COOKIE_SECRET",
"COOKIE_NAME",
"RATE_LIMITER_MAX",
"RATE_LIMITER_EXP_SEC",
"EMAIL_PUB_CHANNEL",
"JWT_ACCESS_PUBLIC_KEY",
"JWT_ACCESS_PRIVATE_KEY",
"JWT_ACCESS_TTL_SEC",
"JWT_ACCOUNT_CREDENTIALS_PUBLIC_KEY",
"JWT_ACCOUNT_CREDENTIALS_PRIVATE_KEY",
"JWT_ACCOUNT_CREDENTIALS_TTL_SEC",
"JWT_REFRESH_PUBLIC_KEY",
"JWT_REFRESH_PRIVATE_KEY",
"JWT_REFRESH_TTL_SEC",
"JWT_CONFIRM_PUBLIC_KEY",
"JWT_CONFIRM_PRIVATE_KEY",
"JWT_CONFIRM_TTL_SEC",
"JWT_RESET_PUBLIC_KEY",
"JWT_RESET_PRIVATE_KEY",
"JWT_RESET_TTL_SEC",
"JWT_OAUTH_PUBLIC_KEY",
"JWT_OAUTH_PRIVATE_KEY",
"JWT_OAUTH_TTL_SEC",
"JWT_2FA_PUBLIC_KEY",
"JWT_2FA_PRIVATE_KEY",
"JWT_2FA_TTL_SEC",
"ACCOUNT_SECRET",
"APP_SECRET",
"USER_SECRET",
"OLD_SECRETS",
}
var optionalVariables = [17]string{
"GITHUB_CLIENT_ID",
"GITHUB_CLIENT_SECRET",
"GOOGLE_CLIENT_ID",
"GOOGLE_CLIENT_SECRET",
"FACEBOOK_CLIENT_ID",
"FACEBOOK_CLIENT_SECRET",
"APPLE_CLIENT_ID",
"APPLE_CLIENT_SECRET",
"MICROSOFT_CLIENT_ID",
"MICROSOFT_CLIENT_SECRET",
"OLD_JWT_ACCESS_PUBLIC_KEY",
"OLD_JWT_ACCOUNT_CREDENTIALS_PUBLIC_KEY",
"OLD_JWT_REFRESH_PUBLIC_KEY",
"OLD_JWT_CONFIRM_PUBLIC_KEY",
"OLD_JWT_RESET_PUBLIC_KEY",
"OLD_JWT_OAUTH_PUBLIC_KEY",
"OLD_JWT_2FA_PUBLIC_KEY",
}
var numerics = [11]string{
"PORT",
"MAX_PROCS",
"JWT_ACCESS_TTL_SEC",
"JWT_ACCOUNT_CREDENTIALS_TTL_SEC",
"JWT_REFRESH_TTL_SEC",
"JWT_CONFIRM_TTL_SEC",
"JWT_RESET_TTL_SEC",
"JWT_OAUTH_TTL_SEC",
"JWT_2FA_TTL_SEC",
"RATE_LIMITER_MAX",
"RATE_LIMITER_EXP_SEC",
}
func NewConfig(logger *slog.Logger, envPath string) Config {
err := godotenv.Load(envPath)
if err != nil {
logger.Error("Error loading .env file")
}
variablesMap := make(map[string]string)
for _, variable := range variables {
value := os.Getenv(variable)
if value == "" {
logger.Error(variable + " is not set")
panic(variable + " is not set")
}
variablesMap[variable] = value
}
for _, variable := range optionalVariables {
value := os.Getenv(variable)
variablesMap[variable] = value
}
intMap := make(map[string]int64)
for _, numeric := range numerics {
value, err := strconv.ParseInt(variablesMap[numeric], 10, 0)
if err != nil {
logger.Error(numeric + " is not an integer")
panic(numeric + " is not an integer")
}
intMap[numeric] = value
}
env := variablesMap["ENV"]
return Config{
port: intMap["PORT"],
maxProcs: intMap["MAX_PROCS"],
databaseURL: variablesMap["DATABASE_URL"],
redisURL: variablesMap["REDIS_URL"],
frontendDomain: variablesMap["FRONTEND_DOMAIN"],
backendDomain: variablesMap["BACKEND_DOMAIN"],
cookieSecret: variablesMap["COOKIE_SECRET"],
cookieName: variablesMap["COOKIE_NAME"],
emailPubChannel: variablesMap["EMAIL_PUB_CHANNEL"],
serviceID: uuid.MustParse(variablesMap["SERVICE_ID"]),
loggerConfig: NewLoggerConfig(
strings.ToLower(variablesMap["DEBUG"]) == "true",
env,
variablesMap["SERVICE_NAME"],
),
tokensConfig: NewTokensConfig(
NewSingleJwtConfig(
variablesMap["JWT_ACCESS_PUBLIC_KEY"],
variablesMap["JWT_ACCESS_PRIVATE_KEY"],
variablesMap["OLD_JWT_ACCESS_PUBLIC_KEY"],
intMap["JWT_ACCESS_TTL_SEC"],
),
NewSingleJwtConfig(
variablesMap["JWT_ACCOUNT_CREDENTIALS_PUBLIC_KEY"],
variablesMap["JWT_ACCOUNT_CREDENTIALS_PRIVATE_KEY"],
variablesMap["OLD_JWT_ACCOUNT_CREDENTIALS_PUBLIC_KEY"],
intMap["JWT_ACCOUNT_CREDENTIALS_TTL_SEC"],
),
NewSingleJwtConfig(
variablesMap["JWT_REFRESH_PUBLIC_KEY"],
variablesMap["JWT_REFRESH_PRIVATE_KEY"],
variablesMap["OLD_JWT_REFRESH_PUBLIC_KEY"],
intMap["JWT_REFRESH_TTL_SEC"],
),
NewSingleJwtConfig(
variablesMap["JWT_CONFIRM_PUBLIC_KEY"],
variablesMap["JWT_CONFIRM_PRIVATE_KEY"],
variablesMap["OLD_JWT_CONFIRM_PUBLIC_KEY"],
intMap["JWT_CONFIRM_TTL_SEC"],
),
NewSingleJwtConfig(
variablesMap["JWT_RESET_PUBLIC_KEY"],
variablesMap["JWT_RESET_PRIVATE_KEY"],
variablesMap["OLD_JWT_RESET_PUBLIC_KEY"],
intMap["JWT_RESET_TTL_SEC"],
),
NewSingleJwtConfig(
variablesMap["JWT_OAUTH_PUBLIC_KEY"],
variablesMap["JWT_OAUTH_PRIVATE_KEY"],
variablesMap["OLD_JWT_OAUTH_PUBLIC_KEY"],
intMap["JWT_OAUTH_TTL_SEC"],
),
NewSingleJwtConfig(
variablesMap["JWT_2FA_PUBLIC_KEY"],
variablesMap["JWT_2FA_PRIVATE_KEY"],
variablesMap["OLD_JWT_2FA_PUBLIC_KEY"],
intMap["JWT_2FA_TTL_SEC"],
),
),
oAuthProvidersConfig: NewOAuthProviders(
NewOAuthProvider(variablesMap["GITHUB_CLIENT_ID"], variablesMap["GITHUB_CLIENT_SECRET"]),
NewOAuthProvider(variablesMap["GOOGLE_CLIENT_ID"], variablesMap["GOOGLE_CLIENT_SECRET"]),
NewOAuthProvider(variablesMap["FACEBOOK_CLIENT_ID"], variablesMap["FACEBOOK_CLIENT_SECRET"]),
NewOAuthProvider(variablesMap["APPLE_CLIENT_ID"], variablesMap["APPLE_CLIENT_SECRET"]),
NewOAuthProvider(variablesMap["MICROSOFT_CLIENT_ID"], variablesMap["MICROSOFT_CLIENT_SECRET"]),
),
rateLimiterConfig: NewRateLimiterConfig(
intMap["RATE_LIMITER_MAX"],
intMap["RATE_LIMITER_EXP_SEC"],
),
encryptionConfig: NewEncryptionConfig(
variablesMap["ACCOUNT_SECRET"],
variablesMap["APP_SECRET"],
variablesMap["USER_SECRET"],
variablesMap["OLD_SECRETS"],
),
}
}
Providers
With the config done, we can start creating the providers:
- cache: Redis cache;
- database: PostgreSQL database;
- mailer: Redis PubSub publisher for emails (the queue will be built on a subsquent article);
- oauth: external authentication providers;
- tokens: jwt signing and verifying;
- encrytion: the provider for envolope encryption (more of an isolation of logic than a proper provider. Encryption logic should always be isolated in production)
Utilities
Observasibility of APIs is important for production debugging, so lets create an utility to build a structure logger.
To make it easy to locate the function and logic, we need to pass the package location, method, and layer. Create the utility on the root utils package. Create a logger.go file and add the following code:
package utils
import (
"log/slog"
)
type LogLayer = string
const (
ControllersLogLayer LogLayer = "controllers"
ServicesLogLayer LogLayer = "services"
ProvidersLogLayer LogLayer = "providers"
)
type LoggerOptions struct {
Layer string
Location string
Method string
RequestID string
}
func BuildLogger(logger *slog.Logger, opts LoggerOptions) *slog.Logger {
return logger.With(
"layer", opts.Layer,
"location", opts.Location,
"method", opts.Method,
"requestId", opts.RequestID,
)
}
Cache
Our cache implementation will just extend the fiber Redis Storage abstraction. On the internal/providers create a /cache directory and add cache.go file with the following struct and new function:
package cache
import (
"context"
"log/slog"
fiberRedis "github.com/gofiber/storage/redis/v3"
"github.com/redis/go-redis/v9"
)
const logLayer string = "cache"
type Cache struct {
logger *slog.Logger
storage *fiberRedis.Storage
}
func NewCache(logger *slog.Logger, storage *fiberRedis.Storage) *Cache {
return &Cache{
logger: logger,
storage: storage,
}
}
With three common methods to reset the cache, get the redist client and ping the cache:
package cache
// ...
func (c *Cache) ResetCache() error {
return c.storage.Reset()
}
func (c *Cache) Client() redis.UniversalClient {
return c.storage.Conn()
}
func (c *Cache) Ping(ctx context.Context) error {
return c.Client().Ping(ctx).Err()
}
Database
The database set-up is a bit more complex. We will write both our migrations and queries in SQL and use migrate to migrate our DB changes, and SQLC to generate safe Go SQL query codes.
Start by installing the necessary dependencies:
$ go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest
$ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
And create a config file for SQLC as sqlc.yaml on the idp root folder:
version: "2"
sql:
- schema: "internal/providers/database/migrations"
queries: "internal/providers/database/queries"
engine: "postgresql"
gen:
go:
package: "database"
out: "internal/providers/database"
sql_package: "pgx/v5"
emit_empty_slices: true
overrides:
- db_type: "timestamptz"
go_type: "time.Time"
- db_type: "uuid"
go_type: "github.com/google/uuid.UUID"
Database Schema
Since we are creating an IDP we will have three main entitites:
-
Accounts(also known as tenants): the main users/admins that can create APPs that provide authentication to their own services. -
Apps: the authentication provider configuration for a given service or services. -
Users: users that have registered for theaccount'sapps.
This leads to the following somewhat complex datamodel:
Initial Migration
With migrate installed run the following command:
$ migrate create --ext sql --dir ./internal/providers/database/migration create_initial_schema
This will generate two files:
YYYYMMDDHHMMSS_create_initial_schema.up.sqlYYYYMMDDHHMMSS_create_initial_schema.down.sql
To the up migration copy the following SQL:
CREATE TABLE "accounts" (
"id" serial PRIMARY KEY,
"first_name" varchar(50) NOT NULL,
"last_name" varchar(50) NOT NULL,
"username" varchar(109) NOT NULL,
"email" varchar(250) NOT NULL,
"password" text,
"version" integer NOT NULL DEFAULT 1,
"is_confirmed" boolean NOT NULL DEFAULT false,
"two_factor_type" varchar(5) NOT NULL DEFAULT 'none',
"created_at" timestamp NOT NULL DEFAULT (now()),
"updated_at" timestamp NOT NULL DEFAULT (now())
);
CREATE TABLE "account_totps" (
"id" serial PRIMARY KEY,
"account_id" integer NOT NULL,
"url" varchar(250) NOT NULL,
"secret" text NOT NULL,
"dek" text NOT NULL,
"recovery_codes" jsonb NOT NULL,
"created_at" timestamp NOT NULL DEFAULT (now()),
"updated_at" timestamp NOT NULL DEFAULT (now())
);
CREATE TABLE "account_credentials" (
"id" serial PRIMARY KEY,
"account_id" integer NOT NULL,
"scopes" jsonb NOT NULL,
"client_id" varchar(22) NOT NULL,
"client_secret" text NOT NULL,
"created_at" timestamp NOT NULL DEFAULT (now()),
"updated_at" timestamp NOT NULL DEFAULT (now())
);
CREATE TABLE "auth_providers" (
"id" serial PRIMARY KEY,
"email" varchar(250) NOT NULL,
"provider" varchar(10) NOT NULL,
"created_at" timestamp NOT NULL DEFAULT (now()),
"updated_at" timestamp NOT NULL DEFAULT (now())
);
CREATE TABLE "apps" (
"id" serial PRIMARY KEY,
"account_id" integer NOT NULL,
"name" varchar(50) NOT NULL,
"client_id" varchar(22) NOT NULL,
"client_secret" text NOT NULL,
"dek" text NOT NULL,
"callback_uris" varchar(250)[] NOT NULL DEFAULT '{}',
"logout_uris" varchar(250)[] NOT NULL DEFAULT '{}',
"user_scopes" jsonb NOT NULL DEFAULT '{ "email": true, "name": true }',
"app_providers" jsonb NOT NULL DEFAULT '{ "email_password": true }',
"id_token_ttl" integer NOT NULL DEFAULT 3600,
"jwt_crypto_suite" varchar(7) NOT NULL DEFAULT 'ES256',
"created_at" timestamp NOT NULL DEFAULT (now()),
"updated_at" timestamp NOT NULL DEFAULT (now())
);
CREATE TABLE "app_keys" (
"id" serial PRIMARY KEY,
"app_id" integer NOT NULL,
"account_id" integer NOT NULL,
"name" varchar(10) NOT NULL,
"jwt_crypto_suite" varchar(7) NOT NULL,
"public_key" jsonb NOT NULL,
"private_key" text NOT NULL,
"created_at" timestamp NOT NULL DEFAULT (now()),
"updated_at" timestamp NOT NULL DEFAULT (now())
);
CREATE TABLE "users" (
"id" serial PRIMARY KEY,
"account_id" integer NOT NULL,
"email" varchar(250) NOT NULL,
"password" text,
"version" integer NOT NULL DEFAULT 1,
"two_factor_type" varchar(5) NOT NULL DEFAULT 'none',
"user_data" jsonb NOT NULL DEFAULT '{}',
"created_at" timestamp NOT NULL DEFAULT (now()),
"updated_at" timestamp NOT NULL DEFAULT (now())
);
CREATE TABLE "user_totps" (
"id" serial PRIMARY KEY,
"user_id" integer NOT NULL,
"url" varchar(250) NOT NULL,
"secret" text NOT NULL,
"dek" text NOT NULL,
"recovery_codes" jsonb NOT NULL,
"created_at" timestamp NOT NULL DEFAULT (now()),
"updated_at" timestamp NOT NULL DEFAULT (now())
);
CREATE TABLE "user_auth_providers" (
"id" serial PRIMARY KEY,
"user_id" integer NOT NULL,
"email" varchar(250) NOT NULL,
"provider" varchar(10) NOT NULL,
"account_id" integer NOT NULL,
"created_at" timestamp NOT NULL DEFAULT (now()),
"updated_at" timestamp NOT NULL DEFAULT (now())
);
CREATE TABLE "blacklisted_tokens" (
"id" uuid PRIMARY KEY,
"expires_at" timestamp NOT NULL,
"created_at" timestamp NOT NULL DEFAULT (now())
);
CREATE UNIQUE INDEX "accounts_email_uidx" ON "accounts" ("email");
CREATE UNIQUE INDEX "accounts_username_uidx" ON "accounts" ("username");
CREATE UNIQUE INDEX "accounts_totps_account_id_uidx" ON "account_totps" ("account_id");
CREATE UNIQUE INDEX "account_credentials_client_id_uidx" ON "account_credentials" ("client_id");
CREATE INDEX "account_credentials_account_id_idx" ON "account_credentials" ("account_id");
CREATE INDEX "auth_providers_email_idx" ON "auth_providers" ("email");
CREATE UNIQUE INDEX "auth_providers_email_provider_uidx" ON "auth_providers" ("email", "provider");
CREATE INDEX "apps_account_id_idx" ON "apps" ("account_id");
CREATE UNIQUE INDEX "client_id_uidx" ON "apps" ("client_id");
CREATE INDEX "app_keys_app_id_idx" ON "app_keys" ("app_id");
CREATE INDEX "app_keys_account_id_idx" ON "app_keys" ("account_id");
CREATE UNIQUE INDEX "app_keys_name_app_id_uidx" ON "app_keys" ("name", "app_id");
CREATE UNIQUE INDEX "users_account_id_email_uidx" ON "users" ("account_id", "email");
CREATE INDEX "users_account_id_idx" ON "users" ("account_id");
CREATE UNIQUE INDEX "user_totps_user_id_uidx" ON "user_totps" ("user_id");
CREATE INDEX "user_auth_provider_email_idx" ON "user_auth_providers" ("email");
CREATE INDEX "user_auth_provider_user_id_idx" ON "user_auth_providers" ("user_id");
CREATE UNIQUE INDEX "user_auth_provider_account_id_provider_uidx" ON "user_auth_providers" ("email", "account_id", "provider");
CREATE INDEX "user_auth_provider_account_id_idx" ON "user_auth_providers" ("account_id");
ALTER TABLE "account_totps" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;
ALTER TABLE "account_credentials" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;
ALTER TABLE "auth_providers" ADD FOREIGN KEY ("email") REFERENCES "accounts" ("email") ON DELETE CASCADE ON UPDATE CASCADE;
ALTER TABLE "apps" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;
ALTER TABLE "app_keys" ADD FOREIGN KEY ("app_id") REFERENCES "apps" ("id") ON DELETE CASCADE;
ALTER TABLE "app_keys" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;
ALTER TABLE "users" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;
ALTER TABLE "user_totps" ADD FOREIGN KEY ("user_id") REFERENCES "users" ("id") ON DELETE CASCADE;
ALTER TABLE "user_auth_providers" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id") ON DELETE CASCADE;
ALTER TABLE "user_auth_providers" ADD FOREIGN KEY ("user_id") REFERENCES "users" ("id") ON DELETE CASCADE;
While on the down drop all tables if they exist:
DROP TABLE IF EXISTS "user_auth_providers";
DROP TABLE IF EXISTS "user_totps";
DROP TABLE IF EXISTS "users";
DROP TABLE IF EXISTS "app_keys";
DROP TABLE IF EXISTS "apps";
DROP TABLE IF EXISTS "auth_providers";
DROP TABLE IF EXISTS "account_credentials";
DROP TABLE IF EXISTS "account_totps";
DROP TABLE IF EXISTS "accounts";
DROP TABLE IF EXISTS "blacklisted_tokens";
SQLC Code generation
To set up the SQLC generated code we need to start by writting a query on the queries folder. Create a accounts.sql file in the queries directory and add the logic to insert an account:
-- name: CreateAccountWithPassword :one
INSERT INTO "accounts" (
"first_name",
"last_name",
"username",
"email",
"password"
) VALUES (
$1,
$2,
$3,
$4,
$5
) RETURNING *;
SQLC knows what to return and what to call to the go method by the code comment on top of the SQL code.
Then in the terminal run the following command:
$ sqlc generate
This will generate the following files:
models.godb.goaccounts.sql.go
These files are auto-generated and auto-updated when you run the sqlc generate command, hence on a new database.go create the logic to connect to the SQLC generated code:
package database
import (
"context"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/tugascript/devlogs/idp/internal/exceptions"
)
type Database struct {
connPool *pgxpool.Pool
*Queries
}
func NewDatabase(connPool *pgxpool.Pool) *Database {
return &Database{
connPool: connPool,
Queries: New(connPool),
}
}
func (d *Database) BeginTx(ctx context.Context) (*Queries, pgx.Tx, error) {
txn, err := d.connPool.BeginTx(ctx, pgx.TxOptions{
DeferrableMode: pgx.Deferrable,
IsoLevel: pgx.ReadCommitted,
AccessMode: pgx.ReadWrite,
})
if err != nil {
return nil, nil, err
}
return d.WithTx(txn), txn, nil
}
func (d *Database) FinalizeTx(ctx context.Context, txn pgx.Tx, err error, serviceErr *exceptions.ServiceError) {
if serviceErr != nil || err != nil {
if err := txn.Rollback(ctx); err != nil {
panic(err)
}
return
}
if commitErr := txn.Commit(ctx); commitErr != nil {
panic(commitErr)
}
if p := recover(); p != nil {
if err := txn.Rollback(ctx); err != nil {
panic(err)
}
panic(p)
}
}
func (d *Database) RawQuery(ctx context.Context, sql string, args []interface{}) (pgx.Rows, error) {
return d.connPool.Query(ctx, sql, args...)
}
func (d *Database) RawQueryRow(ctx context.Context, sql string, args []interface{}) pgx.Row {
return d.connPool.QueryRow(ctx, sql, args...)
}
func (d *Database) Ping(ctx context.Context) error {
return d.connPool.Ping(ctx)
}
Encryption
Encryption is less of a provider, but more of a logic isolation, hence we'll still load it as a provider.
Utilities
For each secret (or key), we will need to derive a KID (Key ID), this is common logic, so in the internal directory lets create a utils package and on a jwk.go folder add the following files:
-
encoders.go:
package utils import ( "math/big" ) func Base62Encode(bytes []byte) string { var codeBig big.Int codeBig.SetBytes(bytes) return codeBig.Text(62) } -
jwk.go:
package utils import ( "crypto/sha256" ) func ExtractKeyID(keyBytes []byte) string { hash := sha256.Sum256(keyBytes) return Base62Encode(hash[:12]) } secrets.go:
package utils
import (
"crypto/rand"
"encoding/base32"
"encoding/base64"
"encoding/hex"
"math/big"
)
func generateRandomBytes(byteLen int) ([]byte, error) {
b := make([]byte, byteLen)
if _, err := rand.Read(b); err != nil {
return nil, err
}
return b, nil
}
func GenerateBase64Secret(byteLen int) (string, error) {
randomBytes, err := generateRandomBytes(byteLen)
if err != nil {
return "", err
}
return base64.RawURLEncoding.EncodeToString(randomBytes), nil
}
func DecodeBase64Secret(secret string) ([]byte, error) {
decoded, err := base64.RawURLEncoding.DecodeString(secret)
if err != nil {
return nil, err
}
return decoded, nil
}
func GenerateBase62Secret(byteLen int) (string, error) {
randomBytes, err := generateRandomBytes(byteLen)
if err != nil {
return "", err
}
randomInt := new(big.Int).SetBytes(randomBytes)
return randomInt.Text(62), nil
}
func GenerateBase32Secret(byteLen int) (string, error) {
randomBytes, err := generateRandomBytes(byteLen)
if err != nil {
return "", err
}
return base32.StdEncoding.WithPadding(base32.NoPadding).EncodeToString(randomBytes), nil
}
func GenerateHexSecret(byteLen int) (string, error) {
randomBytes, err := generateRandomBytes(byteLen)
if err != nil {
return "", err
}
return hex.EncodeToString(randomBytes), nil
}
Provider
Now with the utilites done, create the following providers/encryption package:
package encryption
import (
"encoding/base64"
"log/slog"
"github.com/tugascript/devlogs/idp/internal/config"
"github.com/tugascript/devlogs/idp/internal/utils"
)
const logLayer string = "encryption"
type Secret struct {
kid string
key []byte
}
type Encryption struct {
logger *slog.Logger
accountSecretKey Secret
appSecretKey Secret
userSecretKey Secret
oldSecrets map[string][]byte
backendDomain string
}
func decodeSecret(secret string) Secret {
// Decode the Base64 code to bytes
decodedKey, err := base64.StdEncoding.DecodeString(secret)
if err != nil {
panic(err)
}
return Secret{
// Generate a Key ID per secret
kid: utils.ExtractKeyID(decodedKey),
key: decodedKey,
}
}
func NewEncryption(
logger *slog.Logger,
cfg config.EncryptionConfig,
backendDomain string,
) *Encryption {
oldSecretsMap := make(map[string][]byte)
for _, s := range cfg.OldSecrets() {
ds := decodeSecret(s)
oldSecretsMap[ds.kid] = ds.key
}
return &Encryption{
logger: logger,
accountSecretKey: decodeSecret(cfg.AccountSecret()),
appSecretKey: decodeSecret(cfg.AppSecret()),
userSecretKey: decodeSecret(cfg.UserSecret()),
oldSecrets: oldSecretsMap,
backendDomain: backendDomain,
}
}
DEK Generation
DEKs (Date Encryption Keys) need to be generated by the system, and are saved in the database, for this reason, create a file called dek.go and add the following logic:
- DEK generation & Encryption:
package encryption
import (
"fmt"
"github.com/tugascript/devlogs/idp/internal/utils"
)
const dekLocation string = "dek"
func generateDEK(keyID string, key []byte) ([]byte, string, error) {
base64DEK, err := utils.GenerateBase64Secret(32)
if err != nil {
return nil, "", err
}
encryptedDEK, err := utils.Encrypt(base64DEK, key)
if err != nil {
return nil, "", err
}
dek, err := utils.DecodeBase64Secret(base64DEK)
if err != nil {
return nil, "", err
}
return dek, fmt.Sprintf("%s:%s", keyID, encryptedDEK), nil
}
// ...
func reEncryptDEK(isOldKey bool, dek, key []byte) (string, error) {
if !isOldKey {
return "", nil
}
return utils.Encrypt(base64.RawURLEncoding.EncodeToString(dek), key)
}
- DEK decryption:
package encryption
import (
"context"
"encoding/base64"
"errors"
"fmt"
"log/slog"
"strings"
"github.com/tugascript/devlogs/idp/internal/utils"
)
// ...
type decryptDEKOptions struct {
storedDEK string
secret *Secret
oldSecrets map[string][]byte
}
func decryptDEK(
logger *slog.Logger,
ctx context.Context,
opts decryptDEKOptions,
) ([]byte, bool, error) {
dekID, encryptedDEK, err := splitDEK(opts.storedDEK)
if err != nil {
logger.ErrorContext(ctx, "Failed to split DEK", "error", err)
return nil, false, err
}
key := opts.secret.key
oldKey := dekID != opts.secret.kid
if oldKey {
var ok bool
key, ok = opts.oldSecrets[dekID]
if !ok {
logger.ErrorContext(ctx, "DEK key ID not found")
return nil, false, errors.New("secret key not found")
}
}
base64DEK, err := utils.Decrypt(encryptedDEK, key)
if err != nil {
logger.ErrorContext(ctx, "Failed to decrypt DEK", "error", err)
return nil, false, err
}
dek, err := utils.DecodeBase64Secret(base64DEK)
if err != nil {
logger.ErrorContext(ctx, "Failed to decode DEK", "error", err)
return nil, false, err
}
return dek, oldKey, nil
}
- Methods to decrypt for Account, App & User:
// ...
func (e *Encryption) decryptAccountDEK(ctx context.Context, requestID, storedDEK string) ([]byte, bool, error) {
logger := utils.BuildLogger(e.logger, utils.LoggerOptions{
Layer: logLayer,
Location: dekLocation,
Method: "decryptAccountDEK",
RequestID: requestID,
})
logger.DebugContext(ctx, "Decrypting Account DEK...")
return decryptDEK(logger, ctx, decryptDEKOptions{
storedDEK: storedDEK,
secret: &e.accountSecretKey,
oldSecrets: e.oldSecrets,
})
}
func (e *Encryption) decryptAppDEK(ctx context.Context, requestID, storedDEK string) ([]byte, bool, error) {
logger := utils.BuildLogger(e.logger, utils.LoggerOptions{
Layer: logLayer,
Location: dekLocation,
Method: "decryptAppDEK",
RequestID: requestID,
})
logger.DebugContext(ctx, "Decrypting App DEK...")
return decryptDEK(logger, ctx, decryptDEKOptions{
storedDEK: storedDEK,
secret: &e.appSecretKey,
oldSecrets: e.oldSecrets,
})
}
func (e *Encryption) decryptUserDEK(ctx context.Context, requestID, storedDEK string) ([]byte, bool, error) {
logger := utils.BuildLogger(e.logger, utils.LoggerOptions{
Layer: logLayer,
Location: dekLocation,
Method: "decryptUserDEK",
RequestID: requestID,
})
logger.DebugContext(ctx, "Decrypting User DEK...")
return decryptDEK(logger, ctx, decryptDEKOptions{
storedDEK: storedDEK,
secret: &e.userSecretKey,
oldSecrets: e.oldSecrets,
})
}
Mailer
The mailer will just be a redis publisher, that will publish messages to the email service, hence create a mailer package with the following mailer.go file:
package mailer
import (
"context"
"encoding/json"
"log/slog"
"github.com/tugascript/devlogs/idp/internal/utils"
"github.com/redis/go-redis/v9"
)
const logLayer string = "mailer"
type email struct {
To string `json:"to"`
Subject string `json:"subject"`
Body string `json:"body"`
}
type EmailPublisher struct {
client redis.UniversalClient
pubChannel string
frontendDomain string
logger *slog.Logger
}
func NewEmailPublisher(
client redis.UniversalClient,
pubChannel,
frontendDomain string,
logger *slog.Logger,
) *EmailPublisher {
return &EmailPublisher{
client: client,
pubChannel: pubChannel,
frontendDomain: frontendDomain,
logger: logger,
}
}
type PublishEmailOptions struct {
To string
Subject string
Body string
RequestID string
}
func (e *EmailPublisher) publishEmail(ctx context.Context, opts PublishEmailOptions) error {
logger := utils.BuildLogger(e.logger, utils.LoggerOptions{
Layer: logLayer,
Location: "mailer",
Method: "PublishEmail",
RequestID: opts.RequestID,
})
logger.DebugContext(ctx, "Publishing email...")
message, err := json.Marshal(email{
To: opts.To,
Subject: opts.Subject,
Body: opts.Body,
})
if err != nil {
logger.ErrorContext(ctx, "Failed to marshal email", "error", err)
return err
}
return e.client.Publish(ctx, e.pubChannel, string(message)).Err()
}
It will consume the UniversalClient from the cache, and publish to a given channel.
OAuth
For OAuth 2.0 we can just use the golang.org/x/oauth2 package, just run:
$ go get golang.org/x/oauth2
Scopes
Each external provider has their own scopes, lets start by creating a scopes struct:
package oauth
// ...
type oauthScopes struct {
email string
profile string
birthday string
location string
gender string
}
And for each provider lets create a global scope variable:
-
apple.go:
package oauth // ... var appleScopes = oauthScopes{ email: "email", profile: "name", } -
facebook.go:
package oauth // ... var facebookScopes = oauthScopes{ email: "email", profile: "public_profile", birthday: "user_birthday", location: "user_location", gender: "gender", } -
github.go:
package oauth // ... var gitHubScopes = oauthScopes{ email: "user:email", profile: "read:user", } -
google.go:
package oauth // ... var googleScopes = oauthScopes{ email: "https://www.googleapis.com/auth/userinfo.email", profile: "https://www.googleapis.com/auth/userinfo.profile", birthday: "https://www.googleapis.com/auth/user.birthday.read", location: "https://www.googleapis.com/auth/user.addresses.read", gender: "https://www.googleapis.com/auth/user.gender.read", } -
microsoft.go:
package oauth // ... var microsoftScopes = oauthScopes{ email: "User.Read", profile: "User.ReadBasic.All", }
Provider
Start by creating an oauth2.Config struct for each external provider, using existing defaults when possible:
package oauth
import (
"log/slog"
"golang.org/x/oauth2"
"golang.org/x/oauth2/facebook"
"golang.org/x/oauth2/github"
"golang.org/x/oauth2/google"
"golang.org/x/oauth2/microsoft"
"github.com/tugascript/devlogs/idp/internal/config"
)
// ...
type Config struct {
Enabled bool
oauth2.Config
}
type Providers struct {
gitHub Config
google Config
facebook Config
apple Config
microsoft Config
logger *slog.Logger
}
func NewProviders(
log *slog.Logger,
githubCfg,
googleCfg,
facebookCfg,
appleCfg,
microsoftCfg config.OAuthProviderConfig,
) *Providers {
return &Providers{
gitHub: Config{
Config: oauth2.Config{
ClientID: githubCfg.ClientID(),
ClientSecret: githubCfg.ClientSecret(),
Endpoint: github.Endpoint,
Scopes: []string{gitHubScopes.email},
},
Enabled: githubCfg.Enabled(),
},
google: Config{
Config: oauth2.Config{
ClientID: googleCfg.ClientID(),
ClientSecret: googleCfg.ClientSecret(),
Endpoint: google.Endpoint,
Scopes: []string{googleScopes.email},
},
Enabled: googleCfg.Enabled(),
},
facebook: Config{
Config: oauth2.Config{
ClientID: facebookCfg.ClientID(),
ClientSecret: facebookCfg.ClientSecret(),
Endpoint: facebook.Endpoint,
Scopes: []string{facebookScopes.email},
},
Enabled: facebookCfg.Enabled(),
},
apple: Config{
Config: oauth2.Config{
ClientID: appleCfg.ClientID(),
ClientSecret: appleCfg.ClientSecret(),
Endpoint: oauth2.Endpoint{
AuthURL: "https://appleid.apple.com/auth/authorize",
TokenURL: "https://appleid.apple.com/auth/token",
},
Scopes: []string{appleScopes.email},
},
Enabled: appleCfg.Enabled(),
},
microsoft: Config{
Config: oauth2.Config{
ClientID: microsoftCfg.ClientID(),
ClientSecret: microsoftCfg.ClientSecret(),
Endpoint: microsoft.AzureADEndpoint("common"),
Scopes: []string{microsoftScopes.email},
},
Enabled: microsoftCfg.Enabled(),
},
logger: log,
}
}
For getting the token we will need to pass the correct Scopes, this touches on a concept in go that is passing a value as a value (or copy) of the underlying resource. Since we dynamically add scopes we create a copy of the config each time.
Getting the access token:
package oauth
import (
"context"
// ...
"github.com/tugascript/devlogs/idp/internal/exceptions"
)
// ...
func mapScopes(scopes []Scope, oas oauthScopes) []string {
scopeMapper := make(map[string]bool)
for _, s := range scopes {
switch s {
case ScopeBirthday:
scopeMapper[oas.birthday] = true
case ScopeGender:
scopeMapper[oas.gender] = true
case ScopeLocation:
scopeMapper[oas.location] = true
case ScopeProfile:
scopeMapper[oas.location] = true
}
}
mappedScopes := make([]string, 0, len(scopeMapper))
for k := range scopeMapper {
if k != "" {
mappedScopes = append(mappedScopes, k)
}
}
return mappedScopes
}
// Here we pass by value so we don't update the base configuration
func appendScopes(cfg Config, scopes []string) Config {
cfg.Scopes = append(cfg.Scopes, scopes...)
return cfg
}
// Here we pass by value so we don't update the base configuration
func getConfig(cfg Config, redirectURL string, oas oauthScopes, scopes []Scope) Config {
cfg.RedirectURL = redirectURL
if scopes != nil {
return appendScopes(cfg, mapScopes(scopes, oas))
}
return cfg
}
type getAccessTokenOptions struct {
logger *slog.Logger
cfg Config
redirectURL string
oas oauthScopes
scopes []Scope
code string
}
func getAccessToken(ctx context.Context, opts getAccessTokenOptions) (string, *exceptions.ServiceError) {
opts.logger.DebugContext(ctx, "Getting access token...")
if !opts.cfg.Enabled {
opts.logger.DebugContext(ctx, "OAuth config is disabled")
return "", exceptions.NewNotFoundError()
}
cfg := getConfig(opts.cfg, opts.redirectURL, opts.oas, opts.scopes)
token, err := cfg.Exchange(ctx, opts.code)
if err != nil {
opts.logger.ErrorContext(ctx, "Failed to exchange the code for a token", "error", err)
return "", exceptions.NewUnauthorizedError()
}
opts.logger.DebugContext(ctx, "Access token exchanged successfully")
return token.AccessToken, nil
}
Getting the authorization URL:
package oauth
import (
"context"
// ...
"github.com/tugascript/devlogs/idp/internal/utils"
)
type getAuthorizationURLOptions struct {
logger *slog.Logger
redirectURL string
cfg Config
oas oauthScopes
scopes []Scope
}
func getAuthorizationURL(
ctx context.Context,
opts getAuthorizationURLOptions,
) (string, string, *exceptions.ServiceError) {
opts.logger.DebugContext(ctx, "Getting authorization url...")
if !opts.cfg.Enabled {
opts.logger.DebugContext(ctx, "OAuth config is disabled")
return "", "", exceptions.NewNotFoundError()
}
state, err := utils.GenerateHexSecret(16)
if err != nil {
opts.logger.ErrorContext(ctx, "Failed to generate state", "error", err)
return "", "", exceptions.NewServerError()
}
cfg := getConfig(opts.cfg, opts.redirectURL, opts.oas, opts.scopes)
url := cfg.AuthCodeURL(state)
opts.logger.DebugContext(ctx, "Authorization url generated successfully")
return url, state, nil
}
Getting the user data:
package oauth
import (
"context"
"errors"
"io"
"log/slog"
"net/http"
// ...
"github.com/tugascript/devlogs/idp/internal/config"
"github.com/tugascript/devlogs/idp/internal/exceptions"
"github.com/tugascript/devlogs/idp/internal/utils"
)
func getUserResponse(logger *slog.Logger, ctx context.Context, url, token string) ([]byte, int, error) {
logger.DebugContext(ctx, "Getting user data...", "url", url)
logger.DebugContext(ctx, "Building user data request")
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
logger.ErrorContext(ctx, "Failed to build user data request")
return nil, 0, err
}
req.Header.Set("Accept", "application/json")
req.Header.Set("Authorization", "Bearer "+token)
logger.DebugContext(ctx, "Requesting user data...")
res, err := http.DefaultClient.Do(req)
if err != nil {
logger.ErrorContext(ctx, "Failed to request the user data")
return nil, 0, err
}
if res.StatusCode != http.StatusOK {
logger.ErrorContext(ctx, "Responded with a non 200 OK status", "status", res.StatusCode)
return nil, res.StatusCode, errors.New("status code is not 200 OK")
}
logger.DebugContext(ctx, "Reading the body")
body, err := io.ReadAll(res.Body)
if err != nil {
logger.ErrorContext(ctx, "Failed to read the body", "error", err)
return nil, 0, err
}
defer func() {
if err := res.Body.Close(); err != nil {
logger.ErrorContext(ctx, "Failed to close response body", "error", err)
}
}()
return body, res.StatusCode, nil
}
type UserLocation struct {
City string
Region string
Country string
}
type UserData struct {
Name string
FirstName string
LastName string
Username string
Picture string
Email string
Gender string
Location UserLocation
BirthDate string
IsVerified bool
}
type ToUserData interface {
ToUserData() UserData
}
type extraParams struct {
params string
}
func (p *extraParams) addParam(prm string) {
if p.params != "" {
p.params = p.params + "," + prm
return
}
p.params += prm
}
func (p *extraParams) isEmpty() bool {
return p.params == ""
}
Common options between all providers:
package oauth
// ...
type AccessTokenOptions struct {
RequestID string
Code string
RedirectURL string
Scopes []Scope
}
type AuthorizationURLOptions struct {
RequestID string
RedirectURL string
Scopes []Scope
}
type UserDataOptions struct {
RequestID string
Token string
Scopes []Scope
}
Tokens
Keys Algorithms
When choosing the algorithm of the key pairs to sign and verify JWTs we need to choose the best ones with the best efficient and security ratio in mind.
By my own research the most balanced algorithm would be EdDSC (Edwards-curve Digital Signature Algorithm) with the Ed25519 signature scheme, however this algorithm is not part of the base JWT RFC-7519 standard, only the RFC-8037 which is not widly supported.
Hence for simplicity and compatibility sake for all tokens that require their public keys to be distributed will use the recommended ECDSA (Elliptic Curve Digital Signature Algorithm) algorithm with the P-256 signature scheme.
Utilities
For sharing and saving keys we need to convert them to JWKs so create an encode and decode functions for both Ed25519 and P256 on the utils/jwk.go file:
package utils
import (
"crypto/ecdsa"
"crypto/ed25519"
"crypto/elliptic"
"crypto/sha256"
"encoding/base64"
"fmt"
"math/big"
)
type Ed25519JWK struct {
Kty string `json:"kty"` // Key Type (OKP for Ed25519)
Crv string `json:"crv"` // Curve (Ed25519)
X string `json:"x"` // Public Key
Use string `json:"use"` // Usage (e.g., "sig" for signing)
Alg string `json:"alg"` // Algorithm (EdDSA for Ed25519)
Kid string `json:"kid"` // Key AccountID
KeyOps []string `json:"key_ops"` // Key Operations
}
type P256JWK struct {
Kty string `json:"kty"` // Key Type (EC for Elliptic Curve)
Crv string `json:"crv"` // Curve (P-256)
X string `json:"x"` // X Coordinate
Y string `json:"y"` // Y Coordinate
Use string `json:"use"` // Usage (e.g., "sig" for signing)
Alg string `json:"alg"` // Algorithm (ES256 for P-256)
Kid string `json:"kid"` // Key AccountID
KeyOps []string `json:"key_ops"` // Key Operations
}
// Because of Apple
type RS256JWK struct {
Kty string `json:"kty"`
Kid string `json:"kid"`
Use string `json:"use"`
Alg string `json:"alg"`
N string `json:"n"`
E string `json:"e"`
KeyOps []string `json:"key_ops,omitempty"`
}
const (
kty string = "OKP"
crv string = "Ed25519"
use string = "sig"
alg string = "EdDSA"
verify string = "verify"
p256Kty string = "EC"
p256Crv string = "P-256"
)
// ...
func EncodeEd25519Jwk(publicKey ed25519.PublicKey, kid string) Ed25519JWK {
return Ed25519JWK{
Kty: kty,
Crv: crv,
X: base64.RawURLEncoding.EncodeToString(publicKey),
Use: use,
Alg: alg,
Kid: kid,
KeyOps: []string{verify},
}
}
func DecodeEd25519Jwk(jwk Ed25519JWK) (ed25519.PublicKey, error) {
publicKey, err := base64.RawURLEncoding.DecodeString(jwk.X)
if err != nil {
return nil, err
}
return publicKey, nil
}
func EncodeP256Jwk(publicKey *ecdsa.PublicKey, kid string) P256JWK {
return P256JWK{
Kty: p256Kty,
Crv: p256Crv,
X: base64.RawURLEncoding.EncodeToString(publicKey.X.Bytes()),
Y: base64.RawURLEncoding.EncodeToString(publicKey.Y.Bytes()),
Use: use,
Alg: alg,
Kid: kid,
KeyOps: []string{verify},
}
}
func DecodeP256Jwk(jwk P256JWK) (ecdsa.PublicKey, error) {
x, err := base64.RawURLEncoding.DecodeString(jwk.X)
if err != nil {
return ecdsa.PublicKey{}, err
}
y, err := base64.RawURLEncoding.DecodeString(jwk.Y)
if err != nil {
return ecdsa.PublicKey{}, err
}
return ecdsa.PublicKey{
Curve: elliptic.P256(),
X: new(big.Int).SetBytes(x),
Y: new(big.Int).SetBytes(y),
}, nil
}
func DecodeRS256Jwk(jwk RS256JWK) (*rsa.PublicKey, error) {
nBytes, err := base64.RawURLEncoding.DecodeString(jwk.N)
if err != nil {
return nil, err
}
n := new(big.Int).SetBytes(nBytes)
eBytes, err := base64.RawURLEncoding.DecodeString(jwk.E)
if err != nil {
return nil, err
}
e := big.NewInt(0).SetBytes(eBytes).Int64()
if e <= 0 {
return nil, fmt.Errorf("invalid RSA exponent")
}
return &rsa.PublicKey{N: n, E: int(e)}, nil
}
Provider
For each key we need a key pair, and a reference to a previous public key for keys rotation, create the tokens/tokens.go package:
package tokens
import (
"crypto/ecdsa"
"crypto/ed25519"
"crypto/x509"
"encoding/pem"
"github.com/golang-jwt/jwt/v5"
"github.com/tugascript/devlogs/idp/internal/config"
"github.com/tugascript/devlogs/idp/internal/utils"
)
type PreviousPublicKey struct {
publicKey ed25519.PublicKey
kid string
}
type TokenKeyPair struct {
publicKey ed25519.PublicKey
privateKey ed25519.PrivateKey
kid string
}
type TokenSecretData struct {
curKeyPair TokenKeyPair
prevPubKey *PreviousPublicKey
ttlSec int64
}
// ...
type PreviousEs256PublicKey struct {
publicKey *ecdsa.PublicKey
kid string
}
type Es256TokenKeyPair struct {
privateKey *ecdsa.PrivateKey
publicKey *ecdsa.PublicKey
kid string
}
type Es256TokenSecretData struct {
curKeyPair Es256TokenKeyPair
prevPubKey *PreviousEs256PublicKey
ttlSec int64
}
Now each key is encoded as a PEM in the environment, hence we need to decode the x509 certificates for:
-
Ed25519keys:
package tokens
// ...
func extractEd25519PublicKey(publicKey string) (ed25519.PublicKey, string) {
publicKeyBlock, _ := pem.Decode([]byte(publicKey))
if publicKeyBlock == nil || publicKeyBlock.Type != "PUBLIC KEY" {
panic("Invalid public key")
}
publicKeyData, err := x509.ParsePKIXPublicKey(publicKeyBlock.Bytes)
if err != nil {
panic(err)
}
publicKeyValue, ok := publicKeyData.(ed25519.PublicKey)
if !ok {
panic("Invalid public key")
}
return publicKeyValue, utils.ExtractKeyID(publicKeyValue)
}
func extractEd25519PrivateKey(privateKey string) ed25519.PrivateKey {
privateKeyBlock, _ := pem.Decode([]byte(privateKey))
if privateKeyBlock == nil || privateKeyBlock.Type != "PRIVATE KEY" {
panic("Invalid private key")
}
privateKeyData, err := x509.ParsePKCS8PrivateKey(privateKeyBlock.Bytes)
if err != nil {
panic(err)
}
privateKeyValue, ok := privateKeyData.(ed25519.PrivateKey)
if !ok {
panic("Invalid private key")
}
return privateKeyValue
}
func extractEd25519PublicPrivateKeyPair(publicKey, privateKey string) TokenKeyPair {
pubKey, kid := extractEd25519PublicKey(publicKey)
return TokenKeyPair{
publicKey: pubKey,
privateKey: extractEd25519PrivateKey(privateKey),
kid: kid,
}
}
func newTokenSecretData(
publicKey,
privateKey,
previousPublicKey string,
ttlSec int64,
) TokenSecretData {
curKeyPair := extractEd25519PublicPrivateKeyPair(publicKey, privateKey)
if previousPublicKey != "" {
pubKey, kid := extractEd25519PublicKey(previousPublicKey)
return TokenSecretData{
curKeyPair: curKeyPair,
prevPubKey: &PreviousPublicKey{publicKey: pubKey, kid: kid},
ttlSec: ttlSec,
}
}
return TokenSecretData{
curKeyPair: curKeyPair,
ttlSec: ttlSec,
}
}
-
Es256keys:
package tokens
// ...
func extractEs256KeyPair(privateKey string) Es256TokenKeyPair {
privateKeyBlock, _ := pem.Decode([]byte(privateKey))
if privateKeyBlock == nil || privateKeyBlock.Type != "PRIVATE KEY" {
panic("Invalid private key")
}
privateKeyData, err := x509.ParsePKCS8PrivateKey(privateKeyBlock.Bytes)
if err != nil {
privateKeyData, err = x509.ParseECPrivateKey(privateKeyBlock.Bytes)
if err != nil {
panic(err)
}
}
privateKeyValue, ok := privateKeyData.(*ecdsa.PrivateKey)
if !ok {
panic("Invalid private key")
}
publicKeyValue, err := x509.MarshalPKIXPublicKey(&privateKeyValue.PublicKey)
if err != nil {
panic(err)
}
return Es256TokenKeyPair{
privateKey: privateKeyValue,
publicKey: &privateKeyValue.PublicKey,
kid: utils.ExtractKeyID(publicKeyValue),
}
}
func extractEs256PublicKey(publicKey string) (*ecdsa.PublicKey, string) {
publicKeyBlock, _ := pem.Decode([]byte(publicKey))
if publicKeyBlock == nil || publicKeyBlock.Type != "PUBLIC KEY" {
panic("Invalid public key")
}
publicKeyData, err := x509.ParsePKIXPublicKey(publicKeyBlock.Bytes)
if err != nil {
panic(err)
}
pubKey, ok := publicKeyData.(*ecdsa.PublicKey)
if !ok {
panic("Invalid public key")
}
publicKeyValue, err := x509.MarshalPKIXPublicKey(pubKey)
if err != nil {
panic(err)
}
return pubKey, utils.ExtractKeyID(publicKeyValue)
}
// ...
func newEs256TokenSecretData(privateKey, previousPublicKey string, ttlSec int64) Es256TokenSecretData {
curKeyPair := extractEs256KeyPair(privateKey)
if previousPublicKey != "" {
prevPubKey, kid := extractEs256PublicKey(previousPublicKey)
return Es256TokenSecretData{
curKeyPair: curKeyPair,
prevPubKey: &PreviousEs256PublicKey{publicKey: prevPubKey, kid: kid},
ttlSec: ttlSec,
}
}
return Es256TokenSecretData{
curKeyPair: curKeyPair,
ttlSec: ttlSec,
}
}
Finally finish by creating the full provider:
package tokens
// ...
type Tokens struct {
frontendDomain string
backendDomain string
accessData Es256TokenSecretData
accountCredentialsData Es256TokenSecretData
refreshData TokenSecretData
confirmationData TokenSecretData
resetData TokenSecretData
oauthData TokenSecretData
twoFAData TokenSecretData
jwks []utils.P256JWK
}
func NewTokens(
accessCfg,
accountCredentialsCfg,
refreshCfg,
confirmationCfg,
resetCfg,
oauthCfg,
twoFACfg config.SingleJwtConfig,
frontendDomain,
backendDomain string,
) *Tokens {
accessData := newEs256TokenSecretData(
accessCfg.PrivateKey(),
accessCfg.PreviousPublicKey(),
accessCfg.TtlSec(),
)
accountKeysData := newEs256TokenSecretData(
accountCredentialsCfg.PrivateKey(),
accountCredentialsCfg.PreviousPublicKey(),
accountCredentialsCfg.TtlSec(),
)
jwks := []utils.P256JWK{
utils.EncodeP256Jwk(accountKeysData.curKeyPair.publicKey, accountKeysData.curKeyPair.kid),
utils.EncodeP256Jwk(accessData.curKeyPair.publicKey, accessData.curKeyPair.kid),
}
if accountKeysData.prevPubKey != nil {
jwks = append(jwks, utils.EncodeP256Jwk(
accountKeysData.prevPubKey.publicKey,
accessData.prevPubKey.kid,
))
}
if accessData.prevPubKey != nil {
jwks = append(jwks, utils.EncodeP256Jwk(
accessData.prevPubKey.publicKey,
accessData.prevPubKey.kid,
))
}
return &Tokens{
accessData: accessData,
accountCredentialsData: accountKeysData,
refreshData: newTokenSecretData(
refreshCfg.PublicKey(),
refreshCfg.PrivateKey(),
refreshCfg.PreviousPublicKey(),
refreshCfg.TtlSec(),
),
confirmationData: newTokenSecretData(
confirmationCfg.PublicKey(),
confirmationCfg.PrivateKey(),
confirmationCfg.PreviousPublicKey(),
confirmationCfg.TtlSec(),
),
resetData: newTokenSecretData(
resetCfg.PublicKey(),
resetCfg.PrivateKey(),
resetCfg.PreviousPublicKey(),
resetCfg.TtlSec(),
),
oauthData: newTokenSecretData(
oauthCfg.PublicKey(),
oauthCfg.PrivateKey(),
oauthCfg.PreviousPublicKey(),
oauthCfg.TtlSec(),
),
twoFAData: newTokenSecretData(
twoFACfg.PublicKey(),
twoFACfg.PrivateKey(),
twoFACfg.PreviousPublicKey(),
twoFACfg.TtlSec(),
),
frontendDomain: frontendDomain,
backendDomain: backendDomain,
jwks: jwks,
}
}
func (t *Tokens) JWKs() []utils.P256JWK {
return t.jwks
}
Error Handling
Error handling in Go works differently from most languages, instead of throwing and catching exceptions, values are return as a pointer (error as values) to the error.
This means for ease of use, we need to map/coerce the errors to a known value on the service layer, and map them to an error response on the controller layer.
Service Errors
Create an expections directory on the internal folder for the errors mapping.
Format
The service error is gonna have a standard error class, with a type/code, and a message, but for simplicity we won't add a details slice field.
Create the exceptions/services.go file for the exceptions package:
package exceptions
type ServiceError struct {
Code string
Message string
}
func NewError(code string, message string) *ServiceError {
return &ServiceError{
Code: code,
Message: message,
}
}
// To make `ServiceError` an error type interface
func (e *ServiceError) Error() string {
return e.Message
}
Codes and messages
We need to standerdize the code and message based on HTTP status codes:
package exceptions
// ...
const (
CodeValidation string = "VALIDATION"
CodeConflict string = "CONFLICT"
CodeInvalidEnum string = "INVALID_ENUM"
CodeNotFound string = "NOT_FOUND"
CodeUnknown string = "UNKNOWN"
CodeServerError string = "SERVER_ERROR"
CodeUnauthorized string = "UNAUTHORIZED"
CodeForbidden string = "FORBIDDEN"
CodeUnsupportedMediaType string = "UNSUPPORTED_MEDIA_TYPE"
)
const (
MessageDuplicateKey string = "Resource already exists"
MessageNotFound string = "Resource not found"
MessageUnknown string = "Something went wrong"
MessageUnauthorized string = "Unauthorized"
MessageForbidden string = "Forbidden"
)
func NewNotFoundError() *ServiceError {
return NewError(CodeNotFound, MessageNotFound)
}
func NewValidationError(message string) *ServiceError {
return NewError(CodeValidation, message)
}
func NewServerError() *ServiceError {
return NewError(CodeServerError, MessageUnknown)
}
func NewConflictError(message string) *ServiceError {
return NewError(CodeConflict, message)
}
func NewUnsupportedMediaTypeError(message string) *ServiceError {
return NewError(CodeUnsupportedMediaType, message)
}
func NewUnauthorizedError() *ServiceError {
return NewError(CodeUnauthorized, MessageUnauthorized)
}
func NewForbiddenError() *ServiceError {
return NewError(CodeForbidden, MessageForbidden)
}
Coercion
For the PostgreSQL errors we can coerce them using a mapper:
package exceptions
import (
"errors"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
)
// ...
func FromDBError(err error) *ServiceError {
if errors.Is(err, pgx.ErrNoRows) {
return NewError(CodeNotFound, MessageNotFound)
}
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) {
switch pgErr.Code {
case "23505":
return NewError(CodeConflict, MessageDuplicateKey)
case "23514":
return NewError(CodeInvalidEnum, pgErr.Message)
case "23503":
return NewError(CodeNotFound, MessageNotFound)
default:
return NewError(CodeUnknown, pgErr.Message)
}
}
return NewError(CodeUnknown, MessageUnknown)
}
Controllers Error
ServiceError need to be mappend to a JSON Response, however there are also other types of error responses: body validation and oauth validation.
Error Response
Create the controllers.go and add the error response, which is the same as ServiceError but JSON parsable:
package exceptions
// ...
const (
StatusConflict string = "Conflict"
StatusInvalidEnum string = "BadRequest"
StatusNotFound string = "NotFound"
StatusServerError string = "InternalServerError"
StatusUnknown string = "InternalServerError"
StatusUnauthorized string = "Unauthorized"
StatusForbidden string = "Forbidden"
StatusValidation string = "Validation"
)
type ErrorResponse struct {
Code string `json:"code"`
Message string `json:"message"`
}
func NewErrorResponse(err *ServiceError) ErrorResponse {
switch err.Code {
case CodeServerError:
return ErrorResponse{
Code: StatusServerError,
Message: err.Message,
}
case CodeConflict:
return ErrorResponse{
Code: StatusConflict,
Message: err.Message,
}
case CodeInvalidEnum:
return ErrorResponse{
Code: StatusInvalidEnum,
Message: err.Message,
}
case CodeNotFound:
return ErrorResponse{
Code: StatusNotFound,
Message: err.Message,
}
case CodeValidation:
return ErrorResponse{
Code: StatusValidation,
Message: err.Message,
}
case CodeUnknown:
return ErrorResponse{
Code: StatusUnknown,
Message: err.Message,
}
case CodeUnauthorized:
return ErrorResponse{
Code: StatusUnauthorized,
Message: StatusUnauthorized,
}
case CodeForbidden:
return ErrorResponse{
Code: StatusForbidden,
Message: StatusForbidden,
}
default:
return ErrorResponse{
Code: StatusUnknown,
Message: err.Message,
}
}
}
Validation Response
To validation the body inputs we will use the go validator package, start by installing it:
$ go get github.com/go-playground/validator/v10
Services
Services is where most of the server business logic is located, lets create the services/services.go package, the Services struct needs to encapsulate all providers, and be the only layer talking directly to them.
package services
import (
"log/slog"
"github.com/tugascript/devlogs/idp/internal/providers/cache"
"github.com/tugascript/devlogs/idp/internal/providers/database"
"github.com/tugascript/devlogs/idp/internal/providers/encryption"
"github.com/tugascript/devlogs/idp/internal/providers/mailer"
"github.com/tugascript/devlogs/idp/internal/providers/oauth"
"github.com/tugascript/devlogs/idp/internal/providers/tokens"
)
type Services struct {
logger *slog.Logger
database *database.Database
cache *cache.Cache
mail *mailer.EmailPublisher
jwt *tokens.Tokens
encrypt *encryption.Encryption
oauthProviders *oauth.Providers
}
func NewServices(
logger *slog.Logger,
database *database.Database,
cache *cache.Cache,
mail *mailer.EmailPublisher,
jwt *tokens.Tokens,
encrypt *encryption.Encryption,
oauthProv *oauth.Providers,
) *Services {
return &Services{
logger: logger,
database: database,
cache: cache,
mail: mail,
jwt: jwt,
encrypt: encrypt,
oauthProviders: oauthProv,
}
}
We also need a helper to build the logger, and some common constant, create a helpers.go file:
import (
"log/slog"
"github.com/tugascript/devlogs/idp/internal/utils"
)
const (
AuthProviderEmail string = "email"
AuthProviderGoogle string = "google"
AuthProviderGitHub string = "github"
AuthProviderApple string = "apple"
AuthProviderMicrosoft string = "microsoft"
AuthProviderFacebook string = "facebook"
TwoFactorNone string = "none"
TwoFactorEmail string = "email"
TwoFactorTotp string = "totp"
)
func (s *Services) buildLogger(requestID, location, function string) *slog.Logger {
return utils.BuildLogger(s.logger, utils.LoggerOptions{
Layer: utils.ServicesLogLayer,
Location: location,
Method: function,
RequestID: requestID,
})
}
Health
For our first service, we will create the health endpoint for our API, this service only needs to ping PostgreSQL and ValKey.
Create the health.go file:
package services
import (
"context"
"github.com/tugascript/devlogs/idp/internal/exceptions"
)
const healthLocation string = "health"
func (s *Services) HealthCheck(ctx context.Context, requestID string) *exceptions.ServiceError {
logger := s.buildLogger(requestID, healthLocation, "HealthCheck")
logger.InfoContext(ctx, "Performing health check...")
if err := s.database.Ping(ctx); err != nil {
logger.ErrorContext(ctx, "Failed to ping database", "error", err)
return exceptions.NewServerError()
}
if err := s.cache.Ping(ctx); err != nil {
logger.ErrorContext(ctx, "Failed to ping cache", "error", err)
return exceptions.NewServerError()
}
logger.InfoContext(ctx, "Service is healthy")
return nil
}
Controllers
Controllers are where we map our services to the correct HTTP status response.
Helpers
For the controllor, error handling and logging is the same for all routes, hence create a logger builder and error handling functions on a helpers.go file.
package controllers
import (
"errors"
"fmt"
"log/slog"
"github.com/go-playground/validator/v10"
"github.com/gofiber/fiber/v2"
"github.com/google/uuid"
"github.com/tugascript/devlogs/idp/internal/exceptions"
"github.com/tugascript/devlogs/idp/internal/utils"
)
func (c *Controllers) buildLogger(
requestID,
location,
method string,
) *slog.Logger {
return utils.BuildLogger(c.logger, utils.LoggerOptions{
Layer: utils.ControllersLogLayer,
Location: location,
Method: method,
RequestID: requestID,
})
}
func logRequest(logger *slog.Logger, ctx *fiber.Ctx) {
logger.InfoContext(
ctx.UserContext(),
fmt.Sprintf("Request: %s %s", ctx.Method(), ctx.Path()),
)
}
func getRequestID(ctx *fiber.Ctx) string {
return ctx.Get("requestid", uuid.NewString())
}
func logResponse(logger *slog.Logger, ctx *fiber.Ctx, status int) {
logger.InfoContext(
ctx.UserContext(),
fmt.Sprintf("Response: %s %s", ctx.Method(), ctx.Path()),
"status", status,
)
}
func validateErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, location string, err error) error {
logger.WarnContext(ctx.UserContext(), "Failed to validate request", "error", err)
logResponse(logger, ctx, fiber.StatusBadRequest)
var errs validator.ValidationErrors
ok := errors.As(err, &errs)
if !ok {
return ctx.
Status(fiber.StatusBadRequest).
JSON(exceptions.NewEmptyValidationErrorResponse(location))
}
return ctx.
Status(fiber.StatusBadRequest).
JSON(exceptions.ValidationErrorResponseFromErr(&errs, location))
}
func validateBodyErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, err error) error {
return validateErrorResponse(logger, ctx, exceptions.ValidationResponseLocationBody, err)
}
func validateURLParamsErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, err error) error {
return validateErrorResponse(logger, ctx, exceptions.ValidationResponseLocationParams, err)
}
func validateQueryParamsErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, err error) error {
return validateErrorResponse(logger, ctx, exceptions.ValidationResponseLocationQuery, err)
}
func serviceErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, serviceErr *exceptions.ServiceError) error {
status := exceptions.NewRequestErrorStatus(serviceErr.Code)
resErr := exceptions.NewErrorResponse(serviceErr)
logResponse(logger, ctx, status)
return ctx.Status(status).JSON(&resErr)
}
func oauthErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, message string) error {
resErr := exceptions.NewOAuthError(message)
logResponse(logger, ctx, fiber.StatusBadRequest)
return ctx.Status(fiber.StatusBadRequest).JSON(&resErr)
}
func parseRequestErrorResponse(logger *slog.Logger, ctx *fiber.Ctx, err error) error {
logger.WarnContext(ctx.UserContext(), "Failed to parse request", "error", err)
logResponse(logger, ctx, fiber.StatusBadRequest)
return ctx.
Status(fiber.StatusBadRequest).
JSON(exceptions.NewEmptyValidationErrorResponse(exceptions.ValidationResponseLocationBody))
}
Health
The health controller is pretty simple, since it will only consume the service HealtCheck method directly:
package controllers
import "github.com/gofiber/fiber/v2"
func (c *Controllers) HealthCheck(ctx *fiber.Ctx) error {
requestID := getRequestID(ctx)
logger := c.buildLogger(requestID, "health", "HealthCheck")
logRequest(logger, ctx)
if serviceErr := c.services.HealthCheck(ctx.UserContext(), requestID); serviceErr != nil {
return serviceErrorResponse(logger, ctx, serviceErr)
}
return ctx.SendStatus(fiber.StatusOK)
}
Paths
On the controllers folder create a paths package with the health.go path:
package paths
const Health string = "/health"
Server
The server is where we will initialize the Fiber App, build Services, Controllers and load common middleware.
Routes
Start by creating a server/routes and add routes.go file with the Routes struct:
package routes
import (
"github.com/tugascript/devlogs/idp/internal/controllers"
)
type Routes struct {
controllers *controllers.Controllers
}
func NewRoutes(ctrls *controllers.Controllers) *Routes {
return &Routes{controllers: ctrls}
}
Now create a router method for the health endpoint on health.go:
package routes
import (
"github.com/gofiber/fiber/v2"
"github.com/tugascript/devlogs/idp/internal/controllers/paths"
)
func (r *Routes) HealthRoutes(app *fiber.App) {
app.Get(paths.Health, r.controllers.HealthCheck)
}
Plus also create a common.go with the route Group for API version 1:
package routes
import "github.com/gofiber/fiber/v2"
const V1Path string = "/v1"
func v1PathRouter(app *fiber.App) fiber.Router {
return app.Group(V1Path)
}
Server Instance
Server instance is where we hook up most of our logic, hence it will be a big method where you take the config as a parameter and initialize everything.
On the server.go file inside the server directory, and create the FiberServer instance:
package server
import (
// ...
// ...
"github.com/tugascript/devlogs/idp/internal/server/routes"
// ...
)
type FiberServer struct {
*fiber.App
routes *routes.Routes
}
And load everything in the New function that takes the context.Context, the struter logger (*slog.Logger) and the configuration (config.Config):
package server
import (
"context"
"log/slog"
"time"
"github.com/gofiber/fiber/v2"
"github.com/gofiber/fiber/v2/middleware/cors"
"github.com/gofiber/fiber/v2/middleware/encryptcookie"
"github.com/gofiber/fiber/v2/middleware/helmet"
"github.com/gofiber/fiber/v2/middleware/limiter"
"github.com/gofiber/fiber/v2/middleware/requestid"
fiberRedis "github.com/gofiber/storage/redis/v3"
"github.com/google/uuid"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/tugascript/devlogs/idp/internal/config"
"github.com/tugascript/devlogs/idp/internal/controllers"
"github.com/tugascript/devlogs/idp/internal/providers/cache"
"github.com/tugascript/devlogs/idp/internal/providers/database"
"github.com/tugascript/devlogs/idp/internal/providers/encryption"
"github.com/tugascript/devlogs/idp/internal/providers/mailer"
"github.com/tugascript/devlogs/idp/internal/providers/oauth"
"github.com/tugascript/devlogs/idp/internal/providers/tokens"
"github.com/tugascript/devlogs/idp/internal/server/routes"
"github.com/tugascript/devlogs/idp/internal/server/validations"
"github.com/tugascript/devlogs/idp/internal/services"
)
// ...
func New(
ctx context.Context,
logger *slog.Logger,
cfg config.Config,
) *FiberServer {
logger.InfoContext(ctx, "Building redis storage...")
cacheStorage := fiberRedis.New(fiberRedis.Config{
URL: cfg.RedisURL(),
})
cc := cache.NewCache(
logger,
cacheStorage,
)
logger.InfoContext(ctx, "Finished building redis storage")
logger.InfoContext(ctx, "Building database connection pool...")
dbConnPool, err := pgxpool.New(ctx, cfg.DatabaseURL())
if err != nil {
logger.ErrorContext(ctx, "Failed to connect to database", "error", err)
panic(err)
}
db := database.NewDatabase(dbConnPool)
logger.InfoContext(ctx, "Finished building database connection pool")
logger.InfoContext(ctx, "Building mailer...")
mail := mailer.NewEmailPublisher(
cc.Client(),
cfg.EmailPubChannel(),
cfg.FrontendDomain(),
logger,
)
logger.InfoContext(ctx, "Finished building mailer")
logger.InfoContext(ctx, "Building JWT token keys...")
tokensCfg := cfg.TokensConfig()
jwts := tokens.NewTokens(
tokensCfg.Access(),
tokensCfg.AccountCredentials(),
tokensCfg.Refresh(),
tokensCfg.Confirm(),
tokensCfg.Reset(),
tokensCfg.OAuth(),
tokensCfg.TwoFA(),
cfg.FrontendDomain(),
cfg.BackendDomain(),
)
logger.InfoContext(ctx, "Finished building JWT tokens keys")
logger.InfoContext(ctx, "Building encryption...")
encryp := encryption.NewEncryption(logger, cfg.EncryptionConfig(), cfg.BackendDomain())
logger.InfoContext(ctx, "Finished encryption")
logger.InfoContext(ctx, "Building OAuth provider...")
oauthProvidersCfg := cfg.OAuthProvidersConfig()
oauthProviders := oauth.NewProviders(
logger,
oauthProvidersCfg.GitHub(),
oauthProvidersCfg.Google(),
oauthProvidersCfg.Facebook(),
oauthProvidersCfg.Apple(),
oauthProvidersCfg.Microsoft(),
)
logger.InfoContext(ctx, "Finished building OAuth provider")
logger.InfoContext(ctx, "Building services...")
newServices := services.NewServices(
logger,
db,
cc,
mail,
jwts,
encryp,
oauthProviders,
)
logger.InfoContext(ctx, "Finished building services")
logger.InfoContext(ctx, "Loading validators...")
vld := validations.NewValidator(logger)
logger.InfoContext(ctx, "Finished loading validators")
server := &FiberServer{
App: fiber.New(fiber.Config{
ServerHeader: "idp",
AppName: "idp",
}),
routes: routes.NewRoutes(controllers.NewControllers(
logger,
newServices,
vld,
cfg.FrontendDomain(),
cfg.BackendDomain(),
cfg.CookieName(),
)),
}
logger.InfoContext(ctx, "Loading middleware...")
server.Use(helmet.New())
server.Use(requestid.New(requestid.Config{
Header: fiber.HeaderXRequestID,
Generator: func() string {
return uuid.NewString()
},
}))
rateLimitCfg := cfg.RateLimiterConfig()
server.Use(limiter.New(limiter.Config{
Max: int(rateLimitCfg.Max()),
Expiration: time.Duration(rateLimitCfg.ExpSec()) * time.Second,
LimiterMiddleware: limiter.SlidingWindow{},
Storage: cacheStorage,
}))
server.Use(encryptcookie.New(encryptcookie.Config{
Key: cfg.CookieSecret(),
}))
server.App.Use(cors.New(cors.Config{
AllowOrigins: "*",
AllowMethods: "GET,POST,PUT,DELETE,OPTIONS,PATCH,HEAD",
AllowHeaders: "Accept,Authorization,Content-Type",
AllowCredentials: false, // credentials require explicit origins
MaxAge: 300,
}))
logger.Info("Finished loading common middlewares")
return server
}
Registering the routes
Just create a routes.go file inside the server directory with a RegisterFiberRoutes method:
package server
func (s *FiberServer) RegisterFiberRoutes() {
s.routes.HealthRoutes(s.App)
}
Logging
For logging we will need to create a default logger initially then add the configuration when they are loaded on startup, on a logger.go file:
package server
import (
"log/slog"
"os"
"github.com/tugascript/devlogs/idp/internal/config"
)
func DefaultLogger() *slog.Logger {
return slog.New(slog.NewJSONHandler(
os.Stdout,
&slog.HandlerOptions{
Level: slog.LevelInfo,
},
))
}
func ConfigLogger(cfg config.LoggerConfig) *slog.Logger {
logLevel := slog.LevelInfo
if cfg.IsDebug() {
logLevel = slog.LevelDebug
}
logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{
Level: logLevel,
}))
if cfg.Env() == "production" {
logger = slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: logLevel,
}))
}
return logger.With("service", cfg.ServiceName())
}
Running the app
On the cmd/api folder's main.go file just set-up everything and register the routes:
package main
import (
"context"
"fmt"
"log/slog"
"os/signal"
"runtime"
"syscall"
"time"
"github.com/tugascript/devlogs/idp/internal/config"
"github.com/tugascript/devlogs/idp/internal/server"
)
func gracefulShutdown(
logger *slog.Logger,
fiberServer *server.FiberServer,
done chan bool,
) {
// Create context that listens for the interrupt signal from the OS.
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
// Listen for the interrupt signal.
<-ctx.Done()
logger.InfoContext(ctx, "shutting down gracefully, press Ctrl+C again to force")
// The context is used to inform the server it has 5 seconds to finish
// the request it is currently handling
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := fiberServer.ShutdownWithContext(ctx); err != nil {
logger.ErrorContext(ctx, "Server forced to shutdown with error", "error", err)
}
logger.InfoContext(ctx, "Server exiting")
// Notify the main goroutine that the shutdown is complete
done <- true
}
func main() {
logger := server.DefaultLogger()
ctx := context.Background()
logger.InfoContext(ctx, "Loading configuration...")
cfg := config.NewConfig(logger, "./.env")
logger = server.ConfigLogger(cfg.LoggerConfig())
logger.InfoContext(ctx, "Setting GOMAXPROCS...", "maxProcs", cfg.MaxProcs())
runtime.GOMAXPROCS(int(cfg.MaxProcs()))
logger.InfoContext(ctx, "Finished setting GOMAXPROCS")
logger.InfoContext(ctx, "Building server...")
server := server.New(ctx, logger, cfg)
logger.InfoContext(ctx, "Server built")
server.RegisterFiberRoutes()
// Create a done channel to signal when the shutdown is complete
done := make(chan bool, 1)
go func() {
err := server.Listen(fmt.Sprintf(":%d", cfg.Port()))
if err != nil {
logger.ErrorContext(ctx, "http server error", "error", err)
panic(fmt.Sprintf("http server error: %s", err))
}
}()
// Run graceful shutdown in a separate goroutine
go gracefulShutdown(logger, server, done)
// Wait for the graceful shutdown to complete
<-done
logger.InfoContext(ctx, "Graceful shutdown complete.")
}
Conclusion
In this first article, we introduced the fiber framework, a express inspired go library, as well as how to set up our API with a MSC (model, service, controller) architecture with a centralized configuration.
This code is based on the current ongoing project found on the devlogs repository.
About the Author
Hey there! I am Afonso Barracha, your go-to econometrician who found his way into the world of back-end development with a soft spot for GraphQL. If you enjoyed reading this article, why not show some love by buying me a coffee?
Lately, I have been diving deep into more advanced subjects. As a result, I have switched from sharing my thoughts every week to posting once or twice a month. This way, I can make sure to bring you the highest quality content possible.
Do not miss out on any of my latest articles – follow me here on dev, LinkedIn or Instagram to stay updated. I would be thrilled to welcome you to our ever-growing community! See you around!
Top comments (1)
This post is insane, Thanks!!