Forem: Booster

Booster Framework raises the bar in scalability

Álvaro López Espinosa — Fri, 26 Jan 2024 11:50:19 +0000

Thanks to the collaboration with our partners using Booster Framework in Azure, version 2.2 has completely redesigned how the event processing is handled.

It now uses a combination of the services Azure Cosmos DB change feed and Azure Event Hubs in a way that it can process virtually any number of events in a given time without affecting the rest of the application and still keeping very low processing times.

How it works

Before this change, the event processor was only managed by the “Azure Cosmos DB change feed”, which forced Booster to only use a single processing unit (Azure Function App) for the event processing in most situations. While, with this approach, you can still process almost any number of events at a given time, you can experience long delays in situations where thousands of events are registered in a short period of time (in seconds). This is translated to read models that take tens of seconds to reflect the changes, event handlers that get executed long after the corresponding event is registered, etc.

Booster 2.2 now uses Azure Cosmos DB change feed just to send the event to Azure Event Hubs, where it is distributed optimally among partitions and sent to the corresponding event processor. The great advantage of this approach is that we can have many Azure Function Apps (not just one) processing events in parallel, reducing the latency to the minimum.

Despite this extra level of concurrency and parallelism, all the guarantees Booster offers about data consistency and the order of events are still kept.

Of course, this change doesn’t affect the developer experience or the code of any current Booster application.

Conclusions

Booster has always been focused on scalability (this is one of the reasons that made us take architectural decisions like using event-driven or CQRS), but this new version takes it to the level of the most demanding enterprises of the world.

We keep pushing hard to bring the best of the event-sourcing, CQRS, and serverless worlds to a framework that offers a neat developer experience focused on what matters. Is there something you think can be improved? Go ahead and make a suggestion! Remember that Booster is open-source.

Multi Provider Booster Rockets

Gonzalo Garcia Jaubert — Thu, 26 May 2022 09:22:31 +0000

Overview

Booster framework extensions are called Rockets. A Rocket is nothing more than a node package that implements the public Booster Rocket interfaces to add new functionality to Booster applications.

Multi Provider Rocket

The Booster version 0.24.0 is capable of creating Multi Provider Rockets. Multi-provider Rockets could include implementations for different vendors in the same npm package.

Having all implementations in the same package reduces the need to repeat the same code in different Rocket implementations. Now we only need to publish one Rocket and it can be used by as many providers (AWS, Azure, Local, ...) as we implement support to.

When we build a Multi Provider Rocket we are able to extract the common logic to a package and the specific provider logic to another one. In this case we will have at least as many packages as providers we support. For example, if we create a Rocket that supports Azure and a Local Provider that deploys infrastructure, then we will have the following packages:

Core: The entry point of your Rocket. Where the common logic resides.
Azure: Azure specific logic. For example, code to write to a Cosmos DB table.
Azure infrastructure: Tell Booster how to build the Azure infrastructure. For example, the Terrafom-cdktf code to add a new CosmosDB table.
Local: Local specific logic. For example, code to write to a Nedb table.
Local infrastructure: As with the Azure infrastructure, we need to tell Booster to create a new Nedb table or a new endpoint.
Types: Optionally, we could have a package with shared types to be used by different packages.

Rocket Core is the orchestrator package. It has the responsibility to call the correct provider package, and at the end, the Provider packages will use the infrastructure package as needed.

Rocket example

A common requirement in many projects is to be able to handle files. Booster Core doesn’t support handling files, so a Rocket could help us with this.

Scaffolding

The Rocket files need to provide a basic set of functionalities:

Get a pre-signed url to upload a file to a folder.
Get a pre-signed url to download a file to a folder.
Get a pre-signed url to list all files in a folder.
Following the event sourcing approach, having an event each time a file is uploaded would be useful.
Allow users to define multiple folders.

This Rocket will provide implementation for Azure and the Local Provider. You can find the full code on Github.

If you have any questions about how to start with Rockets, follow the official guide.

The first step will be to create the project scaffolding.

Local Provider infrastructure

Let’s start creating the Local Provider infrastructure. In the index.ts file, export a constant that returns an InfrastructureRocket. This object will contain at least the mountStack method. This method will be called by Booster in the deploy process:

const AzureRocketFiles = (params: RocketFilesParams): InfrastructureRocket => ({
 mountStack: Infra.mountStack.bind(Infra, params),
})

export default AzureRocketFiles

The InfrastructureRocket needs at least one method to mount the stack. We map this method to our Infra class with the RocketFilesParams.

When a user gets a pre-signed get, put or list url, this should point to an Express route that behaves the same as the Azure File Storage.

For the Local Provider, we publish our endpoints by creating Express routes. Because we want to have different endpoints based on the directory parameter, we iterate over our params to create the needed routes.

export class Infra {
 public static mountStack(params: RocketFilesParams, config: BoosterConfig, router: Router): void {
   params.params.forEach((parameter: RocketFilesParam) => {
     router.use(`/${containerName}`, new FileController(parameter.directory).router)
     fsWatch(parameter.directory)
   })
 }
}

For each directory in the parameter list we create a new Router for the /rocketFiles path. This way we can have a single Rocket to handle multiple directories.

The controller will have two endpoints for each directory, one for uploading files and another one to get our files. This is needed to simulate the Azure Blob get/put methods:

constructor(readonly directory: string) {
 this.router.put(`/${directory}/:fileName`, this.uploadFile.bind(this))
 this.router.get(`/${directory}/:fileName`, this.getFile.bind(this))
 this._path = path.join(process.cwd(), containerName, this.directory)
}

We don’t need to add an endpoint for the list endpoint as if the user calls the root url, Express will just return the expected list of files.

Now we only need to define the methods for getting and putting files. Getting a file is as simple as requesting the download method of the express.Response method:

public async getFile(req: express.Request, res: express.Response, next: express.NextFunction): Promise<void> {
 const fileName = req.params.fileName
 const filePath = path.join(this._path, fileName)
 res.download(filePath)
}

Writing a file requires a few steps. First we need to create the destination path and then use the _express.Request _object to write the stream:

const fileName = req.params.fileName
const filePath = path.join(this._path, fileName)
const writeStream = fs.createWriteStream(filePath)
req.pipe(writeStream)
req.on('end', function () {
 const result = {
   status: 'success',
   result: {
     message: 'File uploaded',
     data: {
       name: fileName,
       mimeType: DEFAULT_MIME_TYPE,
       size: DEFAULT_FILE_SIZE,
     },
   },
 } as APIResult
 res.send(result)
})

writeStream.on('error', async function (e) {
 const err = e as Error
 await requestFailed(err, res)
 next(e)
})

Now that we have the main methods, we need to implement the file system watcher. This will detect any changes to a folder and emit the Booster event we want to notify. For this, we implement a fsWatch function that watches the folders and calls the boosterRocketDispatcher method.

export function fsWatch(directory: string): void {
 const _path = path.join(process.cwd(), containerName, directory)
 if (!fs.existsSync(_path)) {
   fs.mkdirSync(_path, { recursive: true })
 }
 fs.watch(_path, async (eventType: 'rename' | 'change', filename: string) => {
   await boosterRocketDispatcher({
     name: filename,
   })
 })
}

The boosterRocketDispatcher is a method Booster provides to interact with the Core functionalities. This method will dispatch the request payload to the function defined in the BOOSTER_ROCKET_FUNCTION_ID environment variable. We defined this variable on the mountStack method for the Local Provider, so let’s add it:

export class Infra {
 public static mountStack(params: RocketFilesParams, config: BoosterConfig, router: Router): void {
   process.env[rocketFunctionIDEnvVar] = functionID
   params.params.forEach((parameter: RocketFilesParam) => {
     router.use(`/${containerName}`, new FileController(parameter.directory).router)
     fsWatch(parameter.directory)
   })
 }
}

The boosterRocketDispatcher function will not only get the request payload, but also will give access to the Booster config variable.

With this call, we can be sure that each time a file is dropped in the directory, Booster will dispatch to our handler. This handler will be defined in the Rocket Core package.

We are connecting our file watcher to our Local Provider logic through the Booster Core dispatcher. All the providers will use this flow to implement the connection:

Local Provider

The Local Provider package (rocket-files-local) implements the logic of the methods we will expose for our Express server:

Get a presigned get url to get a file
Get a presigned put url to** put **a file
List a directory

To return a get and put url we only need to concat the information we have:

export async function presignedGet(config: BoosterConfig, directory: string, fileName: string): Promise<string> {
 return `http://localhost:3000/${containerName}/${directory}/${fileName}`
}

export async function presignedPut(config: BoosterConfig, directory: string, fileName: string): Promise<string> {
 return `http://localhost:3000/${containerName}/${directory}/${fileName}`
}

Those methods will be called from the Core package.

Listing all files in a directory is simply a matter of using the fs module:

export async function list(config: BoosterConfig, directory: string): Promise<Array<ListItem>> {
 const result = [] as Array<ListItem>
 const _path = path.join(process.cwd(), containerName, directory)
 const files = fs.readdirSync(_path)
 files.forEach((file) => {
   const stats = fs.statSync(path.join(_path, file))
   result.push({
     name: file,
     properties: {
       lastModified: stats.ctime,
     },
   })
 })

 return result
}

Core

Once we have the Local Provider implementation, it’s time to build our Core package. This package should call each provider, depending on the application configuration. The first step is to register our Rocket:

export class BoosterRocketFiles {
 public constructor(readonly config: BoosterConfig, readonly params: RocketFilesParams) {
   config.registerRocketFunction(functionID, async (config: BoosterConfig, request: unknown) => {
     return fileUploaded(config, request, params)
   })
 }
}

We register our Rocket using a unique functionID identifier (see rocket-files-params.ts), and a Rocket function that will be called by Booster Core, when our function is called. We will configure this in the Azure provider later. For the Local Provider, it was defined in the fsWatch method.

We also need to provide two methods of setting up the Rocket to our client.

public rocketForAzure(): RocketDescriptor {
 return {
   packageName: '@boostercloud/rocket-files-provider-azure-infrastructure',
   parameters: this.params,
 }
}

public rocketForLocal(): RocketDescriptor {
 return {
   packageName: '@boostercloud/rocket-files-provider-local-infrastructure',
   parameters: this.params,
 }
}

These methods will be used later by the clients.

Next we need to implement the public methods that our clients will use. From the Core point of view, those methods should call the corresponding logic on the provider configured on the application. Now let’s define a class with the get, put and list methods:

public presignedGet(directory: string, fileName: string): Promise<string> {
 this.checkDirectory(directory)
 return this._provider.presignedGet(this.config, directory, fileName)
}

public presignedPut(directory: string, fileName: string): Promise<string> {
 this.checkDirectory(directory)
 return this._provider.presignedPut(this.config, directory, fileName)
}

public list(directory: string): Promise<Array<ListItem>> {
 this.checkDirectory(directory)
 return this._provider.list(this.config, directory)
}

Each method calls the provider method to allow a client using our Rocket Core package to make calls. For example, if a client wants to get a presignedPut url, then it could call our presignedPut method:

 @Returns(String)
  public static async handle(command: FileUploadPut, register: Register): Promise<string> {
    const boosterConfig = Booster.config
    const fileHandler = new FileHandler(boosterConfig)
    return await fileHandler.presignedPut(command.directory, command.fileName)
  }

We only need to implement the fileUploaded method in our Core package to generate a Booster event for each file uploaded. This event will be persisted using the config object. The config object gives us access to the provider events store:

async function processEvent(config: BoosterConfig, metadata: unknown): Promise<void> {
 try {
   const envelop = toEventEnvelop(metadata)
   await config.provider.events.store([envelop], config, console)
 } catch (e) {
   console.log('[ROCKET#files] An error occurred while performing a PutItem operation: ', e)
 }
}

So, for each file uploaded we are going to store an event:

const provider = require(params.rocketProviderPackage)
const metadata = provider.getMetadataFromRequest(request)
if (provider.validateMetadata(params, metadata)) {
 return processEvent(config, metadata)
}

Azure Provider infrastructure

Once we have Local and Core functionality, we could start with the Azure provider infrastructure. As with the Local Provider, we will need to export a constant with the mountStack method that builds the InfrastructureRocket:

const AzureRocketFiles = (params: RocketFilesParams): InfrastructureRocket => ({
 mountStack: Synth.mountStack.bind(Synth, params),
})
export default AzureRocketFiles

We build the needed infrastructure in Azure that consists of:

A storage account
A container
A function

Using cdktf, it's easy to create new TerraformResources and add it to the applicationSynthStack.rocketStack:

const rocketStack = applicationSynthStack.rocketStack ?? []
const rocketStorage = TerraformStorageAccount.build(terraformStack, resourceGroup, appPrefix, utils, config)
rocketStack.push(rocketStorage)

For the FunctionApp we will need to set some specific values to tell Azure how to connect to our blob storage and our functionID:

return new FunctionApp(terraformStack, id, {
 name: functionAppName,
 location: resourceGroup.location,
 resourceGroupName: resourceGroup.name,
 appServicePlanId: applicationServicePlan.id,
 appSettings: {
   FUNCTIONS_WORKER_RUNTIME: 'node',
   AzureWebJobsStorage: storageAccount.primaryConnectionString,
   WEBSITE_CONTENTAZUREFILECONNECTIONSTRING: storageAccount.primaryConnectionString,
   WEBSITE_RUN_FROM_PACKAGE: '',
   WEBSITE_CONTENTSHARE: id,
   WEBSITE_NODE_DEFAULT_VERSION: '~14',
   ...config.env,
   BOOSTER_ENV: config.environmentName,
   BOOSTER_REST_API_URL: `https://${apiManagementServiceName}.azure-api.net/${config.environmentName}`,
   COSMOSDB_CONNECTION_STRING: `AccountEndpoint=https://${cosmosDatabaseName}.documents.azure.com:443/;AccountKey=${cosmosDbConnectionString};`,
   BOOSTER_ROCKET_FUNCTION_ID: functionID,
   ROCKET_FILES_BLOB_STORAGE: rocketStorageAccount.primaryConnectionString,
 },
 osType: 'linux',
 storageAccountName: storageAccount.name,
 storageAccountAccessKey: storageAccount.primaryAccessKey,
 version: '~3',
 dependsOn: [resourceGroup],
 lifecycle: {
   ignoreChanges: ['app_settings["WEBSITE_RUN_FROM_PACKAGE"]'],
 },
})

Next, we need to return our updated applicationSynthStack.

return applicationSynthStack

Finally we have to implement the functionApp code. For this we need to return an Array of our function definitions:

export class RocketFilesFileUploadedFunction {
 static getFunctionDefinition(config: BoosterConfig): BlobFunctionDefinition {
   return {
     name: 'fileupload',
     config: {
       bindings: [
         {
           type: 'blobTrigger',
           direction: 'in',
           name: 'blobUpload',
           path: `${containerName}/{name}`,
           connection: 'ROCKET_FILES_BLOB_STORAGE',
         },
       ],
       scriptFile: config.functionRelativePath,
       entryPoint: config.rocketDispatcherHandler.split('.')[1],
     },
   }
 }
}

We are creating a definition of a function that will be binded to a blob path. The entryPoint and the scriptFile parameters will help Booster to execute our code. Booster provides utilities to connect with the Booster Core in the config object.

The last step for Azure infrastructure providers is to include a method to get the functionApp name. Therefore, our final implementation will be:

const AzureRocketFiles = (params: RocketFilesParams): InfrastructureRocket => ({
 mountStack: Synth.mountStack.bind(Synth, params),
 mountFunctions: Functions.mountFunctions.bind(Synth, params),
 getFunctionAppName: Functions.getFunctionAppName.bind(Synth, params),
})

export default AzureRocketFiles

Azure Provider

Implementing the get, put, list and file uploaded methods in Azure will be done in the Azure provider package.

Let’s build a class to get all the information Azure provides using @azure/storage-blob Package. The get method will be:

public getBlobSasUrl(
 directory: string,
 fileName: string,
 permissions = this.DEFAULT_PERMISSIONS,
 expiresOnSeconds = this.DEFAULT_EXPIRES_ON_SECONDS
): string {
 const key = BlobService.getKey()
 const blobName = BlobService.getBlobName(directory, fileName)
 const credentials = this.getCredentials(key)
 const client = this.getClient(credentials)
 const blobSASQueryParameters = BlobService.getBlobSASQueryParameters(
   blobName,
   permissions,
   expiresOnSeconds,
   credentials
 )
 const containerClient = client.getContainerClient(containerName)
 const blobClient = containerClient.getBlobClient(blobName)
 return blobClient.url + '?' + blobSASQueryParameters
}

We need to build other methods in the same way. The important part here is that we are returning a pre-signed url using the key we have defined in our function:

private static getKey(): string {
 return process.env['ROCKET_STORAGE_KEY'] ?? ''
}

This configuration value is set in the Azure infrastructure package:

applicationSynthStack.functionApp!.addOverride('app_settings', {
 ROCKET_STORAGE_KEY: `${rocketStorage.primaryAccessKey}`,
})

Once we have all the needed functions, we will export them:

export async function presignedGet(config: BoosterConfig, directory: string, fileName: string): Promise<string> {
 const storageAccount = storageName(config.appName)
 return new BlobService(storageAccount).getBlobSasUrl(directory, fileName)
}

export async function presignedPut(config: BoosterConfig, directory: string, fileName: string): Promise<string> {
 const storageAccount = storageName(config.appName)
 return new BlobService(storageAccount).getBlobSasUrl(directory, fileName, WRITE_PERMISSION)
}

export async function list(config: BoosterConfig, directory: string): Promise<Array<ListItem>> {
 const storageAccount = storageName(config.appName)
 return new BlobService(storageAccount).listBlobFolder(directory)
}

A client application

To add this Rocket to our Booster application, first we need to update our dependencies:

npm i --save @boostercloud/rocket-files-core
npm i --save @boostercloud/rocket-files-types
npm i --save @boostercloud/rocket-files-provider-local 
npm i --save @boostercloud/rocket-files-provider-azure

Then we add the dev dependencies for the infrastructure packages:

npm i --save-dev @boostercloud/rocket-files-provider-azure-infrastructure
npm i --save-dev @boostercloud/rocket-files-provider-local-infrastructure

Next, we configure our Rocket in the config.ts file. For Azure:

Booster.configure('production', (config: BoosterConfig): void => {
 config.appName = 'test-rockets-files020'
 config.providerPackage = '@boostercloud/framework-provider-azure'
 config.rockets = [
   new BoosterRocketFiles(config, {
     rocketProviderPackage: '@boostercloud/rocket-files-provider-azure' as RocketProviderPackageType,
     params: [
       {
         directory: 'folder01',
       },
       {
         directory: 'folder02',
       },
     ],
   } as RocketFilesParams).rocketForAzure(),
 ]
})

And for Local:

Booster.configure('local', (config: BoosterConfig): void => {
 config.appName = 'test-rockets-files020'
 config.providerPackage = '@boostercloud/framework-provider-local'
 config.rockets = [
   new BoosterRocketFiles(config, {
     rocketProviderPackage: '@boostercloud/rocket-files-provider-local' as RocketProviderPackageType,
     params: [
       {
         directory: 'folder01',
       },
       {
         directory: 'folder02',
       },
     ],
   } as RocketFilesParams).rocketForLocal(),
 ]
})

To use our Rockets from our application, we create a command that will add a GraphQL mutation. The command to get a pre-signed put url to upload a file will be:

export class FileUploadPut {
 public constructor(readonly directory: string, readonly fileName: string) {}

 @Returns(String)
 public static async handle(command: FileUploadPut, register: Register): Promise<string> {
   const boosterConfig = Booster.config
   const fileHandler = new FileHandler(boosterConfig)
   return await fileHandler.presignedPut(command.directory, command.fileName)
 }
}

And for getting a pre-signed url to** get** a file and to list all the files:

export class FileUploadGet {
 public constructor(readonly directory: string, readonly fileName: string) {}

 @Returns(String)
 public static async handle(command: FileUploadGet, register: Register): Promise<string> {
   const boosterConfig = Booster.config
   const fileHandler = new FileHandler(boosterConfig)
   return await fileHandler.presignedGet(command.directory, command.fileName)
 }
}

export class FileUploadList {
 public constructor(readonly directory: string) {}

 @Returns(String)
 public static async handle(command: FileUploadList, register: Register): Promise<string> {
   const boosterConfig = Booster.config
   const fileHandler = new FileHandler(boosterConfig)
   const listItems = await fileHandler.list(command.directory)
   return '[' + listItems.map((item: ListItem) => JSON.stringify(item)).join(',') + ']'
 }
}

Now that we have all the functionalities ready to handle files, let’s create a ReadModel to project all the file upload events so that we know which files were uploaded. This ReadModel will project the UploadedFileEntity event defined as:

export class UploadedFileEntityReadModel {
 public constructor(public id: string, readonly metadata: unknown) {}

 @Projects(UploadedFileEntity, 'id')
 public static projectUploadedFileEntity(
   entity: UploadedFileEntity,
   currentUploadedFileEntityReadModel?: UploadedFileEntityReadModel
 ): ProjectionResult<UploadedFileEntity> {
   console.log(`ReadModel Projects UploadedFileEntityReadModel ${entity}`)
   return new UploadedFileEntityReadModel(entity.id, entity.metadata)
 }
}

And that’s all. Run your Booster application and you will see the new mutations:

mutation {
    FileUploadPut(input: {directory: "folder01", fileName: "3.txt"})
}
mutation {
    FileUploadGet(input: {directory: "folder01", fileName: "3.txt"})
}
mutation {
  FileUploadList(input: {directory: "folder02"}) 
}

And a query to get the uploaded files:

query{
    UploadedFileEntityReadModels(filter: {}){
        id
        metadata
    }
}

Both are working on Local just as on Azure provider.

Conclusions

With this real world example we have reviewed how to extend Booster logic and infrastructure with the support for any provider.

If you want to know more about how to create Rockets, please go to the official documentation.

Last but not least, if you have any questions about Booster or any other topic related, we will be glad to hear them on our community channel.

Booster authorization in a nutshell

Carlos Dominguez Padron — Thu, 26 May 2022 09:01:46 +0000

Overview

After several successful projects using The Booster Framework, with a variety of clients and internal projects, a common recurring question is How does Booster handle Authorization?
In this article, I will shed some light on this topic, by dissecting Booster’s approach towards authorization.

Standards, standards, standards

By design, The Booster Framework is a role-based authorized framework, which means that commands and read operations, are performed by specific roles created by the user, i.e. Admin, Customer, User, you name it. If you want to keep your API open, you can use the predefined all role, but that would be a bad idea. A better practice would be to limit the access to your API. If you want to know more about how to define roles in Booster, click here.

One widely adopted and standard way of transmitting secure information and authorizing users is the usage of JWT Tokens.

In this article, I don't want to dig deeper into the details of the JWT standard, but long story short, you will need a JWT auth provider, or create your own, which generates valid tokens for your users. Inside of those tokens includes the user roles you need as claims in your Booster application.

In the early stages of Booster development, and since we were using AWS as our first cloud provider, Booster offered endpoints to create, update, and login users, using AWS Cognito. When you created a user, you had to specify that user's role as a part of the associated information. When the user signed in, Cognito returned a JWT token with all the information associated with the user, including the role.

Then, you had to use that token in the Authorization header in your API requests as a bearer token, and Booster internally called Cognito again to verify if the token was valid, and if it contained the right role to perform the API call.

That approach worked for a while, until some Booster users didn't want to use Cognito. Some wanted to use their auth, while others were using a different auth provider like Auth0, Firebase, Cognito, Okta, or whatever.

Once again the standards to the rescue. Even if we were using JWT, Booster was tightly coupled to Cognito to verify the token and get the information associated with it. We decided to extract that part and use a standard token verification inside the Booster core, which works with the JWT tokens, no matter which provider you are using.

Side note: We removed the Cognito dependency from Booster, but if you still want to use it, you can include the AWS Auth Rocket which provides the most common Cognito features out-of-the-box for Booster applications.

Token verifiers

The JWT standard works by signing tokens using private and public key pairs using asymmetric cryptography and different algorithms (mostly RSA or ECDSA). So basically, the public key is well known by the clients, but the private key must be secret and will only be available on the server side.

Ok...but what do I need to use an auth in Booster?

In Booster you will need to specify token verifiers. You can use more than one depending on the use case, but for the moment, let's focus on using one:

Booster.configure('production', (config: BoosterConfig): void => {   
  config.tokenVerifiers = [
    {
      jwksUri: 'https://demoapp.firebase.com/.well-known/jwks.json',
      issuer: 'https://securetoken.google.com/demoapp',
      rolesClaim: 'firebase:groups',
    }
  ]
})

In the example above, we are using Firebase as a provider for a demo app. The jwksUri property contains the public URL where Firebase exposes the public keys as JSON web keys, the issuer specifies who is emitting tokens, and the rolesClaim value is the claim where Firebase adds the roles.

Let's create a configuration using the Cognito provider:

Booster.configure('production', (config: BoosterConfig): void => {   
  config.tokenVerifiers = [
    {
      jwksUri: 'https://cognito-idp.{region}.amazonaws.com/{userPoolId}/.well-known/jwks.json',
      issuer: 'https://cognito-idp.{region}.amazonaws.com/{userPoolId}',
      rolesClaim: 'cognito:groups',
    }
  ]
})

The same here, nothing weird, except provider specific configuration.

That's great, but what if I want to test my roles as a part of the development process and I don't want to stick without any auth provider.

In Booster you can use the awesome local provider for testing purposes and configure it alongside a local token verifier.

Your local kingdom

First, we need to generate our public and private keys for our local JWT auth token generator. For that, you can use this online tool. Attention! Don't use those keys in any production environment.

After that, save both keys into proper files, like private.key and public.key since you will need them later on.

To generate tokens for different users, with different roles, let’s create a simple node.js script:

mkdir testToken;cd testToken;npm init -y;npm install jsonwebtoken

Move the private.key you saved into the project into a folder inside your project i.e. keys folder.

Then copy the following code into the index.js and change it accordingly, depending on your needs.

const fs = require('fs')
const jwt = require('jsonwebtoken')
const path = require('path')

const privateKey = fs.readFileSync(path.join(__dirname, '.', 'keys', 'private.key'))

function forUser(email, role) {
 const keyid = 'booster'
 const issuer = 'booster'
 return jwt.sign(
   {
     id: email,
     demoRole: role,
     email,
   },
   privateKey,
   {
     algorithm: 'RS256',
     subject: email,
     issuer,
     keyid,
     expiresIn: 0
   }
 )
}

console.log('Here is your admin user token: ', forUser('test@boostercloud.com', 'Admin'))

The code above is self explanatory, but basically it’s using the jsonwebtoken library to sign tokens with the private key and it’s adding some data like the email and the role in the demoRole property.

Finally let's run it:

node index.js
Here is your admin user token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImJvb3N0ZXIifQ.eyJpZCI6InRlc3RAYm9vc3RlcmNsb3VkLmNvbSIsImRlbW9Sb2xlIjoiQWRtaW4iLCJlbWFpbCI6InRlc3RAYm9vc3RlcmNsb3VkLmNvbSIsImlhdCI6MTY1MjE5MDE1NiwiZXhwIjoxNjUyMTkwMTU2LCJpc3MiOiJib29zdGVyIiwic3ViIjoidGVzdEBib29zdGVyY2xvdWQuY29tIn0.HMBm_2MPVA_QHKr5hMTW_zmH9BFC5TplOOfSD34NrUUONzOU-1d6gKgNNRV_NX6Nem_4yksnUV64IhhLffmRNljBtIGQ-HaiVQ9S4MnNqyJCQRCArkK4xyu5EQd7RTNtLPS_xetn8kAJYzIlnO1KRNeQphplaeyEMCS5irjR9-A

Woohoo! you have created a valid JWT token to use in Booster.

In order to use that token in Booster, in the API request headers you will need to config the public.key in your token verifier:

Booster.configure('production', (config: BoosterConfig): void => {   
  config.tokenVerifiers = [
    {
      publicKey: fs.readFileSync(path.join(__dirname, '.', 'keys', 'public.key')),
      issuer: 'booster',
      rolesClaim: 'demoRole',

    }
  ]
})

The code above will use the publicKey property instead of the jwksUri since we don’t have a public URL with the keys, as many providers offer.

Take into account that we are using the same issuer we used to sign in the token.

Extra, extra!

Some users asked about token validations that will check other data encoded inside the token, and grant or deny access based on that. For this purpose, Booster config has a property function called extraValidation which receives the decoded token as a parameter, allowing the users to do things like this:

Booster.configure('production', (config: BoosterConfig): void => {   
  config.tokenVerifiers = [
    {
      publicKey: fs.readFileSync(path.join(__dirname, '.', 'keys', 'public.key')),
      issuer: 'booster',
      rolesClaim: 'demoRole',
      extraValidation: (decodedToken) => {
         if (!decodedToken.payload.trust) {
            throw 'We don't trust on you'
         }
      }
    }
  ]
})

Conclusions

As you have seen, Booster provides a standard and easy way of authenticating requests based on user roles which will cover most of the common use cases, since the JWS is widely adopted. For those use cases that won’t fit with the defaults, Booster provides a way of extending the framework thanks to the usage of rockets, but that’s another story. If you want to know more about how to create rockets, please refer to the official documentation.

Last but not least, if you have any questions about Booster or any other topic related, we would be happy to hear your thoughts on our community channel.

Don't waste time building another API, let the machines make them for you with the Booster Framework!

Javier Toledo — Tue, 11 May 2021 17:37:35 +0000

Building a great API is not trivial, you have to design the data structure, choose the right connection mechanism, implement handlers for every endpoint, serialize/deserialize the data and validate it before accepting it in your business logic.

In addition to that, it is often necessary to build API clients for front-end applications or for other services in order to make use of the functionality offered by your service.

If you use GraphQL, the second part can definitely be simplified, as you can use any standard GraphQL client to query the system, but manually building and maintaining schemas and resolvers is still such a hassle that some teams might prefer to stay in REST.

Boost your API with CQRS!

The Booster Framework introduces a very different approach to building APIs: not doing it at all!

This is only possible thanks to the opinionated implementation of the CQRS pattern and Booster's inference capabilities. Let's walk through the structure of a typical Booster application before going back to APIs.

In a Booster application, most functionality is defined in commands. A Command is a data structure that has a handler, like this one that processes a backflip:

@Command({ authorize: 'all' })
export class PerformABackflip {
  public constructor(
    readonly personName: string,
    readonly personAge: number
  ) {}

  public static async handle(command: PerformABackflip, register: Register): Promise<void> {
    const luck = Math.rand()
    if (command.personAge > 35 && luck > 0.5) {
      register.events(PersonSurvivedABackflip(command.personName)
    } else {
      callAnAmbulance()
      register.events(AmbulanceCalled(command.personName)
    }
  }
}

When a command handler is processed, it finishes by writing one or more events into the event store. In this case, a PersonSurvivedABackflip event for lucky or young people, or an AmbulanceCalled for the rest of us 😅.

An Event is just a data structure that looks like this:

@Event
export class PersonSurvivedBackflip {
  public constructor(
    readonly personName: string
  ) {}

  /* We define the `entityID` method to help Booster match 
  the event with the corresponding entity */
  public entityID() {
    return this.personName
  }
}

These events are later reduced into entities that represent the current state. The reduction is carried out by reducer functions that look like these:

@Entity
export class Person {
  public constructor(
    readonly name: string,
    readonly backflipTrials: number,
    readonly backflipSuccesses: number,
  ) {}

  @Reduces(PersonSurvivedBackflip)
  public static reduceSurvival(
    event: PersonSurvivedBackflip, 
    currentPerson?: Person
  ): Person {
    return buildNextPersonObject(currentPerson, true)
  }

  @Reduces(AmbulanceCalled)
  public static reduceFailure(
    event: AmbulanceCalled,
    currentPerson?: Person
  ) {
    return buildNextPersonObject(currentPerson, false)
  }

  private static buildNextPersonObject(
    currentPerson?: Person, 
    success: boolean
  ) {
    const trials = (currentPerson?.backflipTrials ?? 0) + 1
    let successes = (currentPerson?.backflipSuccesses ?? 0)
    if (success) { successes++ }
    return new Person(
      event.personName,
      trials,
      successes
    )
  }
}

Securing queries with Read Models

At this point, the CQRS design helped us to separate the data schema that the system accepts (commands) from the state data schema (entities). This in itself already simplifies the API design: The API schema could just match the command and entity schemas, and we could call that a nice API. But before calling it a day, we need to make an extra consideration; Allowing direct API access to entities would mean no restrictions on data access, so private fields like password hashes or bank accounts would become accessible. That's why Booster adds Read Models to the mix.

ReadModels are eventual consistent caches of the internal state. They're not only a way to filter which fields you want to make accessible, but they can also aggregate related data or make small data transformations to optimize reads. Accessing read models is highly performant and they're updated in real-time when data changes, pushing these changes to the client applications.

A typical read model that projects a single entity would look like this:

@ReadModel({ authorize: 'all' })
export class PersonReadModel {
  public constructor(
    readonly name: string,
    readonly backflipTrials: number,
    readonly backflipSuccesses: number,
  ) {}

  @Projects(Person, 'name')
  public static projectPerson(
    entity: Person,
    currentState?: PersonReadModel
  ): ProjectionResult<PersonReadModel> {
    return new PersonReadModel(
      entity.name,
         entity.backflipTrials,
      entity.backflipSuccesses
    )
  }
}

Inferring APIs from code!

At this point, we could build a very nice, useful, and secure API by just copying the schemas from the classes decorated as Command or ReadModel, and that's exactly what Booster does for you!

Booster analyzes the class structure of all classes decorated as @Command or @ReadModel in compile-time, generating metadata that is used in deploy time to generate a GraphQL schema and provision all the cloud resources required to make the application work, including API gateways, lambda functions, containers, permissions, and even the database tables to store the events.

For some people, this might look similar to maintaining the schema in a regular GraphQL schema file, because, at the end of the day, you're still expressing the schemas as Command and ReadModel classes, but Booster brings many extra advantages to the table:

Commands, Read Models, and all their usages are type-checked in compile-time, reducing or eliminating the likelihood of making mistakes.
No errors can be introduced when serializing/deserializing data because this is done transparently by the framework.
When you update a command or a read model, the API is updated automatically, you don't need to change any other files.
Resolvers are hidden under higher-level abstractions like commands or read models, so you don't need to deal with low-level protocol nuances.
All Read Models support WebSockets by default, so there is no need to implement any extras for real-time support in your applications.

To summarize, by writing highly semantic code and letting the machine do the heavy lifting, Booster allows you to build fully functioning real-time APIs in a breeze, making everything else work out of the box, and saving a ton of time that you can use to add new use cases, write better tests, or manage elusive corner cases.

Try it!

Booster is a 100% open-source project developed by The Agile Monkeys. You can get its full potential for free and with no hidden fees. The typical Booster application can be run on the free tier in AWS, but it also has experimental support for Azure and Kubernetes, so you can run it locally using Minikube.

Learn more about Booster on the official website, the documentation, or the Github project.

If you try it out, be sure to let us know what you thought of it on the project’s Discord channel!

What Booster does for the developer experience

Mario Castro Squella — Mon, 10 May 2021 16:03:01 +0000

Having been a software developer for quite some time now, I've had my fair share of "experiences." The thing is, most of them were just piecing together things instead of writing actual code that solves a business problem. As Stephen O'Grady, co-founder of Redmonk, succinctly put it, "...developers are forced to borrow time from writing code and redirect it towards managing the issues associated with highly complex, multi-factor developer toolchains held together in places by duct tape and baling wire."

I came of age as a developer around the time that frameworks like Ruby on Rails appeared (I did not use Rails since I did my coding in PHP at the time, most of it with the CakePHP framework). Rails inspired many other frameworks for other programming languages that followed its opinionated approach (how about that "MVC craze" in the mid-aughts?). What was empowering about Rails and similar frameworks, though, was that it took care of many things that distracted developers from solving the actual business problem, such as connecting to databases, configuring routes, and so forth. That led to many innovative ideas coming to market faster and shortening the feedback loop. Many went on to become multi-million-dollar businesses and services (GitHub, Airbnb, and Shopify are the poster children for being built on top of Rails). Rails and its clones did this by using their opinionated approach and convention over configuration or, as Bryan Liles describes it in his KubeCon keynote, by being good at providing suitable defaults.

Quite a few years have passed since Rails and similar tools appeared. We now live in a world of public clouds, with many available services that cater to many developer needs. But despite the availability of many tools and solutions, we're still having to put an overwhelming number of things together before writing a single line of business code (and then manage all of it once it's up and running in a production environment). "When you use a tool, it should be easy to use. It should just seamlessly drop into your workflow and elevate you," says Taylor Dolezal, a Developer Advocate at HashiCorp.

All of which begs the question, can we do something to make developing cloud-native applications a joy and not a hassle?

At The Agile Monkeys, we've become aware of those hassles, especially after years of work consulting for many different organizations of many different sizes. Our experiences led to the creation of Booster. Booster is a tool to improve the development experience for those who want to focus their attention on writing actual code that solves a real business problem instead of tacking together cloud infrastructure through multiple configuration files that they then have to integrate into many different configuration management tools.

Booster is a framework for developing software, much like Rails, but it is made for the cloud-native times we live in today. You install the Booster CLI, scaffold and generate boilerplate code, implement your business logic following its opinionated approach, configure your cloud provider, and you're ready to deploy in a cloud-native fashion! To get a better idea, you can watch a 10-minute demo here.

Booster's main contribution to an improved developer experience is inferring everything from the business code. Just using the business code as input, Booster is able to select and provision the required cloud resources (based on your cloud provider of choice), configure the necessary connections and permissions, and return the GraphQL API endpoints, which can then be used by front-end or mobile applications to interact with your system. No more configuration via convoluted YAML files. No more provisioning via vendor-specific SDKs or consoles. The fine folks from the Booster community (be sure to reach out to them on Discord!) have taken the trouble of doing that so devs don't have to.

I believe the future of cloud-native application development is in tools like Booster, tools that will support better development workflows and a more enjoyable developer experience. I recently read this article with the title "Developer experience is the next major competitive front in enterprise tech," and you know what? I could not agree more.

Why a DynamoDB call can take 5 minutes to complete

Álvaro López Espinosa — Fri, 30 Apr 2021 14:21:48 +0000

As part of the work I was doing developing Booster Framework, I was in charge of creating heavy load tests to ensure that its auto-scalability feature works and that the data integrity is preserved (under the AWS provider).

The problem

After many hours of work, load testing Booster with 3,000 requests per second was working pretty well. However, sometimes we were seeing this problem:

The final counter of processed events had to be 60,000 to consider the test successful, but sometimes it got stuck in a random number very close to the right one (for example, 59,489)
We didn't see any error anywhere.
Suddenly, after 5 minutes or so, the right number appeared.

This was unacceptable.

Searching for the root cause

After a lot of investigation, we were seeing that the 0.00001% of all the DynamoDB calls done (it didn't matter if they were a “query”, “get” or “put”) were taking like 4 or 5 minutes to succeed, while the usual time for them was between 5 and 20 milliseconds.

– "Aha! That's the problem!", I thought.

Yes but, Why were those calls taking that long?

The AWS SDK has a built-in retry mechanism. When you do a DynamoDB call, for example, it will automatically redo the request if it gets an error that’s considered “retriable” (like a temporary network error).

It keeps doing retries using an exponential back-off algorithm, meaning that it will wait 50ms before doing the first retry. If it fails again, it will wait twice that time, 100ms, to do the next retry. If it fails again, it will wait 200ms, then 400ms, then 800ms, then 1.6 seconds, etc.

– "That's why calls take so long!!!", I told to myself.
– "Nein, nein, nein!!!", the facts replied to me

It turned out that only 10 retries are attempted by default, and that’s only 51 seconds in total.

Where did the rest of the time (more than 4 minutes) come from?

To the point: It turned out that those 0.00001% DynamoDB calls were done with an HTTP socket in a bad state (because these things happen randomly) and the requests never reached DynamoDB.

The AWS SDK has a default timeout of 2 minutes, so the call was stuck that amount of time before returning an error (a "socket timeout error").

This means that, in order for the first retry to occur, we need to wait 2 minutes. Then, it could be that the second retry also finds the socket in a bad state (it is being reused by thousands of requests), so we wait 2 more minutes.

This explains why a simple DynamoDB call occasionally took 4 or 5 minutes.

Solution

Surprisingly, the solution was just to reduce the timeout to, for example, 5 seconds, instead of 2 minutes. This way we fail much earlier, the AWS SDK can do the next retry (with a new socket connection) sooner, and the chances to succeed are much higher.

This is the code to do this in the Typescript AWS SDK:

const dynamoDB: DynamoDB.DocumentClient = new DynamoDB.DocumentClient({
  maxRetries: 10,
  httpOptions: {
    timeout: 5000,
  },
})

Lessons learned

a) Networking things can fail randomly for no apparent reasons. Even if there are no bugs, just because of network conditions.
b) Timeouts are extremely important. Any request should have a timeout. If not, it can get stuck because of point a) above and there is no way to stop it.
c) Retries are a must in any request. Due to points a) and b) you need to have a retry policy. Not having it is simply a bug, because any network request is expected to fail for no apparent reason. If you don’t have a retry policy, it is similar to having a switch in code with no default and you missed a case condition.
d) Load tests are the most useful thing to find those kinds of errors that you never think of.

Extra tip!

If you find yourself facing similar abnormal long waits when doing any AWS SDK call, try to set the parameter "maxRetries" to 0 and try again. This will expose the error that's causing the retries, so you will be able to act upon it.

Of course, don't forget to enable the retries again after the error is fixed!!

Understanding event-sourcing using the Booster Framework

Mario Castro Squella — Thu, 29 Apr 2021 14:50:18 +0000

I have a confession to make. Before I joined The Agile Monkeys and the Booster project, I had no clue about event-sourcing...

OK, that statement is not entirely accurate. I had been exposed to the concept before (the idea of appending events to a log and using them to reconstitute state) but not in a way related to information systems. Where had I encountered such a concept? Every time I logged in to my bank's website and looked at my account.

You see, in banking, it is vital to keep a record of every event (the most obvious being withdrawals and deposits) that occurred in an account; you don't expect to look at your bank statement and have it simply tell you you have a total of $0 in your account. You want to know how and when you earned or spent money so you can see how you got to $0. Imagine you open an account and start with $0; days later, your first paycheck is deposited and you have $100. You then go to the ATM and withdraw $50 for groceries; a few days later, you withdraw another $50 to pay back a friend who loaned you money. Your total balance after those transactions? $0.
This method of record-keeping and reconstituting state has been around for centuries; it just wasn't called event-sourcing until recently. The bank account ledger is just one example, but there are many others in other areas, such as addendums in legal contracts.

Record-keeping and state in traditional software systems

Event-sourcing appears to be a natural way of modeling data in information systems, but it seems that most of us who learned about storing data when learning about building information systems in software engineering school only learned the CRUD (create, read, update, and delete) method (ALVYS co-founder and CTO Leo Gorodinski even went so far as to say that someone could've come up with event-sourcing by paying no attention to the software engineering literature).

The traditional CRUD style of persisting data and only the current state has its historical reasons; there was a time when computer storage was expensive, so storing every event was unaffordable. But today, in the age of cloud computing, this is not a problem anymore (Adam Dymitruk, creator of Event Modeling, has a great explanation of this on the Event Modeling blog).

An example of CRUD vs. event-sourcing

Let's compare the different approaches with a very simple example: an address book or a list of contacts.

Traditional CRUD approach

For most people, the most obvious way of storing these contacts would probably be in a relational database in a contacts table that would look like something like this:

CREATE TABLE Contacts (
    Id int,
    LastName varchar(255),
    FirstName varchar(255),
    Address varchar(255),
    PhoneNumber varchar(255),
);

Want to add a new contact? Just run an INSERT SQL query. Has your contact changed their address or phone number? Execute an UPDATE SQL query and update those fields. Want to know where a person with a specific ID lives? Execute a SELECT Address query for that ID. This works pretty well most of the time, but what if you need to know the previous addresses where someone lived and not their current address? All those UPDATE queries changed those fields in that ID's row, and the old data is lost forever. That means finding the previous addresses is not possible.

Event-sourcing approach

Instead of updating a row on a database and storing the latest state, with event-sourcing, you would record everything that happens as an event. Let's say that instead of a relational database, we store the events as JSON objects in a NoSQL database, like MongoDB. (However! I am not saying you need to use a NoSQL database for event-sourcing. You can create an event-sourced system using a traditional, relational database, but the schema-less nature of NoSQL databases makes it easier to explain how events are stored.) Let's look at a sample event stream for one of the contacts represented as an array of JSON objects:

[
  {
    "eventType": "ContactCreated",
    "registeredAt": "2021-01-01T00:00:00+00:00",
    "data": {
      "id": 90125,
      "lastName": "Doe",
      "firstName": "John",
      "address": "22 Acacia Avenue, London, UK",
      "phoneNumber": "634-5789"   
    }
  },
  {
    "eventType": "PhoneNumberChanged",
    "registeredAt": "2021-03-21T00:00:00+00:00",
    "data": {
      "id": 90125,
      "phoneNumber": "867-5309"   
    }   
  },
  {
    "eventType": "AddressChanged",
    "registeredAt": "2022-01-01T00:00:00+00:00",
    "data": {
      "id": 90125,
      "address": "2120 South Michigan Avenue, Chicago, IL"    
    }   
  }
]

If we were to query this contact's phone and address, we would see that John Doe, with ID "90125", lives at 2120 South Michigan Avenue, Chicago, IL, and his phone number is 867–5309. But storing this stream of events lets us also easily find out that on March 21, 2021, John Doe changed his phone number from 634–5789 to 867–5309 and then changed his address from 22 Acacia Avenue to 2120 South Michigan Avenue on January 1, 2022. With the traditional CRUD approach that many developers (myself included) have been accustomed to, this information would've been lost.

For the example I just used, the business value of modeling the data around events and storing them is not evident; you pretty much only care about where your contact currently lives and their current phone number, but think of an e-commerce application. Imagine that every time a person modifies their shopping cart, an event is registered with the items that were added to or removed from the cart, and a final event is registered when the person proceeds to checkout. All these events happening between the customer adding the first item to their cart and proceeding to checkout provide a gold mine for analytics. Do people start removing items once they cross a certain price threshold? Which items do people start removing? Do people save items for later? All of these insights can then be translated to better business decisions; do we offer a discount once people cross that price threshold so they don't start removing items from the cart? Do we lower the prices of all the items on our site?

From a business perspective, this is one of the most evident examples of the advantages of modeling your system around events. Another great benefit of this approach is that it provides an audit log right out of the box; this benefit is of particular importance in financial applications. There are also advantages from a technical perspective in terms of scalability, resilience, and decoupling since different applications could tap into a shared event store and derive their own models (I recommend reading Martin Fowler's article on event-sourcing). Of course, like all things in life, nothing is perfect, and there are things you need to watch out for when implementing an event-sourced system, such as eventual consistency and versioning (these topics merit their own articles).

Using Booster to implement an event-sourced application

Seeing as most developers are not accustomed to implementing event-sourced systems, it might appear to most of them that it's technically challenging, especially from the infrastructure and security perspective, and not worth putting forth the effort, especially for those who want to ship their product quickly and require a shorter time-to-market. Luckily, the Booster team has come up with an opinionated framework for coding event-sourced applications using TypeScript, removing many of the barriers to entry of getting such an application up and running quickly. It doesn't just provide you a good starting point to structure your code, but it even generates the whole infrastructure in your cloud provider of choice, ready for production from minute 1.

Booster's cornerstones are event-sourcing, CQRS, and elements of domain-driven design. I'm not going to do a deep dive on this—that's what Booster's documentation is for (and I strongly recommend reading it). Here's the abridged version of what a developer needs to code when building an app with Booster:

Commands: User actions to interact with the application (e.g., AddContact, UpdatePhoneNumber, UpdateAddress)
Events: Things that happened (e.g., ContactCreated, PhoneNumberChanged, AddressChanged)
Entities: Representation of a domain entity's state (e.g., Contact)
Read Models: Cached data the different entities optimized for read operations

Let's look at the code for our sample address book, starting with the commands. Here's what the code for an AddContact command looks like:

@Command({
  authorize: 'all',
})
export class AddContact {
  public constructor(
    readonly id: UUID,
    readonly firstName: string,
    readonly lastName: string,
    readonly address: string,
    readonly phoneNumber: string
  ) {}

  public static async handle(command: AddContact, register: Register): Promise<void> {
    register.events(
      new ContactCreated(command.id, command.firstName, command.lastName, command.address, command.phoneNumber)
    )
  }
}

The @Command class decorator is used to define a command. The class constructor must have the fields that will be part of the request (a GraphQL mutation, as we'll see later) to submit the command. The authorize: 'all' property in the decorator is part of Booster's authorization mechanism. We're not going to talk about that in this tutorial; you can take a look at the documentation to learn more about that.

The command's handle function can do all the necessary logic and validation before registering an event; in this case, it's a new ContactCreated event. Let's look at the code for that event:

@Event
export class ContactCreated {
  public constructor(
    readonly id: UUID,
    readonly firstName: string,
    readonly lastName: string,
    readonly address: string,
    readonly phoneNumber: string,
  ) {}

  public entityID(): UUID {
    return this.id
  }
}

To define an event, a class decorator is also used; in this case, it's the @Event decorator. The code for events is pretty straightforward. The event's structure is defined by its properties in the constructor, and in the entityID function, we need to define which of the properties is the unique identifier for the domain entity affected by this event.

The code for the UpdatePhoneNumber and UpdateAddress commands, as well as for the PhoneNumberChanged, and AddressChanged events follow a similar pattern, so I'm going to skip them here, but you can take a look at them in the GitHub repo.

Now we need to define our domain entity. In the case of the address book, we're managing a bunch of contacts; these contacts all have details for people's first and last names, addresses, phone numbers, and unique IDs to distinguish contacts from each other. The events we've defined make up the source of truth for our system, and we'll use them to reconstitute the state for the different entities. Let's look at the code:

@Entity
export class Contact {
  public constructor(
    public id: UUID,
    public firstName: string,
    public lastName: string,
    public address: string,
    public phoneNumber: string
  ) {}

  @Reduces(ContactCreated)
  public static reduceContactCreated(event: ContactCreated, currentContact?: Contact): Contact {
    if (currentContact) {
      return currentContact // if contact already exists we'll ignore the event
    } else {
      return new Contact(event.id, event.firstName, event.lastName, event.address, event.phoneNumber)
    }
  }

  @Reduces(AddressChanged)
  public static reduceAddressChanged(event: AddressChanged, currentContact: Contact): Contact {
    return new Contact(
      currentContact.id,
      currentContact.firstName,
      currentContact.lastName,
      event.address,
      currentContact.phoneNumber
    )
  }

  @Reduces(PhoneNumberChanged)
  public static reducePhoneNumberChanged(event: PhoneNumberChanged, currentContact: Contact): Contact {
    return new Contact(
      currentContact.id,
      currentContact.firstName,
      currentContact.lastName,
      currentContact.address,
      event.phoneNumber
    )
  }
}

Just like commands and events, we annotate our class with an @Entity decorator to define an entity, and in the constructor, we define the properties we want our Contact entity to have. Then, we need to define a reducer function for each event. You see, the state is reconstituted by taking all the events in an event stream, going from oldest to newest, and applying a function on the state of the entity up to the point of that event and the event itself. For example, take the reducePhoneNumberChangedfunction, which reduces the PhoneNumberChanged event; it's going to return the same first name, last name, and address as the currentContact (i.e., the state of the entity up to that event) and change the phone number to the one registered in the event. Now that we can successfully reconstitute the state of an entity, let's expose them as read models for the world to view them! To do this, we define a read model that projects an entity. Let's look at the code for the ContactReadModel:

@ReadModel({
  authorize: 'all',
})
export class ContactReadModel {
  public constructor(
    public id: UUID,
    public firstName: string,
    public lastName: string,
    public address: string,
    public phoneNumber: string
  ) {}

  @Projects(Contact, 'id')
  public static projectContact(
    entity: Contact,
    currentContactReadModel?: ContactReadModel
  ): ProjectionResult<ContactReadModel> {
    return new ContactReadModel(entity.id, entity.firstName, entity.lastName, entity.address, entity.phoneNumber)
  }
}

Once again, we use decorators, and in our read model class constructor, we define the structure or the properties we'll return to the user when they run a GraphQL query on this read model.

And that's it! The cool thing about Booster is that we can now deploy directly to our cloud provider of choice without having to think about setting up API endpoints, databases, etc., because in Booster, all that is inferred from the code. Once your app is deployed (more on that in the documentation), you can run GraphQL queries and mutations like the following:

Add a new contact:

mutation {
  AddContact(
    input: {id: "90125", firstName: "John", lastName: "Doe", address: "22 Acacia Avenue, London, UK", phoneNumber: "634-5789"}
  )
}

Change that contact's phone number:

mutation {
  UpdatePhoneNumber(
    input: {id: "90125", phoneNumber: "867-5309"}
  )
}

Query that contact's current info:

{
  ContactReadModel(id: "90125") {
    id
    firstName
    lastName
    address
    phoneNumber
  }
}

You can check out the complete code on GitHub and try it out yourself!

Conclusions

Event-sourcing gives us a new way to model data in information systems that's different from the CRUD way of thinking many software developers are accustomed to. This new method offers benefits in terms of scalability, resiliency, decoupling, and access to information previously lost in systems that only handle the latest state.

Booster provides an easy way to harness the benefits of event-sourcing by way of an opinionated framework for developing applications with TypeScript and cloud infrastructure that's inferred from the code.

I encourage you all to try out Booster and modeling your systems around events. Learn more by visiting Booster's website, GitHub repo, or join the conversation on the Booster Discord server!

Event-sourcing and the event-replay mystery

J2RGEZ — Fri, 06 Nov 2020 09:12:50 +0000

It’s been a while since we first started our way into event-sourcing and, from our experience, we really think that there is still space to dig into what event sourcing is and why it’s useful for modern backend applications. But first...

What’s event-sourcing?

Event-sourcing is a collection of patterns that introduces a new way of thinking, where domain events are the source of truth (domain referring to domain-driven design). We need to fulfill the following contract:

The event store persists the full history of domain events. For example:

UserRegistered
UserChangedPassword
UserPublishedAnArticle
Domain events must be chronologically ordered
Domain events are immutable

Given #1 and #2, we can actually replay domain events. That is, reconstructing the state of your application by running the full history of domain events, which can give you the chance to reconstruct the current state of your system and go back to a specific point in time, allowing the following:

Easily find bugs from the system and reconstruct those bugs in different environments.
Deploy different versions of your backend for testing purposes.
Recover from errors by replaying events from a specific point in your system.

Specifically for microservice architectures, there is one case when this comes in handy: Imagine adding a new service, meaning a new read model (local DB) needs to be created. For this purpose, we need to reconstruct the current state of domain models by replaying domain events from the Event Storage. This operation becomes easier when adding snapshotting to the formula.

Snapshotting for event replaying

Snapshots represent the current state of an entity (or aggregate) at a specific time. Snapshots can be configured to be done whenever you want. For example, every X events, X days, or when something special happens in your system.

Replaying events then becomes easier, because you can start rebuilding your service from a specific point of your events’ timeline, and then run all further events from that point. Check the example below.

As you can see, you don’t have to go back to the first UserRegistered event and run all events, but instead, you just take advantage of snapshotting and rollback the User state just before deleting the article.

Not everything is shiny

Definitely, event sourcing advantages were the ones that motivated us to enter this field, but along the way, we found that things weren’t as easy as the theory says:

With high traffic, what would happen if your application goes down? How are you going to recover from the previous failed events plus keep receiving the new ones?
How do you handle side effects? For example, when replaying events imagine that one of the events calls your email service. In that case, we should check that the email was previously sent, or otherwise while we replay the events, the user could end up with a lot of repeated emails.
When should you store a snapshot? Snapshotting depends a lot on the business needs, so we should be open to leave this as configurable as possible, or just generalize it and snapshot every X events.
Is it better to store snapshots synchronously or asynchronously? Synchronous snapshot persistence could happen if you’re not worried about persistence latency.
How do you handle new events coming while replaying events?
How are you going to manage GDPR? As events must be immutable, theoretically you can’t delete events even if they contain personal data.

These problems can be probably solved by adding more control parameters to your events, adding additional checks (to avoid side effects for example), or having a high available queue system like Kafka/RabbitMQ in the middle of your store and your database (for events queuing and system recovery).

Although, what we’ve missed these years is an event-sourcing standard implementation, where the developer doesn’t have to think about the big complexity it has to develop the majority of useful use cases. That’s what we’ve been trying to do with Booster Framework, and we are really excited about what we’ve currently achieved.

Feel free to join our discord or slack if you want to know more about it!

Redefining Event Sourcing

rdoria1 — Fri, 30 Oct 2020 17:40:32 +0000

Event Sourcing is complex and involves multiple concepts, patterns, and architectures. At The Agile Monkeys, we have been working with it for several years and we would like to share our vision of the main concepts around Event Sourcing.

Events

Things that have happened (yes in the past).

e.g. a customer has placed an order.

State

Something that evolves and changes over time with a finite number of possible values.

e.g. A SKU can have multiple states (in stock, out of stock, soon to be available, discontinued, etc.)

Events + State

Events = Things that have happened that changed some state.

e.g. Customer placed an order and therefore the order is in the PLACED status and the inventory of the item decreased by 1 etc.

Event Storage

Permanent Storage where events are persisted.

e.g. Can be a DB (NoSQL, SQL…) or maybe object storage (S3...)

Event Sourcing Contract: 3 Main Concepts

Store the full history of events (aka state changes) in the Event Storage: of course, we mean only the events we intended to store, the ones we designed to persist.
Events are chronologically ordered: This allows to reconstruct the state of the system at any given point in time.
Immutable events: it’s an append-only system (no update nor removal).

Data Removal

Because of data privacy policies (like GDPR in Europe), there are some cases where users want to delete their data. Allowing this would break the Event Sourcing Contract we just described (immutable events).

How can we support this use case?

We can encrypt the data. When users request to delete their data, we are going to delete the encryption key.

This design solves both problems:

By dumping the key we are making data unusable which fulfills the purpose of deleting the data from the user perspective.
We are not opening unwanted doors by actually deleting the data (no permissions, no repo with delete methods, etc.) so we are still respecting the contract we described.

Event Sourcing usually works with

Domain-Driven Design
CQRS
Event-Driven Architecture
MicroServices

Event sourcing excels when used together with the 4 concepts above.

Domain-Driven Design

Uses conceptual maps (called object models) of the domain (easily maps to an actual business concept/idea) that incorporates both behavior and data.

Creates common concepts/vocabulary to easily communicate with other technical teams and business people.

e.g In the e-commerce business, a must-have object model would be an order (defined by its ID, customer ID, date, status, etc..)

1. Domain Event

Captures something that happened that changed the state of a domain model.

e.g. A person changed their address, which changes the Address domain model given an AddressChanged event.

We can then use the knowledge gained from the changes to do something useful with it.

e.g. Since the person moved to a new place, we can send them reminders to update their address for all their bills.

2. Event Sourcing + DDD

Stores the full history of domain events in the Event Storage.

Event Sourcing + Event-Driven Architecture

Misconception: They are not the same!

Event-Driven Architecture: System which is based on components communicating mainly or exclusively through notification events.

When talking about “event” in event sourcing we are actually referring to the state change and not a “notification” as in the event-driven architecture (queues, bus, streams, async communication in general).

domain event != notification event

Event Sourcing + Microservices

When working with microservices architecture, we need to notify multiple services about state changes (domain events) so they can react to them.

Usually, this is done with notification events (so we are using an event-driven architecture).

In this case, notification events are domain events.

e.g. Domain events persisted in the Event Storage. In an event-driven architecture, we need to notify Service A and B about them. In this case, we can call them notification events, as we decided to use async communication for this purpose.

Reading Data

Don’t use your Event Storage as a Read Model.

You might be tempted to query your Event storage to find the latest state for a given event.
This can be slow for large systems. It might work for very small systems but reconstructing the full history to find the latest state for regular usage of your system doesn’t sound like a very good idea.

1. CQRS - Read Model

A pattern that requires having separate classes for reading and writing data.

This allows us to have different models to read and write data.
This also means the repositories (and therefore the DBs) can be different for reading and writing data. For instance, we can write to a relational DB (the writing part is the Event storage) and then read from a NoSQL DB.

CQRS generally has 2 main concepts: commands (write) and queries (read).

A command can return a value: Most likely the returned value will have to do with operation confirmation or identifiers and nothing to do with state changes and this operation is expected to be asynchronous. A domain event will be put into a queue (stream etc..) and be processed at a later stage.
A query does not change any state, its only goal is to read.

This separation makes the following scenario possible.

Let’s say we decide our Event Storage will be a NoSQL DB. Our Read Models will live in our Services and they will be separate DBs, they can be relational DBs.
The Event Storage will remain our source of truth but each Read Model will be designed to render the data as needed per each service. That’s its only goal.

Snapshotting

Since we are storing everything in the Event storage, one very common question is “How do we avoid reading all the data to find the latest state for a given domain model?” The solution is to create snapshots from time to time. Snapshots are actually part of the Event Storage itself. Let’s give an example to make it clearer.

One of the classic examples of event sourcing is the bank card we use to perform withdrawal and credit operations. Domain events will persist these operations. In order to know the current balance of the account, we need to reconstruct all the operations from the beginning. Snapshotting here can be used to have the balance computed every 5 operations. In the worst case, we would need to get the latest snapshot + 4 domain events to know the current balance, thus decreasing the number of reads significantly.

1. Snapshotting vs Read Model: Replay events

If Snapshotting is used to reconstruct the latest state for a given domain model, why would we use Read Models?

When working with a micro-service architecture they are used for different purposes.

Imagine adding a new micro-service, therefore a new ReadModel DB. We need to reconstruct the latest history of domain events from the Event Storage so we’ll use the Replay events feature to achieve this.

Replay Events is a feature of event sourcing and is a byproduct of 2 of the main concepts we defined: full history of domain events + ordered domain events. We can replay the full history of what happened and get the latest state for a given domain model.

This can be a very time-consuming operation if we are talking about millions of domain events. The fastest way to do this is using Snapshotting.

In this case we can seamlessly add a new microservice. We are using snapshots for a different purpose as we are still using read models to read data from a given Microservice and not the snapshot from the Event storage.

The same situation applies if we decide to completely dump a Read Model and change it for a new one. Our source of truth remains the Event Storage and we just need to replay all the events and create our new read model.

Benefits of Event-Sourcing

Data-driven business
When starting a new business/new product etc.. we just don’t know how much our data is worth. Even if the goal is clear at that point in time, it’s always a better idea to keep all the data to possibly use it at some point in the future (storing data is cheap now) Example: Let’s imagine we are creating a new e-commerce site and we are in charge of specifically adding features and then the checkout process to cart items. In this case, what matters the most is the final state of my cart before the user places the order. The fact that the user has added and then removed items doesn’t really affect the checkout process and the correct functioning of the website. Nevertheless, we should keep the full event history of adding/removing items from the cart. Even if not useful now, it can help us analyze the customer behavior and see why they added and deleted those items.
Audit/Logging
By reliably persisting the full history of events we will be able to debug our applications more easily by finding the full history of events for a given domain model.
Since it’s an append-only system, we never lose data. We can’t delete any data.
High-Security Standard naturally by design: We can’t update or remove data. If a system gets hacked, data could still be appended meaning the current state of a given domain model would change. In any case, not losing any data is a great benefit.
Great fit for analytics: Since we have the full history of everything that happened, we analyze the past and use data to drive the business.

Disadvantages of Event-Sourcing

Mental shift for developers: There are a bunch of concepts that need to be mastered here, event sourcing, DDD, CQRS, event-driven architecture etc. It can be overwhelming.
It should not be used for all scenarios. Like any other pattern, it doesn’t make sense in some cases.
Unless we use the Event Storage for the read models (which is definitely not the best practice or not even possible in some cases), we are introducing data redundancy which can lead to potential inconsistencies and also eventual consistency.

This is how we work with event sourcing in Booster.

Emulating the cloud within Booster Framework 💻🌩️

Nick Tchayka — Thu, 29 Oct 2020 18:28:20 +0000

One of the cool things about Booster is that most of it's functionality sits on top of an abstract interface that expects some stuff from the cloud. The framework itself doesn't have a single call to any service from AWS, Azure, or Kubernetes. That's the job of the provider packages.

When you are developing your app, you probably don't wanna think about the very little details of each database, cloud service or whatever. Perhaps you, like me, hate even having to learn each and every library or SDK for the technology/service at hand.

Thanks to this abstraction, you just code by using Booster concepts (command, events, etc.) and forget about the rest. But what happens underneath? Let's take a look 👀

Cloud vs local development

The cloud is cool and all that jazz, but what's better than developing locally and seeing your changes instantly?

Yeah, there are things that emulate the workings of specific services, like DynamoDB, or there are folks who run their entire Kubernetes apps, with all the required processes, like MongoDB, MySQL, Redis, etc. Or even things like Serverless framework that deploy your app relatively quickly, but at the cost of maintaining a huge, messy, YAML file.

Stuff should be simpler, you shouldn't need a beefy computer to develop your app.

Due to many reasons, but along them, the ones I just described, people resolve to coding their app in the simplest way possible, probably an express server, or alike.

What if we had an express server that behaved as our app in the cloud? That's the idea with a local provider.

Implementing a Booster provider to work locally

To implement a Booster provider, you'll need to create two npm packages:

framework-provider-<name of your environment> - This package is in charge of:
- Provide the functions to store/retrieve data from your cloud.
- Transform the specific objects of your cloud into Booster ones, e.g. converting an AWS event into a Booster one.
framework-provider-<name of your environment>-infrastructure - This package is in charge of:
- Provide a deploy function that will set all the required resources in your cloud provider and upload the code correctly, as well as a nuke function that deletes everything deployed, OR
- Provide a start function that will start a server and all the appropriate processes in order to run the project in a specific environment. This one is the one that I'll be using for the local provider.

Given that I'm implementing the local provider, I just named them like:

framework-provider-local
framework-provider-local-infrastructure

To implement the local provider, I'll be using express that will act as the endpoints provided by Booster, and nedb, which is a local, filesystem implementation of a NoSQL database, with an API very similar to MongoDB. It would be the equivalent of SQLite but for NoSQL databases.

Let's start implementing the first package.

The provider interface

Booster's provider interface is a regular TypeScript interface that must have it's methods implemented, an implementation could look like this:

export const Provider = {
  events: {
    rawToEnvelopes: ...,
    forEntitySince: ...,
    latestEntitySnapshot: ...,
    store: ...,
  },
  readModels: {
    rawToEnvelopes: ...,
    fetch: ...,
    search: ...,
    store: ...,
    // ...
  },
  graphQL: {
    rawToEnvelope: ...,
    handleResult: ...,
  },
  api: {
    requestSucceeded,
    requestFailed,
  },
  // ...
}

To begin implementing the basics, let's start with rawToEnvelopes which are functions that convert from the cloud data type to the Booster one.

In the case of the local provider, the data will arrive as it is, as we are in charge of handling it with express, so the implementation is pretty simple:

export function rawEventsToEnvelopes(rawEvents: Array<unknown>): Array<EventEnvelope> {
  return rawEvents as Array<EventEnvelope>
}

export function rawReadModelEventsToEnvelopes(rawEvents: Array<unknown>): Array<ReadModelEnvelope> {
  return rawEvents as Array<ReadModelEnvelope>
}

In the case of the rawToEnvelope function for the graphQL field, we will have to get some more information from the request, like a request ID, a connection ID, or the event type, which will come in the request, to simplify things, let's ignore them:

export async function rawGraphQLRequestToEnvelope(
  request: express.Request
): Promise<GraphQLRequestEnvelope | GraphQLRequestEnvelopeError> {
  return {
    requestID: UUID.generate(),  // UUID.generate() provided by Booster
    eventType: 'MESSAGE',
    connectionID: undefined,
    value: request.body,
  }
}

With these functions implemented, we already have our endpoints connected to Booster, now we just have to teach it how to store/retrieve data!

Creating a local database

Given that we'll be using NeDB to store our Booster app data, we will need to initialize it first. We can do it in the same file as the Provider implementation:

import * as DataStore from 'nedb'
import { ReadModelEnvelope, EventEnvelope } from '@boostercloud/framework-types'

const events: DataStore<EventEnvelope> = new DataStore('events.json')
const readModels: DataStore<ReadModelEnvelope> = new DataStore('read_models.json')

NeDB uses a file for each "table", so we create two DataStores to interact with.

Now we have to implement the methods that the providers require, for example store:

async function storeEvent(event: EventEnvelope): Promise<void> {
  return new Promise((resolve, reject) => {
    events.insert(event, (err) => {
      err ? reject(err) : resolve()
    })
  })
}

async function storeReadModel(readModel: ReadModelEnvelope): Promise<void> {
  return new Promise((resolve, reject) => {
    readModels.insert(readModel, (err) => {
      err ? reject(err) : resolve()
    })
  })
}

Sadly, NeDB doesn't provide a Promise based API, and doesn't play well with promisify, so we have to wrap it manually. The implementation is pretty straightforward.

The rest of the methods are a matter of implementing the proper queries, for example:

async function readEntityLatestSnapshot(
  entityID: UUID, 
  entityTypeName: string
): Promise<EventEnvelope> {
  const queryPromise = new Promise((resolve, reject) =>
    this.events
      .find({ entityID, entityTypeName, kind: 'snapshot' })
      .sort({ createdAt: -1 }) // Sort in descending order
      .exec((err, docs) => {
        if (err) reject(err)
        else resolve(docs)
      })
  )
}

There are some other methods that can be a bit confusing, but they also act as interaction at some point, like managing HTTP responses:

async function requestSucceeded(body?: any): Promise<APIResult> {
  return {
    status: 'success',
    result: body,
  }
}

async function requestFailed(error: Error): Promise<APIResult> {
  const statusCode = httpStatusCodeFor(error)
  return {
    status: 'failure',
    code: statusCode,
    title: toClassTitle(error),
    reason: error.message,
  }
}

After implementing all the methods of the Provider, we are pretty much done with the first package, and we can hop onto the infrastructure train 🚂

Wiring everything up with an Express server

In the same case as the Provider , your Infrastructure object must conform to an interface, which in our case is a start method that initializes everything. Here we will create an express server and wire it into Booster, by calling the functions that the framework core provides.

Let's begin by initializing the express server:

export const Infrastructure = {
  start: (config: BoosterConfig, port: number): void => {
    const expressServer = express()
    const router = express.Router()
    const userProject: UserApp = require(path.join(process.cwd(), 'dist', 'index.js'))
    router.use('/graphql', graphQLRouter(userProject))
    expressServer.use(express.json())
    expressServer.use(router)
    expressServer.listen(port)
  },
}

Here we are importing user's app, in order to gain access to all the public Booster functions (typed in the UserApp type).

You can see that the only endpoint at the moment is /graphql, and that's what we are gonna configure now:

function graphQLRouter(userApp: UserApp) {
  const router = express.Router()
  this.router.post('/', async (req, res) => {
    const response = await userApp.boosterServeGraphQL(req)  // entry point
    res.status(200).json(response.result)
  })
}

And that's it, we only have to call boosterServeGraphQL on the user's app.

Because we already provided all the required methods in the Provider package, Booster has access to all the infrastructure capabilities, and it will use all of them as they need to be, no need to write more code! 🚀

That's all folks!

I'm gonna keep working on improving the local provider, like adding nice logging messages, tests, and more goodies 😉, but you can always check out the complete code in the following folders of the Booster repo:

packages/framework-provider-local
packages/framework-provider-local-infrastructure

Thanks for reading all of this! Have an awesome day,

Nick

Booster is live on Product Hunt!

Nick Tchayka — Wed, 28 Oct 2020 17:00:07 +0000

Booster has been just published to Product Hunt! Check it out here.

In case that you don't know, Booster is a free and open-source framework that allows you writing backends without thinking in servers, databases, and all that stuff that you have to think about when you write a backend. You just write your code following some patterns, write a command, and BOOM, you got a fully cloud-native backend with a GraphQL API ready to be used from your frontend. Isn't that awesome?

If you still are not sure about what's so special about Booster, check out the post Explaining the hype around Booster Framework.

If you have more questions, feel free to join our Discord server, create an issue in the GitHub repo or find tons of information on Booster's website.

See ya!

Explaining the hype around Booster Framework 🔥😈

Nick Tchayka — Wed, 21 Oct 2020 16:34:34 +0000

Let's all agree on one thing: Writing backends is hard, and we suck at it.

Yeah, sure, we're all great at creating our Express server, with our MongoDB or MySQL attached to it, but what comes after?

Are you sure that your dedicated server is ready for when your app starts being successful? Yeah, you could optimize your server, harden your Linux, remove unnecessary packages, stop services, etc. Still at some point, you'll need to scale up your server, buy a more expensive one. But what if that is needed for only that moment when a high influx of users come into your site? Do you downscale to keep your budgets down? Do you keep paying for the server? Have you even thought about adding a load balancer?

The solution is obvious: Serverless and the cloud!!!!1!one!

Sadly, this is not the reality. Why not? Because it's just more stuff to learn, and it's not like learning a new library/tool, it's a ton of info. Have you considered your cloud already? AWS, Azure, Google Cloud Platform?

Let's suppose you chose AWS, here are some more questions: Will you use DynamoDB? Or Aurora? Do you know how to query those databases? Are there any ORMs available for your language+database combination? Are you deploying all of this infrastructure manually? Or are you using something like Terraform or Serverless? By the way, how comfortable are you with YAML? 😂

Not only that, now your code gets executed in separate functions. Don't try packaging your entire Express server into that. Well, at least if you value your wallet 😅

Did you think already about the architecture for your app? Will you communicate the different parts of your app through a queue? Will you do direct calls? Are you gonna store stuff in the database directly? Or is it gonna act more as a cache? 🤔

Also, remember that you have to create a proper SDK on the frontend to interact with your backend.

OH! I nearly forgot! Are you gonna add analytics to your app? Because it'd be good if you thought about that in this moment, where you can start gathering data right from day one.

And as a final note: Good luck maintaining it. 👌

You might be one of those people who see the text above and think "And what's the issue with all of this? It's part of the journey, I'm okay spending 95% of my time learning all this stuff and maintaining it, I'm a developer".

Sorry, but in that case, you're missing a lot of good stuff.

I think that we all agree that tinkering with pointers and registers is not something from nowadays' daily development scene. Sure, there is this small percentage of people that do this in order to optimize the speed of their system, but it's not the vast majority of us. Why? Because of speed of development.

The usage of those things have been abstracted for us, so we write less code that doesn't matter, and write more of the code that matters: The one that makes our app different

Same stuff with fonts, you don't design a font each time for your application, but rather, you just choose the one that fits best for the use-case.

What if I told you that there's a way to abstract all of this hard stuff regarding to backend development, and just focus on what matters in that code that makes your app different?

Enter Booster framework

// Detect dark theme var iframe = document.getElementById('tweet-1281609074975690753-549'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1281609074975690753&theme=dark" }

Booster is this TypeScript framework distilled from years of experience working with companies like eBay, PayPal, Rent the Runway, or Zara. The folks at The Agile Monkeys got tired of reinventing the wheel constantly, and created the ultimate backend framework.

The premise is simple: Make developers' lives easier.

So, what makes Booster special?

It:

Requires no prior knowledge of backend development.
Frees you from thinking about servers.
Eliminates the need to configure your database.
Avoids having to configure endpoints.
Deploys to your favorite cloud, without having to read their docs thoroughly.
Removes any costs for your deployed app while you’re not using it.
Has built-in analytics data gathering for your app.
Is extensible with plugins in case you really want to tweak something (we call these rockets).

Also, you don't have to be thinking in architectures, Booster has thought that already, extracted from the enterprise projects above and many others.

Ok, how does one use it?

I'm not gonna get too deep into actual usage, because this is not a tutorial, but I'll show you enough to see what's going on when you work with Booster.

When you use Booster, you interact a lot with the boost CLI tool that comes with it. Booster has a tiny bit of boilerplate code, but you don't want to write even that one.

How does one configure their app? That's the magic, you don't! The only thing you have to do is write your app around the following concepts:

Commands

A command is a class with some fields that represents an order to your system, along with a handle method that will validate those fields, registering some events (we will talk about them in the next section). Examples of commands are generally the continuation of the sentence "A user can...", e.g.:

A user can deposit money
A user can send a message
A user can create a chat room

How does a command look in practice? Just TypeScript code, and the best part, is that most of it is autogenerated! With a command like boost new:command DepositMoney --fields accountId:UUID amount:number you'll get a file under src/commands
with the following contents:

@Command({
  authorize: ...
})
export class DepositMoney {
  constructor(
    readonly accountId: UUID,
    readonly amount: number
  ) {}

  public async handle(register: Register): Promise<void> {
    register.events(...)
  }
}

You would now specify who can execute this command in the second line, write your validation code inside of the handle method, and register the events. THAT'S IT! No HTTP methods, no routing endpoints, deep controllers, or whatsoever, just the important code.

Events

Same as commands, events' code is autogenerated with a command in your terminal, they are a regular TypeScript class as well.

Remember when I said that your app integrates already the data gathering for analytics? It's because of events! Instead of changing your database, performing queries, etc. You just represent that change as an event. Booster will store all these events infinitely, meaning that at any time, you can check what steps did a user do in order to trigger that bug, see if those users are dropping your registration process due to UX or a bug, think in how to optimize new features, and much more!

Examples of events are facts that you can state after a command, many of them are pretty obvious:

Deposit performed
Message sent
Room created

(Note the ed)

A newly created Booster event looks like this:

@Event
export class DepositPerformed {
  public constructor(
    readonly accountId: UUID,
    readonly amount: number
  ) {}

  public entityID(): UUID {
    return this.accountId
  }
}

Entities

Entities are the internal state of your application, but you don't modify it. Instead, Booster does something like an Array.reduce to this infinite list of events that you have, to convert them into a tangible object. Entities have many uses, but the most important one is snapshotting, so your backend is FAST.

Examples of entities are concepts in your app, stuff like:

Bank account
Conversation
ChatRoom

Here's an entity that calculates its balance from the DepositPerformed event, initializing balance to 0 in the case that the account doesn't exist yet:

@Entity
export class BankAccount {
  public constructor(
    readonly id: UUID,
    readonly balance: number
  ) {}

  @Reduces(DepositPerformed)
  public static reduceDepositPerformed(event: DepositPerformed, current?: BankAccount): BankAccount {
    const currentBalance = current?.balance ?? 0             //    ^--  Note how this is optional, because at the beginning of your app, the account doesnt exist
    return new BankAccount(
      currentBankAccount.id,
      currentBalance + event.amount
    )
  }
}

Read Models

Read models are the way you expose your entities to the public. Read models act as a cache to make sure that your app gets the data ASAP. Also, they allow you to combine different entities, and transform the data in a way that it makes sense for your frontend. Again, a regular TypeScript class:

@ReadModel({
  authorize: ...,
})
export class AccountReadModel {
  public constructor(
    public id: UUID,
    readonly balanceMessage: string,
  ) {}

  @Projects(BankAccount, 'id')
  public static projectBankAccount(entity: BankAccount, current?: AccountReadModel): AccountReadModel {
        let message = "Broke"
    if (entity.balance > 1000) {
      message = "Has sum money here, yo"
    } else if (entity.balance > 10000) {
      message = "Hey, this person is saving for sumthin'"
    } else if (entity.balance > 100000) {
      message = "Whoah, they gonna buy a house or what?"
    } else if (entity.balance > 1000000) {
      message = "They got a shit ton of money, yo!"
    }
    return new AccountReadModel(entity.id, message)
  }
}

That's it!

Now, to deploy this, you just write boost deploy -e production.

AND THAT'S IT. PERIOD.

No configuration, no control panels, no YAML, no anything! Just press one damn button, and you have your app deployed to the cloud

You get your cloud properly configured, with all security measures, architecture, interconnection of services, etc.

Booster spits out a URL which is a GraphQL endpoint for you, and that's it.

Right, but how do I connect this to my frontend?

Because it is GraphQL, you can perfectly use a library like Apollo in order to connect it.

Using React? Vue? Svelte? Elm? No problem, just use the Apollo adapter for one of those. No need to write SDKs/services for talking to the backend.

Commands can be executed through Mutations, Read Models can be queried or subscribed to. That's it, no need to encode URLs, remember parameter orders, create API documentation with Swagger, etc. Just use what you coded.

This is too good to be true, it must cost a lot, right?

Nope, the framework is absolutely free to use and open source. If you are worried about the costs of the cloud, all Booster projects
are eligible for the AWS free tier, and even without it, for 40.000 (that's forty thousand, btw) requests in one month, you'd pay less than one dollar.

(No, the team doesn't get any cut from cloud providers, we are just fed up with tools that create problems instead of solving them 😉)

Ok, what to do now?

If you reached so far, congratulations, and thank you! Nowadays frameworks appear every day, and people just roll their eyes. Booster is truly different and groundbreaking. For learning more, checkout the project's website, where you can find links to documentation, demos, and much more.

I also recommend you to join the project's Discord server and challenge it with your ideas, thoughts, and questions. A project is nothing if it stays in the same place, and by challenging it, you are already helping!

Booster is also part of Hacktoberfest, in case you want to contribute wink wink; nudge nudge. And even if not, a star in our GitHub repo would mean a lot.

Stay tuned for more!

May your day be awesome, and your backends easy 😉,
Nick