Forem: Carl Barrdahl

Journey into Web Monetization - mixtapes

Carl Barrdahl — Sat, 23 May 2020 12:43:41 +0000

I decided to explore the idea about monetizing mixtapes I mentioned in the first post.

To make the proof of concept as simple as possible I use SoundCloud to stream the audio. It has a nice embeddable widget as well as an API with events to detect pause and play progress.

It works like this:

A SoundCloud widget is embedded
A tracklist is defined to keep track of which payment pointer should be used at what time
Get track playing and update payment pointer whenever the event PLAY_PROGRESS is triggered.
Coil is used to monetize the payment pointer in the meta tag

// Initialize SoundCloud widget
const audio = SC.Widget(embedSoundCloud(trackId))

// Fetch tracklist
const tracklist = await fetch("tracklist.json").then(r => r.json())

// Update payment pointer in meta tag as audio is playing
audio.bind(SC.Widget.Events.PLAY_PROGRESS, ({ currentPosition }) => {
  const track = getTrack(currentPosition, tracklist);      
  setMetaTag(track.paymentPointer);
})

// Pause monetization
audio.bind(SC.Widget.Events.PAUSE, (args) => setMetaTag(""));


function setMetaTag(pointer) {
  const monetizationTag = document.querySelector('meta[name="monetization"]');
  if (monetizationTag.content === pointer) {
      return;
  }
  monetizationTag.content = pointer;
}

function getTrack(timestamp, tracklist) {
  const tracks = [...tracklist].reverse() // find last index
  const trackIndex = tracks.findIndex((t) => timestamp / 1000 > t.time)

  return tracks[trackIndex]
}

Some issues that would need to be managed:

Each artist would need to set up a wallet and a payment pointer
Tracklists need to be defined (I crawled from 1001tracklists.com)
Each artist in tracklist need to point to a registered payment pointer
Metatag only supports 1 (artist) payment pointer
Nothing prevents a user to listen if they don't monetize
Monetization stops when browser tab is in background

Future improvements could be:

Move payments to a backend service and stream song position using WebSockets. This could enable payouts to multiple artists (collaborations) and songs playing at the same time (during crossfades and mashups).
Let artists sign up for a wallet and payment pointer
Periodically check if payments are sent to let the user keep listening

You can check it out here:
https://traveling-island-marscapone.glitch.me

And the code:
https://glitch.com/edit/#!/traveling-island-marscapone

Some thoughts on WebMonetization and ILP in general:

Is a browser extension really needed? Why can't the user authorize a backend to send payments to a pointer?
Could we have something like an OAuth2 flow where we allow a site to spend x amount from our wallet?

Journey into Web Monetization - day 4

Carl Barrdahl — Wed, 13 May 2020 14:27:02 +0000

Yesterday I looked at setting up sending payments in the Interledger network. Still not fully grasping how everything works so did some more research.

Here's what I came up with:

ILP Connector (`ilp-connector`)

Also called Interledger Service Provider (ILSP) and routes packets from connected servers
Connectors work as decentralized exchanges
Configured with assetCode (XRP, ETH, BTC, eth) and uses plugins to transact with ledgers
This is what moneyd uses to connect to the network

ILP Stream (`ilp-protocol-stream`)

Creates a server with a connection to the Interledger network
Plugin configures which ILP Connector to connect to
Generates credentials for clients to establish stream and send payments to wallet

ILP Plugin (`ilp-plugin-*`)

Plugins handle the logic of how to perform the transactions by extending sendMoney and _handleMoney
You can write your own plugin to extend these and add custom behaviour to interact with other ledgers
For example ilp-plugin-ethereum uses ethers.io to manage an Ethereum Wallet and connecting to the ledger
Connectors, servers and clients can use these plugins to establish link between them

So basically, Controllers are the routers doing the actual transactions, ILP Streams are receiver (with createServer) and senders (with createConnection). Plugins are used to define how transactions are stored in ledgers.

Here are some ideas of code examples that would be helpful

Self-hosted multi-wallet to allow users to register and create payment pointers
Wallet with Stripe integration to handle subscription and payouts
Browser extension to establish stream to payment pointer in meta tag
Simple plugin to show integration with other libraries
Something to show transaction statistics

Journey into Web Monetization - day 3

Carl Barrdahl — Tue, 12 May 2020 10:58:18 +0000

After the first day of idea generation and the second day of learning, I thought I'd get into the more practial side of it and try to get something up to play around with.

Here's what I did:

Connect to a Interledger testnet with moneyd
Create an ILP stream server to connect to the moneyd instance.
Generate stream credentials and send money

Run npx moneyd local to start the moneyd instance with a local testnet.

Next, set up a simple node project:

npm init -y
npm i -S ilp-protocol-stream ilp-plugin @interledger/connection-tag-utils
mkdir src
touch src/index.js

// src/index.js
const { randomBytes } = require("crypto");
const { createServer, createConnection } = require("ilp-protocol-stream");
const { encode, decode } = require("@interledger/connection-tag-utils");
const createPlugin = require("ilp-plugin");

// Used for encoding and decoding connectionTag
const SECRET_KEY = randomBytes(32);

async function createStreamServer() {
  const server = await createServer({ plugin: createPlugin() });
  server.on("connection", (connection) => {
    // connectionTag contains encrypted data, in this case about the user account
    const { account } = JSON.parse(
      decode(SECRET_KEY, connection.connectionTag)
    );
    connection.on("stream", (stream) => {
      stream.setReceiveMax(10000);
      stream.on("money", (amount) => {
        console.log(`Received payment from ${account} for: ${amount}`);
      });
    });
  });

  return server;
}

async function sendMoney({ amount, destinationAccount, sharedSecret }) {
  const connection = await createConnection({
    plugin: createPlugin(),
    destinationAccount,
    sharedSecret,
  });

  const stream = connection.createStream();
  await stream.sendTotal(amount);
  await connection.end();
}

async function run() {
  const server = await createStreamServer();

  // Encode the user account here so we can use when we receive money
  const data = encode(SECRET_KEY, JSON.stringify({ account: "userAccountId" }));

  const { destinationAccount, sharedSecret } = server.generateAddressAndSecret(
    data
  );

  const amount = 10000;
  await sendMoney({ amount, destinationAccount, sharedSecret });
  await server.close();
}

run().catch((err) => console.log(err));

And finally run it:

npx nodemon src/index.js

Received payment from userAccountId for: 10000

Plugins such as ilp-plugin determines where the server connects. It looks for env variable ILP_BTP_SERVER and defaults to btp+ws://localhost:7768 if not found. This is the local moneyd instance.
Data such as information about user or invoice can be encoded in the connectionTag. This will allow us to update databases for example.
To create a stream client and send money we need: destinationAccount and sharedSecret which we can get from a server.

Possible next steps

Connect to a non-local testnet
Send money between other wallets
Build a REST server and handle different users and payment pointers
Build a browser extension to handle the functionality currently in sendMoney

Journey into Web Monetization - day 2

Carl Barrdahl — Mon, 11 May 2020 10:15:18 +0000

After yesterdays ideas I couldn't decide which one to pursue. I realized I needed a better and deeper understanding of how it works.

Here's what I found out:

WebMonetization (WM) is both push- and pull-based payments. So instead of you push payment from your credit-card you can have websites pull money from your wallet.
Users need wallets. Right now there are only a few options: Coil, Xpring, Stronghold, GateHub and XrpTipbot. No native browser support, only Coil extension.
Payment pointers are urls to wallets ($ instead of https://)
Privacy focused not to leak personal data
OpenPayments is a specification similar to OAuth2 in how wallets should work (API endpoints, .well-know/open-payments manifest etc).

{
  "issuer": "https://wallet.example",
  "authorization_issuer": "https://auth.wallet.example",
  "authorization_endpoint": "https://auth.wallet.example/authorize",
  "invoices_endpoint": "https://wallet.example/invoices",
  "mandates_endpoint": "https://wallet.example/mandates",
  "assets_supported": [
    {"code": "USD", "scale": 2},
    {"code": "EUR", "scale": 2}
  ]
}

When a wallet receive a request from a client (GET $wallet.example/alice) it connects to the ILP network and returns STREAM credentials:

{
  "ilpAddress": "example.ilpdemo.red.bob",
  "sharedSecret": "6jR5iNIVRvqeasJeCty6C+YB5X9FhSOUPCL/5nha5Vs="
}

These credentials are used to start the STREAM (Interledger transport protocol)
"Similar to how the Internet routes packets information, Interledger routes packets of value"
Other names for wallets (SPSP server/client, WM receiver/sender (or provider), acquirer/issuer)
You can set sendMax on clients and receiveMax on servers to end the stream and connection when these are reached.
Right now the eco-system is limited to XRP and USD.
Connectors can be (has been?) built to connect to other ledgers (ETH, BTC etc)

I found these links helpful:

It's quite a lot to take in.

As a developer it's quite difficult to know where to get started.
As a user it should be easier for me to set up a new wallet and use in my browsers
As a creator it should be easy to publish new content with payment pointers, both on established platforms as well as self-hosted

How could one help reduce the friction for users to start using these technologies?

How to make it easier for content creators to create a wallet and get a payment pointer?
How to make it easier for developers to start building with these technologies?
How to make it easier for users to access paid content?

Journey into Web Monetization - day 1

Carl Barrdahl — Sun, 10 May 2020 05:16:32 +0000

I was excited to learn about this new web monetization api and what impact it can have on the future of the web.

My basic understanding of it:

Creators can set up payment pointers (basically a url to a wallet) and receive streaming payments for access to their content.
This payment pointer is added to a meta tag in an html document.
Payment pointers can also be set up in a more dynamic way on the servers.

Pretty quickly after skimming through the documentation, I came up with four potential ideas:

1. Monetize mixtapes

I'm a big fan of mixtapes from artists and labels and find it a great way of discovering new music.

The idea would be to build a stream server which switches the payout depending on timestamps mapped to a table of payment pointers to artists.

This would enable the artist to receive payment exactly for the time their music is being played.

2. Newsletters or articles

Many writers today rely on services such as Medium or Substack to reach their audience. I had this idea of building tools to enable writers to earn money in a more indenpendent way hosting their own content.

Coil describes three strategies of monetizing:

free sample and pay for the rest
some free and some paid
everything free and pay for bonus content

3. SaaS for developers to monetize their apps

I got this idea of creating an sdk that developers could easily add as middleware for their api routes. Instead of the usual subscription-based model, the users would only pay when they use the apps. This could include a granular control of which api's are accessible for free. Another possibility would be to monetize external API's for other apps to use.

4. Photos on IPFS

An embeddable widget would enable photographers or other visual artists to embed their content in their own as well as publishers' websites. They would then receive payment when users view their content (or high-res versions)

As I kept thinking on these ideas I noticed obvious problems with all of them:
Payment streams turn off when webpage goes in the background making music content difficult. Besides, getting users to give up Spotify or other established music services might be challenging.

For newsletters and written content, what would the proof of concept be?
What would happen if readers go offline to read, turning off the payment stream.

I guess it depends on how you look at the monetizing. Is payment mandatory or optional?

Building a REST api using fastify and TypeORM

Carl Barrdahl — Tue, 22 Oct 2019 12:56:15 +0000

A restaurant wants to be able to digitally manage their inventory to more easily keep track of when products expire and work in a more data-driven approach. I got the chance to build a prototype in React Native and Typescript.

This is how I created the backend api using fastify and TypeORM.

You can find an example project on Github: https://github.com/carlbarrdahl/fastify-server-example

Requirements

Inventory should be stored in an MSSQL database
REST api to communicate with database
Only authorized users should be able to access the api

What we will cover

Building a REST api using fastify
Integration testing
Database connection with TypeORM
JSON Schema for client data validation and defining allowed responses
Securing endpoints using JWT
Automatically generated documentation in Swagger

REST api in fastify

I decided to write the api using fastify as a server framework because it's fast, modular as well as easy to use and test. It also has a great ecosystem for its plugin-system and you can easily write your own plugins as we will see later.

A good way to make sure the api behaves as expected is to write integration tests. By developing against a test suite we get a fast feedback loop and don't need to go through the process of manually calling the api to check if it works as expected.

I started by specing out the expected behaviour:



test("GET /products returns list of products", () => {})
test("DELETE /products/:id deletes a product", () => {})
test("GET /inventory returns list of inventory", () => {})
test("POST /inventory/:id creates a product", () => {})
test("DELETE /inventory/:id deletes an inventory", () => {})
test("JWT token is required for endpoints", () => {})

To test endpoints in fastify we can use inject to simulate a request to the server and pass method, url, headers and payload and then make sure the response is what we expect.



// test/server.test.ts
import createServer from "../src/server"
const server = createServer()

test("GET /inventory returns list of inventory", async done => {
  server.inject({ method: "GET", url: `/inventory` }, (err, res) => {
    expect(res.statusCode).toBe(200)
    expect(JSON.parse(res.payload)).toEqual([]) // expect it to be empty for now
    done()
  })
})

By using fastify's plugin system we can make the app modular so we more easily can split into smaller pieces if needed. I opted to go with the following folder structure:



/src
  /modules
    /health
      /routes.ts
      /schema.ts
    /product
      /entity.ts
      /routes.ts
      /schema.ts
    /inventory
      /entity.ts
      /routes.ts
      /schema.ts
  /plugins
    /auth.ts
    /jwt.ts
    /printer.ts
  /server.ts
  /index.ts
/test
  /server.test.ts

Here's what the inventory routes might look like:



// src/modules/inventory/routes.ts
module.exports = (server, options, next) => {
  server.get(
    "/inventory",
    // we will cover schema and authentication later
    { preValidation: [server.authenticate], schema: listInventorySchema },
    async (req, res) => {
      req.log.info(`list inventory from db`)
      const inventory = [] // return empty array for now to make the test green

      res.send(inventory)
    }
  )
  // routes and controllers for create, delete etc.
  next()
}

Our test should now be green, that's a good sign!

However, an inventory api that always return an empty array is not very useful. Let's connect a data source!

Connecting to a database with TypeORM

What is an ORM you might ask? Most databases have different ways of communicating with them. An ORM normalizes this into a unified way so we can easily switch between different types of supported databases without having to change implementation.

First let's create the entity (or model):



// src/modules/inventory/entity.ts
@Entity()
export class Inventory {
  @PrimaryGeneratedColumn("uuid")
  id: string

  // Each product can exist in multiple inventory
  @ManyToOne(type => Product, { cascade: true })
  @JoinColumn()
  product: Product

  @Column()
  quantity: number

  @Column("date")
  expiry_date: string

  @CreateDateColumn()
  created_at: string

  @UpdateDateColumn()
  updated_at: string
}

Next, we'll use the plugin to connect to the database and create a decorator with our data repositories. That way they are easily accessible from our routes.



// src/plugins/db.ts
import "reflect-metadata"
import fp from "fastify-plugin"
import { createConnection, getConnectionOptions } from "typeorm"
import { Inventory } from "../modules/inventory/entity"

module.exports = fp(async server => {
  try {
    // getConnectionOptions will read from ormconfig.js (or .env if that is prefered)
    const connectionOptions = await getConnectionOptions()
    Object.assign(connectionOptions, {
      options: { encrypt: true },
      synchronize: true,
      entities: [Inventory, Product]
    })
    const connection = await createConnection(connectionOptions)

    // this object will be accessible from any fastify server instance
    server.decorate("db", {
      inventory: connection.getRepository(Inventory),
      products: connection.getRepository(Product)
    })
  } catch (error) {
    console.log(error)
  }
})



// ormconfig.js
module.exports = {
  type: "mssql",
  port: 1433,
  host: "<project-name>.database.windows.net",
  username: "<username>",
  password: "<password>",
  database: "<db-name>",
  logging: false
}

We can now add the plugin to createServer and update our route to query the database:



// src/server.ts
server.use(require("./plugins/db"))

// src/modules/inventory/routes.ts
const inventory = await server.db.inventory.find({
  relations: ["product"] // populate the product data in the response
})

Unless we want our tests to query our production database we have to either setup an in-memory test-db or just mock it. Let's create a mock in our test:



// test/server.test.ts
import typeorm = require('typeorm')

const mockProducts = [{...}]
const mockInventory = [{...}]
const dbMock = {
  Product: {
    find: jest.fn().mockReturnValue(mockProducts),
    findOne: jest.fn().mockReturnValue(mockProducts[1]),
    remove: jest.fn()
  },
  Inventory: {
    find: jest.fn().mockReturnValue(mockInventory),
    findOne: jest.fn().mockReturnValue(mockInventory[1]),
    save: jest.fn().mockReturnValue(mockInventory[0]),
    remove: jest.fn()
  }
}
typeorm.createConnection = jest.fn().mockReturnValue({
  getRepository: model => dbMock[model.name]
})
typeorm.getConnectionOptions = jest.fn().mockReturnValue({})

Here's how the test will look for the create inventory route:



test("POST /inventory/:id creates an inventory", done => {
  const body = { product_id: mockProducts[0].id, quantity: 1 }
  server.inject(
    {
      method: "POST",
      url: `/inventory`,
      payload: body,
      headers: {
        Authorization: `Bearer ${token}`
      }
    },
    (err, res) => {
      expect(res.statusCode).toBe(201)
      // assert that the database methods have been called
      expect(dbMock.Product.findOne).toHaveBeenCalledWith(body.product_id)
      expect(dbMock.Inventory.save).toHaveBeenCalled()
      // assert we get the inventory back
      expect(JSON.parse(res.payload)).toEqual(mockInventory[0])
      done(err)
    }
  )
})

How do we know the correct data is being sent when creating an inventory?

Validating requests with JSON schema

Another great thing about fastify is it comes with built in schema validation using json-schema specification.

Why is this important?

We can never know what data a client sends and we don't want to have to manually check the request body in every route. Instead we want to describe what such requests might look like and what kind of responses can be expected. If what the client send doesn't match the schema, fastify will automatically throw an error. This leads to clean, understandable code without cluttering it with unnecessary if statements.

Note: This will not protect against potential xss. A bad actor is still able to send malicious javascript code. We can use fastify-helmet's xssFilter to protect us against this attack.

In addition to validation, we can automatically generate Swagger documentation for our routes based on these specifications so developers know how to use the api. Neat!

These json schemas are defined as simple objects. Here are the schemas for the inventory routes:



const inventorySchema = {
  id: { type: "string", format: "uuid" },
  product_id: { type: "string", format: "uuid" },
  // note the reference to the productSchema ↘
  product: { type: "object", properties: productSchema },
  quantity: { type: "number", min: 1 },
  expiry_date: { type: "string", format: "date-time" },
  created_at: { type: "string", format: "date-time" },
  updated_at: { type: "string", format: "date-time" }
}
export const listInventorySchema = {
  summary: "list inventory",
  response: {
    200: {
      type: "array",
      items: {
        properties: inventorySchema
      }
    }
  }
}
export const postInventorySchema = {
  summary: "create inventory",
  body: {
    // incoming request body
    type: "object",
    required: ["product_id", "quantity"],
    properties: {
      product_id: { type: "string", format: "uuid" },
      quantity: { type: "integer", minimum: 1 }
    }
  },
  response: {
    201: {
      type: "object",
      properties: inventorySchema
    }
  }
}

Fastify will now be very picky about the data it receives and will tell us if something is missing or of incorrect type.

Still, anyone is able to access our API. Next we'll look at how we can restrict this to clients with a valid key using json web token.

Authorization

To secure our api we will use json web token.
https://jwt.io/introduction/

This is what JWT.io has to say:

Authorization: This is the most common scenario for using JWT. Once the user is logged in, each subsequent request will include the JWT, allowing the user to access routes, services, and resources that are permitted with that token. Single Sign On is a feature that widely uses JWT nowadays, because of its small overhead and its ability to be easily used across different domains.

Information Exchange: JSON Web Tokens are a good way of securely transmitting information between parties. Because JWTs can be signed—for example, using public/private key pairs—you can be sure the senders are who they say they are. Additionally, as the signature is calculated using the header and the payload, you can also verify that the content hasn't been tampered with.

This means we can both use it to verify a user is who they say they are and share secret data in a secure way. In our case we'll use it to simply authorize a shared user.

We'll use fastify plugin to import the library and decorate authenticate with a request handler that will verify our token.



// src/plugins/auth.ts
import fp from "fastify-plugin"

export default fp((server, opts, next) => {
  server.register(require("fastify-jwt"), {
    secret: "change this to something secret"
  })
  server.decorate("authenticate", async (req, res) => {
    try {
      await req.jwtVerify()
    } catch (err) {
      res.send(err)
    }
  })

  next()
})

We then run authenticate in the preValidation hook on every request to make sure the jwt is valid.

Internally it retreives the token passed in Authorization header and verifies it has been signed with our secret key.



// src/modules/inventory/routes.ts
server.post(
  "/inventory",
  // authenticate the request before we do anything else
  { preValidation: [server.authenticate], schema: postInventorySchema },
  async (req, res) => {
    const { quantity, product_id } = req.body
    req.log.info(`find product ${product_id} from db`)
    const product = await server.db.products.findOne(product_id)

    if (!product) {
      req.log.info(`product not found: ${product_id}`)
      return res.code(404).send("product not found")
    }

    req.log.info(`save inventory to db`)
    const inventory = await server.db.inventory.save({
      quantity,
      product,
      expiry_date: addDays(product.expires_in)
    })

    res.code(201).send(inventory)
  }
)

Since we don't have any user accounts implemented right now we can generate a temporary token like this:



server.ready(() => {
  const token = server.jwt.sign({ user_id: "<user_id>" })
  console.log(token)
})

As you may have noticed the token is the signed object (with some other stuff) encoded as a Base64 string. We can use this to limit access to specific users or inventory created by a user. Maybe something like this:



// src/modules/inventory/routes.ts
server.get(
  "/inventory/:id",
  { schema: getInventorySchema, preValidation: [server.authenticate] },
  async (req, res) => {
    const inventory = await server.db.inventory.findOne(req.params.id)
    // Make sure the requesting user is the same as the inventory owner
    if (req.user.user_id !== inventory.owner.id) {
      throw new Error("Unauthorized access")
    }
    res.send(inventory)
  }
)

More advanced usage can check the timestamp for when the token has been issued (iat).

Swagger documentation

What's this Swagger documentation I've been going on about? Basically it provides a visual interface for your api allowing you to see how it works, what the request bodies should look like and example responses. Pretty much what we defined in our json schema exposed as documentation.

This is the configuration used in createServer:



server.register(require("fastify-oas"), {

  routePrefix: "/docs",

  exposeRoute: true,

  swagger: {

    info: {

      title: "inventory api",

      description: "api documentation",

      version: "0.1.0"

    },

    servers: [

      { url: "http://localhost:3000", description: "development" },

      { url: "https://<production-url>", description: "production" }

    ],

    schemes: ["http"],

    consumes: ["application/json"],

    produces: ["application/json"],

    security: [{ bearerAuth: [] }],

    securityDefinitions: {

      bearerAuth: {

        type: "http",

        scheme: "bearer",

        bearerFormat: "JWT"

      }

    }

  }

})

Future improvements

User accounts
Caching
Improved error handling
Improved security against XSS and more using fastify-helmet
Load balancing

What did you think of this article?

Did you learn something new? Was something hard to understand? Too much code? Not enough? Am I doing things totally wrong? Tell me in the comments.

Injecting components into re-usable modules with React context?

Carl Barrdahl — Sat, 19 Oct 2019 16:20:28 +0000

Here's an idea I've been pondering for awhile:

What if we passed components through React's context API instead of importing them at the top of each component?

Component libraries are great, however, we still have to piece together the components into modules. This way we could "inject" components making the modules easier to style.

I imagine it looking something like this:

// Shared module for authentication flows
import { useComponents } from "react-use-components"

export const LoginModule = ({ onLogin }) => {
  // Get the components from `React.useContext`
  const { Button, Form, Input, Label } = useComponents()

  const [state, setState] = useState({ email: "" })
  const [error, setError] = useState(null)

  async function handleSubmit(e) {
    /* call api, handle token, errors etc */
  }
  return (
    <Form onSubmit={handleSubmit}>
      <Label htmlFor="email">
        <Input id="email" value={state.email} />
      </Label>
      <Button type="submit">Login</Button>
    </Form>
  )
}

And then set the components to the context in your app:

// App.js
import ComponentProvider from "react-use-components"
import { LoginModule, SignupModule } from "standard-auth-module"

import * as Components from "my-awesome-components"

export default ({ navigate }) => (
  <ComponentProvider components={Components}>
    <LoginModule request={api.login} onLogin={user => navigate("/dashboard")} />
  </ComponentProvider>
)

Here's how the library would look like:


import React, { createContext, useContext } from "react"

const ComponentContext = createContext()

export const useComponents = () => useContext(ComponentContext)

export const ComponentProvider = ({ components: value, ...props }) =>
  React.createElement(ComponentContext.Provider, { value, ...props })

export default ComponentProvider

The point of this would be to build re-usable modules much like an higher abstraction over component libraries. Component libraries would be used for styling and modules for behaviour.

This comes with some obvious challenges like:

How do we control layout?
How do we make sure components in the context have the required prop-types? Limit to single element pattern?
What about performance?
Translations?

What are your thoughts? What kind of re-usable components could be built?

Resources

Demo (https://github.com/carlbarrdahl/react-use-components)
Single Element Pattern (https://github.com/diegohaz/singel)

Generating pdf reports with charts using React and Puppeteer

Carl Barrdahl — Fri, 18 Oct 2019 12:00:36 +0000

In September 2019 I was contacted by a company to build a report api. This company is building a product to measure well-being and levels of stress in organizations by sending out surveys to employees.

Some of the company's clients requested a feature to generate pdf reports based on these surveys. Each survey includes a number of questions with data over multiple periods which will be displayed in charts. Chart data can be displayed in two ways: survey data for periods over time and a summary of all periods.

I was given pretty much free reigns in how to solve the problem, here's how I did it.

Note: The code snippets are trimmed for brevity and does not represent real data or architecture

Requirements

API endpoint should be available on Azure cloud
Endpoint should receive survey data and a template name
Return pdf document with questions asked along with charts of responses

Working with pdf files and generating them dynamically on the server can be challenging. There are libraries like PDFKit (https://pdfkit.org) you can use but you would have to specifically tell it what to draw and where, much like the canvas api:

const pdfKitDoc = new PDFDocument()
questions.forEach((question, i) => {
  pdfKitDoc
    // what's the height of each question?
    .text(question.name, 0, i * ?)
    // draw charts and calculate these x and y values somehow
    .moveTo(100, 150)
    .lineTo(100, 250)
    .lineTo(200, 250)
    .fill('#FF3300')
})

This is not a fun way to build charts.

Instead, I opted to use React as a templating engine to render static html. Using React, it's easy to make changes to styling such as margins, paddings, texts etc and we don't have to worry about positioning and flowing of text. We also get the benefit of the huge ecosystem which includes fantastic libraries for building charts.

Templates can now look like this:

const Template = ({ questions }) => (
  <Layout>
    {questions.map(question => {
      const { type, data } = question.chart
      return (
        <Question key={question.id}>
          <QuestionHeader title={question.name} />
          <Chart type={type} data={data} />
        </Question>
      )
    })}
  </Layout>
)

One limitation is we cannot use canvas to draw the charts since it's dependent on a DOM environment running javascript. We only get to render static html with this approach. Luckily Nivo (https://nivo.rocks) provides beautiful charts with SVG support.

Note: It might be possible to use something like jsdom to work around this but that seems unnecessarily complex. Besides, SVG is great for drawing charts.

To render these templates we use React.renderToStaticMarkup:

export function renderTemplate({ data, language, title }) {
  return ReactDOMServer.renderToStaticMarkup(
    React.createElement(Template, { data, language, title })
  )
}

We now need to convert this html page into a pdf file. For this we can use Google Puppeteer.

Generating pdf with Puppeteer

Puppeteer is a headless Chrome browser which can be told to visit sites and get data from the DOM, commonly used as a scraper or running end-to-end tests. It can also be used to create pdf files.

It works like this:

import puppeteer from 'puppeteer'

export async function renderPDF(html: string) {
  const browser = await puppeteer.launch({
    args: ['--no-sandbox', '--disable-setuid-sandbox']
  })
  const page = await browser.newPage()
  // pass the html string as data text/html so we don't have to visit a url
  await page.goto(`data text/html,${html}`, { waitUntil: 'networkidle0' })
  const pdf = await page.pdf({ format: 'A4' })
  await browser.close()
  return pdf
}

Sometimes (quite often), things don't go as smoothly as expected. Turns out Google Puppeteer has a bug causing an empty pdf to be rendered if any hex colors are used in SVG. To solve this I replaced all occurences of hex colors with rgb values in the html using a regex.

// https://github.com/sindresorhus/hex-rgb
import hexRgb from 'hex-rgb'
export function hexToRgb(str: string) {
  const hexTest = /#[a-f\d]{3,6}/gim
  return str.replace(hexTest, hexColor => {
    const { red, green, blue } = hexRgb(hexColor)
    return `rgb(${red}, ${green}, ${blue})`
  })
}

Mapping data

Each question can be configured to accept different types of answers. These types are:

binary for yes/no
single choice
multi choice
range choice
text for comments

These types need to be represented differently in the report both in terms of chart type but also depending on the template if it should show data over a period of time or an aggregated summary.

// Questions have different answer types and should use different types of charts depending on template
const chartMappers = {
  scale: {
    summary: (responses) => createGroupedBar(responses),
    periodic: (responses) => createPeriodicLine(responses)
  },
  single: {...},
  multi: {...},
  scale: {...},
  text: {...}
}
const templateMappers = {
  summary: periods => mergePeriods(periods),
  periodic: periods => flattenPeriods(periods)
}
export function mapSurveyToCharts({ survey, template }) {
  return {
    questions: survey.questions.map(question => {
      const responses = tempateMappers[template](question.periods)
      const chart = chartMappers[question.Type][template](responses)
      return {
        name: question.Title,
        chart: chart
      }
    })
  }
}

Wrapping it up

We now have all the pieces we need and just have to put everything together:

export async function generateReport({ survey, template, language = 'en_US' }) {
  const data = mapSurveyToCharts({ survey, template })
  const html = renderTemplate({ data, language })
  /*
    Puppeteer is having issues with rendering SVGs with hex colors. Replace all with rgb(R, G, B).
    https://github.com/GoogleChrome/puppeteer/issues/2556
    */
  const replacedHTML = hexToRgb(html)
  const pdf = await renderPDF(replacedHTML)

  return pdf
}

Have you solved this in another way? Something that doesn't make sense? Would love to hear your thoughts and feedback!

Forem: Carl Barrdahl

Journey into Web Monetization - mixtapes

Journey into Web Monetization - day 4

ILP Connector (ilp-connector)

ILP Stream (ilp-protocol-stream)

ILP Plugin (ilp-plugin-*)

Journey into Web Monetization - day 3

Possible next steps

Journey into Web Monetization - day 2

Journey into Web Monetization - day 1

1. Monetize mixtapes

2. Newsletters or articles

3. SaaS for developers to monetize their apps

4. Photos on IPFS

Building a REST api using fastify and TypeORM

Requirements

What we will cover

REST api in fastify

Connecting to a database with TypeORM

Validating requests with JSON schema

Authorization

Swagger documentation

Future improvements

Injecting components into re-usable modules with React context?

Resources

Generating pdf reports with charts using React and Puppeteer

Requirements

Generating pdf with Puppeteer

Mapping data

Wrapping it up

ILP Connector (`ilp-connector`)

ILP Stream (`ilp-protocol-stream`)

ILP Plugin (`ilp-plugin-*`)