Forem: wwayne

Using SAM to deploy an node.js lambda to consume Kinesis stream

wwayne — Sat, 05 Nov 2022 09:54:54 +0000

If you only need inline code for the lambda, the cloudformation should be enough. But if you need to do npm install for your lambda, using SAM would be more suitable and I will show you the steps in the next.

Prerequisite

Install SAM CLI (SAM official install page)
Create a S3 bucket first for storing the lambda

General Plan

Write a sam.yaml file describing a lambda to consume a Kinesis stream
Let the sam.yaml able to invoke a Makefile command to do npm install
Use sam build to do the preparation works before deployment
Use same deploy to upload lambda function into the S3 and setup all the other resources that needed

Details

- sam/
  - sam.yaml
- src/
  - index.js
  - package.json
- Makefile

Step 1: writing saml.yaml

Setup a new lambda
Let the lambda to consume the Kinese stream (SAM will help you to handle role and policy issues automatically)
Tell sam build to make use of makefile for the building

sam/sam.yaml

Transform: 'AWS::Serverless-2016-10-31'

Resources:
  MyLambdaName:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: {YOUR_LAMBDA_FUNCTION_NAME}
      Description: {YOUR_LAMBDA_DESCRIPTION}
      Handler: index.handler
      Runtime: nodejs14.x
      CodeUri: ../  # Because Makefile in '../'
      MemorySize: 128
      Timeout: 10
      Events:
        KinesisStream:
          Type: Kinesis
          Properties:
            BatchSize: 1 # Modify this for batch consuming
            Enabled: true
            Stream: {YOUR_KINESIS_STREAM_ARN}
            MaximumBatchingWindowInSeconds: 0
            StartingPosition: LATEST
    Metadata:
      BuildMethod: makefile

Step 2: writing Makefile

Because in the sam file, we named our lambda resource as MyLambdaName, so we add a command named build-MyLambdaName in the makefile and it will be invoked automatically by running sam build
What the make command do is coping the source file into the tmp folder which created by the sam build and running npm install inside the tmp folder to generate the node_modules

Makefile

build-MyLambdaName:
  @echo "Packaging the source file..."
  @cp -R src/* $(ARTIFACTS_DIR)
  @npm --prefix $(ARTIFACTS_DIR) i --production

Step 3: running `sam build`

My sam file is in the sam/ folder, so I need to specify the file position with the parameter -t
After running the build command, a new folder .aws-sam/ will be created, sam deploy will auto-search and use this folder (don't forget put .aws-sam into the .gitignore)

sam build -t sam/sam.yaml

Step 4: running `sam deploy`

sam deploy will make use of the .aws_sam folder which created by the sam build
Deployment includes uploading lambda function into S3 bucket and setup all the resources in the sam.yaml

sam deploy 
    --region {AWS_REGION} \
    --stack-name {STACK_NAME_IN_CLOUDFORMATION} \
    --s3-bucket {S3_BUCKET_NAME} \
    --s3-prefix {S3 FOLDER NAME IN THE BUCKET}

End

Later when you do any modification to the lambda, you only need to re-run sam build and same deploy.

It will create and upload a new file into your S3 and update lambda function to use the latest file.

I create a tool for helping me generate the API document from test cases

wwayne — Wed, 05 Oct 2022 03:46:31 +0000

Writing an API document is something I hate, and writing an OpenAPI document from scratch is a nightmare.

I checked some auto-api-generation tools and they all need me to modify my codes by adding comments or decorators. I personally don't like it because I wanna keep my code as clean as possible.

Then I was thinking if we could generate an API document based on the unit test because usually the unit test covers all cases of the API, we just need a program to understand the test cases and generate the API document.

So I build Outdoc and here is an example for an express application:

app.js

const express = require('express');
const app = express();

app.get('/projects', (req, res) => {
  res.json([{
    id: '1',
    name: 'project v1'
  }, {
    id: '2',
    name: 'project v2'
  }]);
});

app.test.js

const request = require('supertest');
const app = require('./app.js');

describe('api testing', () => {
  it('should able to find all', (done) => {
    request(app)
      .get('/projects')
      .set('x-api-key', 'outdoc-test')
      .expect(200)
      .end(function(err, res) {
        if (err) throw err;
        done();
      });
  });
});

For using Outdoc, we simply add a few codes into app.js and run the script.

app.js

const express = require('express');
const app = express();

// New added for using Outdoc
if (process.env.NODE_ENV === "test") {
  const { OutDoc } = require('outdoc');
  OutDoc.init();
}

package.json

{
  "scripts": {
    "test": "mocha *.test.js",
    "gen-doc": "outdoc npm test"
}

By running npm run gen-doc, we get an api.yaml as the following

openapi: "3.0.1"
info:
  title: "API Document"
  version: "1.0.0"
paths:
  /projects:
    get:
      parameters:
        - in: "header"
          name: "x-api-key"
          example: "outdoc-test"
          schema:
            type: "string"
      responses:
        200:
          description: "OK"
          content:
            application/json:
              schema:
                type: "array"
                items:
                  type: "object"
                  properties:
                    id:
                      type: "string"
                      example: "1"
                    name:
                      type: "string"
                      example: "project v1"

Outdoc can also auto generate tags for multiple endpoints.

Check more options and examples in Github https://github.com/dapi-labs/outdoc

Sending data between Nodejs processes

wwayne — Fri, 16 Sep 2022 16:03:08 +0000

Last month when I was building an open-sourced project, I met with a situation that needs to run npm test in the child process and send back the data to the main process,

So I did some investigation and figured out my solution in the end. Here is my summary of how many ways we can use for establishing nodejs process communication.

child_process.fork

fork gives you a communication channel by native, specify a file for the child process and they will establish a communication channel after generating the child process, as simple as what is shown in the Nodejs official document.

// index.js
import { fork } from 'node:child_process'
import path from 'path'

const subProcess = fork(path.resolve(process.cwd(),'sub.js'))
subProcess.on('message', (msg) => {
  console.log('PARENT got message: ', msg)
  subProcess.send('parent -> child')
})

// sub.js
process.send("child -> parent")
process.on('message', (msg) => {
  console.log('CHILD got message: ', msg)
})

// output
PARENT got message:  child -> parent
CHILD got message:  parent -> child

child_process.spawn

fork works only if we have the file running in the child process. In some other cases, we might wanna use the child process to execute a command and we use spawn for such cases. I thought the spawned child process can't establish a communication channel until I found the parameters stdio, here is the example:

// index.js
import { spawn } from 'node:child_process'

const subProcess = spawn('node', ['sub.js'], {
  // inherit stdin, stdout and stderr, establish IPC communication channel
  stdio: ['inherit', 'inherit', 'inherit', 'ipc']  
})

subProcess.on('message', (msg) => {
  console.log('PARENT got message: ', msg)
  subProcess.send('parent -> child')
});

// sub.js
process.send("child -> parent")
process.on('message', (msg) => {
  console.log('CHILD got message: ', msg)
})

// output
PARENT got message:  child -> parent
CHILD got message:  parent -> child

Listen stdout or stderr

but for my case, I need my child process to run an npm script, somehow it can't establish the communication channel.

And finally, I have to switch to my tricky solution which passes the data through stdout or stderr, because I found the parent process can listen the data from child process's stdout and stderr if we set stdio as pipe, then what we need to do is just let the child process to output the data with a pre-defined format so the parent process can recognize the data by the format.

// index.js
const subProcess = spawn('node', 'run test', {
  // pipe stderr so we can subsribe the data event on stderr
  stdio: ['inherit', 'inherit', 'pipe']
});

subProcess.stderr.on('data', (buffer) => {
  const data = buffer.toString()
  if (data.startsWith('PREFIX')) {
    // Here we know this is the data sent from the child process manually
  }
})

// testing files
process.stderr.write('PREFIX' + 'message to parent' + "\n")

Conclusion

For establish a communication channel between node process:

Using fork if you have the file for the child process
Using spawn and modify stdio if you only need to run a command in the child process
If you are running an NPM script, we can do communication through stdout and stderr.

I built a SaaS product for API management

wwayne — Sun, 05 Sep 2021 10:02:25 +0000

Hi everyone :)

The idea of building a collaborative tool for API development occurred to me when I was working as a web full-stack developer. I felt we need a tool that serving the frontend and backend to work more closely, so I started building Dapi.

Dapi supply tools like openAPI editor, model visualization, and sequence diagram for the backend team to design API. One-click to deploy an API documentation site, plus a mock API server for the frontend team to do the API integration and review.

And this is a just start, stay tuned and let me know your thoughts, thanks 😁

Website url: https://dapiok.com
Twitter: https://twitter.com/dapi_ok

React avatar generator

wwayne — Wed, 02 Jun 2021 08:26:06 +0000

Hello :)

It has been a while for me to look for an avatar generator for my users that just signed up with an Email on the product.

Last weekend, luckily I found a new avatar template from the Figma community and I really like it at the first glance. So I downloaded the Figma files, writing code based on that. And now here it is, a react library for generating random but nice avatars, including an website for editing and preview.

I hope it helps for you guys :)

Github: https://github.com/dapi-labs/react-nice-avatar
Online Editor: https://nice-avatar.dapi.to/

Open-sourced the API editor of my upcoming product

wwayne — Wed, 26 May 2021 02:33:47 +0000

Hello developers :)

I'm working on my next product which is a collaborative API platform. The idea initially came from my daily works, I was working in a consulting agency that design & build apps for large companies like Nike, Starbucks, Burberry, etc.

Just like other digital teams, we use tools like GitHub, Jira, Figma to manage everything and work together. But as a backend dev, I found the management for the API is chaotic, we use websequencediagrams to draw sequence diagram, using lucidchart to draw some logic diagram and frontend colleagues have to use swagger-editor to check backend's API doc.

We don't have an all-in-one platform to manage everything in the backend. With the project going and become more complicated, it becomes hard for team members to review the logic designing behind each API, which is quite important for those performance-critical APIs. On the other hand, backend devs have to use Slack to notify the frontend team which API has just been changed.

So I started to build a collaborative API platform aim at bringing frontend & backend teams to work together in one place. The product is going to be released in the next month and I just open-sourced the openAPI editor of the product, it might help if you are looking for an alternative of swagger-editor.

Github: https://github.com/chilllab/comet

Happy creating :)