Forem: Vinod S R

dynamoman – the perfect import-export utility for AWS DynamoDB administration

Vinod S R — Sun, 22 May 2022 13:05:20 +0000

It is quite surprising that AWS didn’t provide an import export support for its popular database DynamoDB. When I started using DynamoDB for my project, I found the unavailability of import export feature hounding me during data migration or copying data between environments.

This made me to start looking for options for exporting the tables of DynamoDB. I found there are some options must most of them are more than 1 year old and lacks intuitive features. This made me to think about creating a cli which will handle the import export of tables, thus created a new cli – dynamoman (https://www.npmjs.com/package/dynamoman)

Features

Easily export the dynamoDB table as JSON
Imports the data into any dynamoDB table
Export from one region and import in to another
Supports batch import (chunks of 25 records)
Supports Windows / Linux / Mac

Installation

You can use npm / yarn to install the cli

npm install -g dynamoman

To verify the Installation

dynamoman --help

Getting started

For importing the data use the import command and for exporting use the export command

dynamoman import --file table_dump.json --table <mytable>
dynamoman export --file output.json --table <mytable>

More feature are there in the pipeline including

Export all tables to a zip
Import all tables from a zip

If you like to contribute to this project, please feel free to submit a pull request @ https://github.com/vinodsr/dynamoman

How to enable subscription in Apollo Federation with nestJS graphql

Vinod S R — Sat, 18 Sep 2021 14:01:22 +0000

NestJS with GraphQL

Setting up a graphQL server in nestJS saves 5x-10x of the effort with vanilla nodejs. NestJS uses the standard apollo library for setting up the graphQL. It supports both normal and federation mode. In federation mode, we can setup graphql servers in every microservices, which can be aggregated using a gateway; Just like API-Gateway with REST services. It is a very good technique if you are running microservices and want to separate the responsibility for each service.

NestJS is one of the powerful nodejs framework available as of today. One of the cool features of nestJS is that it allows easy integration with databases, message queues, authentication etc. Further more it allows developers to create application using nodeJS with in a much stricter, robust and flexible framework like Spring Boot / Angular.

If you are new to nestJS head over to https://docs.nestjs.com for getting started. This article expects that you have a prior knowledge of setting up and running nestJS.

the problem
solution
conclusion
bonus tip

the problem

All this comes with some inherent caveats from the apollo library. While setting up the federation you cannot use graphql subscription.

You will probably see this error

Apollo had come up with an approach to over come this problem. But it over complicates the situation. Lets see how to tackle this problem using the features in nestJS. In the example i am using the schema first approach. The same approach can be implemented in the code based approach.

solution

The solution to this problem is to hide the subscription based schema for the federation server and host the subscription based graphql as a separate normal graphql server. Seems very simple huh ? Not that easy :) . We will cover the method with some simple steps.

step 1 : Classify Schema

The primary step is to separate your graphql schema in to different files as per the below conventions.

*.graphql : for graphql syntax supported in both federation and normal mode
*.graphql.federation : for syntax supported only in federation mode (eg : extends)
*.graphql.normal : for syntax supported only in normal mode (eg : subscription)

Save the subscription model in any "graphql.normal" file

You can give any name you would like to. This tutorial follows this naming convention.

step 2 : configure server

Setup the app.module.ts with two graphql modules, one for the normal server and another for the federation server. We need to configure the module in such a way that the only federation module loads the .graphql.federation file and only normal module loads the graphql.normal file. The .graphql file has to be loaded by both modules.

  imports: [
    GraphQLModule.forRoot({
      debug: false,
      playground: true,
      path: '/graphql',
      typePaths: ['./**/*.{graphql,graphql.normal}'],
      installSubscriptionHandlers: true,
    }),
    GraphQLFederationModule.forRoot({
      debug: false,
      playground: false,
      path: '/graphql-federated',
      typePaths: ['./**/*.{graphql,graphql.federation}'],
    }),
  ],

Notice that type paths for the two modules are different as per our convention. The normal graphql with subscription is now available at /grapqhl and the server for federation gateway is available at /graphql-federated

We are not spinning two servers here. Its the same express server with two middleware configured for different paths. So there will not be any performance issues

step 3 : the illusion

This is the most important step. There are some directives in graphql that only works in the federated mode and vice versa. You will finally end up in writing the custom version of the grapqhl model in the federated and the normal files. This will add the headache of duplicate graphql models in the application.

This problem can be tackled in a easy way , using dummy directives !

Define a declarative for “key” in key.directive.ts

import { SchemaDirectiveVisitor } from 'apollo-server-express';
import { GraphQLField } from 'graphql';

/**
 * This is a dummy implementation of the key directive for the normal mode
 */
export class FakeKeyDirective extends SchemaDirectiveVisitor {
  /**
   * Fake Key definition
   * @param _field Field of graphql
   */
  visitFieldDefinition(_field: GraphQLField&lt;any, any&gt;) {
    _field.args;
  }
}

Include it in the module (app.module.ts)

@Module({
  imports: [
    GraphQLModule.forRoot({
      debug: false,
      playground: true,
      path: '/graphql',
      typePaths: ['./**/*.{graphql,graphql.normal}'],
      installSubscriptionHandlers: true,

      directiveResolvers: {
        key: FakeKeyDirective,
      },
    }),
    GraphQLFederationModule.forRoot({
      debug: false,
      playground: false,
      path: '/graphql-federated',
      typePaths: ['./**/*.{graphql,graphql.federation}'],
    }),
  ],
  controllers: [AppController],
  providers: [AppService, UsersResolver, UserService],
})
export class AppModule {}

Define a fake implementation. This only has to work in the normal mode. So the file name has to end with “graphql.normal” say "key.graphql.normal"

directive @key(fields: String) on OBJECT

Now you can define the model using the federation supported @key directive and the model works both in federation and normal graphql server. Let's edit "departments.graphql"

type Department @key(fields: "id") {
  id: ID!
  name: String
}

thats all folks

Now you can start the federation gateway which listens to the the /graphql-federated and the federation works.

For subscription you can use any Websocket enabled gateways like nginx, istio etc and connect directly to the microservices

conclusion

Yes it is possible to enable federation and subscription for graphql in nestJS using a simple trick. More efficient than the Apollo method. You can download the entire code for your reference from my github repo (https://github.com/vinodsr/nestjs-graphql-federation-with-subscription).

bonus tip

Since you are using a separate file extension other than .graphql, your IDE won’t give the native graphql formatting for the graphql.normal and graphql.federation files. There is a fix for this. Just enabled file associations for those extensions.

This is how it looks like in VS Code. Adding the file extensions will give you syntax highlighting and code formatting for the normal and federation graphql.

vinodsr / nestjs-graphql-federation-with-subscription

Sample code showing how to integrate subscription in apollo graphql server using nestJS

CCTV Object Detection using ML - Part 2 with Pi

Vinod S R — Sat, 10 Oct 2020 07:44:50 +0000

This post is the second part of my experiments with CCTV series. If you hadn't read the first part please read it over here.

Mission Pi

After the successful setup of CCTV Object detection in my Laptop, I had decided to implement the feature using my Pi4. Thus achieving 24/7 monitoring under low power consumption ie more savings 💰.

Took my Pi, installed docker and ran the docker compose as per the frigate documentation.

But it failed and I understood that the current version of frigate won't work with ARM base. (containers != virtualization) 😡 .

"Hey Google! Give me some solution"

This time found a fork of frigate, which uses pyarrow image to run the application. But the code base is far behind the actual frigate one. So latest features will be missing if I use this repo.

The only choice I have is to make my own fork and implement the changes. This also gave me the opportunity to add some features. The original version sends the image and details of an event as separate MQTT message. There is no option to link the two messages (no reference ids) to produce a meaningful output.

Sat down with the code for 2 hours and came up with a Pi enabled frigate version. You can download and build the image from my repo

Changes

Added Raspberry Pi support (Tested in Pi 4)
Added a new MQTT event (frigate///event) that will hold the event details and snapshot in base64. (schema)[https://github.com/vinodsr/frigate#frigatecamera_nameobject_nameevent]

During this time I found that my Pi was getting hot > 50 deg. Since the ML process is looking through each frames for identifying the objects it increases the CPU load. I had tweaked the config of frigate, which made the process to look in every 10 frames and this improved the performance and didn't have any problems with the result.

    take_frame: 15

Uploading my config for your reference(https://gist.github.com/vinodsr/edabaf37ca8ab77f68688844ff48124e#file-config-yml)

The Setup

Don't worry it will be easy :)

You need to install a MQTT server(I am using mosquito since it is light weight), compile and run docker and finally setup home automation to trigger some action on the detection.

Install Mosquitto

To install the Mosquitto Broker enter these next commands:

pi@raspberry:~ $ sudo apt update
pi@raspberry:~ $ sudo apt install -y mosquitto mosquitto-clients

To make Mosquitto auto start on boot up enter:

pi@raspberry:~ $ sudo systemctl enable mosquitto.service

Build Frigate for Pi

Clone my repo

git clone https://github.com/vinodsr/frigate.git

Run docker build

 docker build -t frigate:rpi -f Dockerfile.rpi .

Create docker-compose.yml

version: "3.8"
services:
    frigate:
        container_name: frigate
        restart: unless-stopped
        privileged: true
        shm_size: '1g' # should work for 5-7 cameras
        image: frigate:rpi
        volumes:
            - /dev/bus/usb:/dev/bus/usb
            - ./config:/config
            - ./clips:/clips
        ports:
            - "5000:5000"

Run docker compose

docker-compose up -d

Once you build the docker containers, you can toggle the detection by executing docker stop and start comamnds

docker stop frigate
docker start frigate

Further more you can add this as a cronjob to automate the start the stop

crontab -e

Paste the content in the editor to start the detection at 10 pm and stop at 6 am

00 22 * * * docker start frigate
00 06 * * * docker stop frigate

Automate everthing

I use node-red for home automation. Used MQTT, telegram bot nodes to design the automation. (used botmaster to create a telegram bot. You can read more about it from here)

Attaching my flow definition for your reference. You can import it to your node-red flows.

My node-red flow/figcaption>

The Result

After I setup the CCTV automation, when someone 🚶 is at my front door, I will get a telegram message.

Message from telegram bot

Conclusion

This wraps up the CCTV using ML series. It was very fun integrating ML to my normal CCTV cameras. The solution that I achieved is 100% safe and secure. No data is being sent to cloud or stored in any third party server. Hope you like my frigate fork. See you next time with another home IoT project.

CCTV Object Detection using ML

Vinod S R — Sat, 12 Sep 2020 09:51:54 +0000

My CCTV setup always have challenges in terms of motion detection. It simply uses the difference between frames based on a tolerance value and triggers an alert. I need to have a more sophisticated alert mechanism, which can detect person, cat , dog , car etc. But right now i am not going to upgrade the CCTV to a high end system.

Hey google find me a utility which can detect objects from CCTV camera.

After a long search i ended up in a github page of frigate. https://github.com/blakeblackshear/frigate

As per the definition in the README.md

Uses OpenCV and Tensorflow to perform realtime object detection locally for IP cameras. Designed for integration with HomeAssistant or others via MQTT.

* Leverages multiprocessing heavily with an emphasis on realtime over processing every frame
* Uses a very low overhead motion detection to determine where to run object detection
* Object detection with Tensorflow runs in a separate process
* Object info is published over MQTT for integration 
* An endpoint is available to view an MJPEG stream for debugging, but should not be used continuously

WoW !. This is what i was looking for.

Wait a second! Did they say IP camera ? All my camera are analog camera connected to HikVision DVR (DS-7B08HUHI-K1). :(

I didn't give up. Loaded Google and started the search with the query HIKVISION CAMERA STREAM WITHOUT IP CAMERA

After hours of searching. I found that HikVision have the capability to share the rtsp stream of any camera even though non-ip camera.

You need to connect the DVR to your network using a LAN cable
Once it is configured, then the rtsp stream is available at
rtsp://<username>:<password>@<ip of the DVR>:554/Streaming/channels/<x>0<y>

where username and password is same as that you use to login to the CCTV console,
x is the camera number
y can be 1 or 2. 2 will be a low quality stream and 1 will be a high quality stream.

eg :
rtsp://<username>:<password>@<ip of the DVR>:554/Streaming/channels/102

So now i have a IP camera stream. So back to the setup of frigate.

It is very simple. You need to install docker. Head over to the docker.com to install docker in your machine. (https://docs.docker.com/get-docker/)

You can then follow the steps mentioned in the github page (https://github.com/blakeblackshear/frigate#getting-started).

You can also tweak the config.yml for your preference.

    ################
    # Camera level object config. This config is merged with the global config above.
    ################
    objects:
      track:
        - person

You can use this section to tell frigate about the object that needs to be tracked. By default it tracks person. You can add cats, dog etc. The available objects that can be tracked by default can be found at (https://dl.google.com/coral/canned_models/coco_labels.txt)

If you setup a MQTT broker like mosquitto (https://mosquitto.org/), then frigate will send events on each detection to the broker. You can use this for automation.

Make sure you give the rtsp link of the camera correctly and the camera name is correct.

If everything is done correctly, you can start the docker and the site will be available @ port 5000

Load http://localhost:5000/ to see the debug stream of the camera with object recognition.
As per the author the debug stream can reduce the performance of the program

Once a person is in front of the camera, you will see something like this in the debug stream, also an event will be published in the MQTT server, which can be used to trigger some message or anything.

Next Steps

My next step is to integrate this with my home automation to trigger a message on my phone with the picture. Also to run this on raspberry pi, since the original version doesn't support pi.

I will be back once it is ready. Till then take care bye.

Read the second part of the blog here

NestJS : a backend nodeJS framework for the enterprise

Vinod S R — Thu, 09 Jan 2020 08:47:04 +0000

Last week an idea stuck my mind to explore the frameworks available in nodeJS ecosystem to develop backend API, which is scalable and efficient. I was using expressJS long time for my projects. It's time to think about the alternate future 😃.

I started listing down all the features that would be required for a good nodeJS backend framework (purely based on my understanding).

Modularity
Easy to develop
Integration with other applications
Extensibility
Maintainable
Performance
Easy to test and deploy
DevOps support ready
Support for microservices
REST & GraphQL

I took my special analyser googles, fuelled up my Time Machine and started my journey to future in search for my query. Went through frameworks like express, Koa, hapi, molecular, Seneca etc. But none had full-filled my need, until i saw NestJS.

Nest (NestJS) is a futuristic framework for building efficient, scalable Node.js server-side applications. It uses progressive JavaScript, is built with and fully supports TypeScript (yet still enables developers to code in pure JavaScript) and combines elements of OOP (Object Oriented Programming), FP (Functional Programming), and FRP (Functional Reactive Programming). It is build on top of typescript (javascript version is also available). The framework is built by taking the concepts of Angular putting it on top of expressJS or fastify.

Yes you heard it right. I can change the underlying core framework to either express or fastify without changing any API code. Fastify has got proven track record when it comes to performance.

Let's have a look in to how you can setup the nestJS framework.

Chapters ahead

Installation
First API
Exception handling
Guards
Microservices Support
GraphQL
Express/Fastify
More goodies
Wrapping up

Installation

The installation is pretty much straight forward. You can use npm/yarn to install the framework.

npm i -g @nestjs/cli
nest new my-first-project

The cli will ask for the type of package system you want (npm/yarn).
Once the installation is complete you can start the application using the start script.

cd my-first-project
npm run start

Open any browser and try http://localhost:3000. You will now see the "Hello World!"

Hurray!

You can also use npm run start to start in the application in dev mode. This will watch for any changes in the code and restart the application.

First API

In most of the nodeJS backend framework, for writing a API, we will create a callback method for the defined URL. This is not followed in nestJS.

If you have a Angular/Spring boot background, you will find the below features very similar to Angular/Spring boot

In nestJS, in order to create an API we need a controller which is similar to a component in Angular (or a RestController in Springboot).

Let's say i need to create a time api which returns the current server time.

Run the following cli command inside your project directory to create a new controller

nest g controller time

Where time is our controller name

This will create two files. The time.controller.spec.ts is the test file and time.controller.ts is our main file

CREATE /src/time/time.controller.spec.ts (479 bytes)
CREATE /src/time/time.controller.ts (97 bytes)
UPDATE /src/app.module.ts (322 bytes)

You can see that it updates the app.module.ts file to add the reference to the new controller time.controller. This is similar to bootstrapping a component in Angular.

This is how the folder structure looks after you create a controller.

Open the time.controller.ts and replace the code with the below code

import { Controller, Get } from '@nestjs/common';

@Controller('time')
export class TimeController {
    @Get()
    getTime(): any {
      return {
          time : new Date(),
      };
    }
}

Restart the application (If you had started in dev mode, it will restart automatically) and access http://localhost:3000/time using postman / browser / curl

You will now see the below response as json.

{
  "time": "2019-12-31T05:06:31.805Z"
}

If you gave an argument in the @Get decorator then the API url will change accordingly.

So the effective url of an API = + @Controller's url + @Get's url

Similar to @Get, NestJs also supports other REST verb like @post, @Put, @Update, @Delete etc..

What ever object is returned from the method is sent back to the client as the result of the API.

You can get more details about the controller implementation @ https://docs.nestjs.com/controllers

This is one of the simplest API implementation in nodeJS, similar to Java's Spring boot implementation.

Exception handling

In nodeJS we use try/catch or exception middleware to handle the errors in the application.

But in NestJS, the approach is unique.

You can create a filter class that can handle the specific exceptions.

import { ExceptionFilter, Catch, ArgumentsHost, HttpException } from '@nestjs/common';
import { Request, Response } from 'express';

@Catch(HttpException)
export class HttpExceptionFilter implements ExceptionFilter {
  catch(exception: HttpException, host: ArgumentsHost) {
    const ctx = host.switchToHttp();
    const response = ctx.getResponse<Response>();
    const request = ctx.getRequest<Request>();
    const status = exception.getStatus();

    response
      .status(status)
      .json({
        statusCode: status,
        timestamp: new Date().toISOString(),
        path: request.url,
      });
  }
}

This means if an HttpException is thrown from any part of the application, this filter is triggered, the logic executes and a response is sent back to the client.

Easy ? uh ?

More details @ https://docs.nestjs.com/exception-filters

Guards

Security is very critical. Typically in a expressJS application we use middleware to check for the auth level of an API.

In nestJS we can use guards for this purpose.

import { Injectable, CanActivate, ExecutionContext } from '@nestjs/common';
import { Observable } from 'rxjs';

@Injectable()
export class RolesGuard implements CanActivate {
  canActivate(
    context: ExecutionContext,
  ): boolean | Promise<boolean> | Observable<boolean> {
    return true;
  }
}

and in the controller

@Controller('cats')
@UseGuards(RolesGuard)
export class CatsController {}

You can also create global guards in app.module.ts

const app = await NestFactory.create(AppModule);
app.useGlobalGuards(new RolesGuard());

Microservices Support

One of the biggest advantage of this framework is it's out of the box support for microservices and transport layers. It has got the support for different transport layers like TCP, gRPC, MQTT, RabbitMQ etc...

You can read more about microservices @ https://docs.nestjs.com/microservices/basics

GraphQL

NestJS has got out of the box support for GraphQL. It starts with installing the GraphQL support package.

npm i --save @nestjs/graphql apollo-server-express graphql-tools graphql

NestJS offers two methods of integrating GraphQL.

Schema first : where the definition is written in GraphQL Schema Definition Language (SDL)
Code first: where we use decorators to generate the GraphQL schema

You can read more about GraphQL integration @ https://docs.nestjs.com/graphql/quick-start

Express / Fastify ?

Another impressive feature of NestJS is that, it can run on top of express or fastify frameworks. You don't have to change you API's code for that. If you want to access any native framework features of epxress / fastify, it is possible.

By default NestJS uses express framework. If you want to use fastify framework, then you can install the package for fastify.

npm i --save @nestjs/platform-fastify

Once installed you need to modify the module.ts with the below code

import { NestFactory } from '@nestjs/core';
import {
  FastifyAdapter,
  NestFastifyApplication,
} from '@nestjs/platform-fastify';
import { ApplicationModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create<NestFastifyApplication>(
    ApplicationModule,
    new FastifyAdapter()
  );
  await app.listen(3000);
}
bootstrap();

Now your application is ready for fastify.

Benchmark results shows that fastify has got more performance than express

More goodies

Further more it has got out of the box support for Swagger, Jest, Supertest, TypeORM, Sequelize, Terminus, Compodoc, Custom decorators and lots more...

Wrapping up

IMHO NestJS is a framework which can be used in small backend applications to large enterprise level application. It helps you in building an efficient, scalable application. The maintainability of this framework is huge compared to other nodeJS framework. I had seen projects using expressJS where the code maintainability has become a serious concern after some months of development. The ability to switch express and fastify is one of the cool features of NestJS.

Head on to https://nestjs.com to start developing your app.

Happy Coding 👍.

Forem: Vinod S R

dynamoman – the perfect import-export utility for AWS DynamoDB administration

Features

Installation

Getting started

How to enable subscription in Apollo Federation with nestJS graphql

NestJS with GraphQL

Contents

the problem

solution

step 1 : Classify Schema

step 2 : configure server

step 3 : the illusion

thats all folks

conclusion

bonus tip

vinodsr / nestjs-graphql-federation-with-subscription

Sample code showing how to integrate subscription in apollo graphql server using nestJS

CCTV Object Detection using ML - Part 2 with Pi

Mission Pi

Changes

The Setup

Install Mosquitto

Build Frigate for Pi

Automate everthing

The Result

Conclusion

CCTV Object Detection using ML

Next Steps

NestJS : a backend nodeJS framework for the enterprise

Chapters ahead

Installation

First API

Exception handling

Guards

Microservices Support

GraphQL

Express / Fastify ?

More goodies

Wrapping up