Forem: Sandip Basnet

How to find twitter accounts with similar profile picture?

Sandip Basnet — Mon, 09 Jan 2023 09:26:57 +0000

I want to build a service that can tell us what profiles on Twitter have a particular profile picture(Sample: https://www.nftinspect.xyz/twitter-search/nft?chain=ETH&token=4081&contract=0x8a90cab2b38dba80c64b7734e58ee1db38b8992e), I have gone through all references of Twitter APIs (https://www.nftinspect.xyz) but was not able to find one to resolve my problem, and thought about scanning via search but that doesn't sound scalable, any folks with previous or similar experiences? I would really appreciate the help :)
Note, I have found one tool(https://www.nftinspect.xyz), but unfortunately, it is shutting down on 17th Jan.

Create user with read only permission in PostgreSQL

Sandip Basnet — Tue, 03 Jan 2023 10:35:14 +0000

Steps:

create role:

CREATE ROLE readaccess;
grant access to existing tables:

GRANT USAGE ON SCHEMA public TO readaccess; GRANT SELECT ON ALL TABLES IN SCHEMA public TO readaccess;

grant access to future tables:

ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO readaccess;
Create user with password:

CREATE USER grafana WITH PASSWORD 'secretpass@dev';
grant access to user: GRANT readaccess TO grafana;

Migrating postgres db from CLI

Sandip Basnet — Fri, 23 Dec 2022 02:56:07 +0000

Migrating postgres db within same/different server via cli.
1) Export dump from existing db: pg_dump -U postgres -h <host_address> -p 5432 <db_name> > <dump/backup_file_name>.sql
2) Import dump to new db: psql -h <new_db_host_address> -d <new_db_name> -U postgres -f <dump/backup_file_name>.sql

Note, __In both of the cases you will be prompted with password input.

Create a new post from WP REST API

Sandip Basnet — Tue, 06 Sep 2022 03:00:16 +0000

Create a new post from WP REST API

NodeJs CRUD

Sandip Basnet — Sat, 20 Aug 2022 00:30:00 +0000

We will be building REST APIS using node express and sequelize.

create folder: mkdir workshop && cd workshop
initialize the project: npm init
Install required dependencies: npm install express sequelize pg pg-hstore cors --save
Setup express web server: in the root folder create app.js file and add:

const express = require("express");
const bodyParser = require("body-parser");
const cors = require("cors");
const app = express();
var corsOptions = {
  origin: "*"
};
app.use(cors(corsOptions));
// parse requests of content-type - application/json
app.use(express.json());
// parse requests of content-type - application/x-www-form-urlencoded
app.use(express.urlencoded({ extended: true }));
// simple route
app.get("/", (req, res) => {
  res.json({ message: "Hello World!." });
});
// set port, listen for requests
const PORT = process.env.PORT || 5000;
app.listen(PORT, () => {
  console.log(`Server is running on port ${PORT}.`);
});

where:
Express: this module is used for building the Rest apis.
cors: provides Express middleware to enable CORS with various options.
body-parser: parse the request body

connect database: create config folder and add db.config.js file and add:

module.exports = {
  HOST: "localhost",
  PORT: 5432,
  USER: "postgres",
  PASSWORD: "postgres",
  DB: "testdb",
  dialect: "postgres",
  pool: {
    max: 5,
    min: 0,
    acquire: 30000,
    idle: 10000
  }
};

where pool is optional, it will be used for Sequelize connection pool configuration:

max: maximum number of connection in pool
min: minimum number of connection in pool
idle: maximum time, in milliseconds, that a connection can be idle before being released

Initialize Sequelize create database/models and add index.js with:

const dbConfig = require("../config/db.config.js");
const Sequelize = require("sequelize");
const sequelize = new Sequelize(dbConfig.DB, dbConfig.USER, dbConfig.PASSWORD, {
  host: dbConfig.HOST,
  port: dbConfig.PORT,
  dialect: dbConfig.dialect,
  operatorsAliases: false,
  pool: {
    max: dbConfig.pool.max,
    min: dbConfig.pool.min,
    acquire: dbConfig.pool.acquire,
    idle: dbConfig.pool.idle
  }
});
const db = {};
db.Sequelize = Sequelize;
db.sequelize = sequelize;
db.plans = require("./plan.model.js")(sequelize, Sequelize);
module.exports = db;

and update app.js with (put is just below the const app = express();)

const db = require("./database/models");
db.sequelize.sync()
  .then(() => {
    console.log("Synced db.");
  })
  .catch((err) => {
    console.log("Failed to sync db: " + err.message);
  });

In development, you may need to drop existing tables and re-sync database. Just use force: true as following code:
db.sequelize.sync({ force: true }).then(() => { console.log("Drop and re-sync db."); });

Define the Sequelize Model In models folder, create plan.model.js file like this:

module.exports = (sequelize, Sequelize) => {
  const Plan = sequelize.define("plan", {
    title: {
      type: Sequelize.STRING
    },
    description: {
      type: Sequelize.STRING
    },
    completed: {
      type: Sequelize.BOOLEAN
    }
  });
  return Plan;
};

This Sequelize Model represents paln table in PostgreSQL database. These columns will be generated automatically: id, title, description, completed, createdAt, updatedAt.

After initializing Sequelize, we don’t need to write CRUD functions, Sequelize supports all of them:

create a new plan: create(object)
find a Plan by id: findByPk(id)
get all Plans: findAll()
update a Plan by id: update(data, where: { id: id })
remove a Plan: destroy(where: { id: id })
remove all Plans: destroy(where: {})
find all Plans by title: findAll({ where: { title: ... } })
These functions will be used in our Controller.

Add controllers create controllers folder and add plan.controller.js file with following functions:

create
findAll
findOne
update
delete
deleteAll
findAllCompleted
i.e.

const db = require("../database/models");
const Plans = db.plans;
const Op = db.Sequelize.Op;
// Create and Save a new Plan
exports.create = (req, res) => {

};
// Retrieve all P from the database.
exports.findAll = (req, res) => {

};
// Find a single Plan with an id
exports.findOne = (req, res) => {

};
// Update a Plan by the id in the request
exports.update = (req, res) => {

};
// Delete a Plan with the specified id in the request
exports.delete = (req, res) => {

};
// Delete all Plans from the database.
exports.deleteAll = (req, res) => {

};
// Find all completed Plans
exports.findAllCompleted = (req, res) => {

};

Implementations:

create:

exports.create = (req, res) => {
    // Validate request
    if (!req.body.title) {
        res.status(400).send({
            message: "Title can not be empty!"
        });
        return;
    }
    // Create a Plan
    const plan = {
        title: req.body.title,
        description: req.body.description,
        completed: req.body.completed ? req.body.completed : false
    };
    // Save plan in the database
    Plans.create(plan)
        .then(data => {
            res.send(data);
        })
        .catch(err => {
            res.status(500).send({
                message:
                    err.message || "Some error occurred while creating the plan."
            });
        });
};

retrive

exports.findAll = (req, res) => {
    const title = req.query?.title;
    var condition = title ? { title: { [Op.iLike]: `%${title}%` } } : null;
    Plans.findAll({ where: condition })
        .then(data => {
            res.send(data);
        }).catch(err => {
            res.status(500).send({
                message:
                    err.message || "Some error occurred while retrieving plans."
            });
        });
};

Retrieve a single object

exports.findOne = (req, res) => {
    const id = req.params.id;
    Plans.findByPk(id)
        .then(data => {
            if (data) {
                res.send(data);
            } else {
                res.status(404).send({
                    message: `Cannot find Plan with id = ${id}.`
                });
            }
        }).catch(err => {
            res.status(500).send({
                message: "Error retrieving Plan with id=" + id
            });
        });
};

Update an object

exports.update = (req, res) => {
    const id = req.params.id;
    Plans.update(req.body, {
        where: { id: id }
    })
        .then(num => {
            if (num == 1) {
                res.send({
                    message: "Plan was updated successfully."
                });
            } else {
                res.send({
                    message: `Cannot update Plan with id=${id}. Maybe Plan was not found or req.body is empty!`
                });
            }
        }).catch(err => {
            res.status(500).send({
                message: "Error updating Plan with id=" + id
            });
        });
};

Delete an object

exports.delete = (req, res) => {
    const id = req.params.id;
    Plans.destroy({
        where: { id: id }
    })
        .then(num => {
            if (num == 1) {
                res.send({
                    message: "Plan was deleted successfully!"
                });
            } else {
                res.send({
                    message: `Cannot delete Plan with id=${id}. Maybe Plan was not found!`
                });
            }
        }).catch(err => {
            res.status(500).send({
                message: "Could not delete Plan with id=" + id
            });
        });
};

Delete all objects

exports.deleteAll = (req, res) => {
    Plans.destroy({
        where: {},
        truncate: false
    })
        .then(nums => {
            res.send({
                message: `${nums} Plans were deleted successfully!})`
            }).catch(err => {
                res.status(500).send({
                    message:
                        err.message || "Some error occurred while removing all plans."
                });
            });
        });
};

Find all objects by condition

exports.findAllCompleted = (req, res) => {
    Plans.findAll({ where: { completed: true } })
        .then(data => {
            res.send(data);
        }).catch(err => {
            res.status(500).send({
                message:
                    err.message || "Some error occurred while retrieving Plans."
            });
        });
};

*Define Routes
*
When a client sends request for an endpoint using HTTP request (GET, POST, PUT, DELETE), we need to determine how the server will response by setting up the routes.

These are our routes:

/api/plans: GET, POST, DELETE
/api/plans/🆔 GET, PUT, DELETE
/api/plans/completed: GET

Create a plan.routes.js inside routes folder with content like this:

module.exports = app => {
    const plans = require("../controllers/plan.controller.js");
    var router = require("express").Router();

    // Create a new Plan
    router.post("/", plans.create);

    // Retrieve all Plans
    router.get("/", plans.findAll);

    // Retrieve all completed Plans
    router.get("/completed", plans.findAllCompleted);

    // Retrieve a single Plan with id
    router.get("/:id", plans.findOne);

    // Update a Plan with id
    router.put("/:id", plans.update);

    // Delete a Plan with id
    router.delete("/:id", plans.delete);

    // Create a new Plan
    router.delete("/", plans.deleteAll);

    app.use('/api/plans', router);
};

You can see that we use a controller from /controllers/plan.controller.js.

We also need to include routes in app.js (right before app.listen()):

require("./routes/plan.route")(app);

*Test the APIs
*
Run our NodeJs application with command: node app.js.

Using Postman, we’re gonna test all the APIs above

References:
1) https://www.bezkoder.com/node-express-sequelize-postgresql/

Code repository: https://github.com/mondyfy/nodejs-workshop-kist

Sending Emails from NestJs Using the Mailgun API

Sandip Basnet — Wed, 20 Jul 2022 08:49:14 +0000

I had set up the mailing with nodemailer with mailgun few months back, that was working fine but I had got few recurring issues, that's why I have switched the implementation from SMTP to API, and I found it quite simpler than that of the SMTP as no extra library need to be used, apart from the request client axios.

The snippet of the implementation goes like this:

public async sendMail(mail: IMail) { const mailOptions: MailInput = { from: `SON <${configService.get('SENDER_EMAIL')}>`, to: mail.to, subject: mail.subject, } if (mail.template) { const emailTemplateSource = fs.readFileSync(path.join(__dirname, `../../templates/${mail.template}.hbs`), "utf8") const template = handlebars.compile(emailTemplateSource); const htmlToSend = template(mail.templateVariables) mailOptions.html = htmlToSend } else { mailOptions.text = mail.text; } try { const body = Object.keys(mailOptions).map((key, index) => `${key}=${encodeURIComponent(mailOptions[key])}`).join('&'); const response = await axios.post(`https://api.mailgun.net/v3/${configService.get('MAILGUN_DOMAIN')}/messages`, body, { auth: { username: 'api', password: configService.get('MAILGUN_API_KEY') }, headers: { "Content-Type": "application/x-www-form-urlencoded" } }); Logger.log(`Email successfully sent to: ${mail.to}.`) return response; } catch (error) { Logger.warn(`Problem in sending email: ${error}`); throw error; } }

HLS Audio Streaming in NodeJS

Sandip Basnet — Wed, 29 Jun 2022 18:05:08 +0000

In this article, I will discuss the working mechanism of HLS and how we can use HLS(HTTP Live Streaming) for audio/music streaming in NodeJs.

What is HLS?
According to Wikipedia: HLS is an HTTP-based adaptive bitrate streaming communications protocol developed by Apple Inc. It was released in 2009. Support for the protocol is widespread in media players, web browsers, mobile devices, and streaming media servers.

Why HLS?

Fast:
As one large file is divided into segments a few segments in length and those files are fetched on respective player time with the help of the manifest/index file.
Adaptive Bitrate Streaming:
It has the ability to adjust video quality in the middle of a stream as network conditions change. This ability allows videos to keep playing even if network conditions get worse; conversely, it also maximizes video quality to be as high as the network can support.
If the network slows down, the user’s video player detects this, and adaptive bitrate streaming lowers the quality of the stream so that the video does not stop playing. If more network bandwidth becomes available, adaptive bitrate streaming improves the quality of the stream.
Adaptive bitrate streaming is possible because HLS creates several duplicate segmented streams at different quality levels during the segmentation process. The user’s video player can switch from one of those streams to another one during video playback.

Unwanted i.e unwatched or unlistened portions of the file(audio/video) won’t be downloaded.
Less server and client configuration uses HTTP Protocol.

How does HLS work?
Server: An HLS stream originates from a server where (in on-demand streaming) the media file is stored, or where (in live streaming) the stream is created. Because HLS is based on HTTP, any ordinary web server can originate the stream.

Two main processes take place on the server:

Encoding: The audio/video data is reformatted so that any device can recognize and interpret the data. HLS must use H.264 or H.265 encoding.

Segmenting: The audio/video is divided up into segments a few seconds in length.

In addition to dividing the audio/video into segments, HLS creates an index file of the audio/video segments to record the order they belong in.
HLS will also create several duplicate sets of segments at different quality levels: 480p, 720p, 1080p, and so on.

Distribution: The encoded video segments are pushed out to client devices over the Internet when client devices request the stream. Typically, a CDN or OFS like s3 will help distribute the stream to geographically diverse areas.

Client devices: The client device is the device that receives the stream and plays the video — for instance, a user's smartphone or laptop. The client device uses the index file(.m3u8) as a reference for assembling the audio/video in order, and it switches from higher quality to lower quality pictures (and vice versa) as needed.

Example: HLS Audio streaming(NodeJs)

MP3 to chunks:

First of all, initiate a project with npm init and add utils/mp3tochunks.js, the contents of mp2tochunks.js should be:
`const util = require('util');
const exec = util.promisify(require('child_process').exec);
const fs = require('fs');
const path = require('path');

const dir = path.join(dirname, '../songs');
const dest = path.join(dirname, '../temp/chunks');

const startTime = new Date();
console.info('> Start reading files', startTime);

fs.readdir(dir, (readDirError, files) => {
if (readDirError) {
console.error(readDirError);

    return;
}

const countFiles = files.length;
files.map(async (file, index) => { 
    const fileName = path.join(dir, file);

    const { err, stdout, stderr } =
        await exec(`ffmpeg -i ${fileName} -profile:v baseline -level 3.0 -s 640x360 -start_number 0 -hls_time 10 -hls_list_size 0 -f hls  ${dest}/${index}.m3u8`);

    if (err) {
        console.log(err);
    }

    if (countFiles - 1 === index) {
        const endTime = new Date();
        console.info('< End Preparing files', endTime);
    }
});

});`

In the above snippet, I have made a song folder that contains mp3 files that will later be converted into chunks and those chunks are stored on the temp/chunks directory.

The main task in this util is done by ffmpeg, which should be installed on your machine.

Command used:
ffmpeg -i ${fileName} -profile:v baseline -level 3.0 -s 640x360 -start_number 0 -hls_time 10 -hls_list_size 0 -f hls ${dest}/${index}.m3u8

where,
-i : specifies the input file audio/video,
-profile: sets the audio/video encoding codec profiles, setting baseline, as it supports lower-cost applications with limited computing resources,
-level: level as mentioned in Annex A of the H.264 standard,
-s: sets frame size,
-start_number: Set the index of the file matched by the image file pattern to start to read from, the default value is 0.
-hls_time: segment duration in seconds,
-hls_list_size: Set the maximum number of playlist entries. If set to 0 the list file will contain all the segments. The default value is 5,
-f: Set the file format,
${dest}/${index}.m3u8: Sets Path of the chunks generated and manifest file m3u8

The content of the m3u8 file should be something like this:

`#EXTM3U

EXT-X-VERSION:3

EXT-X-TARGETDURATION:10

EXT-X-MEDIA-SEQUENCE:0

EXTINF:10.008778,

00.ts

EXTINF:10.008778,

01.ts

EXTINF:9.985556,

02.ts

EXTINF:10.008778,

03.ts

EXTINF:10.008778,

04.ts

EXTINF:9.985556,

05.ts

EXTINF:10.008778,

06.ts

EXTINF:10.008778,

07.ts

EXTINF:9.985556,

08.ts

EXTINF:10.008778,

09.ts

EXTINF:10.008778,

010.ts

EXTINF:9.985556,

011.ts

EXT-X-ENDLIST`

where,
EXTM3U: this indicates that the file is an extended m3u file. Every HLS playlist must start with this tag.

EXT-X-VERSION: indicates the compatibility version of the Playlist file.

EXT-X-TARGETDURATION: this specifies the maximum duration of the media file in seconds.

EXT-X-MEDIA-SEQUENCE: indicates the sequence number of the first URL that appears in a playlist file. Each media file URL in a playlist has a unique integer sequence number. The sequence number of a URL is higher by 1 than the sequence number of the URL that preceded it. The media sequence numbers have no relation to the names of the files.

EXTINF: tag specifies the duration of a media segment. It should be followed by the URI of the associated media segment — this is mandatory. You should ensure that the EXTINF value is less than or equal to the actual duration of the media file that it is referring to.

Once the manifest file is generated, we have to host it, lets add a node server and test it on hls player i.e. add main.js file that should have:
`var http = require('http');
var fs = require('fs');

const port = 8000

http.createServer(function (request, response) {
console.log('request starting...');

var filePath = './temp/chunks' + request.url;

fs.readFile(filePath, function(error, content) {
    response.writeHead(200, { 'Access-Control-Allow-Origin': '*' });
    if (error) {
        if(error.code == 'ENOENT'){
            fs.readFile('./404.html', function(error, content) {
                response.end(content, 'utf-8');
            });
        }
        else {
            response.writeHead(500);
            response.end('Sorry, check with the site admin for error: '+error.code+' ..\n');
            response.end(); 
        }
    }
    else {
        response.end(content, 'utf-8');
    }
});

}).listen(port);
console.log(Server running at http://127.0.0.1:${port}/);
`

This basically takes the request URL and looks for the corresponding URL if that is generated in chunks, if that is generated it will respond with the contents of the file. for example: for request http://127.0.0.1:8000/0.m3u8 the manifest file will be downloaded, here to test the file you have to add one browser extension i.e. Native HLS Playback, that should play the media you have used. In the meantime, you can inspect the file sequence chunks requested by the client/browser from the server after every 10 seconds the chunk has been loaded.

That sums up the HLS audio/video streaming with nodejs. Here all the files are served locally, in an ideal case they are served through some online file storage service like AWS s3, for that, I have added uploadchunkstos3.js file in the repository of this article.

References

Steps to run any laravel project.

Sandip Basnet — Wed, 25 May 2022 01:38:57 +0000

1) Clone repository
2) composer install
3) copy .env.example to .env cp .env.example .env (if needed)
3) php artisan key:generate
4) php artisan migrate
5) php artisan db:seed (seeders if any)
6) php artisan serve

Backup MySQL database locally

Sandip Basnet — Tue, 24 May 2022 10:06:49 +0000

1) Make sure remote connection is enabled :MySQL remote access

2) If remote access is allowed:

mysqldump --host <host_address or ip> -P <Port Number(3306)> -u <db username> -p <database_name> > offgrid.sql

Remote access of MySQL from CLI

Sandip Basnet — Tue, 24 May 2022 09:53:13 +0000

1) Check if remote access is allowed for user:

SELECT User, Host FROM mysql.user;

if user has % (wildcard just as * ) in host then this user can establish connection from anywhere, you can update this wildcard or specific ip as

UPDATE mysql.user SET HOST='%' WHERE User='<your_username>';

and then flush privileges with:

FLUSH PRIVILEGES;

2) If MySQL is installed on a Ubuntu, you will have to access the file at this location: cd /etc/mysql/mysql.conf.d/mysqld.cnf
Open the file and comment the binded-address i.e.

change: bind-address = 127.0.0.1 to: #bind-address = 127.0.0.1

to bring changes on effect restart the mysql or reboot the system itself:
i.e. restart:

sudo service mysql restart

3) Access from cli:



mysql -u <user_name> -h <host or ip addrsss> -p

List the size of the database and tables: MySQL CLI

Sandip Basnet — Tue, 24 May 2022 09:29:40 +0000

1) Size of the database (in GB):

SELECT table_schema AS "Database", SUM(data_length + index_length) / 1024 / 1024 / 1024 AS "Size (GB)" FROM information_schema.TABLES GROUP BY
table_schema;

2) Size of the database (in MB):

SELECT table_schema AS "Database", SUM(data_length + index_length) / 1024 / 1024 AS "Size (MB)" FROM information_schema.TABLES GROUP BY table_schema;

3) Size of the tables of a database (in MB):

SELECT table_name AS "Table", ROUND(((data_length + index_length) / 1024 / 1024), 2) AS "Size (MB)" FROM information_schema.TABLES WHERE table_schema = "<db_name>" ORDER BY (data_length + index_length) DESC;

Server side implementation of Biometric in NodeJs

Sandip Basnet — Mon, 24 Aug 2020 08:25:57 +0000

Server side implementation of Biometric Authentication

Aug 24 '20 Comments: 1 Answers: 0

We have a microservice-based system and currently users are authenticated by keycloack(jwt) and we need to add biometrics login in this existing system At first, we have designed a biometrics authentication flow as below:

Generate Asymmetric Key Pair at the client end(mobile app).
Share the public key with backend server.
…

Open Full Question