Forem: Karan Janthe

Promises in JS(Personal notes)

Karan Janthe — Fri, 26 Apr 2024 14:11:39 +0000

what are Promise in Javascript?

They are similar to real life promise, how we make promise to someone that i am making promise that i will come meet you, same way we are just telling that, this piece code will get executed in future.

let promise = new Promise (function(resolve, reject) { 
// executor (the producing code, "singer") 
});

Promise object in javascript have two main part, "producing code" and "consuming code", producing code's job is to produce the code, and consuming code's job is tot consume the results from the produced code.

let myPromise = new Promise(function(myResolve, myReject) {  
// "Producing Code" (May take some time)  

  myResolve(); // when successful  
  myReject();  // when error  
});  

// "Consuming Code" (Must wait for a fulfilled Promise)  
myPromise.then(  
  function(value) { /* code if successful */ },  
  function(error) { /* code if some error */ }  
);

function passed to new Promise is called executor, which will execute them and then will make results for consuming code to consume the result.

so in short, executor runs automatically and attempts to do the job, when it finished with the task, it calls resolve if it was successful or reject if there was an error.

Promise Object have this these two internal properties:

resolve example:

let promise = new Promise(function(resolve, reject) {
// the function is executed automatically when the promise is constructed

// after 1 second signal that the job is done with the result "done"
setTimeout(() => resolve("done"), 1000);
});

state: "fulfilled",
result: "done"

reject example:

let promise = new Promise(function(resolve, reject) { 
// after 1 second signal that the job is finished with an error 
setTimeout(() =>  reject(new Error("Whoops!"``)``), 1000); 
});

state: "rejected",
result: "error"

there can be only single result or an error

let promise = new Promise(function(resolve, reject) {
  resolve("done");
  reject(new Error("...")); // ignored
  setTimeout(() => resolve("...")); // ignored
});

Consumer code:

.then part, when promises is either resolved or rejected, to get both values we use .then method on the promise object to get our result or error

let promise = new Promise(function(resolve, reject) {

setTimeout(() => resolve("done!"), 1000);

});

// resolve runs the first function in .then
promise.then(

result => alert(result), // shows "done!" after 1 second

error => alert(error) // doesn't run

);

error example:

let promise = new Promise(function(resolve, reject) {

setTimeout(() => reject(new Error("Whoops!")), 1000);

});

// reject runs the second function in .then
promise.then(

result => alert(result), // doesn't run

error => alert(error) // shows "Error: Whoops!" after 1 second

);

just like normal try catch block, we have same with promises also we have try to get desired result, catch to catch the error, and finally block to execute final code when every thing is done!

catch:

let promise = new Promise((resolve, reject) => {
setTimeout(() => reject(new Error("Whoops!")), 1000);
});

// .catch(f) is the same as promise.then(null, f)
promise.catch(alert); // shows "Error: Whoops!" after 1 second

finally block don't care about the result of promise, whether it was fulfilled or un fulfilled, it only knows to execute the code it is provided, and order matters in finally, if we write finally before the catch or then, it will run first,

new Promise((resolve, reject) => {
throw new Error("error");
})
.finally(() => alert("Promise ready")) // triggers first
.catch(err => alert(err)); // <-- .catch shows the error

what will be result ?

let promise = new Promise(function(resolve, reject) {
    resolve("Good Morning!");
    setTimeout(() => resolve("Good Night!"), 1000);
});

promise.then(alert);

difference between callback and promise ?

callback can be only called once, there is result or some kind of trigger, promises can be called as many times you want even after promise is fulfilled to access the result as many times you want

chaining in promises:

we can chain promises, example:

new Promise(function(resolve, reject) {

setTimeout(() => resolve(1), 1000); // (*)

}).then(function(result) { // (**)

alert(result); // 1

return result * 2;

}).then(function(result) { // (***)  

alert(result); // 2

return result * 2;

}).then(function(result) {

alert(result); // 4

return result * 2;

});

adding many then don't means chaining, they don't forward the result to the next thing, they act as independent promise

 let promise = new Promise(function(resolve, reject) {

setTimeout(() => resolve(1), 1000);

});

promise.then(function(result) {

alert(result); // 1

return result * 2;

});

promise.then(function(result) {

alert(result); // 1

return result * 2;

});

promise.then(function(result) {

alert(result); // 1

return result * 2;

});

Promise Apis:

There are 6 static methods of Promise class:

Promise.all:
we want to execute many promise in parallel and want to wait until all of them are resolved

Promise.all([

new Promise(resolve => setTimeout(() => resolve(1), 3000)), // 1

new Promise(resolve => setTimeout(() => resolve(2), 2000)), // 2

new Promise(resolve => setTimeout(() => resolve(3), 1000)) // 3

]).then(alert);

the order of results will be same as the order of promise, even if the promise got rejected

A popular trick is to map the array of job and pass them into promise all

let urls = [

'https://api.github.com/users/kmj-007',

'https://api.github.com/users/remy',

'https://api.github.com/users/jeresig'

];

// map every url to the promise of the fetch

let requests = urls.map(url => fetch(url));

// Promise.all waits until all jobs are resolved
Promise.all(requests)
.then(responses => responses.forEach(
response => alert(`${response.url}: ${response.status}`)
));

important thing to keep in mind about promise.all

If any of the promises is rejected, the promise returned by Promise.all immediately rejects with that error.

example of error:

Promise.all([

new Promise((resolve, reject) => setTimeout(() => resolve(1), 1000)),

new Promise((resolve, reject) => setTimeout(() => reject(new Error("Whoops!")), 2000)),

new Promise((resolve, reject) => setTimeout(() => resolve(3), 3000))

]).catch(alert); // Error: Whoops!

also promise.all not cancel and does nothing about existing ongoing request, even if promise already got rejected, it will simply no longer watch them or care them, promise got rejected then it's over, promise.all no longer cares about any thing.

in short,

Promise.all = all or nothing

Promise.allSettled:
this one is recently added, old browsers don't support, they need polyfills and transpilors

Promise.allSettled just waits for all promises to settle, regardless of the result.

{status:"fulfilled", value:result} for successful responses,
{status:"rejected", reason:error} for errors.

let urls = [
'https://api.github.com/users/iliakan',
'https://api.github.com/users/remy',
'https://no-such-url'
];



Promise.allSettled(urls.map(url => fetch(url)))
.then(results => {
//
[
  {
    status: 'fulfilled',
    value: /* response data */
  },
  {
    status: 'fulfilled',
    value: /* response data */
  },
  {
    status: 'rejected',
    reason: /* error object */
  }
]
//
}
});
});

polyfill implementation for incompatible browsers:

if (!Promise.allSettled) {
  const rejectHandler = reason => ({ status: 'rejected', reason });
  const resolveHandler = value => ({ status: 'fulfilled', value });

  Promise.allSettled = function(promises) {
    const convertedPromises = promises.map(p => Promise.resolve(p).then(resolveHandler, rejectHandler));
    return Promise.all(convertedPromises);
  };
}

Promise.race:
it's like fastest finger first, whichever gets resolved first wins

Promise.race([

new Promise((resolve, reject) => setTimeout(() => resolve(1), 1000)),

new Promise((resolve, reject) => setTimeout(() => reject(new Error("Whoops!")), 2000)),

new Promise((resolve, reject) => setTimeout(() => resolve(3), 3000))

]).then(alert); // 1

promise.any:
promise any just does what it tells, if any of the promise gets fulfilled, then it returns the result, then ignores the others, if all get rejected, then it combines the error(AggregateError) and returns it

Promise.any([

new Promise((resolve, reject) => setTimeout(() => reject(new Error("Whoops!")), 1000)),

new Promise((resolve, reject) => setTimeout(() => resolve(1), 2000)),

new Promise((resolve, reject) => setTimeout(() => resolve(3), 3000))

]).then(alert); // 1

Promise.resolve/reject:
they are not used that much, because of async/await

Promise.resolve(value) – makes a resolved promise with the given value.
Promise.reject(error) – makes a rejected promise with the given error.

Promise handlers .then/.catch/.finally are always asynchronous.

what does that mean:


let promise = Promise.resolve(); //even if the promise resolved first or immediately

promise.then(() => alert("promise done!"));

alert("code finished"); // this alert shows first

put it more simply, when a promise is ready, its .then/catch/finally handlers are put into the queue; they are not executed yet. When the JavaScript engine becomes free from the current code, it takes a task from the queue and executes it.

That’s why “code finished” in the example above shows first.

internal implementation if you are interested of apis of promise:

Promise.race:

function race(promises) {
  // Return a new promise
  return new Promise((resolve, reject) => {
    promises.forEach(promise => {
      // Wait for any promise to settle (resolve or reject)
      //adding .then to all the promisese to which we are iterating
      promise.then(
      function(result) { resolve(result)},
      function(error) { reject(error) }
      );
    });
  });
}

Promise.all:

function all(promises) {
  return new Promise((resolve, reject) => {
    const result = [];

    if (promises.length === 0) {
      resolve(result);
      return;
    }

    let countPending = promises.length;

    promises.forEach((promise, index) => {
      Promise.resolve(promise).then((value) => {
        result[index] = value;
        countPending--;
        if (countPending === 0)resolve(result); 
      },
      function error(error){
        reject(error);
      });
    });

  });
}

Promise.allSettled:

function allSettled(promises) {
  if (promises.length === 0) {
    return Promise.resolve([]); // Resolve immediately if input array is empty
  }

  return new Promise((resolve, reject) => {
    let result = [];
    let settledCount = 0; // Initialize the settled count

    promises.forEach((promise, idx) => {
      Promise.resolve(promise).then(
        value => {
          result[idx] = { status: 'fulfilled', value };
        },
        reason => {
          result[idx] = { status: 'rejected', reason };
        }
      ).finally(() => {
        settledCount++; // Increment the settled count in finally rather than doing it twice in value or reason function blocks
        // Check if all promises have settled
        if (settledCount === promises.length) {
          resolve(result);
        }
      });
    });
  });
}

Promise.any:

 function any(promises) {
  // your code here
  if (!promises.length) throw new AggregateError("No Promise passed");

  return new Promise((resolve, reject) => {
    let settledCount = 0, let errors = [];
    promises.forEach((promise, index) => promise
      .then(data => resolve(data))
      .catch(err => {
        errors[index] = err;
        settledCount++;
        if (settledCount === promises.length) reject(new AggregateError(
          'No Promise in Promise.any was resolved',
          errors
        ))
      })
    )
  })
}

That's it for now, will add more things when i learn more!

Chat with your Github Repo using llama_index and chainlit

Karan Janthe — Tue, 16 Apr 2024 22:40:38 +0000

By the end of this blog, you will be able to chat and ask question to your codebase, with just 100 lines of code!

Before we begin, let's first address two big elephants in the room!

llama_index

in simpler terms, LlamaIndex is tool that makes our life easy by working as a bridge between our custom data and large language models(LLM), processing manually different types of data which can be structured, unstructured, extracting them, processing them, all headache is gone with LlamaIndex, there are so many already community plugins ready on llamaHub, no need to reinvent the wheel again, thanks to huge open source community!

chainlit

chainlit is open source project that makes it very easy to build frontend interfaces like chatgpt and other features that are required for conversational ai app, so we can focus on the core part and don't need to worry about basic things, and it is dead simple to work with

let's begin!!

basic setup:

openai_key = os.getenv("OPENAI_API_KEY")
GITHUB_TOKEN = os.getenv("GITHUB_TOKEN")

If you don’t have a Github Token, go here

we need to create github client which will fetch the repo's content and download them for us:

import os
from llama_index.readers.github import GithubRepositoryReader
from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    load_index_from_storage,
    PromptHelper,
    ServiceContext,
)
from langchain_community.embeddings import HuggingFaceEmbeddings
import chainlit as cl
from llama_index.readers.github import GithubRepositoryReader, GithubClient
from langchain_openai import ChatOpenAI


openai_key = os.getenv("OPENAI_API_KEY")
GITHUB_TOKEN = os.getenv("GITHUB_TOKEN")
github_client = GithubClient(GITHUB_TOKEN)

owner = "KMJ-007"
repo = "RustyScheme"
branch = "main"

# We currently don't support images or videos, so we filter them out to reduce the load time.
IGNORE_MEDIA_EXTENSIONS = [
    ".png",
    ".jpg",
    ".jpeg",
    ".gif",
    ".mp4",
    ".mov",
    ".avi",
    ".PNG",
    ".JPG",
    ".JPEG",
    ".GIF",
    ".MP4",
    ".MOV",
    ".AVI",
]

ignore_directories = []

don't worry about other imports, they will make sense in a minute

here comes our first function which's purpose is to load index for our github repo, and store them in dir, so we don't need to do it every time we run the project, we can use vector database or other methods to store vectors also

@cl.cache
def load_index():
    try:
        # rebuild the storage context
        storage_context = StorageContext.from_defaults(
            persist_dir="./rusty_scheme_vectors"
        )
        # load the index
        index = load_index_from_storage(
            storage_context=storage_context
        )
    except Exception as e:
        print(f"Error loading index: {str(e)}")
        # storage not found; create a new index
        print("Storage not found, creating a new index")
        documents = GithubRepositoryReader(
            github_client=github_client,
            owner=owner,
            repo=repo,
            filter_directories=(
                ignore_directories,
                GithubRepositoryReader.FilterType.EXCLUDE,
            )
            if ignore_directories
            else None,
            filter_file_extensions=(
                IGNORE_MEDIA_EXTENSIONS,
                GithubRepositoryReader.FilterType.EXCLUDE,
            ),
            verbose=True,
            concurrent_requests=768,
        ).load_data(
            branch=branch,
        )
        llm = ChatOpenAI(
            temperature=0,
            model_name="gpt-3.5-turbo",
            streaming=True,
            openai_api_key=openai_key,
        )
        prompt_helper = PromptHelper(
            context_window=4095,
            num_output=256,
            chunk_overlap_ratio=0.1,
            chunk_size_limit=None,
        )
        embed_model = HuggingFaceEmbeddings()
        service_context = ServiceContext.from_defaults(
            llm=llm,
            embed_model=embed_model,
            prompt_helper=prompt_helper,
        )
        index = VectorStoreIndex.from_documents(
            documents, service_context=service_context
        )
        index.storage_context.persist("./rusty_scheme_vectors")
    return index

let's understand what above code does,

@cl.cache

just tells chainlit to cache this function,

    try:
        # rebuild the storage context
        storage_context = StorageContext.from_defaults(
            persist_dir="./rusty_scheme_vectors"
        )
        # load the index
        index = load_index_from_storage(
            storage_context=storage_context
        )

code is two blocks, try and except, try blocks first tries to find the vectors from the dir, and if it finds then it loads them

if try block fails to load the index from the dir, except code block gets called

documents = GithubRepositoryReader(
            github_client=github_client,
            owner=owner,
            repo=repo,
            filter_directories=(
                ignore_directories,
                GithubRepositoryReader.FilterType.EXCLUDE,
            )
            if ignore_directories
            else None,
            filter_file_extensions=(
                IGNORE_MEDIA_EXTENSIONS,
                GithubRepositoryReader.FilterType.EXCLUDE,
            ),
            verbose=True,
            concurrent_requests=768,
        ).load_data(
            branch=branch,
        )

above code fetchs the repo we mentioned and loads them as documents to process

llm = ChatOpenAI(
            temperature=0,
            model_name="gpt-3.5-turbo",
            streaming=True,
            openai_api_key=openai_key,
        )
        prompt_helper = PromptHelper(
            context_window=4095,
            num_output=256,
            chunk_overlap_ratio=0.1,
            chunk_size_limit=None,
        )

after loading documents, we tell about the llm which we want to use in ChatOpenAi, prompt_helper makes it easy to provide context window and putting constraint.

embed_model = HuggingFaceEmbeddings()
        service_context = ServiceContext.from_defaults(
            llm=llm,
            embed_model=embed_model,
            # node_parser=node_parser,
            prompt_helper=prompt_helper,
        )
        index = VectorStoreIndex.from_documents(
            documents, service_context=service_context
        )
        index.storage_context.persist("./rusty_scheme_vectors")

HuggingFaceEmbeddings is a function that we use for converting our documents to vector which is called embedding, you can use any embedding model from huggingface, it will load the model on your local computer and create embeddings(you can use external api/service to create embeddings), then we just pass this to context and create index and store them into folder so we can reuse them and don't need to recalculate it.

chat_start

we need to load index on the chat start, this is exactly what the following function does:

@cl.on_chat_start
async def on_chat_start():
    index = load_index()

    query_engine = index.as_query_engine(
        streaming=True,
        similarity_top_k=2,
    )

    cl.user_session.set("query_engine", query_engine)

    await cl.Message(author="Assistant", content="Hello! How may I help you?").send()

we load the index, make a query engine and in that we pass similarity top k to 2, it basically tells how top similar answers get from, then we set the query engine in user session and send the assistant message

on_message

when user sends a message, following function gets called:

@cl.on_message
async def main(message: cl.Message):
    query_engine = cl.user_session.get("query_engine")
    msg = cl.Message(content="", author="Assistant")
    print(message.content)
    res = query_engine.query(message.content)

    for text in res.response_gen:
        token = text
        await msg.stream_token(token)

    await msg.send()

so the above code gets the query_engine, we pass the user asked a question to query engine, which under the hood calls the query engine which uses index to get similar embeddings and give it to LLM to get structured normal language output which normal user can understand.

whole code(app.py):

import os
from llama_index.readers.github import GithubRepositoryReader
from llama_index.core import (
    VectorStoreIndex,
    StorageContext,
    load_index_from_storage,
    PromptHelper,
    ServiceContext,
)
from langchain_community.embeddings import HuggingFaceEmbeddings
import chainlit as cl
from llama_index.readers.github import GithubRepositoryReader, GithubClient
from langchain_openai import ChatOpenAI


openai_key = os.getenv("OPENAI_API_KEY")
GITHUB_TOKEN = os.getenv("GITHUB_TOKEN")
github_client = GithubClient(GITHUB_TOKEN)

owner = "KMJ-007"
repo = "RustyScheme"
branch = "main"

# We currently don't support images or videos, so we filter them out to reduce the load time.
IGNORE_MEDIA_EXTENSIONS = [
    ".png",
    ".jpg",
    ".jpeg",
    ".gif",
    ".mp4",
    ".mov",
    ".avi",
    ".PNG",
    ".JPG",
    ".JPEG",
    ".GIF",
    ".MP4",
    ".MOV",
    ".AVI",
]

ignore_directories = []


# function to load the index from storage or create a new one
# todo: add the support for vector database
@cl.cache
def load_index():
    try:
        # rebuild the storage context
        storage_context = StorageContext.from_defaults(
            persist_dir="./rusty_scheme_vectors"
        )
        # load the index
        index = load_index_from_storage(
            storage_context=storage_context
        )
    except Exception as e:
        print(f"Error loading index: {str(e)}")
        # storage not found; create a new index
        print("Storage not found, creating a new index")
        documents = GithubRepositoryReader(
            github_client=github_client,
            owner=owner,
            repo=repo,
            filter_directories=(
                ignore_directories,
                GithubRepositoryReader.FilterType.EXCLUDE,
            )
            if ignore_directories
            else None,
            filter_file_extensions=(
                IGNORE_MEDIA_EXTENSIONS,
                GithubRepositoryReader.FilterType.EXCLUDE,
            ),
            verbose=True,
            concurrent_requests=768,
        ).load_data(
            branch=branch,
        )
        llm = ChatOpenAI(
            temperature=0,
            model_name="gpt-3.5-turbo",
            streaming=True,
            openai_api_key=openai_key,
        )
        prompt_helper = PromptHelper(
            context_window=4095,
            num_output=256,
            chunk_overlap_ratio=0.1,
            chunk_size_limit=None,
        )
        embed_model = HuggingFaceEmbeddings()
        service_context = ServiceContext.from_defaults(
            llm=llm,
            embed_model=embed_model,
            # node_parser=node_parser,
            prompt_helper=prompt_helper,
        )
        index = VectorStoreIndex.from_documents(
            documents, service_context=service_context
        )
        index.storage_context.persist("./rusty_scheme_vectors")
    return index


@cl.on_chat_start
async def on_chat_start():
    index = load_index()

    query_engine = index.as_query_engine(
        streaming=True,
        similarity_top_k=2,
    )

    cl.user_session.set("query_engine", query_engine)

    await cl.Message(author="Assistant", content="Hello! How may I help you?").send()


@cl.on_message
async def main(message: cl.Message):
    query_engine = cl.user_session.get("query_engine")
    msg = cl.Message(content="", author="Assistant")
    print(message.content)
    res = query_engine.query(message.content)

    for text in res.response_gen:
        token = text
        await msg.stream_token(token)

    await msg.send()

Run the app by:

chainlit run app.py -w

it will open the frontend, wait for index to load and ask the question

that's it, hope i explained things clearly, we can integrate vector database, and external embeddings via api and many more things, i wanted to go in more depth, we are still playing on top of lot of abstractions, maybe in future posts i will go in more depth, but for this post i wanted to keep things in scope to explain for beginners and show how easy it is.

How we improved our client-side PDF generation by 5x

Karan Janthe — Sun, 17 Mar 2024 00:43:06 +0000

At Smartagent, we faced a challenge that might sound familiar to many developers: our client-side PDF generation feature was struggling under the weight of massive user demands. Our customers love customization—they want to generate various reports, tweaking everything from formats to the data columns they see. To give you an idea, here's a glimpse into the level of customization we provide:

Using react-pdf, we crafted a solution that allowed users to manipulate their reports with an impressive degree of flexibility. But, as data grew (imagine trying to cram an entire financial year's worth of invoices, up to 22,000 rows, into one PDF), our solution began to falter, especially on older PCs with limited resources.

The "Aw, Snap!" error became a common sight, frustrating users trying to generate their reports:

The Quest for a Solution

Initially, we experimented with the idea of shifting PDF generation to the server side. However, this approach would require us to rewrite our entire PDF generation logic a daunting task given the time-sensitive nature of the issue.

So, we turned our attention back to the client side, aiming for a solution that could gracefully handle large datasets even on low-spec PCs.

Diving into Experiments

Our journey led us to explore two promising technologies: WebAssembly (Wasm) and web workers.

Experiment 1: Wasm-PDF

We combined Wasm with Faker to create a demo for high-performance PDF generation. The results were eye-opening: generating a PDF with 1,000 rows took just 200ms, a task that previously took over a minute and often led to crashes. For 22,000 rows, the process took a mere 6 seconds.

Experiment 2: Web-worker

We discovered that our application was crashing because PDF generation was blocking the main thread. By leveraging web workers, we could offload this task, allowing the app to remain responsive.

With web workers, generating 1,000 rows took only 3 seconds.

What We Learned

Our exploration revealed valuable insights:

Performance vs. Practicality: While Wasm offered incredible speed, it required a significant time to rewrite our PDF logic. Given our tight schedule and the learning curve associated with Wasm and Rust, we had to consider alternative solutions.
The Power of Web Workers: Web workers provided a practical way to improve performance without needing to rewrite our existing logic. By moving PDF generation off the main thread, we could prevent crashes and improve user experience, even if the process took a bit longer.

Our Choice and Why

In the end, we opted for the web worker solution. It addressed our immediate needs without sacrificing the user experience. Adding a progress bar gave users visual feedback, reassuring them that their reports were on their way.

This decision was a compromise, shaped by our constraints and our commitment to delivering a reliable service. However, our exploration of Wasm opened up exciting possibilities for future optimizations.

Observability & Monitoring in Nodejs using Signoz

Karan Janthe — Wed, 21 Feb 2024 20:16:51 +0000

Hey there! If you've ever been deep into coding your Node.js app, tweaking and tuning every bit, only to find your server still can't handle the load, you're not alone. It's like everything looks perfect, but something's still off, and it's super annoying, right?

Spotting the Issue
So, how do you figure out what's wrong? Well, there's this saying that goes, "You can't improve what you don't measure." That's spot on because the first step to solving our mystery is to shine a light on those hidden problems.

When we hit this wall, we tried a bunch of stuff:

flame graphs: These are super cool for seeing where your CPU time is going. It's like a detective tool for your functions.
Memory Snapshots:
When your app is guzzling memory like there's no tomorrow, memory snapshots are your best bet. Chrome's dev tools are pretty handy for this.

But, what if you want to keep a constant eye on your app?
Like having a health check that tells you everything's cool or if it's time to dive back in and fix stuff.

Discovering a Lifesaver: OpenTelemetry and SigNoz
That's when we stumbled upon OpenTelemetry. It's this awesome open-source project that helps you track all sorts of data from your app. Think of it as a Swiss Army knife for telemetry data (stuff like logs, metrics, traces).

Now, while OpenTelemetry is like having the right tools, we still needed a place to see all this info easily. That's where SigNoz comes in. Imagine finding a treasure chest in your backyard. That's how we felt when we found SigNoz. It's this platform where you can see everything going on with your app - logs, metrics, traces, you name it.

The best part? We set it all up for free using Oracle Cloud's VM.

in this blog, i am mostly going to cover how to host signoz for free and better monitor and observe your backend.

Create free oracle cloud account and create free VM and get access to the VM using ssh:

this tutorial will help you through each and every setup, i choose ubunutu as os, which is easy to work with and have lot of online guides when you are stuck!

now you have free VM with, now we are going to use docker to take care of everything for us, if you are not that much familiar with docker, it provides secure and reliable environment for applications to run,

just run following command on the VM in the dir in which you want to keep all the signoz related data

git clone -b main https://github.com/SigNoz/signoz.git && cd signoz/deploy/

run install script, it will install docker and required things as own it's own

./install.sh

this will look something like this when it is fetching the images from docker hub:

it will take some time, so relax, and wait till it is complete

When it's complete, you can now visit the dashboard by going to your

<IP>:3301

it will look something like this, where you first need to create account

you can see there are lot of features which will be helpful when monitoring and for observability, you can also invite other team members to the same dashboard,

now let's configure opentelemtry in our nodejs

// tracing.js
'use strict'
const process = require('process');
const opentelemetry = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const signoz_url="https://localhost"

const exporterOptions = {
   url: `https:localhost:4318/v1/traces`
   }

const traceExporter = new OTLPTraceExporter(exporterOptions);
const sdk = new opentelemetry.NodeSDK({
   traceExporter,
   instrumentations: [getNodeAutoInstrumentations()],
   resource: new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: 'node_app'
      })
});

// initialize the SDK and register with the OpenTelemetry API
// this enables the API to record telemetry

sdk.start()

// gracefully shut down the SDK on process exit
process.on('SIGTERM', () => {
   sdk.shutdown()
 .then(() => console.log('Tracing terminated'))
 .catch((error) => console.log('Error terminating tracing', error))
 .finally(() => process.exit(0));
 });

run your node backend with this script using below command

node -r ./tracing.js index.js

after usage, you will be able to see your service in the dashboard, you can see traces also, logs, and filter them and create your own custom dashboard if you want, and create custom alerts also.

Wrapping Up

Fixing performance issues in Node.js can feel like finding a needle in a haystack. But with the right tools - OpenTelemetry for gathering data and SigNoz for making sense of it all. It's like having a friend who's got your back, showing you exactly where the problem lies so you can fix it and move on.

if you have a question or just want to discuss things

reach me out on twitter

How to do Open Source Contribution?

Karan Janthe — Thu, 25 Jan 2024 03:38:45 +0000

If you're new to tech or just getting started, you'll often hear the advice to contribute to open source. It's great advice advice that everyone should consider. There are many reasons why this advice is commonly given and many people ask me how should i start contributing to open source, so this is my answer.

Firstly, we all build upon each other's work in the tech industry. We stand on the shoulders of giants and without this collective knowledge-sharing, we wouldn't be able to dream of the possibility of developing AGI. Sharing knowledge is the force that moves the world and humanity forward.

This "pay it forward" mentality is needed more than ever to make more world a better place That's why people suggest contributing to open source. Many companies appreciate this ethos, and some even have a strong open-source culture. You can stand out among other people by highlighting your open-source contribution.

However, I recently had an interesting conversation with a friend who asked how to contribute to open source, I told him, to pick an interesting project to which you want to contribute, which you like, find an issue, create PR, be polite, and don't hesitate to ask for help. But I wanted to know where he learned about this because he doesn't even have any kind of IDE installed on his PC, for beginners I usually recommend focusing on skills because without them you can't to any meaningful contribution.

I found out, that many YouTubers and DevRel on the internet are promoting open source, but the issue is they are doing it in the wrong way. They make it sound like a silver bullet: "Contribute to open source and get a job.". This is sometimes misleading because if you just make PR and mention it on your resume, recruiters aren't easily fooled. Sometimes they check what you contributed, and if it's just a small readme change and you mention that 3 times in your resume, that creates a negative impression.

Open source is a philosophy in itself. There's nothing wrong with using open-source tools and libraries without actively contributing. there's no pressure or obligation to participate, it's about what you believe in, there are many reasons why one thinks to contribute to open source,

learning new skills, facing new challenges
helping/supporting other people/project
building a personal portfolio, resume...
personal satisfaction

Many people want to contribute to open source but don't know the starting point. I suggest going about doing your normal work, and opportunities will naturally arise, If you encounter a bug in some library that you are using for your project, check if there is an open issue, If not, create one and create PR to fix it, This way, your contribution will be more meaningful and relevant to your experiences.

Rather than actively seeking out open source projects to contribute to, let it happen organically, Trying to find projects to contribute to can feel like having a hammer and seeing every problem as a nail. This isn't always the case.

Start with meaningful open-source contributions that you can relate to, especially if you're just starting. Don't chase after opportunities, they will come to you naturally. As you gain a deeper understanding of fundamentals, you'll be better equipped to contribute to more advanced projects and contribute to more impactful ideas.

PS: this blog was originally published here