Forem: Naineel Soyantar

A Date with Daytona: Exploring AstroJS and Sanity CMS

Naineel Soyantar — Sat, 28 Dec 2024 07:16:50 +0000

Recently, while working on my blog application, I discovered Daytona—a powerful development environment manager—through the ongoing Quest 23 on quira.sh. In this blog, we’ll explore how AstroJS, Sanity, and Daytona significantly enhanced the development process of this blog application.

What is Daytona?

Daytona is an open-source Development Environment Manager (DEM) designed to streamline the management and deployment of development environments, known as Workspaces. These Workspaces are built on Docker containers and are fully compatible with the DevContainer standard.

Key Features of Daytona:

Modular Architecture: Enables compatibility with configuration standards such as Dockerfiles, Docker Compose, Nix, and Devfile.
Simplified Deployment: Deploy fully functional development environments using a single command, while Daytona manages all underlying complexities for you.
DevContainers Foundation: Utilizes the versatile DevContainers standard for defining and managing cloud-based development environments.

DevContainers and Configuration

The DevContainers standard allows you to configure your development environment with ease using a devcontainer.json file placed under the .devcontainer folder on your repository.

Benefits of DevContainers:

DevContainers allow you to define custom, feature-rich, and optimized development environments tailored to your specific needs. It also supports Dockerfile and Docker Compose configurations for seamless integration.

For more details, explore DevContainers and their capabilities.

Example of a `devcontainer.json`

Here’s an example configuration for a project using AstroJS and Sanity:

{
    "name": "Astro + Sanity Dev Container",
    "image": "mcr.microsoft.com/devcontainers/typescript-node:latest",
    "features": {
        "ghcr.io/devcontainers/features/node:1": {
            "version": "20",
            "pnpmVersion": "9.12.2"
        },
        "ghcr.io/devcontainers/features/git:1": {}
    },
    "forwardPorts": [4321, 3333],
    "customizations": {
        "vscode": {
            "extensions": [
                "astro-build.astro-vscode",
                "esbenp.prettier-vscode",
                "dbaeumer.vscode-eslint",
                "eamodio.gitlens",
                "styled-components.vscode-styled-components",
                "ms-vscode.vscode-typescript-next"
            ],
            "settings": {
                "astro.enableAutoImport": true,
                "editor.formatOnSave": true,
                "typescript.tsdk": "/node_modules/typescript/lib",
                "eslint.format.enable": true,
                "files.eol": "\n",
                "prettier.requireConfig": true
            }
        }
    },
    "containerEnv": {
        "NODE_ENV": "development",
        "PNPM_HOME": "/tmp/pnpm-store"
    },
    "postCreateCommand": {
        "install-dependencies": "pnpm config set store-dir /tmp/pnpm-store && pnpm install",
        "env-file-create": "/bin/bash ./create-env-file.sh"
    },
    "portsAttributes": {
        "3333": { "label": "Sanity Studio" },
        "4321": { "label": "Astro Dev Port" }
    }
}

Key Sections:

image: Specifies the base image for the development environment.
features: Adds pre-configured tools or utilities to the container.
forwardPorts: Maps ports from the container to the host machine.
customizations: Specifies editor configurations, such as extensions, settings, and formatting preferences, particularly for VS Code.
containerEnv: Sets environment variables for the container.
postCreateCommand: Runs commands after the container is created (e.g., install dependencies).
portsAttributes: Labels for specific ports used in development.

For more configuration options, check the DevContainer JSON reference.

Prerequisites for Daytona

Before downloading Daytona, there are some prerequisites to be fulfilled:

Docker
Text Editor/IDE like VSCode, JetBrains, etc.
SSH Client, required for accessing remote servers

Download Daytona

Daytona is available for all major operating systems. Follow the installation guide to set it up.

Then use the following command, to start daytona:

daytona

Starting Daytona

For starting Daytona, you would need to start its server process either in daemon mode or in the current terminal session.

Note: Make sure the Docker Daemon is also running in the background.

Start the Daytona server process in Daemon Mode:

daytona server start

Note: Daytona Daemon mode is not supported on Windows machines. You would need to use the following command to start the server in your current terminal session:

daytona serve

Adding Your Git Providers

To connect your Git provider (e.g., GitHub, GitLab), use the following command:

daytona git-providers add

Follow the on-screen instructions to integrate your Git profiles. Once completed, you can access projects hosted on your Git repositories.

Setting up your IDE

Daytona allows the development environment to be run on any of the popular IDEs. Just run the below command and choose the IDE of your choice:

daytona ide

Using Daytona for Development

To utilize Daytona efficiently, you’ll need a project with a devcontainer.json file placed under a .devcontainer folder in your repository.

For ease of use, we will be using this project.

Commands to Get Started with Daytona:

Create a development container with granular control on starting out your environment like a specific project or trying out one of the sample projects:

   daytona create

Directly start a development container from a Git repository:

   daytona create https://github.com/daytonaio/sample-astro-blog

You will start seeing the logs of the development container in the console:

After waiting for a while, you will see a success message and a new window of your chosen IDE open, with an SSH connection to the development container already set up where you can start out developing your project like you would do on your native PC or laptop.

Developing with Daytona

We will follow the README.md of our sample AstroJS + Sanity Blog to start out developing with Daytona.

Next Steps: Generating Credentials for Sanity and Setting Up Environment Variables

Sign in to Sanity: Begin by logging into your Sanity account and creating a new project to get started.
Generate an API Token: Navigate to the API section in your Sanity project settings and create a new token. This will provide the key required for setting up the environment variables.

Once the API token is generated, use it to configure your project’s environment variables as described in the documentation.

Running the project

Now, run the below commands in a separate terminal windows:

To start our AstroJS application, run below command:

pnpm run dev

The AstroJS application is the frontend where we can read the blogs that we publish from the backend i.e. Sanity CMS!

To access our Sanity Studio, run below command:

pnpm run sanity-dev

The Sanity Studio is used for managing content for our AstroJS application. We can create, delete and manipulate the data for our AstroJS application right from the Sanity Studio.

Conclusion

With just one tool, we can set up a fully-fledged development environment that offers all the features of a native setup. Whether you're a seasoned developer or just starting out, Daytona's robust features and seamless integration with modern tools like AstroJS and Sanity are worth exploring.

My development journey with Daytona has been exceptional, and I’ve gained in-depth knowledge about DevContainers, Docker, and development environments in general.

Do check them out and star them – they are Open Source: Daytona!

Also, feel free to explore my sample project on daytona: AstroJS + Sanity Blog.

Enhancing Cyclops: Adding Cache Metrics to Prometheus – My First OSS Contribution

Naineel Soyantar — Tue, 30 Jul 2024 08:41:38 +0000

Hola Mis Amigos,

Recently, I learned about Cyclops - A Developer friendly Kubernetes tool.

What is Cyclops

Cyclops is a tool for managing Kubernetes (K8s) clusters and offers a GUI solution for developers to deploy and manage these clusters, making the management of containerized applications super easy! It also allows developers and DevOps teams to create custom Helm charts for deploying any application with specific requirements.

Prerequisites for using Cyclops

Cyclops is an easy-to-use tool but requires some additional knowledge about the technology it uses and the problem it solves.

Kubernetes - for container orchestration of your application.
Docker - for containerization of your application.
Minikube - for managing the local Kubernetes cluster.

Good First Issues on Cyclops GitHub

As a newcomer to Cyclops, I wanted to contribute and make a meaningful impact. While browsing their repository, I came across a "good-first-issues" that seemed like a perfect starting point. I followed the "CONTRIBUTING.md" to set up Cyclops on my localhost. After searching through the issues I found a good issue to work upon (this)

The Task

The task was to extend the current implementation of Prometheus monitor to include the cache metrics from the templateCache utilization.

I began by understanding the requirements and the codebase; I tried to find the starting point of the application and build an understanding of its workflow.

The ristretto.Cache was located in the internal > template > cache > inmemory.go file. The first step was to make the field Metrics to true while initializing the cache. This made sure the cache metrics were being generated.

The next step was to include the cache metrics in the existing monitor:

m := Monitor{
    ModulesDeployed: prometheus.NewGauge(prometheus.GaugeOpts{
        Name:      "modules_deployed",
        Help:      "No of modules Inc or Dec",
        Namespace: "cyclops",
    }),
}

I identified a few key metrics provided by ristretto and included them in the monitor as Gauge metrics. A Gauge represents a single numerical value that can go up and down arbitrarily, making it a good choice for the cache metrics.

The monitor now contained:

m := Monitor{
    ModulesDeployed: prometheus.NewGauge(prometheus.GaugeOpts{
        Name:      "modules_deployed",
        Help:      "No of modules Inc or Dec",
        Namespace: "cyclops",
    }),
    CacheHits: prometheus.NewGauge(prometheus.GaugeOpts{
        Name:      "cache_hits",
        Help:      "No of cache hits",
        Namespace: "cyclops",
    }),
    CacheMisses: prometheus.NewGauge(prometheus.GaugeOpts{
        Name:      "cache_misses",
        Help:      "No of cache misses",
        Namespace: "cyclops",
    }),
    CacheKeysAdded: prometheus.NewGauge(prometheus.GaugeOpts{
        Name:      "cache_keys_added",
        Help:      "No of cache keys added",
        Namespace: "cyclops",
    }),
    CacheCostAdded: prometheus.NewGauge(prometheus.GaugeOpts{
        Name:      "cache_cost_added",
        Help:      "No of cache cost added",
        Namespace: "cyclops",
    }),
    CacheKeysEvicted: prometheus.NewGauge(prometheus.GaugeOpts{
        Name:      "cache_keys_evicted",
        Help:      "No of cache keys evicted",
        Namespace: "cyclops",
    }),
    CacheCostEvicted: prometheus.NewGauge(prometheus.GaugeOpts{
        Name:      "cache_cost_evicted",
        Help:      "No of cache cost evicted",
        Namespace: "cyclops",
    }),
}

Then, made a metrics list array and looped it to register the metrics to the prometheus registry for storing metrics during controller runtime.

metricsList := []prometheus.Collector{
    m.ModulesDeployed,
    m.CacheHits,
    m.CacheMisses,
    m.CacheKeysAdded,
    m.CacheCostAdded,
    m.CacheKeysEvicted,
    m.CacheCostEvicted,
}

for _, metric := range metricsList {
    if err := metrics.Registry.Register(metric); err != nil {
        logger.Error(err, "unable to connect prometheus")
        return Monitor{}, err
    }
}

Next, I created an update method on Prometheus monitor which will update the metric of the cache in the Prometheus monitor, allowing us to get the latest metrics in the /metrics route.

func (m *Monitor) UpdateCacheMetrics(cache *ristretto.Cache) {
    cacheMetrics := cache.Metrics

    m.CacheHits.Set(float64(cacheMetrics.Hits()))
    m.CacheMisses.Set(float64(cacheMetrics.Misses()))
    m.CacheKeysAdded.Set(float64(cacheMetrics.KeysAdded()))
    m.CacheCostAdded.Set(float64(cacheMetrics.CostAdded()))
    m.CacheKeysEvicted.Set(float64(cacheMetrics.KeysEvicted()))
    m.CacheCostEvicted.Set(float64(cacheMetrics.CostEvicted()))
}

Now as per the requirements, the only thing remaining was to write a cron job which will update the cache metric in the Prometheus monitor periodically for a fixed time delta. If you don't know about a cron job - here's an amazing article to understand this interesting topic.

func StartCacheMetricsUpdater(m *Monitor, cache *ristretto.Cache, interval time.Duration, logger logr.Logger) {
    go func() {
        ticker := time.NewTicker(interval)
        defer ticker.Stop()

        logger.Info("starting cache metrics updater")

        for range ticker.C {
            m.UpdateCacheMetrics(cache)
        }
    }()
}

The StartCacheMetricsUpdater function takes a pointer to the Prometheus monitor, a pointer to the cache, a time interval, and a logger. Inside the function, a goroutine is started which runs at every tick of the interval, calling the UpdateCacheMetrics method to update the cache metrics in the Prometheus monitor.

Finally, in the main.go file, I called the StartCacheMetricsUpdater function to kickstart the updating of cache metrics to the Prometheus monitor.

Before raising a PR for this feature, I thoroughly tested the code to ensure that it doesn't break any existing functionality. This might be the most important step while contributing.

Conclusion

Contributing to the Cyclops project by adding cache metrics to Prometheus was a rewarding experience. It allowed me to deepen my understanding of both Cyclops and Prometheus while making an improvement to the tool. I learned the importance of thoroughly understanding the existing codebase, strictly following the contribution guidelines, and testing changes rigorously.

And I'll sign-off now and will see you ...

Analyzing logs - the lame way

Naineel Soyantar — Mon, 08 Jul 2024 06:06:40 +0000

In the previous blog post, we saw how to create a production-ready logging system. In this blog post, we will see how to analyze the logs generated by the system.

🫱🫱🫱🫱Let's build a production-ready logger using Winston🫲🫲🫲🫲

While many advanced tools are available for log analysis, such as the ELK stack and Splunk, we will see how to do it the lame way using JavaScript🥱🥱.

The Log Format

The logs generated by the logger are in the following format:

Understanding The Log Format

The log format includes the following fields:

timestamp: The time at which the log was generated
level: The log level (DEBUG, INFO, WARN, ERROR)
Source IP: The IP address of the source of the client making the request
kind of request: The kind of request made by the client (GET, POST, PUT, DELETE)
response time: The time taken by the server to respond to the request
path: The path of the request made by the client
status code: The status code of the response sent by the server
response time: The time taken by the server to respond to the request
total time: The total time taken by the server to process the request
Remote IP: The IP address of the client making the request

The Log Analyzer

Laying The Basis For The Log Analyzer

First, create a regular expression to parse the log format.

const logLevelPattern = /(?<date>\d{4}-\d{2}-\d{2})\s+(?<time>\d{2}:\d{2}:\d{2})\s+(?<level>\w+):\s+(?<client_ip>::ffff:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|::1)\s+(?<method>\w+)\s+(?<path>\/\S*)\s+(?<status>\d{3})\s+(?<response_time_1>\d+\.\d{3})\s+ms\s+(?<response_time_2>\d+\.\d{3})\s+ms\s+(?<other_ip>::ffff:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|::1)/g;

The regular expression seems daunting at first sight, but it's not that hard to understand. It consists of named groups for each field in the log format. The named groups are enclosed in (?<name>pattern).

Using named groups makes it easier to extract the fields from the log line. For example, to extract the level field from the log line, we can use the following code:

const logLine = '2024-07-04 10:42:09 error: ::ffff:72.200.26.133 GET /api/v1/resource 201 0.082 ms 0.177 ms ::ffff:72.200.26.133';

const match = logLevelPattern.matchAll(logLine);

for (const m of match) {
    //We are interested in the groups from the match object because we have used named groups in the regex.
    console.log(m.groups);
    console.log(m.groups.level);
}

The match object will have this kind of output:

The groups property of the match object will have the named groups extracted from the log line. We can access the level field using m.groups.level.

Analyzing The Logs Using The Log Analyzer

Now that we have the regular expression to parse the log format, let's create a log analyzer that reads the logs from a file and analyzes them.

import { EventEmitter } from 'events';
import fs from 'fs';
import path from 'path';

class LogAnalyzer extends EventEmitter {
    // initialize the log analyzer with the log file stream and the log level pattern and initialize the objects to store the analysis results
    constructor(logFileStream, logLevelPattern) {
        super();
        this.logFileStream = logFileStream;
        this.logLevelPattern = logLevelPattern;
        this.time = {};
        this.paths = {};
        this.ips = {};
        this.responseTime = [];
        this.totalResponseTime = [];
        this.count = 0;
    }

    //method to start the analysis
    analyze() {
        this.logFileStream.on('ready', () => console.log('================START======================='));
        this.logFileStream.on('data', this.processChunk.bind(this));
        this.logFileStream.on('end', this.finishAnalysis.bind(this));
    }

    //method to process each chunk of data read from the log file
    processChunk(chunk) {
        console.log('Processing chunk:', this.count);

        this.logFileStream.pause();

        const output = chunk.toString().matchAll(this.logLevelPattern);

        for (const match of output) {
            //extract the named groups from the match object
            const { groups } = match;
            //update the objects with the extracted fields
            this.updateObjects(groups);
        }
        this.count++;

        this.logFileStream.resume();
    }

    //method to update the objects with the extracted fields
    updateObjects(groups) {
        const hourKey = groups.time.split(':')[0] + ':00';
        this.time[hourKey] = (this.time[hourKey] || 0) + 1;
        this.updateObject(this.paths, groups.path);
        this.updateObject(this.ips, groups.client_ip);
        this.responseTime.push(parseFloat(groups.response_time_1));
        this.totalResponseTime.push(parseFloat(groups.response_time_2));
    }

    //method to update an object with a key
    updateObject(obj, key) {
        obj[key] = (obj[key] || 0) + 1;
    }

    //method to finish the analysis
    finishAnalysis() {
        console.log('================END=========================');
        console.log("Let's perform some analysis on the log file");

        this.emit('analysisComplete', this.getAnalysisResults());
    }

    //method to sort an object based on the count of the keys
    sortingObject(obj, max = true) {
        //sorting is based on the count of the keys to get the most used or least used keys
        if (max) {
            return Object.entries(obj).sort((a, b) => b[1] - a[1]);
        } else {
            return Object.entries(obj).sort((a, b) => a[1] - b[1]);
        }
    }

    //method to get the analysis results
    getAnalysisResults() {
        return {
            timeDistribution: this.sortingObject(this.time, true).slice(0, 4),
            mostUsedPathDistribution: this.sortingObject(this.paths, true).slice(0, 4),
            leastUsedPathDistribution: this.sortingObject(this.paths, false).slice(0, 4),
            ipDistribution: this.sortingObject(this.ips, true).slice(0, 4),
            avgResponseTime: this.average(this.responseTime).toFixed(5),
            avgTotalResponseTime: this.average(this.totalResponseTime).toFixed(5),
        };
    }

    average(arr) {
        return arr.reduce((a, b) => a + b, 0) / arr.length;
    }
}

// Usage:
const logLevelPattern = /(?<date>\d{4}-\d{2}-\d{2})\s+(?<time>\d{2}:\d{2}:\d{2})\s+(?<level>\w+):\s+(?<client_ip>::ffff:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|::1)\s+(?<method>\w+)\s+(?<path>\/\S*)\s+(?<status>\d{3})\s+(?<response_time_1>\d+\.\d{3})\s+ms\s+(?<response_time_2>\d+\.\d{3})\s+ms\s+(?<other_ip>::ffff:\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|::1)/g;

const logFileStream = fs.createReadStream(path.resolve('./logs/combined/generated_logs.log'), {
    encoding: 'utf-8',
    highWaterMark: 10 * 1024, // 10KB
});

const analyzer = new LogAnalyzer(logFileStream, logLevelPattern);

analyzer.on('analysisComplete', (results) => {
    console.log('Analysis complete. Results:', results);
});

analyzer.analyze();

Workflow:

The logFileStream is created using fs.createReadStream to read the log file in chunks.
The LogAnalyzer class is created with the logFileStream and logLevelPattern as arguments.
The analyze method is called on the LogAnalyzer instance to start the analysis.
The processChunk method is called when a chunk of data is read from the log file. It processes the chunk and extracts the fields from the log line using the logLevelPattern.
In the processChunk method, updateObjects is called to update the objects with the extracted fields.
The finishAnalysis method is called when the analysis is complete. It emits an event analysisComplete with the analysis results.
The sortingObject method is used to sort the objects based on the count of the keys.
The getAnalysisResults method returns the analysis results.
The average method calculates the average of an array of numbers.
The analysisComplete event is listened to and the analysis results are logged to the console.
The analyze method is called to start the analysis.

The LogAnalyzer class reads the log file in chunks and processes each chunk to extract the fields from the log line. It updates the objects with the extracted fields and calculates the average response time and total response time. Finally, it emits an event analysisComplete with the analysis results.

The analysis results include the time distribution, most used path distribution, least used path distribution, IP distribution, average response time, and average total response time.

Output:

This way we can analyze the logs generated by the logger using the log analyzer in an absolutely lame way😂😂

You can find the complete code for the log analyzer here

Generating random logs

Let's address the elephant in the room: not everyone has a log file from a real production system to analyze. So, let's generate some random logs using the faker library.

fakerjs documentation

import fs from 'fs';
import { faker } from '@faker-js/faker';

// Sample data
const methods = ["GET", "POST", "PUT", "DELETE"];
const paths = ["/", "/api/v1/resource", "/login", "/register"];
const ips = ['45.94.188.156', '238.249.31.148', '91.113.1.90', '113.232.207.105', '96.129.247.250', '105.171.179.234', '144.42.125.14', '109.111.74.178', '72.200.26.133', '83.65.134.149'];
const statusCodes = [200, 201, 400, 404, 500];
const levels = ["info", "error", "warn", "debug"];

// Function to generate a random timestamp
const randomTimestamp = (start, end) => {
    const startDate = start.getTime();
    const endDate = end.getTime();
    return new Date(startDate + Math.random() * (endDate - startDate));
};

// Generate log entries
const logEntries = [];
const startDate = new Date(2024, 6, 4, 10, 41, 0); // Months are 0-based in JavaScript
const endDate = new Date(2024, 6, 4, 10, 45, 0);

for (let i = 0; i < 1000; i++) {  // Generate 500 log entries

    //Generating random log entries
    const timestamp = randomTimestamp(startDate, endDate);
    const dateStr = timestamp.toISOString().replace('T', ' ').substring(0, 19);
    const level = faker.helpers.arrayElement(levels);
    const clientIp = faker.helpers.arrayElement(ips);
    const method = faker.helpers.arrayElement(methods);
    const path = faker.helpers.arrayElement(paths);
    const status = faker.helpers.arrayElement(statusCodes);
    const responseTime1 = (Math.random() * (0.1 - 0.05) + 0.05).toFixed(3);
    const responseTime2 = (Math.random() * (0.3 - 0.1) + 0.1).toFixed(3);
    const otherIp = faker.helpers.arrayElement(ips);

    logEntries.push(`${dateStr} ${level}: ::ffff:${clientIp} ${method} ${path} ${status} ${responseTime1} ms ${responseTime2} ms ::ffff:${otherIp}`);
}

// Write to log file
fs.writeFileSync('./logs/combined/generated_logs.log', logEntries.join('\n'));

console.log("Log file generated successfully.");

This script generates 1000 random log entries and writes them to a log file generated_logs.log. The log entries are generated with random timestamps, log levels, client IPs, methods, paths, status codes, response times, and other IPs.

You can find the complete code for generating random logs here

Conclusion

In this blog post, we saw how to analyze the logs generated by the logger using the log analyzer. We created a log analyzer that reads the logs from a file, parses the log format, and analyzes the logs. We also generated random logs using the faker library to test the log analyzer.

While the log analyzer is a simple way to analyze logs, it is not suitable for large log files or real-time log analysis. For large log files or real-time log analysis, you can use advanced log analysis tools like ELK stack, Splunk, etc. But for small log files or testing purposes, the log analyzer is a good starting point.

So... Thanks for reading this blog post. I hope you enjoyed it. If you have any questions or feedback, feel free to leave a comment below. And I'll...

PS:

I typically write these articles in the TIL form, sharing the things I learn during my daily work or afterward. I aim to post once or twice a week with all the things I have learned in the past week.

Let's build a production-ready logger using Winston

Naineel Soyantar — Mon, 01 Jul 2024 13:13:07 +0000

Hi guys, long time no see...

Time to learn about another awesome concept: logging

What does Wikipedia say about logging?

In computing, logging is the act of keeping a log of events that occur in a computer system, such as problems, errors, or just information on current operations. A message or log entry is recorded for each such event. These log messages can then be used to monitor and understand the operation of the system, to debug problems, or during an audit. (Logging (computing))

So going by the definition, logging simply means keeping a log of events that occur in a computer system. This can be used to monitor and understand the operation of the system, to debug problems, or during an audit. Many types of logs can be generated by a system, but we will be focusing on Server Logs in this article.

What Are Server Logs?

Server Logs are logs that are generated by a server. These logs are used to monitor and understand the server's operation. Typical examples of server logs are:

Page Requests Logs
Error Logs
Access Logs
Information Logs, etc.

Logs help in understanding the behavior of the server, how many requests are being made to the server, what kind of requests are being made, what kind of errors are being generated, etc. Understanding the patterns in the logs can help optimize the server's performance and debug issues.

Winston is one such tool that can be used to generate logs in a Node.js application. Its various modifications can be used to generate logs in different formats and at different levels.

Installing Winston

To install Winston, run the following command:

npm install winston

Using Winston

Let's create the simplest logger using Winston:

const winston = require('winston');

const logger = winston.createLogger({
    level: 'info',
    format: winston.format.json(),

    transports: [
        new winston.transports.Console()
    ]
});

// now we can use the logger to log messages as we want
logger.info('Hello, Winston!');
logger.error('An error occurred!');
logger.warn('Warning: This is a warning!');

Output:

Understanding the components of the Winston Logger

We have created a simple logger using Winston. Let's understand the components of the logger:

Levels - Levels are used to specify the severity of the log. Winston is pre-configured with the following levels:

error: Severity 0 - This is an error message.
warn: Severity 1 - This is a warning message.
info: Severity 2 - This is an informational message.
http: Severity 3 - This is an HTTP log.
verbose: Severity 4 - This is a verbose log.
debug: Severity 5 - This is a debug message.
silly: Severity 6 - This is a silly message.

The above example of the logger is configured to log messages at the info, warn, and error levels. We can also create custom levels for the Winston logger.

We can use the syslog levels as well. The syslog levels can be used from winston.config.syslog.levels. The default levels are from winston.config.npm.levels.

import winston from 'winston';

const customLevels = {
    levels: {
        error: 0,
        warn: 1,
        info: 2,
    },
    colors: {
        error: 'red',
        warn: 'yellow',
        info: 'green',
    }
};

// add colors to the custom levels
winston.addColors(customLevels.colors);

// define a logger with custom levels
const logger = winston.createLogger({
    levels: customLevels.levels,
    format: winston.format.combine(
        winston.format.colorize(),
        winston.format.simple()
    ),

    transports: [
        new winston.transports.Console()
    ]
});

// now we can use the logger to log messages as we want
logger.error('Hello, Winston!');
logger.warn('An error occurred!');
logger.info('Warning: This is a warning!');

Output:

Here, we have defined custom levels for the logger and added colors to the custom levels. We have to inform Winston about the custom levels and colors using the addColors method.

Formats - Formats are used to specify the format of the log message. Winston is pre-configured with the following formats:

JSON - This format logs the message in JSON format.
simple - This format logs the message in a simple format.
colorize - This format logs the message in color.
printf - This format logs the message in a custom format.
timestamp - This format logs the message with a timestamp.
combine - This format combines multiple formats.

We can also create custom formats for the Winston logger.

import winston from 'winston';

// define a logger with custom format
const logger = winston.createLogger({
    level: 'info',
    format: winston.format.combine(
        winston.format.colorize(),
        winston.format.timestamp({
            format: 'YYYY-MM-DD HH:mm:ss',
        }),
        winston.format.printf(({ level, message, timestamp }) => {
            return `${timestamp} [${level}]: ${message}`;
        })
    ),
    transports: [
        new winston.transports.Console()
    ]
});

// now we can use the logger to log messages as we want
logger.info('Hello, Winston!');
logger.error('An error occurred!');
logger.warn('Warning: This is a warning!');

Output:

Here, we have defined a custom format for the logger. The custom format logs the message in the format [timestamp] [level]: message

Transports - Transports are used to specify the destination of the log message. Winston is pre-configured with the following transports:
- Console - This transport logs the message to the console. Mostly used for development purposes.
- File - This transport logs the message to a file. We can specify the filename, the maximum size of the file, and the maximum number of files.
- HTTP - This transport logs the message to another server using HTTP. Mostly used for remote logging services.
- Stream - This transport logs the message to a NodeJS writable stream.
Daily Rotate File - Daily Rotate File is an important way to prevent the accumulation of unnecessary old log files that are of no use in a live server and can be replaced automatically, ensuring the latest log files are available always and the redundant old files are removed/rotated to save the space on the live server

Let's install the Daily Rotate File package:

  npm install winston-daily-rotate-file

  import winston from 'winston';
  import 'winston-daily-rotate-file';

  const fileFormat = winston.format.combine(
    winston.format.timestamp({
        format: 'YYYY-MM-DD HH:mm:ss',
    }),
    winston.format.printf(({ timestamp, level, message }) => {
        return `${timestamp} [${level}]: ${message}`;
    })
  );

  const combinedFileTransport = new winston.transports.DailyRotateFile({
    filename: '%DATE%_combined.log',
    datePattern: 'YYYY-MM-DD-HH',
    maxSize: '2m',
    dirname: './logs/combined',
    maxFiles: '14d',
    format: fileFormat
  });

Here, we have defined a Daily Rotate File transport for the logger. The Daily Rotate File transport logs the message to a file with the filename combined.log and rotates the file daily.

The maximum size of the file is 2m i.e. 2MB and the maximum number of days the files will be kept is 14d, where d is days. The %DATE% is a placeholder that will be replaced with the current date in the format YYYY-MM-DD-HH.

Designing a production-ready logger using Winston, Morgan and Daily Rotate File

Let's design a production-ready logger using Winston, Morgan and Daily Rotate File:

// winston.config.js
import winston from 'winston';
import 'winston-daily-rotate-file';

const fileFormat = winston.format.combine(
    winston.format.colorize(),
    winston.format.uncolorize(),
    winston.format.timestamp({
        format: 'YYYY-MM-DD HH:mm:ss',
    }),
    winston.format.prettyPrint({
        depth: 5
    }),
    winston.format.printf((info) => `${info.timestamp} ${info.level}: ${info.message}`),
);

const consoleFormat = winston.format.combine(
    winston.format.colorize(),
    winston.format.timestamp({
        format: 'YYYY-MM-DD HH:mm:ss',
    }),
    winston.format.prettyPrint({
        depth: 5
    }),
    winston.format.printf((info) => `${info.timestamp} ${info.level}: ${info.message}`),
);

const consoleTransport = new winston.transports.Console({
    format: consoleFormat,
});

const combinedFileTransport = new winston.transports.DailyRotateFile({
    filename: '%DATE%_combined.log',
    format: fileFormat, //format means - how the log should be formatted
    datePattern: 'YYYY-MM-DD-HH',
    maxSize: '2m',
    dirname: './logs/combined',
    maxFiles: '14d',
});

const errorFileTransport = new winston.transports.DailyRotateFile({
    filename: '%DATE%_error.log',
    level: 'error',
    format: fileFormat, //format means - how the log should be formatted
    datePattern: 'YYYY-MM-DD-HH',
    maxSize: '2m',
    dirname: './logs/errors',
    maxFiles: '14d',
});

const httpTransport = new winston.transports.Http({
    format: winston.format.json(),
    host: 'localhost',
    port: 4000,
    path: '/logs',
    ssl: false,
    batch: true,
    batchCount: 10,
    batchInterval: 10000,
});


const logger = winston.createLogger({
    levels: winston.config.syslog.levels,
    transports: [
        errorFileTransport,
        combinedFileTransport,
        consoleTransport,
        httpTransport,
    ],
});

export default logger;

Here, we have defined a logger using the Winston library with the following functionalities:

Console Transport - Logs the message to the console.
Daily Rotate File Transport - Logs the message to a file and rotates the file daily.
HTTP Transport - Logs the message to another server using HTTP.
Error File Transport - Logs the error message to a file and rotates the file daily.
Combined File Transport - Logs the message to a file and rotates the file daily.
Syslog Levels - We have used the syslog levels for the logger.

// index.js
import express from 'express';
import morgan from 'morgan';
import logger from './config/winston.config.js';

const app = express();

app.use(express.json());

app.use(morgan('dev', {
    stream: {
        write: message => {
            logger.info(message);
        }
    }
}));

app.get('/', (_req, res) => {
    return res.status(200).json({ message: 'Hello World' });
});

function handleFatalError(err) {
    logger.error(err);
    process.exit(1);
}

app.listen(process.env.PORT || 3000, () => {
    logger.info('Server is running on http://localhost:' + (process.env.PORT || 3000) + '/');

    process.on('unhandledRejection', handleFatalError);
    process.on('SIGTERM', handleFatalError);
});

To further enhance the functionality of our logger, we have integrated it with the Morgan library. Morgan is a middleware that logs the HTTP requests to the server. We have used the dev format of Morgan to log the HTTP requests to the server.

The stream option of Morgan is used to specify the destination of the log message. We have used the Winston logger to log the message to the console.

Output:

Files created by the Daily Rotate File Transport:
Logs generated in the console:
Logs generated by the HTTP Transport:

The following logs were generated by our logger's HTTP Transport. The logs were sent to another server running on localhost:4000/logs. The requests were made in batches as specified in the configuration.
Logs generated by the Error File Transport:
Logs generated by the Combined File Transport:

Conclusion

In this article, we have learned about the concept of logging and how to create a logger using Winston in a Node.js application. We have also learned about the various components of the Winston logger, such as levels, formats, and transports.

Try out the Winston logger setup in your Node.js applications and let me know your thoughts in the comments. If you have any questions or suggestions, feel free to ask.

So... that's it for today.

I would like to thank you 🫡🎉 for reading here and appreciate your patience. It would mean the world to me if you give feedback/suggestions/recommendations below.

See you in the next article, which will be published on......

PS:

I typically write these articles in the TIL form, sharing the things I learn during my daily work or afterward. I aim to post once or twice a week with all the things I have learned in the past week.

Iterators and Generators again... (asynchronous ones)

Naineel Soyantar — Sun, 17 Mar 2024 17:23:45 +0000

Last time around here, we learned about Iterators and Generators in JavaScript. We also gazed on why it is needed, how to utilize this functionality, and a tutorial example of implementing the iterators and generators.

If you are not familiar with this concepts then I would request you all to check out my previous article on Iterators and Generators.

Now that you are familiar with iterators and generators, let's look at asynchronous ones (I swear this won't be a huge one :D).

Async Iterators

In JavaScript, async iterators are an extended functionality of the normal iterators. To make an object an async iterable, it must implement the async iterable protocol, which is indicated by the presence of the Symbol.asyncIterator method. This method should return a next method which should return a Promise which will essentially fetch the data from the desired endpoint by performing asynchronous operation.

Let's take a look at coded example of the same to grasp the concept in better manner -

const someKindOfAsyncOperation = async (id) => {
    return new Promise((res, rej) => {
        setTimeout(() => {
            res(`Returning data of User: ${id}`)
        })
    })
}

const obj = {
    [Symbol.asyncIterator] : async function* (){
        for(let i = 0; i < 5; i++){
            const value = await someKindOfAsyncOperation(i);
            yield value;
        }
    }
};

(async () => {
  for await (const value of obj) {
    console.log(value); //Returning data of User: 0 ...
  }
})();

Here, someKindOfAsyncOperation is an asynchronous operation which loosely relates to the API calls for fetching the user details. We have used generators way of implementing the iterators because of its usability.

During the asynchronous operation, the function will be halted at the yield keyword until the for...of loop calls for the next method of the object obj. The code structure remain very similar, the major changes are the introduction of the asynchronous operations.

Async Generators

Async Generators are very useful for implementing Async Iterators. It removes the boilerplate code that we had to write every time implementing Iterators or Async Iterators.

Well for the spoilers, we have already made use of the Async Generator earlier in this article. So let's just dive right into it -

const someKindOfAsyncOperation = async (id) => {
    return new Promise((res, rej) => {
        setTimeout(() => {
            res(`Returning data of User: ${id}`)
        })
    })
};


async function* asyncGenerator() {
  for (let i = 0; i < 5; i++) {
    const value = await someKindOfAsyncOperation(i);
    yield value;
  }
};

(async () => {
  for await (const value of asyncGenerator()) {
    console.log(value); //Returning data of User: 0 ...
  }
})();

Here in the code, the IIFE will be called, which will call the next method obtained from the asyncGenerator() and it will call the asynchronous function someKindOfAsyncOperation which upon resolving gives the desired output. and then for await...of loop will call the next function again which will yield the next value from the asyncGenerator generator function and so on and so forth, all of the desired output will be obtained in the way we want.

For simplicity sake we have used a mock asynchronous operation, we can easily extend this kind of functionality to fetch sequential or custom data.

So... THANK YOU!!🎉🫡

I would like to thank you for reading till here and appreciate your patience. It would mean the world to me if you give feedbacks/suggestions/recommendations below.

PS:

I typically write these articles in the TIL form, sharing the things I learn during my daily work or afterward. I aim to post once or twice a week with all the things I have learned in the past week.

Iterators and Generators in Javascript🔬

Naineel Soyantar — Sat, 02 Mar 2024 14:25:16 +0000

While programming in JavaScript, you must have wondered about how do the iterable types in JavaScript like Strings, Arrays, Sets and, Maps know what will be the next element of their object. Also, why if essentially all the different iterable data types in JavaScript are of type Object (except String) but a primitive Object is not iterable? We will learn all about this today.

Iterators

Iterators are those objects that implement the iterator_protocol along with their other properties. Now you must be wondering, what is iterator_protocol?

MDN defines it as -

Iterator protocol defines a standard way to produce a sequence of values (either finite or infinite), and potentially a return value when all values have been generated

To simplify it further, it defines a way for an object to produce values that may or may not be sequential from that object. Iterator protocol states that an object must implement a method @@iterator via a constant Symbol.iterator.

Now, what is this `Symbol.iterator` and what it should return?

Symbol.iterator is a constant that basically tells JavaScript's for...of loop that an object is iterable. The iterator_protocol states that Symbol.iterator constant must return an object with property named next. This next method must return an object having properties -

done - [Boolean] - denoting if the iteration has been finished or not.
value - [any] - value to be returned for this iteration. [Optional, if done - true]

Also, MDN mentions this also -

In practice, neither property is strictly required; if an object without either property is returned, it's effectively equivalent to { done: false, value: undefined }.
If an iterator returns a result with done: true, any subsequent calls to next() are expected to return done: true as well, although this is not enforced on the language level.

Now enough of theory - let's see some live examples of built-in iterators

let arr = [1,2,3]
let map = new Map([[1, 'a'], [2, 'b'], [3, 'c']])
let str = "abcdefgh"

console.log(arr[Symbol.iterator]()) // Object [Array Iterator] {}
console.log(map[Symbol.iterator]()) // [Map Entries] { [ 1, 'a' ], [ 2, 'b' ], [ 3, 'c' ] }
console.log(str[Symbol.iterator]()) // Object [String Iterator] {}

console.log(arr[Symbol.iterator]().next()) // { value: 1, done: false }
console.log(map[Symbol.iterator]().next()) // { value: [ 1, 'a' ], done: false }
console.log(str[Symbol.iterator]().next()) // { value: 'a', done: false }

As we can see in the output of a property Symbol.iterator - it returns an iterator object with next method on it (these are built-in and were initialized while initializing the built-in objects), calling which returns an object with the first value of the data defined earlier.

Now, lets build some custom Iterators as well-

const obj = {
    0: "a",
    1: "b",
    2: "c",
    3: "d",

    [Symbol.iterator]: () => {
        let start = 0;
        return {
            next: () => {
                if(obj[start] != undefined){
                    return {
                        value: obj[start++],
                        done: false
                    }
                }else{
                    return {
                        done: true
                    }
                }
            }
        }
    }
}

console.log(obj[Symbol.iterator]()) // { next: [Function: next] }

for(let elem of obj){
    console.log(elem) // a b c d
}

Above code defines a custom object with properties that are generally defined in array and a Symbol.iterator object which returns a object method next and some custom logic to generate the elements.

Generators

Custom iterators for objects sounds very honey-nectorish but the implementation of it is similar to honey bee's sting. To simplify custom iterators, generator functions (syntax - function*(){}) are utilized. These functions are a special type that, upon invocation, do not execute their entire function body. Instead, they halt execution only until the yield keyword is encountered. At that point, execution pauses, and the function returns an object with properties defined by the iterator protocol. -

done - [Boolean] - denoting if the iteration has been finished or not.
value - [any] - value to be returned for this iteration. [Optional, if done - true] Upon calling next again, the execution of the function will resume from the previous yield keyword until it reaches the next yield.

Let's code some examples to become familiar with generator functions -

function* generatorFunc(){
    yield 1;
    yield 2;
    yield 3;
    yield 4;
}

//calling this function returns new generators
const iter = generatorFunc(); // Object [Generator] {}
console.log(iter.next()) // { value: 1, done: false }
console.log(iter.next()) // { value: 2, done: false }
console.log(iter.next()) // { value: 3, done: false }
console.log(iter.next()) // { value: 4, done: false }
console.log(iter.next()) // { value: undefined, done: true }

Now, let's build a custom object iterator using generator function-

const obj = {
    0: "a",
    1: "b",
    2: "c",
    3: "d",

    [Symbol.iterator]: function* (){
        let start = 0;
        while(true){
            if(obj[start] == undefined){
                break;
            }

            yield obj[start++];
        }
    }
}

for(let elem of obj){
    console.log(elem); // a b c d
}

Now, if we compare the implementation of Symbol.iterator in both objects we created, you can clearly observe the disparity in the amount of boilerplate code required to construct an iterator manually versus using generator functions. Also if we talk about usability of generator functions on its own, then it will be in my idolo Lord Shikamaru's words-

There are some advanced utilities of generator functions, you check it out on MDN Docs - Advanced Generators

For the next iterations, we will look at topics like -

asynchronous iterators and asynchronous generators in javascript
authentication and authorization workflow

THANK YOU!!🎉🫡

I would like to thank you for reading till here and appreciate your patience. It would mean the world to me if you give feedbacks/suggestions/recommendations below.

PS:

I typically write these articles in the TIL form, sharing those things I learn during my daily work or after that. I aim to post once or twice a week with all things I have learned in the past week.

Forem: Naineel Soyantar

A Date with Daytona: Exploring AstroJS and Sanity CMS

What is Daytona?

Key Features of Daytona:

DevContainers and Configuration

Benefits of DevContainers:

Example of a devcontainer.json

Key Sections:

Prerequisites for Daytona

Download Daytona

Starting Daytona

Adding Your Git Providers

Setting up your IDE

Using Daytona for Development

Commands to Get Started with Daytona:

Developing with Daytona

Next Steps: Generating Credentials for Sanity and Setting Up Environment Variables

Running the project

Conclusion

Enhancing Cyclops: Adding Cache Metrics to Prometheus – My First OSS Contribution

What is Cyclops

Prerequisites for using Cyclops

Good First Issues on Cyclops GitHub

The Task

Conclusion

Analyzing logs - the lame way

The Log Format

Understanding The Log Format

The Log Analyzer

Laying The Basis For The Log Analyzer

Analyzing The Logs Using The Log Analyzer

Generating random logs

Conclusion

Let's build a production-ready logger using Winston

What Are Server Logs?

Installing Winston

Using Winston

Understanding the components of the Winston Logger

Designing a production-ready logger using Winston, Morgan and Daily Rotate File

Conclusion

Iterators and Generators again... (asynchronous ones)

Async Iterators

Async Generators

Iterators and Generators in Javascript🔬

Iterators

Now, what is this Symbol.iterator and what it should return?

Generators

THANK YOU!!🎉🫡

Example of a `devcontainer.json`

Now, what is this `Symbol.iterator` and what it should return?