Forem: Desmond Obisi

Chronos Synapse is a telemetry-first job analytics platform that turns your scheduled/triggered code into real-time operational insight — without running code on the server.

SDK-first ingestion: a lightweight Node SDK (chronos-synapse-sdk) that registers jobs and streams execution telemetry (status, duration, errors, stdout/stderr, code snippet/language, versions).
Backend ingestion service: Fastify API that stores job definitions and execution events in Redis Stack and emits real-time updates via Socket.IO.
Read-only dashboard: a Next.js 14 app for visualizing jobs, executions, and analytics with live updates and user-scoped stats.

Why this approach? Many teams already run jobs in their own apps. Chronos keeps the execution where it belongs and specializes in visibility, metrics, and real-time insights.

Key capabilities:

Deterministic job IDs and schedules (cron or one-time runAt) with server-driven job:trigger events for recurring jobs.
Rich telemetry capture out of the box (error stack, stdout/stderr, snippet+language, app and job versions) with batching, backoff, and retries.
Real-time dashboard updates powered by Redis Pub/Sub → Socket.IO.

Demo

Screenshots (replace with your links):

Jobs list with “Last run” indicators
Execution details with inline logs & code snippet
Analytics with success/failed stacked bars + total executions line

Quick SDK example:

import ChronosRunner from 'chronos-synapse-sdk';

const runner = new ChronosRunner({
 apiKey: process.env.CHRONOS_API_KEY!,
 // endpoint defaults to http://localhost:3001
 captureConsole: true,
});

await runner['client'].registerJobs([
 {
  id: 'job:daily-report',
  name: 'Daily Report',
  schedule: '0 * * * *',
  runMode: 'recurring',
 },
 {
  id: 'job:launch-once',
  name: 'One-time Launch',
  schedule: '',
  runMode: 'once',
  runAt: '2025-09-01T12:00:00Z',
 },
]);

runner.register('job:daily-report', async () => {
 // your work
});
runner.register('job:launch-once', async () => {
 // runs once based on runAt or first cron match
});

runner.start();

How I Used Redis 8

Chronos Synapse leans on Redis Stack as the system of record and real-time transport — far beyond caching. We combine multiple modules/features:

RedisJSON (primary store)
- Jobs and executions are stored as JSON documents. Execution JSON includes rich telemetry fields (stdout/stderr, errorType/stack, code snippet/language, versions, labels/metadata).
RediSearch (fast querying & analytics)
- Two indices: idx:jobs and idx:executions.
- Executions index uses a TAG field for jobId (normalized, hyphens removed) so we can query with @jobId:{id1|id2|...} per user scope.
- We sort/filter by startedAt and aggregate client-side for charts and daily/weekly/monthly bucketing.
RedisTimeSeries (metrics and trends)
- Time series keys for counters and duration distributions: ts:jobs:executions, ts:jobs:success, ts:jobs:failed, ts:jobs:duration_ms.
- Writes use TS.INCRBY (counters) and TS.ADD ... ON_DUPLICATE LAST (durations) with DUPLICATE_POLICY=LAST to avoid collisions.
- This enables a fast, low-overhead trend API for the analytics page.
Pub/Sub (realtime fan-out)
- Ingestion publishes a lightweight execution:ingested event.
- A managed Redis subscriber relays events to Socket.IO (with keepalive PING every 20s, auto-reinit on failure).
- Dashboard receives instant updates for “Recent Executions” and stats.
Operational hardening (Redis client)
- connectTimeout, keepAlive, noDelay, and a reconnect strategy that avoids 0ms delay on first retry.
- Regular PING health checks, and defer to re-init subscriber on ping failure.

Patterns beyond cache:

RedisJSON + RediSearch = document DB + search for job/execution browsing & scoped analytics.
RedisTimeSeries = metrics store suitable for charts and time-windowed KPIs.
Pub/Sub = real-time event bus for dashboard/SDK triggers.

Data model & queries (sketch)

Job JSON (id, name, schedule, runMode, userId, org/app, tags, timestamps)
Execution JSON (id, jobId TAG, status, started/finished, duration, exitCode, stdout, stderr, snippet/language, versions)
Search examples:
- FT.SEARCH idx:executions "@jobId:{<id>|<id2>}" SORTBY startedAt DESC LIMIT 0 1000
- User-scoped analytics derived by intersecting a user’s job IDs and time window.

Scheduling & triggers

A small server-side scheduler scans enabled jobs every 30s, parses minute-level cron, and emits job:trigger events (per minute de-dupe via a Redis key). For one-time jobs, we either respect runAt (single fire) or fire on the first cron match.

Team: Solo submission (credit: @desmondsanctity )

Thanks for reading and for organizing the challenge!

Copilot vs. Cody: All you need to know

Desmond Obisi — Wed, 11 Dec 2024 21:23:50 +0000

The advancement of artificial intelligence (AI) has impacted the software industry, especially with the introduction of AI coding assistants. A McKinsey study showed that developers using AI tools perform 20%-50% faster on average than those not using it. There has also been a debate about whether these AI coding assistant tools can truly increase or decrease developer's performance, function, productivity ratio, and overall experience.

As a programmer, I want to be productive and accomplish tasks faster with the help of these AI tools. I also want to use the one that provides the overall best experience regarding integrations, how secure it is, if the code suggestions are of great quality, and how easily I can extend its features and capabilities. This article will look at two of the most used AI coding tools from a developer's point of view.

While this article isn’t part of this debate, it will highlight the core features, strengths, weaknesses, benchmarks, etc., of these widely used AI coding assistant tools—Sourcegraph Cody and GitHub Copilot—to lead us to an objective conclusion on which is more reliable. Let’s get into the breakdown and comparison of these tools.

Overview of Copilot vs. Cody

In this section, you’ll learn the background of both tools, highlighting their core features and functionalities to help you understand them and their uses.

What is GitHub Copilot?
GitHub Copilot was officially launched in June 2021, resulting from a collaboration between GitHub and OpenAI. Powered by OpenAI’s Codex model, Copilot was designed to help developers write code faster by providing real-time suggestions and auto-completions.

It was released as a plugin on the JetBrains marketplace on October 29, 2021. On June 21, 2022, GitHub announced that Copilot was out of "technical preview" and is available as a subscription-based service for individual developers after one year of beta testing from developers who got early access.

Core features of Copilot

Real-time code suggestions: Copilot offers contextual code suggestions as developers type, including entire functions and complex code snippets.
GitHub Copilot chat: Copilot now offers a chat interface where you can run prompts and generate things like unit tests, code snippets, etc.
Multi-language support: It supports many programming languages, from popular ones like Python, JavaScript, and TypeScript to less common languages.
Natural language to code: Copilot can convert natural language comments into actual code, which is particularly useful for prototyping and quick development tasks.
Learning from open source: The AI-powered tool is trained on billions of lines of code from public repositories, making it adept at generating relevant code snippets.

What is Sourcegraph Cody?
Cody, introduced by Sourcegraph, is an AI-powered coding assistant designed to use advanced search and codebase context to help you understand, write, and fix code faster. Launched in 2023, Cody aims to provide deeper context and more accurate code suggestions, particularly for complex and large-scale projects.

Unlike Copilot, which is heavily geared toward individual developers, Cody is built with a focus on team collaboration and enterprise-level projects.

Core features of Cody

Contextual code intelligence: Cody is designed to understand the broader context of the codebase, making it more effective than many AI coding assistants for large projects with interconnected code files.
Multi-file awareness: Cody can analyze and provide suggestions based on multiple files within a project, ensuring its suggestions align with the entire codebase's structure.
Code search integration: Cody integrates with Sourcegraph's powerful code search capabilities, allowing you to quickly navigate and find relevant code snippets within their projects.
Custom commands: Cody has an option for creating custom commands that you can use to write large language model (LLM)- powered instructions. These instructions automate boring or repetitive tasks in their day-to-day work, such as documentation, comment writing, etc.
Multi LLM support: Cody is designed to use different LLMs for code generation, error fixing, optimization, etc. Cody also allows developers to add their LLM keys, and the usage will be on their account.
Multi-language support: Cody supports many programming languages, from the popular ones like Python, Javascript, Rust, C, and C++ to the less common ones like Julia or Zig.
Enterprise-grade security: Cody is tailored for enterprise environments, with enhanced security and privacy features, making it suitable for organizations with strict data protection requirements.

Now that we’ve explored the histories and the core features of the two AI assistant tools, let’s compare them in terms of functions, integrations, security, and developer experience.

Comparison between Sourcegraph Cody and GitHub Copilot

This comparison will evaluate how Sourcegraph Cody and GitHub Copilot perform against the chosen criteria, reflecting the expectations developers have when using these tools. The table below will provide an overview, helping to determine which tool stands out as the best option.

Cost comparison
Cody: Cody is also available on a subscription basis, with pricing significantly better than its counterpart, Copilot. It also has a forever free tier, unlike the 30-day free trial from Copilot, allowing individual developers to have a thorough experience before purchasing.

Copilot: Copilot is available on a subscription basis, with individual and team plans. The pricing is generally affordable for individual developers, but the costs can add up for larger teams.

Ease of setup and use
Cody: Setting up Cody requires developers to install the extension from the supported IDEs marketplace, log in with their Sourcegraph account, and start using the tool. Cody can also be accessed on the web. The user interface is clean and integrates seamlessly with supported IDEs, providing an additional chat window interface and shortcut keys for handy commands.

Copilot: Setting up Copilot is relatively straightforward, especially for developers already using Visual Studio Code or JetBrains IDEs. The process involves:

Installing the Copilot extension from the marketplace.
Logging in with a GitHub account.
Enabling the tool.

Once set up, Copilot provides real-time suggestions as the developer writes code. The user interface is intuitive, with suggestions appearing as semi-transparent text within the IDE, allowing developers to accept, modify, or reject them easily.

Supported IDEs and environment
Cody: Cody is designed to work best with IDEs that integrate with Sourcegraph’s ecosystem. It supports popular IDEs like the VS Code editor, some JetBrains IDEs like IntelliJ IDEA, PyCharm, etc., and Neovim, which is still in the experimental phase.

Copilot: Copilot is primarily designed for Visual Studio Code editor and Visual Studio IDE, but it also supports others like Neovim and JetBrains IDEs like IntelliJ IDEA, PyCharm, etc. This wide IDE support makes it accessible to a broad range of developers.

Compatibility with programming languages
Cody: Cody supports a wide range of programming languages, including JavaScript, TypeScript, PHP, Python, Java, C/C++, C#, Ruby, Go, SQL, Swift, Objective-C, Perl, Rust, Kotlin, Dart, etc., and shell scripting languages (like Bash, PowerShell).

Copilot: Copilot boasts compatibility with various programming languages, from mainstream languages like Python, JavaScript, and Ruby to more niche languages. This versatility makes it a go-to tool for developers working across different projects and languages.

LLM Integrations
Cody: Cody defaults to the Claude 3 model from Anthropic for its code generation, autocomplete, and chat features, but its strengths lie in its diverse LLM integration(Claude, Mistral, GPT, and Gemini), unlike Copilot, which solely relies on the GPT models. It also allows users to bring their own API key for the supported LLMs.

Copilot: Copilot stems from a joint effort between GitHub and OpenAI, so Copilot is limited to only the GPT models for its code generation, chat, and autocomplete feature.

Quality of code suggestions
Cody: Cody shines in providing high-quality code suggestions that consider the broader context of the project. Its multi-file awareness and deep codebase understanding enable it to offer more precise and contextually appropriate suggestions, particularly in large projects. It also works well with popular code hosting platforms like GitLab, Bitbucket, etc., with respect to code generation and context awareness.

Copilot: Copilot’s suggestions are generally accurate and contextually relevant, especially for common coding patterns and standard libraries. However, its effectiveness can diminish in more complex scenarios or when dealing with highly customized codebases or codebases whose host is not GitHub.

Context understanding and accuracy
Cody: Cody’s ability to analyze and understand the entire codebase gives it an edge in providing accurate suggestions that align with the project’s overall structure. This makes Cody particularly valuable in enterprise environments where code complexity and scale are significant considerations.

Copilot: While Copilot is good at understanding the immediate context, the difference here is that its suggestions can sometimes lack the depth needed for complex codebases, particularly when the code spans multiple files or involves intricate dependencies.

Speed and efficiency improvements
Cody: While Cody may not generate suggestions as instantaneously as Copilot, its focus on accuracy and context often results in fewer errors and less time spent correcting code. This can translate into significant efficiency gains for developers working on large, complex projects.

Copilot: Copilot is designed to boost developer productivity by providing quick suggestions and auto-completions. This is particularly beneficial in rapid development environments. One of its strengths is its speed in generating suggestions.

Data handling policies
Cody: Cody is built with enterprise users in mind and, as such, places a strong emphasis on security and privacy. Sourcegraph offers more control over how data is handled, including options for on-premises deployment, which can alleviate concerns about data exposure in cloud environments.

Copilot: Copilot has faced scrutiny regarding its data handling practices, particularly concerning the training data derived from public repositories. While GitHub has implemented measures to protect user privacy, concerns remain about how user data is used and whether Copilot’s suggestions might inadvertently reproduce copyrighted code.

Intellectual Property(IP) concerns and business implications
Cody: Cody’s approach to code suggestions, which focuses on understanding the developer’s existing codebase rather than drawing from a broad corpus of public code, reduces the risk of IP issues. This makes Cody a safer option for businesses concerned about the legal implications of AI-generated code.

Copilot: The potential for Copilot to suggest code that closely mirrors existing open-source code has raised IP concerns. Developers and organizations need to be cautious about the legal implications of using code suggestions that might be derived from copyrighted material.

Value for money
Cody: Cody offers better value for money because it supports free users with a good feature list. Paid users even get more value and access to a wider range of features like flexible LLM choice, tight security, and context filters, which prevent LLMs from reading sensitive files in the codebase.

Copilot: For individual developers or small teams working on diverse projects, Copilot offers great value for money, especially given its fast autocomplete, broad language support, and ease of use.

From comparing features, functions, and differences, Cody emerges at least in my opinion, the clear favorite and a better option. The next section outlines how we reached this conclusion.

Sourcegraph Cody's strengths over GitHub Copilot

The comparison table above considers the differences and similarities between GitHub Copilot and Sourcegraph Cody. Undoubtedly, Cody shows to be a better tool when we examine all the criteria. Here are some of the strengths of Cody over Copilot that are worthy of highlighting:

Superior code suggestions: Cody’s ability to use Code Search, another Sourcegraph tool under the hood, allows it to provide more accurate and contextually relevant code suggestions. It also has a feature for prompts and custom commands where you can tailor it to a particular prompt and reuse it in the codebase for code suggestions or related tasks.
Better context understanding: Cody’s multi-file awareness gives it a significant advantage in understanding the broader context of the project. It scans through open files in the editor and files in the same folder and can link a file anywhere in the codebase when using the chat feature.
Enhanced security and privacy: Cody’s focus on enterprise-grade security and privacy makes it a superior choice for organizations with strict data protection requirements. Admins on the Sourcegraph Enterprise instance can use the Cody Context Filters to determine which repositories Cody can use as the context in its requests to third-party LLMs. Other users can also use the Cody Ignore File, an experimental feature yet to set files they want Cody to ignore when picking context in the codebase.
Data handling and IP protection: Cody generates its suggestions by relying on the existing codebase and general LLM knowledge. It does not scan or use open-source codes or other people’s codes. This is a significant advantage for enterprises that must ensure compliance with IP laws and avoid potential legal disputes.
Cost-effectiveness: While Cody’s pricing is fairly lower than Copilot, it offers substantial value for large teams and enterprises and that is a good advantage for the tool.

Addressing common questions and concerns about Copilot

Users have asked these common questions, to know which is better for them between Copilot and other AI coding tools. This section will answer some of these questions from a user perspective to help you understand if Copilot is worth the bargain for your development workflow. Some of these questions are:

Is Copilot worth buying?
Copilot is worth considering for individual developers or small teams looking for an affordable AI-assisted coding tool. It provides quick suggestions and supports various programming languages, making it a versatile tool for various projects. However, its effectiveness can vary depending on the complexity of the codebase and the developer’s specific needs.

What are the disadvantages of Copilot?
The main disadvantages of Copilot include:

Limited context awareness: Copilot’s suggestions are primarily based on the immediate written code, which can lead to less accurate suggestions in complex projects.
IP concerns: The potential for Copilot to suggest code derived from open-source repositories raises legal and ethical concerns, particularly for businesses.
Security and privacy: As a cloud-based tool, Copilot may not meet the security requirements of all organizations, especially those handling sensitive data.

Is it OK to use Copilot?
Copilot is generally safe to use for individual developers and small teams, provided they know the potential IP risks and limitations in context understanding. However, organizations should carefully consider these factors, especially if they work with proprietary or sensitive code.

Conclusion

Both Cody and Copilot are powerful AI-assisted coding tools, each with pros and cons. Copilot is an excellent choice for individual developers and small teams looking for an affordable, easy-to-use tool with broad language support, while Cody goes above and beyond in context flexibility, better security and data compliance, access to different LLMs, etc.

I have explored both GitHub Copilot and Sourcegraph Cody and have seen that each tool has its strengths. After diving into both, I found Cody’s deeper context understanding, access to different LLMs, and a strong focus on security to be game-changers for projects of any size. If you’re curious, I’d recommend signing up for Cody and seeing how it fits into your workflow. However, if Copilot aligns better with your needs, it’s also a solid option. Both tools offer valuable features, so exploring what works best for you is worth exploring.

Athene AI - Voice Journaling Assistant

Desmond Obisi — Sun, 24 Nov 2024 16:00:17 +0000

This is a submission for the AssemblyAI Challenge : No More Monkey Business.

What I Built

The project Athene AI is a digital journal note powered by AssemblyAI's speech-to-text model. It allows user to record journal entries and it auto transcribes it, generate summaries, proper speech segmentation, does mood analysis and auto create tags for characterizing the note.

Demo

The app is live and can be used here:
Athene AI

The code is also available here:
Frontend
Backend

Video Demo

Journey

I figured that I would journal more if i can just talk to something and it gets transcribed to readable words, organized by tags and also have a bit of categorization that will help me remember what the mood or environment was when I entered that journal.

With LeMUR, when i transcribe the audio of the journal, I used the inbuilt summary method to get summary. I also created two task that is processed by LeMUR: to generate a title for the journal entry and to auto tag it based on categories it fall into.

Note: This submission also qualifies for the Sophisticated Speech-To-Text prompt. Using Universal-2, AssemblyAI’s latest and most accurate speech-to-text model, I built the transcription engine for the journal app. It processes the journal entries smoothly while maintaining a good user experience.

The whole engine runs on Cloudflare workers deployment with Hono web framework. It also uses KV for storage of the transcribed data.

Fullstack Serverless Starter with Hono.js, Cloudflare Workers, and Neon Postgres

Desmond Obisi — Sun, 01 Sep 2024 22:50:45 +0000

This is a submission for the Neon Open Source Starter Kit Challenge : Ultimate Starter Kit

My Kit

This is a starter project for a fullstack serverless application using Hono.js, Cloudflare Workers, and Neon Postgres. It is a simple CRUD application that allows you to create, read, update, and delete posts. It also has provisions for building user interfaces using HTML and CSS.

Link to Kit

This project is available on GitHub. If you find it helpful, star it and share it with others. Contributions are also accepted.

DesmondSanctity / neon-serverless-starter

This is a starter project for a fullstack serverless application using Hono.js, Cloudflare Workers and Neon Postgres. It is a simple CRUD application that allows you to create, read, update and delete posts. It also has provisions for building user interface using HTML and CSS.

Fullstack Serverless Starter with Hono.js, Cloudflare Workers and Neon Postgres.

Prerequisites

Node.js and npm installed
Cloudflare Workers account
Neon Postgres account and database

Project structure

The project structure is as follows:

---docs
    |-- schemas.ts
---prisma
    |-- schema.prisma
---public
    |-- images.*png,jpg,svg
    |-- favicon.ico
---src
    |-- configs
    |-- controllers
    |-- middlewares
    |-- queries
    |-- routes
    |-- validations
    |-- views
    |-- index.ts
---utils
    |-- constants.js
---tsconfig.json
---wrangler.toml
---package.json
---package-lock.json
---README.md

The docs folder contains the OpenAPI schemas for generating documentation for the application. It uses @hono/zod-openapi to generate the schemas. Only the schemas is documented in this starter.
The prisma folder contains the database schema and migration files.
…

View on GitHub

Your Journey

Building a fullstack serverless app the conventional way always does not pan out well. You have to make tweaks here and there everytime when you want to build something. To make it easy, I have created this starter project that has most of the things you need for your serverless application, saving you hours of configurations and setups.

I used Hono.js and Cloudflare workers to create and deploy the APIs, Neon for a serverless database, and integration of Workers AI so to show how easy it is to run AI models in a serverless application. The starter app also uses Twind via SSR to serve Tailwind CSS files, helping us to have a fullstack app with APIs and user interfaces in the same project.

What is included?

Authentication using JWT.
Database interaction using Prisma.
Middleware for authentication and logging.
Validation using Zod.
Secure Headers.
Error handling.
Deployment to Cloudflare.
Pagination for getAll endpoints.
Search functionality for getAll endpoints.
Sorting for getAll endpoints.
Filtering for getAll endpoints.
AI Bindings to use Workers AI.
Twind configuration that allows use of Tailwind CSS in Cloudflare Workers through SSR.
Serving of static assets.
Support for OpenAPI documentation.
Example user interfaces using HTML and CSS.

AI-powered market analysis for currency pairs with Hono.js, Cloudflare Workers and Twilio

Desmond Obisi — Sat, 22 Jun 2024 21:17:03 +0000

This is a submission for Twilio Challenge v24.06.12

What I Built

Nomisma is a cutting-edge AI-driven application that delivers real-time market analysis and updates on currency pairs directly to your WhatsApp. This innovative tool leverages the power of Hono.js, Twilio serverless functions, WhatsApp API, and Cloudflare Workers AI to provide users with timely and insightful financial information.

Key Features:

Real-Time Market Analysis: Receive the latest updates on currency pairs, including bid, ask, and mid prices. Detailed analysis to help you make informed decisions on currency trading.
WhatsApp Integration: Get all your updates and analysis sent directly to your WhatsApp for convenient and immediate access. Ensures you are always in the loop, no matter where you are.
User-Friendly Interface: Simple and intuitive UI for setting up your preferences and managing notifications. Built using Hono.js for a smooth and responsive user experience.
Serverless and Scalable: Utilizes Twilio's serverless functions for efficient and scalable communication handling. Cloudflare Workers AI ensures robust and fast processing of market data and analysis.

Use Cases:

Traders: Stay updated with the latest market trends and data for informed trading decisions.
Financial Analysts: Get regular insights into currency pairs to aid in market research and analysis.
Investors: Receive timely updates on currency pairs to monitor investments and market movements.

Demo

DesmondSanctity / tyche-nomisma

An AI tool that you can use to get currency pair update via Whatsapp. Built with Twilio and Cloudflare Workers AI.

tyche-nomisma

An AI tool that you can use to get currency pair update via Whatsapp. Built with Hono.js, Twilio(serverless functions & whatsapp) and Cloudflare Workers AI.

prerequisite to run project

Cloudflare account(with your terminal authenticated with wrangler CLI) Read more here
Cloudflare API KEY and ACCOUNT ID
Twilio account(with Twilio WhatsApp sandbox and phone number)
Tradermade API Key for live rates of currency pairs. See more here

step 1

set up a file called .dev.vars where all your environment variables will be added.
it will look like this:

TRADER_MADE_API = ""
CLOUDFLARE_API_KEY = ""
CLOUDFLARE_ACCOUNT_ID = ""
TWILIO_PHONE_NUMBER = ""

step 2 (run locally)

run the following commands to install dependencies and start the dev server respectively.

npm install
npm run dev

step 3 (deploy to prod)

you should store your secret to cloudflare workers for production use using the command below. where key is…

View on GitHub

You can find the live solution here - Nomisma

Twilio and AI

How It Works...

Setup:

Users sign up and provide their WhatsApp number for receiving notifications.
Select the currency pairs you are interested in and the intervals at which you want to receive updates.

Notification and Analysis:

The AI processes market data and generates insightful analysis for the selected currency pairs.
Notifications are sent to your WhatsApp, ensuring you have accurate information at your fingertips.

Technology Stack:

Hono.js: Provides a fast, flexible, and scalable web framework for building the application.
Twilio (Serverless Functions & WhatsApp API): Handles the communication and delivery of notifications via WhatsApp.
Cloudflare Workers AI: Ensures efficient data processing and AI-driven market analysis. I used the @cf/meta/llama-3-8b-instruct for the analysis generation.

Additional Prize Categories

This submission qualifies for:

Twilio Times Two: I used Twilio serverless function to host the messaging service to make it easily scalable when there are many users. I also used the WhatsApp API for the communication between the app and the users.
Impactful Innovators: This app is very innovative for getting deep market insight and analysis using AI and APIs that offer historical data. It can be used by traders, investors, and analysts to keep up to date with fundamental, technical, and trend analysis without having to do much work.

Faster Postgres Queries with Cloudflare Hyperdrive and Neon

Desmond Obisi — Wed, 07 Feb 2024 14:36:14 +0000

If you have a database that gets queried from multiple regions (e.g. via edge functions) and you want to improve overall performance, this guide may help. To demonstrate, we’ll use Neon, a fully managed serverless PostgreSQL, and Cloudflare Hyperdrive, a lightning-fast globally distributed database acceleration platform. When you put Hyperdrive between your database and globally distributed applications and functions that query it, Cloudflare will route and cache queries to reduce latency and improve overall performance.

This guide will walk you through each step of the process, from setting up Neon Postgres and Cloudflare Hyperdrive to writing custom API services and deploying your application. We'll also cover essential topics such as utilizing Wrangler CLI for Hyperdrive setup, learning how Hyperdrive makes queries faster, and benchmarking native Postgres queries against the Hyperdrive accelerated queries.

Prerequisites

A Cloudflare account.
Wrangler CLI installed on your local machine.
Your local machine should have Node.js and npm installed. Wrangler CLI requires a Node version of 16.13.0 or later to avoid permission issues.
A basic knowledge of SQL.

Setting up Neon Postgres

Setting up your first Neon Postgres project is straightforward—follow the steps below to get started!

Visit Neon to sign up for an account.

After a successful registration, the system will redirect you to create a new project. Here, you can fill in what Postgres version you want to use, the project name, the database name, and the region you want the database instance to run on.

A pop-up with the connection link to your Neon Postgres database will appear when your project is successfully created. Select the psql option from the dropdown list, copy the string, and save it somewhere. You will use it to connect to Hyperdrive in your demo application later.

Finally, create a table and insert some data into your database using the Neon SQL Editor. From the side menu, locate the SQL Editor option, click on it, replace the command with the one below, and run it.

    CREATE TABLE users(id SERIAL PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL, phone TEXT NOT NULL);
    INSERT INTO users (name, email, phone) VALUES
      ('Kehinde', 'kh@nmail.com', '1209867574'),
      ('Desmond', 'des@nmail.com', '4584930999'),
      ('Franklin', 'frk@nmail.com', '3458976048'),
      ('Aisha', 'as@nmail.com', '0987954890'),
      ('Adaeze', 'az@nmial.com', '1234567890');
    SELECT * FROM users;

This SQL command does three things specifically:

The first command, CREATE TABLE users, creates a table in your database called "users” with columns: id, name, email, and phone. The id is set to be the primary key and should follow a series, while the other three are text strings and should not be null; that is to say, they shouldn’t be empty when creating a user record.
The second command, INSERT INTO users, inserts data into each column of the table as specified in the VALUES section.
The last one, SELECT * FROM users, returns all the data in the users table.

Upon successfully running the query above, you’ll get a screen like this one below. This screen is found in the Tables Menu. Click on Tables from the side menu and select users to see the new content created.

Setting up Cloudflare Hyperdrive and Workers

In this section, you will learn how to set up a Cloudflare Worker project with the Wrangler CLI, bind your Hyperdrive proxied database, and use it in your application. You will be creating the application from scratch.

If you already have Wrangler CLI installed, you don’t need to worry about this first step. If you don’t have Wrangler CLI installed, run this command in a new terminal to install it globally:

    npm install wrangler@latest

Next, navigate to the directory where you want to store your project on your machine and run this command to set up a Cloudflare Workers application.

    npm create cloudflare@latest

Workers is a Cloudflare service that provides an execution environment that allows you to create new applications or augment existing ones without configuring or maintaining infrastructure.
Workers run on Cloudflare’s global network in hundreds of cities worldwide.

The above command will prompt you to install the create-cloudflare package and lead you through the setup steps.

For this guide, set up a basic Workers application by:

Naming your new Worker directory by specifying where you want to create your application.
Selecting the "Hello World" script as the type of application you want to create.
Answering "yes" to using TypeScript.
You will be asked if you want to deploy the project to Cloudflare. Choose not to deploy and go to the newly created project directory to begin writing code. You will do the deployment later.

When the setup is done and before creating your Hyperdrive binding, log in with your Cloudflare account for authentication by running the command below:

    cd <your-worker-project> && npx wrangler login

This command will open your browser to log into Cloudflare and acquire access to your terminal. Upon a successful connection, you will get a screen like the one below:

Let’s connect to Hyperdrive now that you have configured Wrangler CLI and have logged in to your Cloudflare account. Run the command below to connect your Neon Postgres to Hyperdrive:

    npx wrangler hyperdrive create pg-hyperdrive --connection-string="your-neon-db-connecion-string"

A few things to note in the above command:

The wrangler hyperdrive create command initializes the Hyperdrive creation.
The pg-hyperdrive is the name of your Hyperdrive. It can be any name of your choice.
The --connection-string flag="your-neon-db-connection-string" is for passing your database connection string as a parameter to the create command.

You will get a response in your terminal like the one below upon a successful connection.

The final step for this section is to set the generated Hyperdrive ID to your wrangler.toml file, which you will use to connect to Neon Postgres in the next section, as shown below. You must remove the initial content from the package and update your wrangler.toml file like the one below. Remember to replace the placeholder with your Hyperdrive ID!

Writing the API services

If you did everything above, you should have a folder structure like the one below. In this section, you will set up the database connection, create custom query functions, and update the /src/index.ts file to match your configurations. Also, since you are deploying to Workers, you will use the Hono framework instead of Express.

Hono is an Ultrafast web framework for Cloudflare Workers, Fastly Compute, Deno, Bun, Vercel, Lagon, Node.js, and others. You can learn more about Hono here.

First, install two key packages: the Postgres driver and Hono. Also, you’ll need to install the Postgres driver types as a dev dependency since you are using Typescript.

    npm i pg hono
    npm i --save-dev @types/pg

Create a dbconnect.ts file in the src folder and paste the code below.

    // src/dbconnect.ts
    import { Context } from 'hono';
    import { Client } from 'pg';

    export const db = async (c: Context) => {
        const client = new Client({ connectionString: c.env.HYPERDRIVE.connectionString });
        // Connect to your database
        await client.connect();
        return client
    };

In the above code, you set up a database client from the Postgres driver to use the link generated by Hyperdrive as a proxy to your database and return the client instance for your app to use in reading and writing to the database.

Next, you will write some helper functions that will be used to perform read and write operations to your database through Hyperdrive proxy. You will create a queries.ts file in the src directory and paste the code below.

    // src/queries.ts
    import { Context } from "hono";
    import { db } from "./dbconnect";

    type User = {
        name: string;
        email: string;
        phone: string;
    }
    export const getUsers = async (c: Context) => {
        const limit = c.req.query('limit') || 10;
        const offset = c.req.query('offset') || 0;
        const dbClient = await db(c);
        let result = await dbClient.query
            (`SELECT * FROM users LIMIT ${limit} OFFSET ${offset}`);
        return result;
    };

    export const getUser = async (c: Context) => {
        const id = parseInt(c.req.param('id'))
        const dbClient = await db(c);
        let result = await dbClient.query
            (`SELECT * FROM users WHERE id = ${id}`);
        return result;
    }
    export const createUser = async (c: Context) => {
        const user = await c.req.json<User>()
        const dbClient = await db(c);
        let result = await dbClient.query
            (`INSERT INTO users (name, email, phone) VALUES ('${user.name}', '${user.email}', '${user.phone}')`);
        return result;
    }

Here, you are writing three different functions that get all the users, get a single user, and create a user in the database. For each function, you are passing the context parameter, which contains the request body or parameters, to the database client. The database client will instantiate with these parameters. After that, you can use the database client to make your queries or insertion. After a successful query or insertion, it returns the result to be read in the application.

Finally, update the contents of the index.ts file in the src folder to match the code below.

    // src/index.ts
    import { Hono } from 'hono';
    import { cors } from 'hono/cors'
    import { prettyJSON } from 'hono/pretty-json'
    import { getUsers, getUser, createUser } from './queries'
    import { db } from './dbconnect';


    type Env = {
        HYPERDRIVE: Hyperdrive;
    };
    const app = new Hono();
    app.get('/', (c) => c.text('Pretty Users API'))
    app.use('*', prettyJSON())
    app.notFound((c) => c.json({ message: 'Route Not Found', ok: false }, 404))

    const api = new Hono<{ Bindings: Env }>()
    api.use('/users/*', cors())
    api.get('/users', async (c) => {
        const users = await getUsers(c)
        return c.json({ users })
    })
    api.get('/users/:id', async (c) => {
        const user = await getUser(c)
        return c.json({ user })
    })
    api.post('/users', async (c) => {
        const ok = createUser(c)
        return c.json({ ok })
    })
    app.route('/api', api)

    export default app;

In the code above, you obtained the Hyperdrive data from wrangler.toml. You first created an app instance with Hono and set the basic endpoint that you can use to check your routing works fine and make the JSON response formatted with the prettyJSON() method.

After that, you created another instance of Hono called API, which has the Env binding. This is so that you can read the value of the Hyperdrive ID from the context anywhere in the app. Hopefully, it now makes sense why you passed the context to your query functions and database connection function too.

Finally, you defined each route to call their respective query functions and pass the context down to them. The API route is now defined so that your endpoint calls will be to api/users/…

Remember, this is a Cloudflare Worker application using Hono, so in the index, you can read the data in your .toml file as a type of Env.

Deploying to Cloudflare Workers

Now that everything is set up, the next step is to deploy your application to Cloudflare Workers, the serverless execution environment. To deploy your application, run

    npx wrangler deploy

This will deploy your application to Workers. If you are not logged in to the Wrangler CLI, the command will prompt you to log in for authentication; your application will then be deployed. You will get a response upon a successful deployment, as shown in the image below.

Testing and benchmarking with vs without Hyperdrive

The core reason for using Hyperdrive to proxy the database instead of using the Neon connection URL directly in your application is the optimization for speed and performance when making queries. This section compares the response time for a large query from across the world, first using our new API on Hyperdrive and then using a traditional single-region API server. As you can see below, the Hyperdrive route is significantly faster.

The link to the applications used for the demo can be found here on GitHub.

Here’s a query to fetch all users in the database for Neon Postgres and Cloudflare Hyperdrive proxy. It responded at 780ms on the first query and between 300ms and 450ms on subsequent queries.

Here’s a similar query to fetch all users in the database that’s routing through a single API server. It responded at 1888ms at the first query and between 600ms and 3s on subsequent queries.

This video demonstrates the speed test between Neon Postgres connected through Hyperdrive and Neon Postgres connected through an API App in a single region.

Speed Benchmarking For Databases Queries | Opentape

Desmond Obisi - Feb 7th, 2:21pm

app.opentape.io

Why was the Workers + Hyperdrive approach faster?

A few reasons:

Workers are deployed globally, close to users - for most of the world, the network latency between local device and worker function is lower than when using a single API server.
Hyperdrive keeps a connection to the database alive - Hyperdrive keeps a “warm” connection to the database alive across function calls. This eliminates the multiple round trips usually required to establish a new connection.
Hyperdrive caches certain responses - For SELECT queries, Hyperdrive caches responses for 60 seconds by default, so repetitive queries will never even hit the database.

Conclusion

Congratulations! If you followed along, you learned to use Neon Serverless Postgres with Cloudflare Hyperdrive to achieve faster database queries. You also learned about Workers, Wrangler CLI for interfacing with Cloudflare's products, and Hono for writing Node.js APIs that can be deployed to Workers.

Other things that can be explored on this subject leveraging Hyperdrive for intensive database operations are transactions and query caching. You can check the reference section for a feel of other things Cloudflare offers with the Hyperdrive developer tool.

References

Implement OTP Verification using Redis and Node.js

Desmond Obisi — Tue, 30 May 2023 11:11:38 +0000

Introduction

In today's digital space, user authentication and verification are crucial components of any application's security measure. One-time password (OTP) systems provide an extra layer of security by generating unique codes that can be used once and are valid for a short period.

This article will guide you through the process of building an OTP and verification feature using Redis and Node.js. Redis, a powerful in-memory data structure store, will be utilized to store and manage the OTPs efficiently.

Overview of OTP-based Authentication

In this section, we will go through an overview of OTP-based authentication and discuss its advantages over traditional password-based systems.

What is OTP-based Authentication?

One-time password (OTP) authentication is a mechanism that provides an additional layer of security during an authentication process.

Unlike traditional password-based authentication, which relies on fixed passwords, OTP-based authentications generate random and unique codes that are valid for a short period (typically a few minutes). Users are required to provide the OTP along with their username or email to gain access to the system.

Advantages of OTP-based Authentication:

OTP-based authentication offers several advantages over traditional password-based systems, a few of which are:

Increased Security
Mitigation of Password-related Risks
Two-factor Authentication (2FA)

By implementing OTP-based authentication and verification in your applications, you can significantly enhance the security of user authentication while providing a seamless, user-friendly experience.

Let's move on to the next section, where we will set up the development environment for our project.

Setting Up the Development Environment

In this section, we will go through the process of setting up the development environment for building the OTP and verification feature with Redis and Node.js. We will first cover the installation of Node.js, setting up Redis, and creating a new Node.js project.

Setting Up Node.js:

To begin, you need to install Node.js on your system. Node.js is available for multiple operating systems and can be downloaded from the official Node.js website. Follow these steps to install Node.js:

Visit the Node.js website using your web browser.
Download the appropriate Node.js installer for your operating system (Windows, macOS, or Linux).
Run the installer and follow the on-screen instructions to complete the installation process.

To verify that Node.js is installed correctly, open a command prompt or terminal and run the following command:

    node --version

You should see the version number of Node.js printed in the console.

Setting Up Redis

Next, you need to set up Redis, the in-memory data structure store that will be used to store and manage the OTPs. Follow these steps to install and configure Redis:

Visit the Redis website using your web browser.
Download the Redis distribution package for your operating system. Since Redis does not run directly on Windows, you can either run it through WSL2 (Windows Subsystem for Linux) or by downloading this port of Redis for Windows here and installing it. In this tutorial, we will be using the latter.
Open a command prompt or terminal and run the Redis server by executing the following command:

    redis-cli

This will start the Redis server on the default port 6379.

Creating a New Node.js Project

Now that you have Node.js and Redis all setup, let's create a new Node.js project for building the OTP and verification feature.

Follow these steps:

Open a command prompt or terminal and create or navigate to the directory where you want to create your project.

    mkdir tutorials && cd tutorials

Run the following command to clone the template that I made for this project in my GitHub repository and after that, run it.

    git clone https://github.com/DesmondSanctity/node-redis-otp.git && npm install

This will give you most of the project setup except the various logic for the OTP generation, Redis caching and data retrieval, which we will be implementing in the next section.

Generating and Caching OTPs with Redis

In this section, we will explore the implementation details of generating OTPs and caching the data with Redis. We will use a user signup/login system to demonstrate this concept. We would also use a Node.js client library redis and otp-generator to interact with the local Redis instance and generate unique OTPs respectively.

Connecting to the Local Redis Instance

To utilize the caching capabilities of Redis, set up and connect to the local Redis instance you installed on your machine. In the server.js file, set up the connection as shown below and export it for use in other files.

    ...
    // create a client connection
    export const client = redis.createClient();

    // on the connection
    client.on("connect", () => console.log("Connected to Redis"));

    await client.connect();
    ...

Generating OTP for User and Storing to Redis

Now that you have established a connection with your local Redis installation, you can test generating OTPs and storing them in Redis. In the userController.js file where the user logic is, add a method to generate OTP and store OTP using the packages we installed earlier.

    ...
    export async function register(req, res) {

        try {
            const { username, password, email, firstName, lastName } = req.body;

            // Generate a random OTP using the otp-generator package
            const otp = otpGenerator.generate(4, {
                lowerCaseAlphabets: false,
                upperCaseAlphabets: false,
                specialChars: false
            });

            // check the existing user
            const existUsername = new Promise((resolve, reject) => {
                UserModel.findOne({ username: username }, function (err, user) {
                    if (err) reject(new Error(err))
                    if (user) reject({ error: "Please use unique username" });

                    resolve();
                })
            });

            // check for existing email
            const existEmail = new Promise((resolve, reject) => {
                UserModel.findOne({ email: email }, function (err, email) {
                    if (err) reject(new Error(err))
                    if (email) reject({ error: "Please use unique Email" });

                    resolve();
                })
            });

            await Promise.all([existUsername, existEmail])
                .then(() => {
                    if (password) {
                        bcrypt.hash(password, 10)
                            .then(hashedPassword => {

                                const user = new UserModel({
                                    username,
                                    password: hashedPassword,
                                    email,
                                    firstName,
                                    lastName
                                });

                                // Store the OTP in Redis, with the user's email as the key
                                client.set(email, otp);

                                const { password, ...responseUser } = user._doc;
                                // return save result as a response
                                user.save()
                                    .then(result => res.status(201).send({
                                        msg: "User Register Successfully",
                                        OTP: otp, User: responseUser
                                    }))
                                    .catch(error => res.status(500).send({ error }))

                            }).catch(error => {
                                return res.status(500).send({
                                    error: "Enable to hashed password"
                                })
                            })
                    }
                }).catch(error => {
                    return res.status(500).send({ error })
                })

        } catch (error) {
            return res.status(500).send(error);
        }

    }
    ...

From the above code, the otpGenerator function from otp-generator package was used to generate four unique integers for the OTP and to store them in the Redis client using client.set(key, value); where value is the OTP we are storing and key is a unique identifier we will use to identify each stored value. In this case, the email of the user will be used as the key because it is unique per user.

You can check your local instance to see if the OTP is saved by querying Redis with the key/identifier. Run the command below to see the resulting OTP for a user with email example@gmail.com

    redis-cli
    GET example@gmail.com

Verifying OTP for User

So far, a unique OTP has been generated on user registration, now we need to verify the user by writing another block of code. The logic here is to check the OTP to be supplied to the code block with the one in the local Redis instance, if they match, the user will be verified. If there is no match, the user will be notified that the OTP is incorrect.

    ...
    export async function verifyUser(req, res, next) {
        try {

            const { username, email, otp } = req.body;

            // check the user existance
            let exist = await UserModel.findOne({ username: username });
            if (!exist) return res.status(404).send({ error: "Can't find User!" });

            // Retrieve the stored OTP from Redis, using the user's email as the key
            const storedOTP = await client.get(email);

            if (storedOTP === otp) {
                // If the OTPs match, delete the stored OTP from Redis
                client.del(email);

                // Update the user's isVerified property in the database
                await UserModel.findOneAndUpdate({ username }, { isVerified: true });

                // Send a success response
                res.status(200).send('OTP verified successfully');
            } else {
                // If the OTPs do not match, send an error response
                res.status(400).send('Invalid OTP');
            }
            next();

        } catch (error) {
            return res.status(500).send({ error });
        }
    }
    ...

As you can see from the code, the client.get(key) method was used to get the stored OTP and to compare it with the one input by the user. If there is a match, the OTP will be deleted using client.del(key) method to avoid any vulnerabilities and the user info isVerified will be updated to true.

Detecting Missed Verification

Sometimes, a user might register successfully but will fail to complete OTP verification. In such a case, the database will not record them as verified users yet, any attempt by such a user to log in will initiate a response with a notification for them to complete OTP verification. Also, a new code will be generated for them to use. An example code is below:

    ...
    export async function login(req, res) {

        const { email, password } = req.body;

        try {

            UserModel.findOne({ email })
                .then(user => {
                    bcrypt.compare(password, user.password)
                        .then(passwordCheck => {

                            if (!passwordCheck) return res.status(400).send({
                                error: "Don't have Password"
                            });

                            if (!user.isVerified) {
                                // Generate a random OTP using the otp-generator package
                                const otp = otpGenerator.generate(4, {
                                    lowerCaseAlphabets: false,
                                    upperCaseAlphabets: false,
                                    specialChars: false
                                });

                                // Store the OTP in Redis, with the user's email as the key
                                client.set(email, otp);

                                return res.status(400).send({
                                    error: "User is not verified, please finish verification using this OTP",
                                    OTP: otp
                                })
                            }

                            // create jwt token
                            const token = jwt.sign({
                                userId: user._id,
                                username: user.username
                            }, process.env.JWT_SECRET, { expiresIn: "24h" });

                            return res.status(200).send({
                                msg: "Login Successful...!",
                                user: user,
                                token
                            });

                        })
                        .catch(error => {
                            return res.status(400).send({ error: "Password does not Match" })
                        })
                })
                .catch(error => {
                    return res.status(404).send({ error: "Username not Found" });
                })

        } catch (error) {
            return res.status(500).send({ error });
        }
    }
    ...

In the code block above, after the login credentials has been checked and ascertained correct, the isVerified parameter will also be checked to see if the user is verified. If they are not verified, a new OTP is generated for them with a notification to complete the verification using the verify endpoint that was done initially. Once they are verified, they can successfully log in to the system.

This is a short video demonstrating how the whole code put together works.

Conclusion

Implementing OTP-based authentication and verification is an effective way to strengthen the security of your applications. By using Redis as a data store and Node.js as the backend, you can easily build a robust OTP system.

Throughout this article, we went through step-by-step instructions and code examples to guide you in building this feature. With the knowledge gained from this tutorial, you are now equipped with adequate knowledge to implement OTP-based authentication and verification in your own projects, ensuring a safer user experience.

Resources

Setting up Node.js Email Server with Nodemailer and Mailtrap

Desmond Obisi — Wed, 22 Mar 2023 01:15:27 +0000

Introduction

The email service is one of the services common to every software application. Emails are used to communicate pieces of information mostly about downtime, updates and verification of activities by the user in a software application.
In this tutorial, we will be demonstrating how to set up an email server in a Node.js application using Nodemailer and Mailtrap. First of all, let us learn the basics!

Nodemailer

Nodemailer is a module for Node.js applications that allows easy-as-cake email sending. The project started back in 2010 when there was no better option for sending email messages, today it is still the solution most Node.js users turn to by default. Underneath the hood, Nodemailer uses the Node.js core module net to create a connection to the SMTP server specified in the transport object. Once a connection is established, Nodemailer uses the SMTP protocol to send email messages by sending various commands and responses over the connection.

Here's a simplified example of how Nodemailer sends an email message:

Create a transport object that defines the SMTP server, authentication credentials, and other options.
Create a message object that defines the email message, including the sender, recipient, subject, and body.
Use the transporter.sendMail() method to send the message, passing in the message object and a callback function to handle any errors or responses.
Nodemailer uses the transport object to establish a connection to the SMTP server and send the message using the SMTP protocol. If the message is sent successfully, Nodemailer returns a response object containing information about the message, like the message ID and the response from the SMTP server.

Overall, Nodemailer provides a simple and powerful interface for sending email messages in Node.js, taking away the complexities of the SMTP protocol and allowing developers to focus on the content and delivery of their messages.

Mailtrap

Mailtrap is a service that provides a mock SMTP server for testing and development purposes. It allows developers to test their email-sending functionality without actually sending emails to real recipients. Mailtrap works by creating a temporary SMTP server that is accessible through a unique hostname and port number. When an email is sent to the Mailtrap server, it is intercepted and stored in your Mailtrap inbox instead of being sent to the intended recipient. Mailtrap also provides a web interface that allows you to view and manage your test email messages, as well as features like email forwarding and spam testing.

Here are the following steps to enable Mailtrap to work with Nodemailer:

Sign up for a Mailtrap account and create a new inbox.
Retrieve your SMTP credentials from the Mailtrap inbox settings.
Configure Nodemailer to use the Mailtrap SMTP server by creating a transport object with the SMTP settings from Mailtrap.
Use the transporter.sendMail() method to send email messages, just like you would with any other SMTP server.
Check your Mailtrap inbox to see the test email message. The message will be displayed in the inbox with its headers and content, allowing you to verify that it was sent correctly.

Overall, Mailtrap is a useful tool for testing and debugging email functionality in a safe and controlled environment. By using Mailtrap with Nodemailer, developers can test their email-sending functionality without the risk of accidentally sending test emails to real recipients or getting blocked by spam filters.

Setting up the Email Server

Now that we’ve had the basics out of the way, let us delve into a more interesting part: we will be setting up the email server using Node.js. The process follows this form;

Creating Mailtrap Account

Go to the Mailtrap website and sign up. When the sign up is completed, you will see a screen like the one below.
Click on Email Testing to get your secret credentials. Select Nodemailer as an integration option and you will get a screen like the one below.
At this point, you’d have to save the keys, it would be needed to set up the application later.

Setup Node.js Project

To set up our Node.js server, we need to create a project folder that will contain our application. For this tutorial, we will call the folder node-email-server.

To begin, we will run the following commands from our terminal.

    cd desktop && mkdir node-email-server
    cd node-email-server
    code .    /** this is to open the folder in our code editor (VSCode) */

In the root folder, run the following command to install the packages we will be using

    npm install nodemon nodemailer dotenv cors express mailgen

Nodemon—automatically restarts the node application when file changes in the directory are detected.
Cors—for providing an Express middleware that can be used to enable CORS with various options.
Express—Node.js web application framework that provides a robust set of features for web and mobile applications.
Mailgen—Node.js package that generates clean, responsive HTML e-mails for sending transactional mail.

Create a .env file in the root folder—this is where we will store the Mailtrap keys we generated earlier.

    EMAIL=<sender's email address>
    USER=<mailtrap user key>
    PASSWORD=<mailtrap password>

We will proceed to create the server file, in our case, it is the index.js file. Add the following code to the server file

    import express from 'express';
    import cors from 'cors';
    import dotenv from 'dotenv';
    import nodemailer from 'nodemailer';
    import Mailgen from 'mailgen';

    dotenv.config()   // configure our app to use env variables
    const app = express();
    /** middlewares */
    app.use(express.json());
    app.use(cors());
    app.disable('x-powered-by'); // less hackers know about our stack

    const port = process.env.PORT || 8080;
    /** HTTP GET Request */
    app.get('/', (req, res) => {
        res.status(201).json("Health Check PASS");
    });


    // create reusable transporter object using the default SMTP transport
    const transporter = nodemailer.createTransport({
        host: 'smtp.mailtrap.io',
        port: 2525,
        auth: {
            user: process.env.USER, // generated mailtrap user
            pass: process.env.PASSWORD, // generated mailtrap password
        }
    });
    // generate email body using Mailgen
    const MailGenerator = new Mailgen({
        theme: "default",
        product : {
            name: "Test Email",
            link: 'https://mailgen.js/'
        }
    })
    // define a route for sending emails
    app.post('/send-email', (req, res) => {
        // get the recipient's email address, name and message from the request body
        const { to, name, message } = req.body;
        // body of the email
        const email = {
            body : {
                name: name,
                intro : message || 'Welcome to Test Mail! We\'re very excited to have you on board.',
                outro: 'Need help, or have questions? Just reply to this email, we\'d love to help.'
            }
        }
        const emailBody = MailGenerator.generate(email);
        // send mail with defined transport object
        const mailOptions = {
            from: process.env.EMAIL,
            to: to,
            subject: 'Test Email',
            html: emailBody
        };
        transporter.sendMail(mailOptions, (error, info) => {
            if (error) {
                console.log(error);
                res.status(500).send('Error sending email');
            } else {
                console.log('Email sent: ' + info.response);
                res.send('Email sent successfully');
            }
        });
    });
    // start the server
    app.listen(port, () => {
        console.log(`Server started on port ${port}`);
    });

Firstly, we imported the packages we will be needing and created an app instance using express. We defined our middleware and port. The interesting parts are:

The email transport instance was created with nodemailer and using the SMTP details that mailtrap provided us
The function MailGenerator which we used to generate clean, responsive HTML e-mails.
The route which houses those logics we need for sending email requests to the server whenever the send-email endpoint is called.
Finally, we started the server and let the app listen to the defined port which we will use to access and test our email API.

Testing our Email Server

This is my favourite part of the whole process, testing our endpoint to see if our application actually works. For this tutorial, we will be using Postman a HTTP client used to test, debug or document API. You can set up Postman as a desktop application or use it as a web app without installing it on your local machine. Check here for more details on how to set it up with any option you want.

Making a request

Our app is running on http://localhost:8080 so our endpoint is exposed to this url: http://localhost:8080/send-email We will send a POST request to the endpoint with three parameters in the body of the request which are: to, the email address of the recipient of the mail, name, the name of the recipient and message, which is optional if we do not want to overwrite the default message in our application. Here is what our Postman request and response will look like.

If we check our Mailtrap account, we will see the mail we just sent and viola! our app works as expected. Here is what mine looks like.

Conclusion

We have learnt how to use Nodemailer and Mailtrap to set up a Node.js email server and also how these technologies work behind the scene. The code repository for this project can be found here.
Note that Nodemailer works with other SMTP providers like Gmail, Hotmail, MailGun, SendGrid, SendinBlue, etc. The list of supported services can be found here too.

If you enjoyed the article, feel free to leave a comment, give it a shout-out and share it with others. Bueno!

Documenting Node.js API using Swagger

Desmond Obisi — Mon, 20 Mar 2023 09:56:43 +0000

Introduction

API documentation is an integral part of developing software. It is an instruction manual that explains how to use an API and its services. This manual might contain tutorials, code examples, screenshots, and anything else that helps users better understand how to work with the API.

In this article, we will be learning how to document API written in Node.js using a tool called Swagger. Swagger allows you to describe the structure of your APIs so that machines can read them. The ability of APIs to describe their own structure is the root of all awesomeness in Swagger. Why is it so great? Well, by reading our API’s structure, swagger can automatically build beautiful and interactive API documentation. It can also automatically generate client libraries for your API in many languages and explore other possibilities like automated testing. Swagger does this by asking our API to return a YAML or JSON that contains a detailed description of your entire API. This file is essentially a resource listing of our API which adheres to OpenAPI Specification s.

Building our API with Node.js and Express

To start writing the API specs, we will build our API with Node.js which is a back-end JavaScript runtime environment that runs on the V8 JavaScript Engine and executes JavaScript code outside a web browser. For the purpose of simplicity, I have set up the project and it can be cloned from this GitH ub repository. To get the backend running on our local machine, we will follow these steps:

Create a new folder for the project and run this command in the root folder to clone the repository

git clone https://github.com/DesmondSanctity/node-js-swagger.git

To successfully run the code, we will need a database connection. I used a MongoDB Atlas cluster for the database and we can follow this tutorial to set up one, it is quite simple to set up. After the set up, we will get our URL and that is all we need to connect to our database from our app.
We are using JSON Web Token (JWT) to authenticate access to our API, so we will generate a secret key that will be used by our app to send and verify requests. To generate that, we will run this command anywhere in our terminal. This script will generate a random 64-bit ASCII string, that can be used for encrypting JWT tokens.

node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

We will now create a file named .env where we will store our MongoDB Atlas cluster URL and JWT secret as an environment variable. The file should look like this:

    JWT_SECRET=<your JWT secret>
    ATLAS_URI=<your MongoDB Atlas cluster URL>

Now, we are ready to run the application but first, we will install some packages and then start the application. If you cloned the GitH ub repository earlier, you just need to run these commands below:

    npm install   // To install the neccessary packages
    npm start    //  To start the application

If you are successful at this point, you will see the following message in your terminal

    > mini-blog@1.0.0 start
    > nodemon server.js

    [nodemon] 2.0.20
    [nodemon] to restart at any time, enter `rs`
    [nodemon] watching path(s): *.*
    [nodemon] watching extensions: js,mjs,json
    [nodemon] starting `node server.js`
    Database Connected
    Server connected to http://localhost:8080

Adding Swagger UI and Configurations

Now that we have our API ready, we will start defining the Swagger specs for them. There are two ways we can build Swagger documentation for our API:

Manually write the specs inside the router files or a dedicated json or yaml file in our application.
Using existing developer’s tools or packages to generate the documentation automatically.

In this tutorial, we will use the manual approach to ensure accuracy in our definitions and specifications. First, we will install two packages called "swagger-jsdoc" and "swagger-ui-express" as dependencies using this command:

npm install swagger-jsdoc swagger-ui-express --save-dev

After the installation, we will create a new file called swagger.js in the root directory of our application and paste the following code into it

    import swaggerJsdoc from 'swagger-jsdoc'
    import swaggerUi from 'swagger-ui-express'
    const options = {
      definition: {
        openapi: '3.0.0',
        info: {
          title: 'Mini Blog API',
          description: "API endpoints for a mini blog services documented on swagger",
          contact: {
            name: "Desmond Obisi",
            email: "info@miniblog.com",
            url: "https://github.com/DesmondSanctity/node-js-swagger"
          },
          version: '1.0.0',
        },
        servers: [
          {
            url: "http://localhost:8080/",
            description: "Local server"
          },
          {
            url: "<your live url here>",
            description: "Live server"
          },
        ]
      },
      // looks for configuration in specified directories
      apis: ['./router/*.js'],
    }
    const swaggerSpec = swaggerJsdoc(options)
    function swaggerDocs(app, port) {
      // Swagger Page
      app.use('/docs', swaggerUi.serve, swaggerUi.setup(swaggerSpec))
      // Documentation in JSON format
      app.get('/docs.json', (req, res) => {
        res.setHeader('Content-Type', 'application/json')
        res.send(swaggerSpec)
      })
    }
    export default swaggerDocs

We can see from the code where we defined the Open API Specification (OAS) that we will be using for our documentation: the information or details about the API, the servers we will expose it to and the routes in our app where Swagger should look for specifications for each of our API.

We can also see our swaggerDocs function that allows the app instance and port to be able to generate the documentation for using the swaggerUi and swaggerJsdoc package we installed earlier and serve it at /docs route. We can also get the JSON format using /docs.json. Lastly, we will update our server.js file to include our swaggerDocs function to always generate and update our docs whenever we run the project.

The final step before writing out the specification for each endpoint is to add the swaggerDocs function to our server.js file so we initiate swagger whenever our app starts.

    import express from 'express';
    import cors from 'cors';
    import morgan from 'morgan';
    import dotenv from 'dotenv';
    import connect from './database/conn.js';
    import userRouter from './router/user.js';
    import postRouter from './router/post.js';
    import swaggerDocs from './swagger.js'

    dotenv.config()
    const app = express();
    /** middlewares */
    app.use(express.json());
    app.use(cors());
    app.use(morgan('tiny'));
    app.disable('x-powered-by'); // less hackers know about our stack

    const port = process.env.PORT || 8080;
    /** HTTP GET Request */
    app.get('/', (req, res) => {
        res.status(201).json("Home GET Request");
    });

    /** api routes */
    app.use('/api/user', userRouter)
    app.use('/api/post', postRouter)
    /** start server only when we have valid connection */
    connect().then(() => {
        try {
            app.listen(port, () => {
                console.log(`Server connected to http://localhost:${port}`);
            })
            swaggerDocs(app, port)
        } catch (error) {
            console.log('Cannot connect to the server')
        }
    }).catch(error => {
        console.log("Invalid database connection...!");
    })

Writing our API Specifications

Now to the main task, we will write the specifications for our API that Swagger will use to generate the documentation for us. We currently have two controllers and router files in our app for user and post. Our controllers contain logic for the app while the router carries the endpoint and payload to the controller in form of requests. Let’s start defining the spec!

For User router, we will first import the packages we will be needing in the router file user.js:

    import { Router } from "express";
    const userRouter = Router();
    /** import all controllers */
    import * as controller from '../controllers/userController.js';
    import Auth from '../middleware/auth.js';

    /** All the API routes comes here*/

    export default userRouter;

Then we will write the spec for each request type viz GET, POST, PUT and DELETE. The specification is a yaml file embedded at the beginning of the route we want to document. Some key points to note are:

Open API specification instance—written at the beginning of the yaml file.
The API endpoint—the URL we will use for the request.
Request Type—indicates if it is a GET, POST, PUT or DELETE request.
Request Body—for passing our payload to the API.
Parameters—data we pass through the URL or params to the backend
Content Type—the type of content we are sending. It is passed to the HTTP header.
Schema—it contains the type of our request body, the required fields and the properties that the request body can accept.
Response—the result of the API call we made, it tells us if it failed or succeded and reports errors as well.

POST:

    /** POST Methods */
    /**
     * @openapi
     * '/api/user/register':
     *  post:
     *     tags:
     *     - User Controller
     *     summary: Create a user
     *     requestBody:
     *      required: true
     *      content:
     *        application/json:
     *           schema:
     *            type: object
     *            required:
     *              - username
     *              - email
     *              - password
     *            properties:
     *              username:
     *                type: string
     *                default: johndoe 
     *              email:
     *                type: string
     *                default: johndoe@mail.com
     *              password:
     *                type: string
     *                default: johnDoe20!@
     *     responses:
     *      201:
     *        description: Created
     *      409:
     *        description: Conflict
     *      404:
     *        description: Not Found
     *      500:
     *        description: Server Error
     */
    userRouter.route('/register').post(controller.register); // register user

    /**
     * @openapi
     * '/api/user/login':
     *  post:
     *     tags:
     *     - User Controller
     *     summary: Login as a user
     *     requestBody:
     *      required: true
     *      content:
     *        application/json:
     *           schema:
     *            type: object
     *            required:
     *              - username
     *              - password
     *            properties:
     *              username:
     *                type: string
     *                default: johndoe
     *              password:
     *                type: string
     *                default: johnDoe20!@
     *     responses:
     *      201:
     *        description: Created
     *      409:
     *        description: Conflict
     *      404:
     *        description: Not Found
     *      500:
     *        description: Server Error
     */
    userRouter.route('/login').post(controller.verifyUser,controller.login); // login in app

    /**
     * @openapi
     * '/api/user/verify':
     *  post:
     *     tags:
     *     - User Controller
     *     summary: Verify a user
     *     requestBody:
     *      required: true
     *      content:
     *        application/json:
     *           schema:
     *            type: object
     *            required:
     *              - username
     *            properties:
     *              username:
     *                type: string
     *                default: johndoe
     *     responses:
     *      201:
     *        description: Created
     *      409:
     *        description: Conflict
     *      404:
     *        description: Not Found
     *      500:
     *        desccription: Server Error
     */
    userRouter.route('/verify').post(controller.verifyUser, (req, res) => res.end()); // authenticate user

GET:

    /** GET Methods */
    /**
     * @openapi
     * '/api/user/{username}':
     *  get:
     *     tags:
     *     - User Controller
     *     summary: Get a user by username
     *     parameters:
     *      - name: username
     *        in: path
     *        description: The username of the user
     *        required: true
     *     responses:
     *      200:
     *        description: Fetched Successfully
     *      400:
     *        description: Bad Request
     *      404:
     *        description: Not Found
     *      500:
     *        description: Server Error
     */
    userRouter.route('/:username').get(controller.getUser) // user with username

PUT:

/** PUT Methods */
    /**
     * @openapi
     * '/api/user/update':
     *  put:
     *     tags:
     *     - User Controller
     *     summary: Modify a user
     *     requestBody:
     *      required: true
     *      content:
     *        application/json:
     *           schema:
     *            type: object
     *            required:
     *              - userId
     *            properties:
     *              userId:
     *                type: string
     *                default: ''
     *              firstName:
     *                type: string
     *                default: ''
     *              lastName:
     *                type: string
     *                default: ''
     *     responses:
     *      200:
     *        description: Modified
     *      400:
     *        description: Bad Request
     *      404:
     *        description: Not Found
     *      500:
     *        description: Server Error
     */
    userRouter.route('/update').put(controller.updateUser); // is use to update the user profile

DELETE:

/** DELETE Methods */
    /**
     * @openapi
     * '/api/user/{userId}':
     *  delete:
     *     tags:
     *     - User Controller
     *     summary: Delete user by Id
     *     parameters:
     *      - name: userId
     *        in: path
     *        description: The unique Id of the user
     *        required: true
     *     responses:
     *      200:
     *        description: Removed
     *      400:
     *        description: Bad request
     *      404:
     *        description: Not Found
     *      500:
     *        description: Server Error
     */
    userRouter.route('/:userId').delete(controller.deleteUser);

Testing our APIs on Swagger

After finishing our API documentation, we should be able to view our Swagger documentation and test our API using it as well. If you followed till this point, you should have a view like the one below. Our documentation is served on the /docs route.

We will make some requests using the Swagger documentation UI and see the results.

To create a User:

To create a Post:

As we have seen from the examples above, we can use the Swagger documentation to test our API, to create, read and modify data in our database. This helps to make the API understandable and easy to integrate by others. Following these examples, we can go ahead and test for other request methods like PUT to update a user or post, GET to read a user or post and DELETE to delete them from the database.

Conclusions

We can finally conclude that API documentation is a very important part of the software development cycle and that it aids collaboration and makes user experience seamless. Some of the advantages of Swagger include but are not limited to:

Synchronizes the API documentation with the server and client at the same pace.
Allows us to generate REST API documentation and interact with the REST API. The interaction with the REST API using the Swagger UI Framework gives clear insight into how the API responds to parameters.
Provides responses in the format of JSON and XML.
Implementations are available for various technologies.