Forem: Slaknoah

How to 10X Your Development Workflow with Cursor MCPs

Slaknoah — Mon, 28 Jul 2025 10:13:19 +0000

If you've read our beginner's guide to MCPs, you now understand the basics of MCPs and have even built your own weather server.

But here's where things get really exciting—the MCP ecosystem is exploding with specialized servers that are transforming how developers work.

We're not talking about basic file operations anymore. Developers are building MCPs that connect AI directly to browsers, design tools, databases, APIs, and entire development pipelines. The result? AI assistants that can actually understand and work with your entire tech stack.

The MCP Developer Ecosystem is Taking Off

The developer community has embraced MCPs in a big way. Here's what's happening:

Browser Automation: MCPs like Playwright let AI assistants control browsers, test web applications, and scrape data—all through natural language commands—granting developers an unprecedented level of productivity while testing or automating processes.

Design-to-Code: Tools like Framelink's Figma MCP connect AI directly to design files, enabling instant translation from mockups to working code.

Database Integration: MCPs for PostgreSQL, SQLite, and other databases let AI query and analyze your data without you writing a single SQL statement.

Development Tools: Git MCPs handle version control, deployment MCPs manage infrastructure, and testing MCPs run your entire test suite.

API Integrations: MCPs for GitHub, Slack, Jira, and dozens of other services mean AI can interact with your entire toolchain.

This isn't just about convenience—it's fundamentally changing how we work with AI in development environments. Instead of describing what you want and hoping the AI understands, you can point it directly at your actual systems and let it figure out the details.

What You'll Learn

In this guide, we'll walk through using two powerful MCPs that showcase what's possible: the Playwright MCP for browser automation and Framelink's Figma MCP for design-to-code workflows. Both integrate seamlessly with Cursor, giving you AI-powered superpowers for web development and design implementation.

By the end, you'll see how MCPs are transforming developer workflows and have hands-on experience with two tools that can immediately improve your productivity.

Prerequisites

Before diving into these advanced MCPs, make sure you have:

Cursor IDE installed and configured with MCP support
Node.js (version 16 or higher) for running MCP servers
Basic familiarity with MCPs (You can learn from our beginner's guide if unfamiliar)
A Figma account (free tier works fine) for the design workflow section
Chrome or Chromium browser installed for Playwright automation

If you haven't set up MCPs in Cursor before, check out our previous guide for the complete setup process.

1. Browser Automation with Playwright MCP

The Playwright MCP brings full browser automation capabilities to your AI assistant.

Instead of manually testing web applications or writing complex automation scripts, you can simply describe what you want to test or automate—and it works like an absolute charm!

Why Playwright MCP is a Game-Changer

Traditional browser automation requires writing detailed scripts with selectors, wait conditions, and error handling. The Playwright MCP changes this completely:

Natural Language Control: Control browsers with simple commands like "Click the login button, fill in the form, and check if the dashboard loads"
Smart Element Detection: Uses accessibility snapshots for greater reliability
No Vision Models Needed: Works with structured data, not screenshots, making it faster and more reliable
Full Browser Capabilities: Handle file uploads, PDFs, network requests, console logs, and multi-tab workflows

Setting Up Playwright MCP in Cursor

First, let's get the Playwright MCP connected to Cursor. Open your Cursor Settings, click Tools & Integrations and navigate to the MCP configuration section.

Click New MCP Server and add this configuration to your mcp.json file:

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": [
        "@playwright/mcp@latest"
      ]
    }
  }
}

The Playwright MCP will automatically install when first used. If you want to run it in headless mode (no browser GUI), add the --headless flag:

{
  "mcpServers": {
    "playwright": {
      "command": "npx",
      "args": [
        "@playwright/mcp@latest",
        "--headless"
      ]
    }
  }
}

The MCP should become active after taking a while to load (look for a green dot).

Restart Cursor if this does not happen for a while or you see a red dot. Read the official docs for more help and setup instructions.

Your First Browser Automation

Now let's try some basic browser automation. Start a chat with Cursor's AI and try these commands:

Basic Navigation and Testing:

Navigate to https://google.com and take a screenshot of the page

Form Interaction:

Go to a Youtube and search for "MCPs for Beginners," return details about the top results in JSON format

Multi-step Workflows:

Open GitHub, search for "model context protocol", and take a screenshot of the first repository in the results

The AI will use Playwright's tools to:

browser_navigate - Navigate to URLs
browser_snapshot - Get page structure (better than screenshots for interactions)
browser_click - Click elements based on descriptions
browser_type - Fill forms and input fields
browser_take_screenshot - Capture visual results

Advanced Browser Automation Examples

Testing a Shopping Cart Flow:

Test the complete shopping experience on an e-commerce site:
1. Navigate to the homepage
2. Search for "laptop"
3. Add the first product to cart
4. Go to checkout
5. Fill out the shipping form (use test data)
6. Take a screenshot of the order summary

API Testing Through the Browser:

Use the browser's network tools to:
1. Go to a web application
2. Perform some actions that trigger API calls
3. Show me all the network requests that were made
4. Check if any returned errors

Multi-tab Workflow:

Open three tabs:
1. GitHub repository page
2. The project's documentation site  
3. A related Stack Overflow question
Take screenshots of all three and summarize what each shows

Understanding How Playwright MCP Works

The Playwright MCP operates in two modes:

Snapshot Mode (Default): Uses Playwright's accessibility tree to understand page structure. This is faster, more reliable, and doesn't require visual models. The AI gets a structured representation of the page that includes element roles, text content, and interaction possibilities.

Vision Mode: Uses screenshots for visual-based interactions. Enable this with the --vision flag if you need pixel-perfect interactions or are working with complex visual elements.

The accessibility snapshot approach is particularly powerful because it:

Works consistently across different screen sizes and themes
Focuses on interactive elements and semantic structure
Provides rich context about what each element does
Eliminates issues with dynamic CSS classes or changing layouts

2. Design-to-Code with Framelink's Figma MCP

The Figma MCP represents something that seemed impossible just months ago: direct integration between design tools and AI coding assistants.

Instead of manually translating designs to code, you can literally point your AI at a Figma frame and get working implementation.

You can even use MCPs to get Figma designs from images!

Why Design-to-Code MCPs are a Game-changer

Traditional design-to-code workflows are painfully manual:

Designer creates mockups in Figma
Developer inspects designs, measures spacing, extracts colors
Developer writes CSS/HTML by hand
Back-and-forth to match the design exactly
Repeat for every design change

The Figma MCP eliminates most of this friction:

Direct Design Access: AI reads Figma files directly through the API
Automatic Code Generation: Converts design elements to working code
Design System Awareness: Understands colors, fonts, spacing, and component patterns
Real-time Updates: Works with the latest version of designs automatically

Setting Up Framelink's Figma MCP

First, you'll need a Figma access token. Here's how to get one:

Open Figma and click your profile icon in the top left
Select "Settings" from the dropdown
Click the "Security" tab
Scroll to "Personal access tokens" and click "Generate new token"
Name your token and ensure you have read permissions for "File content" and "Dev resources"
Copy the generated token

Image Source: Figma

Now add the Figma MCP to your Cursor configuration:

{
  "mcpServers": {
    "figma": {
      "command": "npx",
      "args": [
        "-y",
        "figma-developer-mcp",
        "--figma-api-key=YOUR_FIGMA_TOKEN",
        "--stdio"
      ]
    }
  }
}

Replace YOUR_FIGMA_TOKEN with the token you just generated.

Just like before, wait for the yellow dot to turn green to indicate a successful load.

Check your mcp.json configuration and restart Cursor if any errors.

Your First Design Implementation

Now for the fun part—implementing a Figma design. Here's the workflow:

1. Choose a Design Section

The MCP works best with focused sections rather than entire pages. In Figma, right-click on a specific frame or component you want to implement. Select "Copy/Paste as" → "Copy link to selection".

2. Request Implementation

In Cursor, start a new chat and paste the Figma link along with a request like:

Implement this Figma frame as a React component: [paste Figma link]

Please:

- Use Tailwind CSS for styling
- Make it responsive
- Match the design exactly including colors, spacing, and typography
- Add hover states for interactive elements

3. Watch the Magic Happen

The AI will:

Call the get_figma_data function with your link
Analyze the design structure, colors, fonts, and spacing
Use the download_figma_images tool to download the correct images automatically
Generate clean, working code that matches the design
Include proper semantic HTML and accessibility attributes

Advanced Figma MCP Workflows

Component System Implementation:

Analyze this Figma design system and create reusable React components: [link]

Extract:
- Color palette as CSS variables
- Typography scale as Tailwind classes  
- Button variants as a component with props
- Card layouts as flexible components

Responsive Design Translation:

Implement this mobile design and make it responsive for desktop: [mobile frame link]

Consider:
- Mobile-first approach
- Appropriate breakpoints
- Touch-friendly interaction areas
- Performance optimization

Design Iteration Handling:

Compare this updated design with the current implementation and show me what needs to change: [updated Figma link]

Highlight:
- Color changes
- Spacing adjustments  
- New or removed elements
- Typography updates

Understanding the Figma MCP Data Flow

When you make a request, here's what happens behind the scenes:

1. API Data Retrieval: The MCP calls Figma's API to get the design data for your specific frame or component. This includes vector information, text content, colors, effects, and layout properties.

2. Data Compression: The raw Figma API response can be massive. The MCP compresses this data by ~90% while preserving all the essential information for code generation.

3. Structured Analysis: The AI receives a clean, structured representation of your design including:

Element hierarchy and nesting
Exact colors (hex codes) and gradients
Typography (fonts, sizes, weights, line heights)
Spacing and positioning data
Layer effects (shadows, borders, etc.)
Component instances and overrides

4. Code Generation: Using this structured data, the AI generates appropriate code that matches the design specifications.

Best Practices for Design-to-Code with MCPs

Start Small: Begin with individual components rather than full pages. Cards, buttons, and form elements work particularly well.
Provide Context: The more context you give about the intended use, the better the results.

Specify Technology Stack: Be explicit about your preferences:

Convert this design to a Vue 3 component using:

- Composition API
- TypeScript
- Tailwind CSS
- Emit events for user interactions

Iterate and Refine: Use the MCP for rapid iteration:

The spacing looks off compared to the Figma design. Please adjust the margins and padding to match exactly: [same link]

Common Issues and Solutions

403 Forbidden Errors: This usually means permission issues:

Ensure your Figma token has the right permissions
Check that the file is shared with your account
Verify the link points to a valid, accessible frame

Overwhelming Design Complexity: For complex layouts:

Break designs into smaller sections
Focus on one component at a time
Use design system components as building blocks

Version Mismatches: If designs change frequently:

Always use the latest Figma links
Mention when you're working with updated designs
Consider implementing a design review process

The Future of MCP-Powered Development

What we've explored today—browser automation and design-to-code workflows—represent just the beginning.

The key insight is that MCPs don't just add capabilities to AI—they fundamentally change the relationship between developers and their tools.

The MCP ecosystem is growing rapidly, and we're seeing innovations that will reshape development:

AI-Native Development Tools: MCPs that understand entire codebases, manage dependencies, and handle deployment pipelines.

Cross-Platform Integration: MCPs that connect web development with mobile platforms, desktop and CLI applications, and even hardware.

Collaborative Workflows: MCPs that enable AI assistants to work together on complex projects, sharing context and coordinating tasks.

Real-Time Data Integration: MCPs that connect to live systems, enabling AI to work with current data, user behavior, and system performance.

Building Your Own Specialized MCPs

As you get comfortable with existing MCPs, consider building your own for your specific tech stack:

Custom API MCPs for your company's internal services
Database MCPs tailored to your schema and queries
Deployment MCPs that understand your infrastructure
Testing MCPs that know your application's specific requirements

The patterns you learned building the Weather MCP server in the first article apply to any domain.

What's Next

MCPs are moving from experimental tools to essential parts of the developer toolkit. The developers and teams adopting them early are seeing significant productivity gains—not just from automation, but from the ability to work at a higher level of abstraction.

Start incorporating MCPs into your daily development workflow, and watch your productivity skyrocket!

The future of development is AI-driven, and MCPs are the bridge that makes it possible. The question isn't whether this will transform how we build software—it's how quickly you'll adapt to take advantage of it.

Ready to dive deeper? Check out the official MCP server gallery and other directories such as MCP Server Finder to discover more specialized tools.

Getting Started with Model Context Protocol (MCP)—A Beginner's Guide

Slaknoah — Thu, 24 Jul 2025 13:04:43 +0000

We've all been there—your favorite AI model is either hallucinating or giving answers irrelevant to your particular problem due to a lack of context and/or tools. Talk about frustrating.

That's precisely the problem the Anthropic team set out to solve with their Model Context Protocol (MCP) standard.

In this short guide, you'll learn all about MCPs, how to start using them, and even build one of your own.

Let's get into it!

What is MCP?

MCP is an open standard that connects AI models to external tools and data sources. Instead of AI assistants being limited to their training data, MCP lets them interact with databases, file systems, APIs, and custom applications in real time.

Think of it as USB-C for AI applications—a standardized way to plug AI models into different data sources, tools, and services.

Benefits of Using MCPs

Connect to Real Tools and Data: MCP breaks down the walls between AI models and your actual work environment. Your AI assistant can read files, query databases, call APIs, and execute scripts—all with your permission.

Build Once, Use Everywhere: The key power of MCP is interoperability. Build an MCP server once, and it works with any MCP-compatible AI client. Whether you're using Claude Desktop, Cursor, VS Code, or dozens of other applications, they can all leverage the same MCP servers.

Enable Multi-System Communication: MCP servers can act as bridges between different systems and AI models. You can create workflows where AI agents communicate with each other through shared MCP servers, enabling complex multi-step automation.

Extend Without Rebuilding: Instead of every AI application building its own integrations, MCP provides a standard interface. Developers can extend AI capabilities without rebuilding the same connections over and over.

Maintain Control and Security: You control what data and tools your AI can access. MCP servers run with the permissions you grant them, and many clients ask for approval before executing potentially sensitive operations.

Access Real-Time Information: Unlike static training data, MCP enables AI models to fetch current information from live systems—current weather, recent emails, latest database records, or real-time system status.

What You'll Learn

This guide takes you from zero MCP knowledge to building your own functional MCP server. You'll start by using pre-built servers with Claude Desktop to understand the basics. Then we'll switch to Cursor for development work and build a weather server that fetches real-time forecasts and alerts.

By the end, you'll understand the core MCP concepts, know how to debug common issues, and have the foundation to build servers for your specific needs.

MCP Basics: Using Pre-Built Servers with Claude Desktop

Let's start with the easiest possible introduction to MCP. We'll use Claude Desktop because it has excellent MCP support and requires zero coding to get started.

Setting Up Claude Desktop with MCP

First, download Claude Desktop if you haven't already. Choose either macOS or Windows (Linux isn't supported yet). Make sure you're running the latest version by clicking the Claude menu and selecting "Check for Updates..."

Now we'll add a filesystem MCP server that lets Claude read and write files on your computer. Don't worry - it will ask for permission before doing anything.

Configuring the MCP Server

Open the Claude menu on your computer and select "Settings..." (not the Claude Account Settings in the app window).

Click on "Developer" in the left sidebar, then click "Edit Config". This will create and open the configuration file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

If the file doesn't open automatically or you can't find it, expand and follow these steps:

Option 1: Use the default text editor

# On Mac
open ~/Library/Application\ Support/Claude/claude_desktop_config.json

# On Windows  
notepad %APPDATA%\Claude\claude_desktop_config.json

Option 2: Navigate with Finder/File Explorer

Mac: Press Cmd+Shift+G in Finder, type ~/Library/Application Support/Claude/
Windows: Press Win+R, type %APPDATA%\Claude\, press Enter

Replace the file contents with this JSON configuration:

On Mac/Linux:

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-filesystem",
        "/Users/yourusername/Desktop",
        "/Users/yourusername/Downloads"
      ]
    }
  }
}

On Windows:

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-filesystem",
        "C:\\Users\\yourusername\\Desktop",
        "C:\\Users\\yourusername\\Downloads"
      ]
    }
  }
}

Replace yourusername with your actual username (run whoami in a CLI if you're not sure). The paths should point to directories you want Claude to access and modify.

Install Node.js (Required)

This MCP server uses Node.js. Verify you have it installed by opening Terminal (Mac) or Command Prompt (Windows) and running:

node --version

If you get "command not found" or "node is not recognized", download Node.js from nodejs.org.

Save the configuration file and restart Claude Desktop completely.

Testing Your First MCP Connection

After restarting, you should see a slider icon in the bottom left corner of the input box. Click it to see your available tools.

Image Source: MCP website

Debugging errors:

If you get an error such as "Could not attach MCP server Filesystem," or "MCP Filesystem: Server disconnected," follow these steps to debug:

Run this command to view Claude desktop logs:

tail -n 20 -F ~/Library/Logs/Claude/mcp*.log

Tip: Paste your logs into an LLM to get debugging help

Now try these commands with Claude:

"Can you write a short poem and save it to my desktop?"
"What files are in my Downloads folder?"
"Can you create a new folder called 'MCP-Test' on my desktop?"

When Claude suggests using tools, you'll see a permission prompt. Click "Allow" to let it proceed. It should now be able to access your files.

Congratulations! You just experienced MCP in action. Claude can now interact with your filesystem because you connected it to an MCP server that provides file operations.

What Just Happened?

Here's the simple breakdown:

MCP Server: The filesystem server you installed provides tools for file operations
MCP Client: Claude Desktop connects to your server and gets a list of available tools
AI Integration: When you ask Claude to work with files, it automatically uses the appropriate tools
User Control: You approve each action before it happens

This client-server setup means you can connect Claude to databases, APIs, other applications, or custom tools you build yourself.

The diagram below gives a visual representation of what's happening.

Image Source: MCP website

How to Use MCPs with Cursor for AI-powered Development

Claude Desktop is great for general use, but to better leverage MCPs in your coding workflows, you want to use them with a client like Cursor (everyone's favorite AI-powered Code Editor).

You can use VSCode too if you're a boomer.

Setting Up MCP in Cursor

Cursor has built-in MCP support. To enable it:

Download and install Cursor
Open Cursor's settings
Click on "Tools & Integrations"
Click the "Add New MCP" button, an mcp.json file should open
Paste the JSON configuration snippet from earlier (for the filesystem server) into the mcp.json file

This allows you to connect to the same filesystem server you used with Claude Desktop, but this time it's integrated into your coding environment.

The power here is that Cursor's AI can now:

Read your project files to understand context
Create new files and folders
Edit existing code
Run scripts and commands
All while maintaining full awareness of your project structure

Try opening a project in Cursor and asking it to reference a file outside somewhere on your computer for context. You'll see how MCP transforms the AI from a general assistant into a project-aware coding partner.

Time to Build: Building Your Own MCP Server with Cursor

Now for the fun part—building your own MCP server. We'll create a weather server that can fetch forecasts and alerts. This example works well because it's practical, simple to understand, and shows all the core MCP concepts.

Setting Up the Project

Open Cursor and create a new folder for your project:

mkdir weather-mcp-server
cd weather-mcp-server

We'll use Python for this example. Create a new file called weather.py and paste in this code:

from typing import Any
import httpx
from mcp.server.fastmcp import FastMCP

# Initialize FastMCP server
mcp = FastMCP("weather")

# Constants for the National Weather Service API
NWS_API_BASE = "https://api.weather.gov"
USER_AGENT = "weather-mcp-server/1.0"

Here's what each import does:

typing.Any: Helps with type hints (makes code clearer)
httpx: Modern HTTP client library for making web requests (like requests but async-friendly)
mcp.server.fastmcp: The MCP framework that handles all the protocol details for us

The FastMCP class automatically converts your Python functions into MCP tools using function signatures and docstrings.

We're using the NWS (National Weather Service) API (https://api.weather.gov) and the USER_AGENT identifies our server to the API (good API citizenship).

Next, install the required dependencies:

pip install mcp httpx

Adding Helper Functions

Add these helper functions to handle API requests—these do the heavy lifting of talking to the weather API:

async def make_nws_request(url: str) -> dict[str, Any] | None:
    """Make a request to the NWS API with proper error handling."""
    headers = {
        "User-Agent": USER_AGENT,
        "Accept": "application/geo+json"
    }
    async with httpx.AsyncClient() as client:
        try:
            response = await client.get(url, headers=headers, timeout=30.0)
            response.raise_for_status()
            return response.json()
        except Exception:
            return None

def format_alert(feature: dict) -> str:
    """Format an alert feature into a readable string."""
    props = feature["properties"]
    return f"""
Event: {props.get('event', 'Unknown')}
Area: {props.get('areaDesc', 'Unknown')}
Severity: {props.get('severity', 'Unknown')}
Description: {props.get('description', 'No description available')}
Instructions: {props.get('instruction', 'No specific instructions provided')}
"""

The make_nws_request function is our generic API caller that:

Sets proper headers (User-Agent identifies us, Accept tells the API we want GeoJSON format)
Uses async with to properly manage the HTTP connection
Includes error handling - if anything goes wrong, it returns None instead of crashing
Has a 30-second timeout to prevent hanging requests

The format_alert function takes the raw API response and makes it human-readable. The NWS API returns complex JSON objects, but we only need the key information formatted nicely.

Creating Your First Tool

Now let's add a tool that fetches weather alerts for a given state:

@mcp.tool()
async def get_alerts(state: str) -> str:
    """Get weather alerts for a US state.

    Args:
        state: Two-letter US state code (e.g. CA, NY)
    """
    url = f"{NWS_API_BASE}/alerts/active/area/{state}"
    data = await make_nws_request(url)

    if not data or "features" not in data:
        return "Unable to fetch alerts or no alerts found."

    if not data["features"]:
        return "No active alerts for this state."

    alerts = [format_alert(feature) for feature in data["features"]]
    return "\n---\n".join(alerts)

This is our first MCP tool! Here's how it works:

The @mcp.tool() decorator tells FastMCP "this function should be available as a tool to AI clients." The framework automatically:

Reads the function signature to understand what parameters it needs
Uses the docstring as the tool description
Converts the type hints into a proper schema for AI models

The function itself builds a URL for the NWS alerts endpoint, makes the request, and processes the response. Notice how we handle different scenarios:

API failure → return an error message
No alerts found → return a friendly "no alerts" message
Alerts found → format each one and join them with separators

Why this works well: AI models get a clear description ("Get weather alerts for a US state") and know exactly what parameter to provide (a two-letter state code). The return value is always a human-readable string.

Adding a Second Tool

Let's add another tool for weather forecasts. This one is slightly more complex because it requires two API calls:

@mcp.tool()
async def get_forecast(latitude: float, longitude: float) -> str:
    """Get weather forecast for a location.

    Args:
        latitude: Latitude of the location
        longitude: Longitude of the location
    """
    # First get the forecast grid endpoint
    points_url = f"{NWS_API_BASE}/points/{latitude},{longitude}"
    points_data = await make_nws_request(points_url)

    if not points_data:
        return "Unable to fetch forecast data for this location."

    # Get the forecast URL from the points response
    forecast_url = points_data["properties"]["forecast"]
    forecast_data = await make_nws_request(forecast_url)

    if not forecast_data:
        return "Unable to fetch detailed forecast."

    # Format the periods into a readable forecast
    periods = forecast_data["properties"]["periods"]
    forecasts = []
    for period in periods[:5]:  # Only show next 5 periods
        forecast = f"""
{period['name']}:
Temperature: {period['temperature']}°{period['temperatureUnit']}
Wind: {period['windSpeed']} {period['windDirection']}
Forecast: {period['detailedForecast']}
"""
        forecasts.append(forecast)

    return "\n---\n".join(forecasts)

Here's how this tool works:

Step 1: Get the Grid Point - The NWS API doesn't take coordinates directly for forecasts. You first need to convert lat/lng coordinates to their internal grid system by calling the /points/ endpoint.

Step 2: Get the Actual Forecast - The first API call returns a URL for the forecast endpoint specific to that location. We then call that URL to get the actual forecast data.

Data Processing - The forecast comes back as an array of "periods" (tonight, tomorrow, tomorrow night, etc.). We:

Take only the first 5 periods (to avoid overwhelming output)
Extract the key information from each period
Format it in a readable way

Running the Server

Add this at the end of your file:

if __name__ == "__main__":
    mcp.run(transport='stdio')

Now you can test your server directly in Cursor's terminal:

python weather.py

If you get no errors, your MCP server is live!

Complete code:

from typing import Any
import httpx
from mcp.server.fastmcp import FastMCP

# Initialize FastMCP server
mcp = FastMCP("weather")

# Constants for the National Weather Service API
NWS_API_BASE = "https://api.weather.gov"
USER_AGENT = "weather-mcp-server/1.0"

async def make_nws_request(url: str) -&gt; dict[str, Any] | None:
    """Make a request to the NWS API with proper error handling."""
    headers = {
        "User-Agent": USER_AGENT,
        "Accept": "application/geo+json"
    }
    async with httpx.AsyncClient() as client:
        try:
            response = await client.get(url, headers=headers, timeout=30.0)
            response.raise_for_status()
            return response.json()
        except Exception:
            return None

def format_alert(feature: dict) -&gt; str:
    """Format an alert feature into a readable string."""
    props = feature["properties"]
    return f"""
    Event: {props.get('event', 'Unknown')}
    Area: {props.get('areaDesc', 'Unknown')}
    Severity: {props.get('severity', 'Unknown')}
    Description: {props.get('description', 'No description available')}
    Instructions: {props.get('instruction', 'No specific instructions provided')}
"""

@mcp.tool()
async def get_alerts(state: str) -&gt; str:
    """Get weather alerts for a US state.

    Args:
        state: Two-letter US state code (e.g. CA, NY)
    """
    url = f"{NWS_API_BASE}/alerts/active/area/{state}"
    data = await make_nws_request(url)

    if not data or "features" not in data:
        return "Unable to fetch alerts or no alerts found."

    if not data["features"]:
        return "No active alerts for this state."

    alerts = [format_alert(feature) for feature in data["features"]]
    return "\n---\n".join(alerts)

@mcp.tool()
async def get_forecast(latitude: float, longitude: float) -&gt; str:
    """Get weather forecast for a location.

    Args:
        latitude: Latitude of the location
        longitude: Longitude of the location
    """
    # First get the forecast grid endpoint
    points_url = f"{NWS_API_BASE}/points/{latitude},{longitude}"
    points_data = await make_nws_request(points_url)

    if not points_data:
        return "Unable to fetch forecast data for this location."

    # Get the forecast URL from the points response
    forecast_url = points_data["properties"]["forecast"]
    forecast_data = await make_nws_request(forecast_url)

    if not forecast_data:
        return "Unable to fetch detailed forecast."

    # Format the periods into a readable forecast
    periods = forecast_data["properties"]["periods"]
    forecasts = []
    for period in periods[:5]:  # Only show next 5 periods
        forecast = f"""
    {period['name']}:
    Temperature: {period['temperature']}°{period['temperatureUnit']}
    Wind: {period['windSpeed']} {period['windDirection']}
    Forecast: {period['detailedForecast']}
"""
        forecasts.append(forecast)

    return "\n---\n".join(forecasts)

if __name__ == "__main__":
    mcp.run(transport='stdio')

Connecting to Cursor

To use your weather server with Cursor's AI, you need to configure Cursor to connect to it. Add this to the mcp.json file from earlier in your Cursor settings (new entry under mcpServers):

{
  "weather": {
    "command": "python",
    "args": ["/absolute/path/to/your/weather.py"]
  }
}

Now when you chat with Cursor's AI, you can ask questions like:

"What are the current weather alerts in California?"
"Get me the forecast for San Francisco (latitude 37.7749, longitude -122.4194)"

Common Gotchas and Fixes

Path Issues: Always use absolute paths in your configuration. Relative paths often break because the working directory might not be what you expect.

API Errors: The National Weather Service API only works for US locations. If you're testing with international coordinates, you'll get errors.

Permission Problems: Make sure your Python script is executable and all dependencies are installed.

Tool Not Showing: If Cursor doesn't see your tools, restart Cursor after updating the configuration.

Understanding the Core Concepts

Now that you've built a working MCP server, let's understand the three main primitives MCP provides:

Tools: Let AI Models Take Actions

Tools are functions that AI models can call to perform actions. In our weather server, get_alerts and get_forecast are tools. Tools are model-controlled - the AI decides when to use them based on user requests.

Use tools when you want the AI to:

Fetch data from APIs
Execute commands
Perform calculations
Interact with external services

Resources: Give AI Models Access to Data

Resources are files or data that AI models can read for context. Unlike tools, resources are application-controlled - the user explicitly selects which resources to include.

Use resources when you want to provide:

File contents
Database records
API responses
Documentation

Prompts: Reusable Templates for Common Tasks

Prompts are predefined templates that help users accomplish specific tasks. They're user-controlled and appear as quick actions or slash commands in client applications.

Use prompts for:

Common workflows
Standardized formats
Multi-step processes
Team templates

Our MCP used all three of these.

Debugging and Troubleshooting

Things will go wrong when you're building MCP servers. Here's how to debug them effectively in Cursor.

Common errors:

Your server doesn't appear in Cursor's MCP list
Tools don't execute when the AI tries to use them
Error messages in the terminal when running your server
Cursor can't connect to your server

Essential Debugging Tools

MCP Inspector is your best friend for debugging. Install it globally:

npx @modelcontextprotocol/inspector path/to/your/weather.py

The Inspector lets you:

Test your server without an AI client
See exactly what tools and resources your server provides
Execute tools manually with custom inputs
View detailed error messages

Quick Fixes That Usually Work

Restart Cursor after changing MCP configuration
Use absolute paths everywhere in your config
Check the terminal for error messages when your server starts
Test with Inspector before connecting to Cursor
Verify your Python environment has all required packages

What's Next?

You've now experienced MCP as a user and built your own server. Here's how to keep growing your MCP skills.

Browse Existing MCP Servers for Inspiration

Check out the official MCP servers repository to see what others have built:

Database connectors (PostgreSQL, SQLite)
API integrations (GitHub, Slack, Google Drive)
Development tools (Git operations, code analysis)
Productivity tools (calendar, email, file management)

Each server shows different patterns and approaches you can learn from and draw inspiration for your own MCPs!

Learn how to use LLMs to build MCPs

Congrats! You've gone from knowing nothing about MCP to building a working server that can work with various clients to make AI more useful.

This guide has provided you with the foundational knowledge to start building servers for your specific needs. The AI tools you use every day are about to become much more powerful!

In the next article, we'll explore how to leverage MCP in Cursor to 10x your productivity as a developer!

AI Agents for Beginners: Building Your First AI Agent

Slaknoah — Sat, 07 Jun 2025 15:30:54 +0000

AI agents are everywhere right now, and for good reason. They're transforming how we interact with software, moving beyond simple question-answer systems to tools that can actually do things for us.

From customer support bots that can process refunds to coding assistants that can fix bugs across multiple files, agents are becoming the new interface between humans and complex systems.

But here's the thing: most "agents" you see aren't really agents at all. They're just chatbots following a script. Today, we're going to build a real agent—one that can think, decide, and act on its own. And don't worry, we'll keep it simple so even beginners can easily follow along!

Let's dive in! But first,

What Makes a True AI Agent?

What do we mean by a "real agent"?

The key distinction between a true agent and a simple automation lies in autonomy and dynamic decision-making. Let's illustrate with an analogy:

Workflows are like a GPS with a fixed route—if there's a roadblock, it can't adapt
Agents are like having a local guide who knows all the shortcuts and can change plans on the fly

Workflows are great for simple, well-defined tasks, but they often can't handle the complexity of very dynamic queries, unlike agents.

What we'll build

With the aid of the OpenAI SDK, we'll build a simple stock information agent that can answer questions about stocks and companies. It will be able to:

Fetch real-time stock prices using the Yahoo Finance API
Find company CEOs from stock data
Identify ticker symbols from company names
Ask for clarification when queries are ambiguous

What makes this a true agent is that it autonomously decides:

Which tool to use for each query
When to chain multiple tools together
When to ask the user for more information
How to handle errors and retry with different approaches

Prerequisites

Here's what you'll need:

Python 3.7 or higher
An OpenAI API key (get one at platform.openai.com)
Basic Python knowledge

Create a project directory and install the required packages:

mkdir stock-info-agent
cd stock-info-agent
pip install openai yfinance python-dotenv

Create a .env file in your project directory:

OPENAI_API_KEY=your_api_key_here

{/* */}

Building the Agent: A Step-by-Step Walkthrough

Let's build this agent from the ground up, understanding each component along the way.

1. Setting Up the Agent Class

First, we create our agent class with OpenAI integration:

import json
from typing import Optional, Dict, Any, List
from openai import OpenAI
import yfinance as yf
from dotenv import load_dotenv
import os

load_dotenv()

class StockInfoAgent:
    def __init__(self):
        self.client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))
        self.conversation_history = []

The conversation_history is crucial – it allows our agent to maintain context across multiple interactions, understanding, for example, that "their CEO" refers to the company discussed earlier.

2. Creating the Tools

Our agent needs tools to interact with the real world. Let's create them:

Stock Price Tool

def get_stock_price(self, ticker_symbol: str) -> Optional[str]:
    """Fetches the current stock price for the given ticker_symbol."""
    try:
        stock = yf.Ticker(ticker_symbol.upper())
        info = stock.info
        current_price = info.get('currentPrice') or info.get('regularMarketPrice')

        if current_price:
            return f"{current_price:.2f} USD"
        return None
    except Exception as e:
        print(f"Error fetching stock price: {e}")
        return None

This tool uses Yahoo Finance to fetch real-time stock prices. Notice the error handling—robust agents must handle failures gracefully.

CEO Finder Tool

def get_company_ceo(self, ticker_symbol: str) -> Optional[str]:
    """Fetches the name of the CEO for the company associated with the ticker_symbol."""
    try:
        stock = yf.Ticker(ticker_symbol.upper())
        info = stock.info

        # Look for CEO in various possible fields
        ceo = None
        for field in ['companyOfficers', 'officers']:
            if field in info:
                officers = info[field]
                if isinstance(officers, list):
                    for officer in officers:
                        if isinstance(officer, dict):
                            title = officer.get('title', '').lower()
                            if 'ceo' in title or 'chief executive' in title:
                                ceo = officer.get('name')
                                break
        return ceo
    except Exception as e:
        print(f"Error fetching CEO info: {e}")
        return None

This tool searches through different data structures to find the CEO, as Yahoo Finance doesn't have a standardized format.

Ticker Symbol Finder Tool

This tool is crucial for handling natural language queries about companies:

def find_ticker_symbol(self, company_name: str) -> Optional[str]:
    """Tries to identify the stock ticker symbol for a given company_name."""
    try:
        # Use yfinance Lookup to search for the company
        lookup = yf.Lookup(company_name)

        stock_results = lookup.get_stock(count=5)

        if not stock_results.empty:
            return stock_results.index[0]

        # If no stocks found, try all instruments
        all_results = lookup.get_all(count=5)

        if not all_results.empty:
            return all_results.index[0]

    except Exception as e:
        print(f"Error searching for ticker: {e}")

    return None

This tool allows users to refer to companies by name ("Apple", "Tesla", "that EV company") instead of needing to know ticker symbols.

The Clarification Tool

This is what makes our agent truly interactive, allowing it to ask the user for clarification when needed.

def ask_user_for_clarification(self, question_to_user: str) -> str:
    """Poses the question_to_user to the actual user and returns their typed response."""
    print(f"\nAgent needs clarification: {question_to_user}")
    response = input("Your response: ")
    return response

3. Defining Tools for OpenAI

The agent needs to know what tools are available and how to use them. We define them in OpenAI's function calling format:

def create_tool_definitions(self) -> List[Dict[str, Any]]:
    """Creates OpenAI function calling definitions for the tools."""
    return [
        {
            "type": "function",
            "function": {
                "name": "get_stock_price",
                "description": "Fetches the current stock price for the given ticker symbol",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "ticker_symbol": {
                            "type": "string",
                            "description": "The stock ticker symbol (e.g., 'AAPL', 'MSFT')"
                        }
                    },
                    "required": ["ticker_symbol"]
                }
            }
        },
        # ... (other tool definitions)
    ]

These definitions are like instruction manuals for the AI—they tell it what each tool does and what parameters it needs.

4. The Brain: Processing User Queries

Here's where the magic happens—the agent's decision-making loop:

def process_user_query(self, user_query: str) -> str:
    """Processes a user query using the OpenAI API with function calling."""
    self.conversation_history.append({"role": "user", "content": user_query})

    system_prompt = """You are a helpful stock information assistant. You have access to tools that can:
                    1. Get current stock prices
                    2. Find company CEOs
                    3. Find ticker symbols for company names
                    4. Ask users for clarification when needed

                    Use these tools to help answer user questions about stocks and companies. 
                    If information is ambiguous, ask for clarification."""

    while True:
        messages = [
            {"role": "system", "content": system_prompt},
            *self.conversation_history
        ]

        # Call OpenAI API with function calling
        response = self.client.chat.completions.create(
            model="gpt-4-turbo-preview",
            messages=messages,
            tools=self.create_tool_definitions(),
            tool_choice="auto"  # Let the model decide which tool to use
        )

        response_message = response.choices[0].message

        # If no tool calls, we're done
        if not response_message.tool_calls:
            self.conversation_history.append({"role": "assistant", "content": response_message.content})
            return response_message.content

        # Execute the tool the agent chose
        tool_call = response_message.tool_calls[0]
        function_name = tool_call.function.name
        function_args = json.loads(tool_call.function.arguments)

        print(f"\nExecuting tool: {function_name} with args: {function_args}")

        # Execute the tool
        result = self.execute_tool(function_name, function_args)

        # Add everything to conversation history
        self.conversation_history.append({
            "role": "assistant",
            "content": None,
            "tool_calls": [{
                "id": tool_call.id,
                "type": "function",
                "function": {
                    "name": function_name,
                    "arguments": json.dumps(function_args)
                }
            }]
        })

        self.conversation_history.append({
            "tool_call_id": tool_call.id,
            "role": "tool",
            "name": function_name,
            "content": str(result) if result is not None else "No result found"
        })

The key insight here is the while True loop—the agent keeps thinking and acting until it has a complete answer. It might use one tool, or five, or ask for clarification multiple times. This is true autonomy.

See Our Agent In Action

Here's a real conversation that demonstrates our agent's capabilities very well:

You: Who is the CEO of the EV company from China and what is its stock price?

Executing tool: ask_user_for_clarification with args: {'question_to_user': 'Are you referring to NIO, XPeng, or another Chinese EV company?'}
Agent needs clarification: Are you referring to NIO, XPeng, or another Chinese EV company?

Your response: BYD

Executing tool: find_ticker_symbol with args: {'company_name': 'BYD'}
Executing tool: get_company_ceo with args: {'ticker_symbol': 'BYDDF'}
Executing tool: get_stock_price with args: {'ticker_symbol': 'BYDDF'}

Agent: The CEO of BYD, the Chinese EV company, is Mr. Chuan-Fu Wang, and its current stock price is $59.50 USD.

The agent autonomously:

Recognized "EV company from China" was ambiguous
Asked which specific company
Found the ticker symbol for BYD
Retrieved the CEO information
Fetched the current stock price
Composed a complete answer

How to Use the Agent

Running the agent is simple, navigate to the project directory and run the following command:

python stock_agent.py

Try these example queries to see different behaviors:

Simple query:

You: What's Apple's stock price?
Agent: Apple's current stock price is $182.63 USD.

Ambiguous query requiring clarification:

You: Who runs that big EV company?
Agent: Are you asking about the CEO of Tesla?
You: Yes
Agent: The CEO of Tesla is Mr. Elon R. Musk.

Complex multi-tool query:

You: Compare the stock prices of Microsoft and Apple
Agent: Microsoft (MSFT) is currently trading at $415.26 USD, while Apple (AAPL) is trading at $182.63 USD.

Extending Your Agent

The modular design makes it easy to add new capabilities:

Add Market Analysis

def get_price_change(self, ticker_symbol: str, period: str = "1d") -> Dict[str, Any]:
    """Get price change over a period"""
    stock = yf.Ticker(ticker_symbol)
    hist = stock.history(period=period)
    if len(hist) >= 2:
        start_price = hist['Close'].iloc[0]
        end_price = hist['Close'].iloc[-1]
        change = end_price - start_price
        percent_change = (change / start_price) * 100
        return {
            "change": f"${change:.2f}",
            "percent": f"{percent_change:.2f}%"
        }

Add News Integration

def get_company_news(self, ticker_symbol: str) -> List[Dict[str, str]]:
    """Get recent news about the company"""
    stock = yf.Ticker(ticker_symbol)
    news = stock.news
    return [{"title": item['title'], "link": item['link']} for item in news[:5]]

Best Practices for Building Agents

Clear Tool Descriptions: Write descriptions as if explaining to a colleague
Graceful Error Handling: Always handle API failures and missing data
Conversation Context: Maintain history for natural interactions
User Transparency: Show which tools are being executed
Start Simple: Add complexity only when needed

For more helpful tips, check out Anthropic's guide.

The Complete Code

Click to expand the full implementation

import json
from typing import Optional, Dict, Any, List
from openai import OpenAI
import yfinance as yf
from dotenv import load_dotenv
import os

# Load environment variables
load_dotenv()

class StockInfoAgent:
    def __init__(self):
        self.client = OpenAI(api_key=os.getenv('OPENAI_API_KEY'))
        self.conversation_history = []

    def get_stock_price(self, ticker_symbol: str) -&gt; Optional[str]:
        """Fetches the current stock price for the given ticker_symbol."""
        try:
            stock = yf.Ticker(ticker_symbol.upper())
            info = stock.info
            current_price = info.get('currentPrice') or info.get('regularMarketPrice')

            if current_price:
                return f"{current_price:.2f} USD"
            return None
        except Exception as e:
            print(f"Error fetching stock price: {e}")
            return None

    def get_company_ceo(self, ticker_symbol: str) -&gt; Optional[str]:
        """Fetches the name of the CEO for the company associated with the ticker_symbol."""
        try:
            stock = yf.Ticker(ticker_symbol.upper())
            info = stock.info

            # Look for CEO in various possible fields
            ceo = None
            for field in ['companyOfficers', 'officers']:
                if field in info:
                    officers = info[field]
                    if isinstance(officers, list):
                        for officer in officers:
                            if isinstance(officer, dict):
                                title = officer.get('title', '').lower()
                                if 'ceo' in title or 'chief executive' in title:
                                    ceo = officer.get('name')
                                    break

            # Fallback to general company info
            if not ceo and 'longBusinessSummary' in info:
                ceo = None  

            return ceo
        except Exception as e:
            print(f"Error fetching CEO info: {e}")
            return None

    def find_ticker_symbol(self, company_name: str) -&gt; Optional[str]:
        """Tries to identify the stock ticker symbol for a given company_name."""
        try:
            # Use yfinance Lookup to search for the company
            lookup = yf.Lookup(company_name)

            stock_results = lookup.get_stock(count=5)

            if not stock_results.empty:
                return stock_results.index[0]

            # If no stocks found, try all instruments
            all_results = lookup.get_all(count=5)

            if not all_results.empty:
                return all_results.index[0]

        except Exception as e:
            print(f"Error searching for ticker: {e}")

        return None

    def ask_user_for_clarification(self, question_to_user: str) -&gt; str:
        """Poses the question_to_user to the actual user and returns their typed response."""
        print(f"\nAgent needs clarification: {question_to_user}")
        response = input("Your response: ")
        return response

    def create_tool_definitions(self) -&gt; List[Dict[str, Any]]:
        """Creates OpenAI function calling definitions for the tools."""
        return [
            {
                "type": "function",
                "function": {
                    "name": "get_stock_price",
                    "description": "Fetches the current stock price for the given ticker symbol",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "ticker_symbol": {
                                "type": "string",
                                "description": "The stock ticker symbol (e.g., 'AAPL', 'MSFT')"
                            }
                        },
                        "required": ["ticker_symbol"]
                    }
                }
            },
            {
                "type": "function",
                "function": {
                    "name": "get_company_ceo",
                    "description": "Fetches the name of the CEO for the company associated with the ticker symbol",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "ticker_symbol": {
                                "type": "string",
                                "description": "The stock ticker symbol"
                            }
                        },
                        "required": ["ticker_symbol"]
                    }
                }
            },
            {
                "type": "function",
                "function": {
                    "name": "find_ticker_symbol",
                    "description": "Tries to identify the stock ticker symbol for a given company name",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "company_name": {
                                "type": "string",
                                "description": "The name of the company"
                            }
                        },
                        "required": ["company_name"]
                    }
                }
            },
            {
                "type": "function",
                "function": {
                    "name": "ask_user_for_clarification",
                    "description": "Poses a question to the user and returns their response",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "question_to_user": {
                                "type": "string",
                                "description": "The question to ask the user"
                            }
                        },
                        "required": ["question_to_user"]
                    }
                }
            }
        ]

    def execute_tool(self, tool_name: str, arguments: Dict[str, Any]) -&gt; Any:
        """Executes the specified tool with given arguments."""
        if tool_name == "get_stock_price":
            return self.get_stock_price(arguments["ticker_symbol"])
        elif tool_name == "get_company_ceo":
            return self.get_company_ceo(arguments["ticker_symbol"])
        elif tool_name == "find_ticker_symbol":
            return self.find_ticker_symbol(arguments["company_name"])
        elif tool_name == "ask_user_for_clarification":
            return self.ask_user_for_clarification(arguments["question_to_user"])
        else:
            return None

    def process_user_query(self, user_query: str) -&gt; str:
        """Processes a user query using the OpenAI API with function calling."""
        # Add user message to conversation history
        self.conversation_history.append({"role": "user", "content": user_query})

        system_prompt = """You are a helpful stock information assistant. You have access to tools that can:
                        1. Get current stock prices
                        2. Find company CEOs
                        3. Find ticker symbols for company names
                        4. Ask users for clarification when needed

                        Use these tools to help answer user questions about stocks and companies. If information is ambiguous, ask for clarification."""

        while True:
            messages = [
                {"role": "system", "content": system_prompt},
                *self.conversation_history
            ]

            # Call OpenAI API with function calling
            response = self.client.chat.completions.create(
                model="gpt-4-turbo-preview",
                messages=messages,
                tools=self.create_tool_definitions(),
                tool_choice="auto"
            )

            response_message = response.choices[0].message

            # If no tool calls, we're done
            if not response_message.tool_calls:
                self.conversation_history.append({"role": "assistant", "content": response_message.content})
                return response_message.content

            # Execute the first tool call
            tool_call = response_message.tool_calls[0]
            function_name = tool_call.function.name
            function_args = json.loads(tool_call.function.arguments)

            print(f"\nExecuting tool: {function_name} with args: {function_args}")

            # Execute the tool
            result = self.execute_tool(function_name, function_args)

            # Add the assistant's message with tool calls to history
            self.conversation_history.append({
                "role": "assistant",
                "content": None,
                "tool_calls": [{
                    "id": tool_call.id,
                    "type": "function",
                    "function": {
                        "name": function_name,
                        "arguments": json.dumps(function_args)
                    }
                }]
            })

            # Add tool result to history
            self.conversation_history.append({
                "tool_call_id": tool_call.id,
                "role": "tool",
                "name": function_name,
                "content": str(result) if result is not None else "No result found"
            })

    def chat(self):
        """Interactive chat loop."""
        print("Stock Information Agent")
        print("Ask me about stock prices, company CEOs, or any stock-related questions!")
        print("Type 'quit' to exit.\n")

        while True:
            user_input = input("You: ")

            if user_input.lower() in ['quit', 'exit', 'bye']:
                print("Goodbye!")
                break

            try:
                response = self.process_user_query(user_input)
                print(f"\nAgent: {response}\n")
            except Exception as e:
                print(f"\nError: {e}\n")

if __name__ == "__main__":
    agent = StockInfoAgent()
    agent.chat()

Conclusion

Congratulations! You've built a true AI agent that can think, decide, and act autonomously. This isn't just a chatbot following a script—it's an intelligent system that can handle ambiguous queries, ask for clarification, and chain multiple tools together to solve complex problems.

The key takeaways:

True agents make their own decisions about how to solve problems
Simple, composable patterns beat complex frameworks
Start with basic tools and add complexity only when needed
Conversation memory and error handling are crucial

With this foundation, you can build agents for any domain, from financial analysis to customer support to personal assistants. The possibilities are endless when you give AI the ability to think and act autonomously.

Best of luck, and we can't wait to see what you build!

Set Up An E-Commerce Backend With Medusa In Just A Few Minutes

Slaknoah — Wed, 12 Mar 2025 12:59:25 +0000

Medusa is an open-source, customizable digital e-commerce platform designed to give you full control over your e-commerce backend. Unlike traditional platforms that provide a rigid, out-of-the-box solution, Medusa is built with flexibility in mind, allowing you to tailor every aspect of your store to suit your business needs.

Medusa comes with three main components:

Commerce Modules: A suite of core commerce functionalities like inventory tracking, cart totals, payment processing, order management, and more. These modules provide the essential tools to get your store up and running.
Customization Framework: Medusa is designed for customization. You can easily add custom API endpoints, integrate third-party services, introduce business logic, and build data models that fit your specific requirements. Additionally, you can create workflows and automations to streamline your operations.
Admin Dashboard: A fully customizable admin dashboard that empowers store owners and merchants to configure, manage, and operate their stores efficiently.

When you install Medusa, you’re not just getting an out-of-the-box e-commerce platform; you're getting a robust foundation that you can extend and modify to fit your business's vision.

This guide teaches you how to use Medusa to set up a standard full-stack E-commerce app in just a few minutes.

Let's get into it!

Who Should Use Medusa?

Medusa is ideal for businesses and teams that need a digital e-commerce platform that supports unique requirements not typically found in other out-of-the-box solutions. Whether you’re building a Direct-to-Consumer (D2C) brand, an Order Management System (OMS), a custom marketplace, or a global B2B platform, Medusa gives you the tools to bring your ideas to life.

Medusa works for businesses of all sizes, from startups to large enterprises. It’s also well-suited for development teams of all sizes, with a developer-friendly architecture that allows for easy management and deployment of projects.

Here are a few examples of how companies use Medusa:

Medusa empowers businesses to scale and customize their digital commerce solutions without being restricted by traditional e-commerce platforms, providing the flexibility to create truly unique and innovative experiences.

Prerequisites

Before you begin, ensure you have the following tools and technologies set up:

Node.js v20+: Medusa requires Node.js, so ensure you're using version 20 or higher. You can check your Node.js version with:

  node -v

If you need to install or update Node.js, visit the official website.

Git: Git is required to clone repositories and manage your project versioning.
Install Git from git-scm.com.
PostgreSQL: Medusa uses PostgreSQL as its default database. Ensure PostgreSQL is installed and running on your computer.

How to Set Up an E-commerce Application with Medusa

Follow these simple steps to set up your Medusa backend application—complete with a working admin dashboard and NextJS frontend client.

Step 1: Create Your Database

Before you start setting up Medusa, you’ll need to create a PostgreSQL database for your project. The name of your database should match the name of your Medusa project to ensure proper configuration and smooth operation.

Create a PostgreSQL Database

Log into PostgreSQL: Open your terminal and log into PostgreSQL by running the following command:

   psql -U postgres

This will open the PostgreSQL interactive terminal. You may be prompted for your PostgreSQL password.

Create a New Database: Once logged in, run the following command to create your new database. Replace my-medusa-store with your desired database name (this should match the name of your Medusa project):
```
CREATE DATABASE my_medusa_store;
```
This will create a new PostgreSQL database. You may also create a new database via a GUI tool such as pgAdmin.
Exit PostgreSQL: After creating the database, you can exit PostgreSQL by typing:
```
\q
```
Now that you’ve created your database, you’re ready to proceed with installing Medusa and configuring it to connect to this database.

Step 2: Create a New Medusa Project

With the database set up, it's time to create your new project. Use the create-medusa-app command to set up the backend:

npx create-medusa-app@latest my-medusa-store

Here, my-medusa-store is the name of your project’s directory. During installation, you will be prompted to choose whether you want to install the Next.js storefront along with the Medusa backend. You can choose to install it now or skip and add it later.

We'll assume you chose to install now for the rest of this guide.

The installation process will:

Set up a PostgreSQL database and connect it to your Medusa app.
Install Medusa and related dependencies.
Install a Next.js storefront if you chose to include it.

If successful, Medusa will run at http://localhost:9000, and the Medusa Admin dashboard will be available at http://localhost:9000/app.

The Next.js storefront will be available at http://localhost:8000.

Step 3: Run Medusa Application in Development

At this stage, your Medusa app should already be running so you may skip to the next step.

However, to start the Medusa application locally at any point, run the following command from your project directory:

npm run dev

This command starts both the backend at http://localhost:9000 and the Admin dashboard at http://localhost:9000/app.

Any changes made to the src directory will automatically restart the server, making development seamless.

Step 5: Set Up Medusa Admin User

You'll be prompted to set up an Admin User account the first time you run your Medusa app. Simply fill out the form to do so.

You may also do this using the Medusa CLI:

npx medusa user -e admin-medusa@test.com -p supersecret

Replace admin-medusa@test.com with your preferred email address and supersecret with a strong password.

This user will be able to log into the Admin dashboard at http://localhost:9000/app to manage products, orders, and customers.

Project Structure

Your Medusa project includes several important directories and files—here's a breakdown:

src/: The core of your custom development, containing sub-directories like:
- admin/: Custom UI widgets and routes for the Admin dashboard.
- api/: Custom API routes that extend Medusa’s functionality.
- modules/: Custom business logic for your store.
- jobs/: Scheduled tasks and cron jobs.
- scripts/: Custom scripts executed via the Medusa CLI.
- workflows/: Automated flows executed from anywhere in your application.
medusa-config.ts: The file where you define and configure the database and other settings for your Medusa backend.

Step 6 (Optional): Add a Product

Once your Medusa project is up and running, you can start adding products to your store. The Medusa Admin Dashboard provides a simple interface to manage your inventory, and adding a product is quick and easy.

Add a Product to Your Inventory

Access the Admin Dashboard: Open your browser and navigate to the Medusa Admin Dashboard at http://localhost:9000/app. Log in with the admin credentials you created during the setup process.
Navigate to the Inventory Section: On the left sidebar of the Admin Dashboard, click on Inventory. This will take you to the section where you can manage products.
Click the Create Button: In the Inventory section, click the Create button to add a new product to your store.
Fill in Product Details: A form will appear where you can input details for your new product. This includes:
- Title: Enter the name of the product.
- Description: Provide a description that gives customers more information about the product.
- SKU: Optionally, add a SKU (Stock Keeping Unit) for inventory tracking.
- Attributes: Various attributes describing the product.
Save the Product: Once you’ve filled out the product details, click Save to add the product to your inventory.

Your product is now part of your Medusa store, and you can view or edit it at any time from the Inventory section of the Admin Dashboard.

Step 7: Visit the Storefront

If you chose to install the NextJS storefront in the earlier step, you can visit http://localhost:8000/ to see it and interact with your e-commerce store.

If you didn't install earlier, follow these instructions if you'd like to now.

The storefront could serve as a great starting point for your custom e-commerce application.

Customize Your Medusa Store

Now that your Medusa application is up and running, you can start customizing it to fit your needs. Medusa is highly extendable, allowing you to:

Add custom modules: Extend Medusa’s functionality with your own business logic.
Create custom API endpoints: Add APIs to meet specific requirements for your store.
Install third-party plugins: Integrate payment gateways, shipping providers, and more.
Customize the admin dashboard: Modify the admin interface to meet your business needs.

Refer to the Official Medusa Documentation for instructions on how to do these.

Conclusion

Medusa is a powerful, open-source e-commerce platform designed to offer complete flexibility and control over your e-commerce store. Its modular architecture and customization capabilities make it an ideal solution for businesses seeking to go beyond traditional e-commerce setups.

With Medusa, you can easily manage inventory, orders, and customer data while customizing every aspect of your store to align with your unique business requirements. Whether you're building a Direct-to-Consumer platform, a marketplace, or a global B2B solution, Medusa equips you with the tools needed to innovate and scale.

By following this guide, you've set up a robust e-commerce application, added products to your inventory, accessed a highly-customizable storefront, and explored the various customization options available. Dive deeper into Medusa's features and continue to tailor your store to provide a truly unique shopping experience for your customers.

Seamless Payment Processing with Stripe and NestJS

Slaknoah — Wed, 26 Feb 2025 15:48:20 +0000

If you’ve been around the internet long enough, you’ve probably encountered Stripe in one way or another. It’s used by countless startups, indie creators, and massive companies to handle payments with minimal fuss. Today, it’s arguably one of the most popular choices for developers who want to accept payments online without dealing with the mountain of complexities that come with payment systems.

Meanwhile, NestJS has quickly gained momentum for being a Node.js framework that sports strong architectural patterns and TypeScript support—two attributes that many modern developers find attractive. Put them together, and you get a powerful combo for building dependable, structured, and maintainable payment-related features.

This article takes you through the basics of integrating Stripe with a NestJS application. You’ll learn how to set up your NestJS project and configure a Stripe Module that manages your payment flows in a clean and reusable way.

You’ll see how to handle common tasks like creating payment intents, listing products, and issuing refunds. Although we’ll go step by step, a basic familiarity with NestJS and TypeScript will be helpful—but don’t worry if you’re just starting out. We’ll keep things very simple.

Let's jump right in, and by the end, you should be able to create and manage payments or subscriptions in your own NestJS application using the official Stripe SDK.

What You’ll Learn

Project Setup: How to create a new NestJS project and install the necessary dependencies for Stripe.
Configuring Stripe: Methods for storing and injecting the Stripe API key using NestJS’s powerful dependency injection system.
Core Stripe Operations: How to handle essential flows in Stripe, including:
- Listing products
- Creating customers
- Creating payment intents
- Subscriptions
- Issuing refunds
- Generating payment links
Organizing Code: Strategies for placing Stripe logic in dedicated service and controller files for better maintainability.
Next Steps: Ways to expand your integration, from leveraging webhooks to handle advanced business logic to better securing your integration.

If that sounds interesting to you, let’s get building!

What You’ll Need

Before you write any code, ensure you have the following:

Node.js and npm: You should have Node.js installed (preferably the latest LTS version). npm (or Yarn, if you prefer) will be used to install the project’s dependencies.
Nest CLI: While not strictly mandatory, using the Nest CLI (@nestjs/cli) makes it simpler to generate new modules, controllers, and services. Install it globally if you want to generate boilerplate code with commands like nest g. Install by running:
```
$ npm i -g @nestjs/cli
```
Stripe Account: You’ll need a Stripe account to get an API key.
Basic NestJS Understanding: Some familiarity with NestJS modules, services, and controllers will help. We’ll still explain the code as we go, so if you’re a bit new, you can still keep up.

With these things in place, you’re ready to create a fresh NestJS project or add Stripe to an existing one.

Project Setup

Let’s begin by setting up a new NestJS project. You can do this using the Nest CLI:

$ nest new stripe-nest-tutorial

This command scaffolds a new NestJS project named stripe-nest-tutorial. Next, navigate into that directory:

$ cd stripe-nest-tutorial

Inside your newly created project, you’ll install Stripe and the NestJS Config Module. The Config Module helps with environment variables:

npm install stripe @nestjs/config

At this point, you have a simple NestJS application with a standard structure (e.g., an AppModule, AppController, etc.) and dependencies stripe and @nestjs/config.

Next, you want to create or edit an .env file at the root of the project for our environment variables. In that file, place your Stripe secret key:

STRIPE_API_KEY=sk_test_123456...

⚠️ Note: Don’t commit real secret keys to public repositories.

This environment variable will be read by your NestJS app via @nestjs/config.

Introducing the Stripe Module

Within NestJS, one of the best practices for integrating third-party services is to create a dedicated module. This module can handle all the configuration logic, controllers, and services related to Stripe. That way, you keep the rest of your application’s modules clean and free from payment-specific clutter.

Let’s break down the files that make up our Stripe integration. You’ll have:

stripe.module.ts: A module that sets up Stripe and provides the Stripe API key to the rest of the app.
stripe.service.ts: A service that talks directly to the Stripe SDK. This is where you’ll create payment intents, fetch products, and so on.
stripe.controller.ts: A controller that exposes our Stripe-related endpoints. You can call these endpoints from your frontend or from other parts of your application.

Run the following command in your terminal to generate these files:

$ nest g module stripe && nest g controller stripe && nest g service stripe

Let's now edit these boilerplate files to fit our needs. Edit stripe.module.ts as follows:

import { DynamicModule, Module } from '@nestjs/common';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { StripeController } from './stripe.controller';
import { StripeService } from './stripe.service';

@Module({})
export class StripeModule {
  static forRootAsync(): DynamicModule {
    return {
      module: StripeModule,
      controllers: [StripeController],
      imports: [ConfigModule.forRoot()],
      providers: [
        StripeService,
        {
          provide: 'STRIPE_API_KEY',
          useFactory: async (configService: ConfigService) =>
            configService.get('STRIPE_API_KEY'),
          inject: [ConfigService],
        },
      ],
    };
  }
}

Module Explanation

ConfigModule.forRoot(): Initializes the configuration system in NestJS, letting us use environment variables inside our modules and services.
providers: We define two providers here:
1. StripeService, which contains all the business logic.
2. An object that provides the actual 'STRIPE_API_KEY'. Notice the use of useFactory and the injection of ConfigService. This is a convenient way to retrieve the STRIPE_API_KEY from our .env file.

forRootAsync() is a pattern used in NestJS that allows for asynchronous or dynamic configuration. It’s handy when you need to load environment variables or perform other tasks during initialization.

Once our module is set up, we can import it in our app.module.ts:

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { StripeModule } from './stripe/stripe.module';

@Module({
  imports: [StripeModule.forRootAsync()],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

This line ensures that the Stripe module is available to the entire application.

Building the Stripe Service

Now let’s dive into the stripe.service.ts. This is where you’ll see how NestJS interacts with the official Stripe Node library. Here is the code:

import { Inject, Injectable, Logger } from '@nestjs/common';
import Stripe from 'stripe';

@Injectable()
export class StripeService {
  private stripe: Stripe;
  private readonly logger = new Logger(StripeService.name);

  constructor(
    @Inject('STRIPE_API_KEY')
    private readonly apiKey: string,
  ) {
    this.stripe = new Stripe(this.apiKey, {
      apiVersion: '2024-12-18.acacia', // Use latest API version, or "null" for your default
    });
  }

  // Get Products
  async getProducts(): Promise<Stripe.Product[]> {
    try {
      const products = await this.stripe.products.list();
      this.logger.log('Products fetched successfully');
      return products.data;
    } catch (error) {
      this.logger.error('Failed to fetch products', error.stack);
      throw error;
    }
  }

  // Get Customers
  async getCustomers() {
    try {
      const customers = await this.stripe.customers.list({});
      this.logger.log('Customers fetched successfully');
      return customers.data;
    } catch (error) {
      this.logger.error('Failed to fetch products', error.stack);
      throw error;
    }
  }

  // Accept Payments (Create Payment Intent)
  async createPaymentIntent(
    amount: number,
    currency: string,
  ): Promise<Stripe.PaymentIntent> {
    try {
      const paymentIntent = await this.stripe.paymentIntents.create({
        amount,
        currency,
      });
      this.logger.log(
        `PaymentIntent created successfully with amount: ${amount} ${currency}`,
      );
      return paymentIntent;
    } catch (error) {
      this.logger.error('Failed to create PaymentIntent', error.stack);
      throw error;
    }
  }

  // Subscriptions (Create Subscription)
  async createSubscription(
    customerId: string,
    priceId: string,
  ): Promise<Stripe.Subscription> {
    try {
      const subscription = await this.stripe.subscriptions.create({
        customer: customerId,
        items: [{ price: priceId }],
      });
      this.logger.log(
        `Subscription created successfully for customer ${customerId}`,
      );
      return subscription;
    } catch (error) {
      this.logger.error('Failed to create subscription', error.stack);
      throw error;
    }
  }

  // Customer Management (Create Customer)
  async createCustomer(email: string, name: string): Promise<Stripe.Customer> {
    try {
      const customer = await this.stripe.customers.create({ email, name });
      this.logger.log(`Customer created successfully with email: ${email}`);
      return customer;
    } catch (error) {
      this.logger.error('Failed to create customer', error.stack);
      throw error;
    }
  }

  // Product & Pricing Management (Create Product with Price)
  async createProduct(
    name: string,
    description: string,
    price: number,
  ): Promise<Stripe.Product> {
    try {
      const product = await this.stripe.products.create({ name, description });
      await this.stripe.prices.create({
        product: product.id,
        unit_amount: price * 100, // amount in cents
        currency: 'usd',
      });
      this.logger.log(`Product created successfully: ${name}`);
      return product;
    } catch (error) {
      this.logger.error('Failed to create product', error.stack);
      throw error;
    }
  }

  // Refunds (Process Refund)
  async refundPayment(paymentIntentId: string): Promise<Stripe.Refund> {
    try {
      const refund = await this.stripe.refunds.create({
        payment_intent: paymentIntentId,
      });
      this.logger.log(
        `Refund processed successfully for PaymentIntent: ${paymentIntentId}`,
      );
      return refund;
    } catch (error) {
      this.logger.error('Failed to process refund', error.stack);
      throw error;
    }
  }

  // Payment Method Integration (Attach Payment Method)
  async attachPaymentMethod(
    customerId: string,
    paymentMethodId: string,
  ): Promise<void> {
    try {
      await this.stripe.paymentMethods.attach(paymentMethodId, {
        customer: customerId,
      });
      this.logger.log(
        `Payment method ${paymentMethodId} attached to customer ${customerId}`,
      );
    } catch (error) {
      this.logger.error('Failed to attach payment method', error.stack);
      throw error;
    }
  }

  // Reports and Analytics (Retrieve Balance)
  async getBalance(): Promise<Stripe.Balance> {
    try {
      const balance = await this.stripe.balance.retrieve();
      this.logger.log('Balance retrieved successfully');
      return balance;
    } catch (error) {
      this.logger.error('Failed to retrieve balance', error.stack);
      throw error;
    }
  }

  // Payment Links
  async createPaymentLink(priceId: string): Promise<Stripe.PaymentLink> {
    try {
      const paymentLink = await this.stripe.paymentLinks.create({
        line_items: [{ price: priceId, quantity: 1 }],
      });
      this.logger.log('Payment link created successfully');
      return paymentLink;
    } catch (error) {
      this.logger.error('Failed to create payment link', error.stack);
      throw error;
    }
  }
}

Service Explanation

Constructor: We inject the STRIPE_API_KEY, which was registered in stripe.module.ts. The Stripe object is initialized here with the provided API key.
Logger: We’re using Nest’s Logger to give quick feedback about success or failure in each function. This can be replaced with your own logging strategy, but the built-in one is convenient for many use cases.
createPaymentIntent: This function is used to initiate a payment. It tells Stripe how much the payment is for and in which currency.
createSubscription: Subscriptions allow you to charge customers on a recurring basis. This is especially useful if you’re selling a SaaS product.
createCustomer: If you want to save user data and manage recurring billing or one-click payments, you’ll want to create customers in Stripe.
createProduct: Products are the items or services that you bill customers for. If you’re building an e-commerce platform, you’ll typically create a product first, then a price for that product.
refundPayment: Refunds are inevitable. This function uses the payment_intent identifier to issue a refund.
attachPaymentMethod: This function is handy when you want to attach a payment method (e.g., a credit card) to a customer for future billing or subscription creation.
getBalance: This is used to retrieve your Stripe balance. Useful for checking how much money is in your account.
createPaymentLink: Payment Links are a simplified way to get a ready-to-use link that you can share with your customers. When they open that link, they can handle the entire checkout flow without any extra backend code.

All these methods rely on this.stripe, which is your connected Stripe client. By wrapping them in a service, you can inject this logic wherever you need it inside your application. If you want to add or modify more functionality, you have a centralized place to do so.

Editing the Stripe Controller

Finally, we have the stripe.controller.ts file, which sets up our API routes, replace its code with this:

import { Body, Controller, Get, Post } from '@nestjs/common';
import { StripeService } from './stripe.service';

@Controller('stripe')
export class StripeController {
  constructor(private readonly stripeService: StripeService) {}

  @Get('products')
  async getProducts() {
    return this.stripeService.getProducts();
  }

  @Get('customers')
  async getCustomers() {
    return this.stripeService.getCustomers();
  }

  @Post('create-payment-intent')
  async createPaymentIntent(@Body() body: { amount: number; currency: string }) {
    const { amount, currency } = body;
    return this.stripeService.createPaymentIntent(amount, currency);
  }

  @Post('subscriptions')
  async createSubscription(@Body() body: { customerId: string; priceId: string }) {
    const { customerId, priceId } = body;
    return this.stripeService.createSubscription(customerId, priceId);
  }

  @Post('customers')
  async createCustomer(@Body() body: { email: string; name: string }) {
    return this.stripeService.createCustomer(body.email, body.name);
  }

  @Post('products')
  async createProduct(@Body() body: { name: string; description: string; price: number }) {
    return this.stripeService.createProduct(body.name, body.description, body.price);
  }

  @Post('refunds')
  async refundPayment(@Body() body: { paymentIntentId: string }) {
    return this.stripeService.refundPayment(body.paymentIntentId);
  }

  @Post('payment-links')
  async createPaymentLink(@Body() body: { priceId: string }) {
    return this.stripeService.createPaymentLink(body.priceId);
  }

  @Get('balance')
  async getBalance() {
    return this.stripeService.getBalance();
  }
}

Controller Explanation

@Controller('stripe'): This sets up a base route for all endpoints in this controller. That means every route in this file will start with /stripe.
@Get('products'): Fetches products. If you hit GET /stripe/products, you’ll get a list of your Stripe products.
@Post('create-payment-intent'): Creates a payment intent. Send a JSON body with the amount and currency.
@Post('subscriptions'): Creates a subscription. Send a JSON body with a customerId and a priceId.
@Post('customers'): Creates a new customer. The body should have an email and name.
@Post('products'): Creates a product with a price. The body takes name, description, and price (in dollars).
@Post('refunds'): Issues a refund. Send a paymentIntentId in the body to point Stripe to the correct payment.
@Post('payment-links'): Generates a payment link (with a pre-built frontend) you can share. Send a priceId in the body.
@Get('balance'): Retrieves the current Stripe account balance.

Notice how each route calls the corresponding function in our StripeService. This design keeps your code organized: the controller handles incoming requests, while the service manages the integration with Stripe. If you need to alter business rules or add extra logging, you can do so in the service without messing around in the controller.

Testing Your Routes

By this point, you have all the pieces of a functioning Stripe integration. You can run your NestJS application with:

npm run start:dev

Then, test any of your endpoints with your preferred tool—Postman, cURL, or even a frontend client:

Fetching Products:
- GET http://localhost:3000/stripe/products
Creating a Payment Intent:
- POST http://localhost:3000/stripe/create-payment-intent
- Body:
```
{
  "amount": 2000,
  "currency": "usd"
}
```
If everything’s set up properly, Stripe will respond with a JSON object containing the new payment intent. You can repeat this process for the other routes.

Improving and Expanding

After verifying that the basics work, you can expand your Stripe integration with additional steps:

Webhook Handling: Stripe can send events (webhooks) to your server whenever payments succeed or fail, or when a subscription is canceled, and more. Handling these webhooks allows you to automate tasks in your app (e.g., unlocking a feature after a successful payment). You can do this by creating a WebhookController in the same or different module and verifying the signature from Stripe.
Security Enhancements: NestJS’s guards, interceptors, and built-in tools can help protect your routes. For example, you might only let admins create products or retrieve the full list of customers. Read this guide to learn how.

The arrangement we have now—module, service, and controller—should give you a solid foundation to add all these features without your code becoming messy.

Stripe and NestJS are a powerful duo for projects that require robust, maintainable payment flows.

Check Stripe’s official documentation for more advanced operations, and if you'd like to build an entire e-commerce store with NestJS—complete with a sleek frontend and robust backend—checkout this our other guide.

Secure Your NestJS App: Implementing Authentication with Auth0

Slaknoah — Tue, 18 Feb 2025 12:02:29 +0000

With NestJS, a powerful framework for building scalable server-side applications, and Auth0, a leading provider of authentication services, you can easily implement robust authentication mechanisms.

This guide will walk you through securing your NestJS app using Auth0.

Let's get into it!

Introduction to Auth0

Auth0 is a flexible, drop-in solution to add authentication and authorization services to your applications.

Besides the standard username-password system, it supports various identity providers, including social login providers like Google and GitHub, and enterprise identity providers such as Microsoft Active Directory.

Auth0 simplifies the process of securing your application by handling user authentication, session management, and more for you, so you can focus on other parts of your app.

Prerequisites

Before we dive in, ensure you have the following:

Node.js.
Basic knowledge of NestJS and Javascript/Typescript.

Setting Up the Project

First, we set up a basic NestJS project.

1. Creating a New NestJS Project

Start by creating a new NestJS project:

$ npm i -g @nestjs/cli
$ nest new my-nestjs-app

Navigate into the project directory:

$ cd my-nestjs-app

2. Install Dependencies

Install the necessary packages:

$ npm install passport @nestjs/passport passport-jwt jwks-rsa

Configuring Auth0

Now let's set up Auth0.

First, create a free Auth0 account if you don't already have one.

After creating your account, set up an Auth Tenant. This tenant acts as a container for your identity service configuration and user data, isolated from other Auth0 customers, similar to individual apartments in a building.

Next, create an Auth0 API within your tenant. This API will handle authentication and authorization requests from your applications. Navigate to the APIs section of the Auth0 Dashboard and click 'Create API.'

Fill out the form as follows:

Name: NestJS API

Identifier: https://nestjs.demo.com

Signing Algorithm: RS256.

Auth0 allows you to create multiple APIs, each with a unique identifier (often structured as a URL) for easy differentiation. Note that Auth0 does not call these URLs.

After creating the API, you will see your Auth0 API details. Select the Quick Start tab to find guidance on setting up various backend technologies. Choose Node.js from the code box.

You will need two values from the code snippet in the box to configure Passport: an Auth0 Issuer URL and an Auth0 Audience.

These values should be stored in a .env (environment variables) file at the root of your project:

AUTH0_ISSUER_URL=https://<AUTH0-TENANT-NAME>.auth0.com/
AUTH0_AUDIENCE=https://nestjs.demo.com

Implementing Authentication

Now that we've set up Auth0, let's write the code to make use of it.

Authentication Flow

So, here's how our Authentication flow will work:

A User attempts a login from the frontend client.
Our NestJS backend receives the request and forwards it to Auth0 for authentication (Auth0 handles user authentication and manages users for you).
Auth0 issues a JWT to our client which it can then check for on any subsequent requests to grant access to protected resources.

Let's get into it!

1. Creating the Auth Module

Create a new Auth module. This module holds our core authentication logic:

$ nest g module auth

In auth/auth.module.ts, set up the module as follows:

import { Module } from '@nestjs/common';
import { PassportModule } from '@nestjs/passport';
import { JwtStrategy } from './jwt.strategy';

@Module({
  imports: [PassportModule.register({ defaultStrategy: 'jwt' })],
  providers: [JwtStrategy],
  exports: [PassportModule],
})
export class AuthModule {}

What's going on?

PassportModule: We import the PassportModule to integrate Passport.js, which is a popular middleware for handling various authentication strategies. In this case, we configure it to use JWT as the default authentication strategy.
JwtStrategy: This is the "strategy" that handles JWT validation. We will define it next, and it will be used to authenticate requests that include a valid JWT.
The AuthModule exports PassportModule so that we can use it elsewhere in the application.

2. Implementing the JwtStrategy

In NestJS, strategies are a core concept used to handle different types of authentication. A strategy defines how a specific kind of authentication is handled, such as verifying a JWT, handling OAuth tokens, or managing session cookies.

The JwtStrategy here is used to validate JSON Web Tokens (JWTs) received from the client, ensuring that only authorized requests can access protected resources.

Create a new strategy file auth/jwt.strategy.ts:

import { Injectable } from '@nestjs/common';
import { PassportStrategy } from '@nestjs/passport';
import { ExtractJwt, Strategy } from 'passport-jwt';
import { passportJwtSecret } from 'jwks-rsa';

@Injectable()
export class JwtStrategy extends PassportStrategy(Strategy) {
  constructor() {
    super({
      secretOrKeyProvider: passportJwtSecret({
        cache: true,
        rateLimit: true,
        jwksRequestsPerMinute: 5,
        jwksUri: `${process.env.AUTH0_DOMAIN}/.well-known/jwks.json`,
      }),
      jwtFromRequest: ExtractJwt.fromAuthHeaderAsBearerToken(),
      audience: process.env.AUTH0_AUDIENCE,
      issuer: `${process.env.AUTH0_DOMAIN}/`,
      algorithms: ['RS256'],
    });
  }

  validate(payload: unknown): unknown {
    return payload;
  }
}

What's going on?

secretOrKeyProvider: This is a function from jwks-rsa that fetches the public key from Auth0's JWKS (JSON Web Key Set) endpoint. This key is used to validate the JWT's signature.
jwtFromRequest: This tells Passport how to extract the JWT from the incoming request. We use ExtractJwt.fromAuthHeaderAsBearerToken(), which looks for the token in the Authorization header in the form of Bearer <token>.
audience and issuer: These are set to match the values that Auth0 uses to issue the tokens. They help ensure that the token is intended for our application and was issued by the correct source.
validate: This method is called after the JWT is validated. In this case, we simply return the payload, which will now contain information about the authenticated user, and will be the attached to the request.

Securing Routes

Now that we have our authentication logic in place, let’s secure our application routes.

1. Updating the App Module

We need to import the AuthModule into our main application module so that we can use it throughout the app.

In app.module.ts, import the AuthModule like this to make the authentication functionality available throughout our app:

import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { AuthModule } from './auth/auth.module';

@Module({
  imports: [AuthModule],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

2. Securing the Controller

Now, let’s protect the routes in the controller (which simply handles request routing) using the AuthGuard, which is essentially a gatekeeper for your routes.

In app.controller.ts, update the controller as follows:

import { Controller, Get, UseGuards } from '@nestjs/common';
import { AppService } from './app.service';
import { AuthGuard } from '@nestjs/passport';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {}

  @Get()
  getHello(): string {
    return this.appService.getHello();
  }

  @UseGuards(AuthGuard('jwt'))
  @Get('protected')
  getProtectedResource() {
    return { message: 'This is a protected resource' };
  }
}

What's going on?

@UseGuards(AuthGuard('jwt')): This decorator applies the JWT authentication guard to the getProtectedResource route. It ensures that any request to this route must include a valid JWT. If the token is invalid or missing, the request will be rejected with an unauthorized error.
AuthGuard('jwt'): This specifies that the JwtStrategy (the 'jwt' strategy) should be used for authentication.
getProtectedResource: This route will only be accessible to authenticated users. If the JWT is valid, the user will be able to access this resource.

We’ve now set up JWT-based authentication and protected certain routes to ensure only authorized users can access them.

So, let's test it.

Testing Authentication

Before we can test, we'll need to first register a client application with Auth0.

1. Register Client with Auth0

Follow these steps to register a client with Auth0:

Open the Auth0 Applications section of the Auth0 Dashboard.
Click on the Create Application button.
Provide a Name such as "Postman".
Choose Single Page Web Applications as the application type.
Click on the Create button.
Finally, note down your Domain, Client ID, and Client Secret. These will be used later.

💡 Tip

Under the Authentication tab in the left sidebar of your Auth0 dashboard, you can manage the different ways users can authenticate. 

For example, you can easily set up social login to allow users authenticate with their social profiles rather than just a username and password.

2. Test with Postman

To test authentication, you'll need a frontend client application. Here, we don't have one built so we'll be using Postman—an API testing platform—instead.

Step 1: Set Up Postman

Ensure you have Postman installed. If not, download it from the Postman website.

Step 2: Configure Postman to Get a Token from Auth0

Open Postman:
- Launch Postman and create a new request.
Go to the Authorization Tab:
- In your request, navigate to the "Authorization" tab.
- From the "Type" dropdown, select "OAuth 2.0".
Configure the Token Details:
- Click on the "Get New Access Token" button.
- Fill in the following details:
  - Token Name: Give your token a name (e.g., Auth0 Token).
  - Grant Type: Choose "Authorization Code".
  - Access Token URL: Use https://YOUR_AUTH0_DOMAIN/oauth/token (replace YOUR_AUTH0_DOMAIN with your actual Auth0 domain).
  - Auth URL: Use https://YOUR_AUTH0_DOMAIN/authorize
  - Client ID: Enter the Client ID from your Auth0 application settings that you noted down earlier.
  - Client Secret: Enter the Client Secret from your Auth0 application settings.
  - Audience: Use the audience specified in your API settings (your AUTH0_AUDIENCE in the .env file).
  - Client Authentication: Select "Send as Basic Auth Header".
```
ℹ️ Note

You need to add the URL in the `Callback URL` field to the "Allowed Callback URLs" field of your Auth0 Application's setting. 
```

Get the Token:
- Click "Request Token" to generate a new access token.
- Postman will send a request to Auth0 and fetch a token after you log in. Once successful, you'll see your token listed under "Available Tokens".
Use the Token:
- Click "Use Token" to attach it to your request.
- Postman will automatically add the token to the Authorization header as a Bearer token.

Step 3: Test Your API Endpoints

Make a request to the /protected http:localhost:3000/protected from Postman.

If everything is set up correctly, you'll receive a response indicating successful access to the protected resource (in this case a response containing a This is a protected resource message ).


text
{ message: 'This is a protected resource' }

Conclusion

And that's it! You have successfully secured your NestJS application using Auth0 and tested your secure API with Postman.

With these steps, you've built the groundwork for a robust authentication system. You may now explore additional Auth0 features like Role-Based Access Control (RBAC) to further enhance security.

What is a Cache Stampede? How to Prevent It Using Redis

Slaknoah — Tue, 21 Jan 2025 20:20:28 +0000

A cache stampede is an issue that businesses might not encounter often, but those who have experienced it have plenty of horror stories. Just ask Facebook — in 2010, they faced one of their worst outages, lasting four hours, due to a cache stampede.

So, what exactly is a cache stampede, and how can we prevent or mitigate it? Let’s dive in!

First, What is a Cache?

For those unfamiliar, a cache is a temporary storage layer that holds frequently accessed data. Think of it as keeping readers’ favorite books at the front counter of a library for easy and quick access, rather than having each user get them from the shelves every time.

A cache helps reduce the load on backend servers by providing quick access to commonly requested information.

PS: We’ll be using this library analogy throughout the article, so follow along!

What, then, is a Cache Stampede?

Now, imagine you have a high-traffic website that relies heavily on cached data to ensure smooth performance. Everything works seamlessly — until the cached data expires. Cached data needs periodic refreshing to stay current, and when it expires, the backend can suddenly become overwhelmed with requests.

It’s like all the readers discovering their favorite books are missing from the counter and rushing to the bookshelves at once — causing a “stampede.”

This surge overwhelms the library, just as expired cache data prompts a flood of requests to the backend, causing a massive spike in load and potentially crippling the system. This, in simple terms, is what’s known as a cache stampede.

More Technically Speaking…

Now let’s give a slightly more technical, yet more accurate explanation.

Cached data requires periodic regeneration to ensure it remains current, which naturally involves fetching updated data from the backend.

During this regeneration phase, clients querying the cache may encounter a “cache miss” — meaning the requested data is absent from the cache — unlike a “cache hit,” where the desired data is readily available.

The problem arises when too many clients experience a cache miss and all turn to the backend for data at once.

The result is a cascade of requests that can snowball into a race condition, where multiple threads compete for the same resource, ultimately degrading performance and possibly leading to system collapse.

How to Prevent a Cache Stampede Using Redis

Now that we know what a cache stampede is, let’s explore the ways we can prevent it using everyone’s favorite open-source caching server — Redis!

Redis?

Redis is an open-source, in-memory data store often used as a database, cache, and message broker. Its primary advantage lies in its ability to store and access data with lightning-fast speed, making it an ideal choice for caching.

Redis helps your application quickly retrieve frequently accessed data, reducing the need to constantly query the backend.

Preventing a Cache Stampede with Redis

Redis, with advanced features such as distributed locks, provides several ways to mitigate cache stampedes.

Let’s dive into these techniques—we’ll briefly explain each one, explore its tradeoffs, and then give a small code snippet depicting basic implementation.

1- Mutex Locking

Mutex locking is a method to ensure that only one process can regenerate a piece of cache data at a time.

This would be like allowing only one reader to fetch a favorite book or a few favorite books on behalf of multiple others.

In a cache stampede scenario, mutex locks prevent multiple clients from overwhelming the backend by ensuring that only one client fetches the data, while the others wait for the cache to be updated.

The idea here is simple: when a piece of data needs to be regenerated, only one client does the work while others wait.

How It Works:

When a cache miss occurs, the client attempts to acquire a lock.
If the lock is acquired, the client proceeds to regenerate the cache and updates Redis with the new data.
Once the data is refreshed, the lock is released.
If the lock is not acquired (because another client is already regenerating the cache), the client can either wait for the lock to be released or return a default response.

Pros:

Prevents multiple clients from simultaneously hitting the backend, reducing load and preventing a stampede.
Simple to understand and implement with Redis.

Cons:

Requires careful management of the lock to avoid deadlocks (a situation where two or more processes are unable to proceed because each is waiting for the other to release resources) or delays if a client fails to release it.
May introduce slight latency for clients waiting for the lock to be released.
If the lock TTL (time-to-release) is not managed properly, it could lead to stale locks, where no process can acquire the lock again.

Implementation:

if (redis.setnx('lock:key', '1')) {

    // The lock was successfully acquired

    try {

        let data = fetchDataFromBackend();

        redis.setex('cache:key', 3600, data); // Cache the data for 1 hour

    } finally {

        redis.del('lock:key'); // Always release the lock

    }

} else {

    // Lock not acquired; handle accordingly (e.g., wait or return stale data)

}

2- Cache Warming

Instead of placing a fixed number of favorite books at the counter every morning, imagine you — being the hardworking librarian that you are — replenished the supply at the counter from time to time.

This proactive approach ensures that readers will always find their favorites at the counter, never needing to make their way to the shelves.

In the world of caching, this proactive strategy is known as Cache Warming. By refreshing the cache with frequently requested data before it expires, you reduce the likelihood of a cache stampede.

Instead of multiple clients bombarding the backend when a cache miss occurs, the system preloads critical data, ensuring smooth and efficient access.

How It Works:

Identify the data most frequently accessed or critical to your application’s performance.
Use background jobs, cron tasks, or application logic to periodically refresh these cache entries before they expire.

Pros:

Significantly reduces the risk of cache misses, thus preventing a stampede.
Ensures that users experience minimal latency as the data they request is almost always ready in the cache.

Cons:

Requires accurate prediction of cache expiration and usage patterns to be effective.
Can lead to unnecessary backend calls and resource usage if the data isn’t as frequently requested as anticipated.
Refreshing the cache in the background requires additional computational resources.

Implementation:

setInterval(() => {

    let data = fetchDataFromBackend();

    redis.setex('cache:key', 3600, data); // Cache the data for 1 hour

}, refreshInterval);

3- Stale-While-Revalidate

The Stale-While-Revalidate (SWR) strategy offers a practical balance between speed and freshness.

In this approach, when a client requests data, it serves the existing cached version (even if it’s slightly stale) while simultaneously triggering a background process to update the cache with fresh data from the backend.

Following our library analogy, this would be like offering users slightly older copies of favorites while your assistant fetches a new batch of the latest copies from the shelves.

How It Works:

When a request comes in, the system first checks the cache.
If the cache has data, even if it’s stale, it immediately serves this data to the client to ensure low latency.
In parallel, a background process fetches fresh data from the backend and updates the cache for future requests.

Pros:

Provides immediate data to users, minimizing wait time.
Asynchronous updating keeps the cache relevant without blocking user requests.

Cons:

Requires a tolerance for slightly outdated data being served.
Additional complexity in managing the synchronization of background refresh tasks.

Implementation:

let cacheData = redis.get('cache:key');

if (cacheData) {

    // Serve stale data and trigger a refresh in the background

    refreshCacheInBackground('cache:key');

    return cacheData;

} else {

    // Fallback to backend data fetching and cache update

    let data = fetchDataFromBackend();

    redis.setex('cache:key', 3600, data);

    return data;

}

function refreshCacheInBackground(key) {

    setTimeout(() => {

        let freshData = fetchDataFromBackend();

        redis.setex(key, 3600, freshData);

    }, 0);

}

4- Distributed Caching and Load Balancing

Distributed caching involves spreading cached data across multiple servers or regions to ensure high availability, fault tolerance, and load distribution.

Think of it as setting up multiple counters in different locations in the library, each holding a collection of reader favorites, thus distributing the “demand” and reducing the traffic at any one counter.

In a distributed caching system, load balancing ensures that requests are efficiently routed to the cache node best suited to serve them, preventing any single server from becoming a bottleneck. This strategy leverages geographic proximity and network efficiency to optimize response times and reduce the load on any individual backend server.

Pros:

Improved Availability: Data is available even if some nodes fail.
Scalability: Easily handle increased traffic by adding more nodes.
Reduced Latency: Users are served by the nearest cache node, reducing response times.

Cons:

Complexity: Requires careful configuration and management of distributed nodes.
Data Consistency: Ensuring consistency across multiple nodes can be challenging.

Redlock

In distributed systems, the use of locks—as you can imagine—can quickly get messy. This is why a reliable lock mechanism, such as Redlock, is crucial to ensure that only one process can modify a shared resource at a time.

Redlock is an algorithm designed for distributed locking using Redis, providing fault tolerance and ensuring that the lock is correctly managed across multiple Redis nodes.

How It Works:

Acquire Lock: The client tries to acquire the lock on the majority of Redis nodes.
Set Expiry: Each lock has an expiration to prevent deadlocks if the client fails to release it.
Consensus: The lock is considered acquired if the client manages to lock it on a majority of nodes within a given timeframe.
Release Lock: Once the operation is complete, the client releases the lock across all nodes.

Pros:

Fault Tolerance: Works even if some nodes fail.
Avoids Single Point of Failure: The lock isn’t dependent on a single Redis instance.

Cons:

Complexity: More complex to implement compared to single-node locking.
Slight Overhead: Involves multiple Redis nodes, which could introduce slight delays.

Basic NestJS-based Implementation of Redlock:

import { Injectable } from '@nestjs/common';

import Redis from 'ioredis';

import Redlock from 'redlock';

@Injectable()

export class LockService {

    private redisClients: Redis[];

    private redlock: Redlock;

constructor() {
    this.redisClients = [
        new Redis({ host: '127.0.0.1', port: 6379 }),
        new Redis({ host: '127.0.0.2', port: 6379 }),
        // Add more nodes as needed
    ];

    this.redlock = new Redlock(
        this.redisClients,
        {
            driftFactor: 0.01, // time drift factor
            retryCount: 10,    // max number of retries
            retryDelay: 200,   // time in ms between retries
        }
    );
}

async acquireLock(resource: string, ttl: number): Promise&lt;any&gt; {
    try {
        const lock = await this.redlock.acquire([resource], ttl);
        return lock;
    } catch (err) {
        console.error('Failed to acquire lock:', err);
        throw err;
    }
}

async releaseLock(lock: any): Promise&lt;void&gt; {
    try {
        await lock.release();
    } catch (err) {
        console.error('Failed to release lock:', err);
    }
}



}

5- Proactive Cache Updates

Proactive cache updates involve updating the cache whenever a write operation occurs in the database.

This ensures that the cache always reflects the latest state of the data, akin to a librarian immediately replacing old copies of reader favorites at the counter with the latest editions (in our imaginary library books get outdated fast!).

This approach, also known as write-through, eliminates cache misses for updated data, as the cache is refreshed in real-time alongside database updates.

Pros:

Consistency: Cache always reflects the latest data, reducing stale data risks (users will never find an outdated copy at the counter).
Reduced Latency: Users receive updated data without waiting for cache regeneration.

Cons:

Increased Write Load: Every database write triggers a cache update, potentially increasing the load.
Complexity: Requires integration between database write operations and cache updates.

Implementation:

db.on('write', (data) => {

  redis.setex(cache:key:${data.id}, 3600, JSON.stringify(data));

});

6- Rate Limiting

Rate limiting controls the number of requests a client can make in a given time frame.

It’s like imposing a limit on how many books a reader can pick per hour or per day. This prevents any single client or group of clients from overwhelming the system with too many requests in a short period.

Rate limiting helps mitigate the risk of a stampede by throttling requests, ensuring the backend and cache can handle the load gracefully.

Pros:

Protection Against Overload: Prevents system overload by limiting excessive requests.
Fair Resource Distribution: Ensures all clients get fair access to resources.

Cons:

Potential User Frustration: Users may be denied service if they hit the limit, potentially harming their experience.
Complex Configuration: Requires careful tuning to balance user experience and system protection.

Implementation:

const limit = 10; // Max 10 requests per minute

const ttl = 60; // Time-to-live in seconds

async function rateLimit(clientId) {

    const current = await redis.incr(rate:${clientId});

    if (current === 1) {

        await redis.expire(rate:${clientId}, ttl);

    }

    if (current > limit) {

        throw new Error('Rate limit exceeded');

    }

}

7- Probabilistic Early Expiration

Probabilistic early expiration involves renewing cache entries before they expire based on a probabilistic decision.

This would be like replenishing the supply of some randomly selected favorites at the front counter before it actually runs out.

This essentially means some cached data would be refreshed at random before it even expired, smoothing out cache regeneration and avoiding sudden bursts of traffic.

Pros:

Reduced Stampede Risk: Spreads cache regeneration over time, preventing sudden backend spikes.
Improved System Stability: Reduces the chance of simultaneous cache misses leading to a stampede.

Cons:

Complexity: Requires careful tuning of the probabilistic model.
Potential Resource Waste: Some cache entries may be renewed unnecessarily.

Implementation:

function shouldExpireEarly(ttl) {

    const probability = Math.min(1, (1 - (ttl / maxTtl)) * scalingFactor);

    return Math.random() < probability;

}

if (shouldExpireEarly(cacheTtl)) {

    let data = fetchDataFromBackend();

    redis.setex('cache:key', 3600, data);

}

Conclusion

There you have it! We’ve explored various ways you can mitigate the cache stampede problem, leveraging the time-tested and trusted Redis.

The right approach for you will of course depend on your system’s requirements such as latency and tolerance for stale data, as well as the complexity you’re willing to manage.

By implementing these techniques, you can protect your backend from sudden load spikes and ensure a smooth experience for your users (or perhaps maintain a quiet library!).

Step-by-Step Guide to Queue Processing in a NestJS Application

Slaknoah — Fri, 17 Jan 2025 17:12:39 +0000

Resource-intensive tasks such as processing files or generating reports can quickly weigh down a NestJS application, causing slow response times and poor user experiences. Studies show that 53% of users will abandon a site if it takes longer than three seconds to load. You can employ a queue processing system to keep your application responsive while offloading heavy lifting

In this guide, we’ll explore how to set up queue processing in a NestJS application using Bull (a popular Node.js queue library) and Redis. Whether you’re optimizing an existing app or building a new one that needs to scale, this approach will help you efficiently manage background tasks without slowing down the user interface

What Are Queues and Jobs, and Why Do You Need Them?

NestJS is a progressive Node.js framework that leverages TypeScript to build robust, reliable, and scalable server-side applications. It brings together concepts from object-oriented programming (OOP), functional programming (FP), and functional reactive programming (FRP) into a well-structured architecture that makes it easier to create maintainable and modular code.

A queue can be incredibly helpful for handling heavy tasks in a NestJS application. Think of a queue like a conveyor belt in a factory: tasks are added to the belt (queue) and processed one by one. This allows your application to stay responsive because it doesn’t have to process every task immediately, as time-consuming tasks are offloaded to specialized workers in the background.

By handling these tasks over to a queue and its workers, your main application can continue serving requests without bottlenecks or long wait times.

What Are Jobs?

A job is a specific task in the queue. For example, a job could be "Send an email to user@example.com." or "Generate a report for order #12345."

Each job contains the data needed for processing and is executed independently. This modularity ensures flexibility and scalability.

Benefits of Using Queues in Applications

Queues are not just for handling heavy tasks; they bring significant advantages to your application, such as:

Improved Performance: Offloading time-consuming tasks ensures the main app remains fast and responsive.
Scalability: You can deploy multiple workers to process tasks in parallel, enabling your app to handle more users and jobs.
Error Handling and Retries: Failed jobs can be retried automatically, reducing manual intervention.
Scheduling and Prioritization: You can schedule tasks for later (e.g., sending reminders) or prioritize critical jobs.
Resilience: With queues, your app can recover from failures without losing data, as jobs persist until they are processed.

Prerequisites and Setting Up Queue Processing

Before we start, make sure you have:

Node.js Installed - Confirm Node.js and npm (Node Package Manager) are set up on your machine.
NestJS Knowledge - Basic understanding of NestJS, particularly modules and services.
Redis Installed - Redis is essential for managing queues. Install it using one of these methods:

macOS:

brew install redis  
brew services start redis

Linux:

sudo apt update  
sudo apt install redis-server  
sudo systemctl start redis

Docker:

docker run -d -p 6379:6379 redis

Step 1: Installing Dependencies

First, install the required libraries:

npm install @nestjs/bull bull  
npm install -save-dev @types/bull

@nestjs/bull: Integrates Bull with NestJS, making queue management easy.
bull: A powerful queue library built on Redis.
@types/bull: Adds TypeScript support for Bull, enabling better development practices.

Step 2: Configuring Redis in AppModule

To set up queue processing in NestJS, you need to configure Redis in the AppModule. The BullModule.forRoot method is used to establish a global connection to Redis, ensuring all queues in your application can interact with it. In this implementation, the connection details like host, port, username, and password are fetched using NextJS configService to pull environment variables for flexibility and security.

import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bull';

@Module({
  imports: [
    BullModule.forRoot({
      redis: {
        host: configService.get('REDIS_HOST'),
        port: configService.get('REDIS_PORT'),
        username: configService.get('REDIS_USERNAME'),
        password: configService.get('REDIS_PASSWORD'),
      },
    }),
  ],
})
export class AppModule {}

Step 3: Registering a Queue in Your Module

Once Redis is configured, you need to register specific queues in the module where they will be used. For instance, if you’re handling tasks related to processing sheet data, you can register the sheets queue in the SheetsModule. This is achieved using the BullModule.registerQueue method, where you can also define default job options like retries and back-off strategies.

import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bull';
import { SheetsService } from './sheets.service';
import { SheetsProcessor } from './sheets.processor';

@Module({
  imports: [
    BullModule.registerQueue({
      name: 'sheets',
      defaultJobOptions: {
        attempts: 3,
        backoff: {
          type: 'exponential',
          delay: 1000,
        },
      },
    }),
  ],
  providers: [SheetsService, SheetsProcessor],
})
export class SheetsModule {}

Step 4: Creating a Queue Service

A Queue Service is like a “manager” that adds tasks (jobs) to the queue. You may ask, Why do you need a queue service? Without a dedicated service, adding tasks to the queue would require repetitive logic in multiple places, leading to messy and hard-to-maintain code.

Here’s how to create the service in tasks.service.ts:

import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bull';
import { Queue } from 'bull';

@Injectable()
export class TasksService {
  constructor(@InjectQueue('tasks-queue') private readonly tasksQueue: Queue) {}

  async addTask(data: any) {
    await this.tasksQueue.add('process-task', data);
  }
}

Injecting the Queue: @InjectQueue('tasks-queue') gives access to the 'tasks-queue'.
Adding a Job: tasksQueue.add('process-task', data) adds a job named 'process-task' to the queue, along with the data needed for processing.

Example Usage:
If a user submits a contact form, you can add the task like this:

await tasksService.addTask({ email: '<user@example.com>', message: 'Hello!' });

Step 5: Creating a Queue Processor

The Queue Processor is responsible for handling jobs in the queue. Here’s how to create the processor in tasks.processor.ts:

import { Processor, Process } from '@nestjs/bull';
import { Job } from 'bull';

@Processor('tasks-queue')
export class TasksProcessor {
  @Process('process-task')
  async handleTask(job: Job) {
    console.log('Processing task:', job.data);
    // Write the job logic here
  }
}

Processor Annotation: @processor('tasks-queue') tells NestJS this class handles jobs in the 'tasks-queue.’
Process Annotation: @process('process-task') specifies the type of job this method will handle.
Processing Logic: handleTask(job: Job) receives the job and executes the required logic, such as sending an email or processing a file.

Step 6: Enhancing with Advanced Features

Let’s add more functionality to make your queue processing robust.

Error Handling and Retries

Configure retries for failed jobs:

BullModule.registerQueue({
  name: 'tasks-queue',
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 1000,
    },
  },
});

Scheduling Jobs

Schedule tasks for later execution:

await tasksQueue.add('reminder', { userId: 123 }, { delay: 60000 }); // 1-minute delay

Common Challenges in Queue Processing

Queue processing is a powerful way to handle background tasks, but it comes with its own set of challenges. Recognizing and addressing these challenges ensures a more reliable and efficient system.

Handling Job Failures

One of the most common issues in queue processing is job failure. Failures can occur for various reasons, including invalid input data, network disruptions, or external API outages. If not addressed, failed jobs can clog the queue or result in incomplete processes.

To mitigate this, it’s important to implement robust error-handling mechanisms. For example, you can configure retries for transient errors such as a network timeout. Additionally, failed jobs should be logged and monitored to identify recurring issues. Tools like Bull allow you to define retry strategies, including exponential backoff, to handle failures gracefully without overwhelming your system.

Stuck Jobs

Stuck jobs occur when a task remains unprocessed because of a misconfiguration, worker crash, or dependency issue. For instance, a queue worker might lose connection to Redis, leaving the job in a pending state indefinitely.

Detecting and clearing stuck jobs is critical to maintaining queue health. Monitoring tools like Bull Dashboard or custom logging systems can help identify jobs in the queue for too long. A proactive approach is to implement job timeouts. Setting a maximum processing time for each job ensures unresponsive tasks are marked as failed and re-queued or logged for manual intervention.

Scaling Bottlenecks

As traffic increases, a single queue worker might become insufficient to handle the load. This leads to longer queue lengths and slower processing times, which can degrade the user experience.

A key solution is scaling queue workers dynamically based on load. For example, during peak hours in an e-commerce platform, you can increase the number of workers to process payment confirmations faster. Leveraging tools like Kubernetes or AWS Auto Scaling allows you to automate this scaling process.

Debugging Complexity

Debugging issues in queue processing can be challenging because jobs are executed asynchronously. Identifying why a job failed or analyzing its payload can be time-consuming without proper logging or monitoring.

Implementing structured logs for each step of the job’s lifecycle, such as creation, processing, success, or failure, can simplify debugging. Attaching metadata to jobs (such as timestamps and worker identifiers) also helps trace their execution history.

Best Practices for Queue Management

Efficient queue management is essential for maintaining a scalable and reliable system. By following industry best practices, you can optimize performance, reduce failures, and ensure smooth operations.

Establish Clear Job Naming and Data Structures

Each job should have a descriptive name and well-defined data schema. For example, instead of naming a job “task1,” name it “sendEmailNotification.” This makes it easier to track job types and understand their purpose.

Additionally, a structured format for job data should be used. For instance, if a job involves sending an email, the payload should include fields like recipient, subject, and message. Clear naming and structured data make debugging and monitoring more effective.

Monitor Queue Performance

Regularly monitoring queue health is crucial for detecting and resolving issues early. Tools like Bull Dashboard or custom monitoring solutions can provide insights into:

The number of pending, active, and completed jobs.
Average job processing time.
Failure rates and error patterns.

Implement Scalable Architectures

As your application grows, so will the volume of jobs. A scalable queue architecture ensures your system can handle this growth without degradation in performance.

One effective strategy is to distribute job processing across multiple workers. For example, if your queue involves tasks like sending emails, processing invoices, and updating databases, you can assign specific workers to each job type. This specialization reduces contention and improves overall throughput.

Use Retry and Backoff Strategies

Retries are essential for handling transient failures, such as temporary API outages or network issues. However, blindly retrying a job without delay can exacerbate the problem by overwhelming external systems.

Backoff strategies, such as exponential delays between retries, prevent this scenario. For example, if an API call fails, you might retry after 2 seconds, then 4 seconds, and so on. This gradual approach increases the chances of success while reducing strain on the system.

Key Metrics to Track for Queue Optimization

Tracking key performance metrics ensures that your queue system runs efficiently and can scale to meet demand. Here are some of the most important metrics:

Queue Length

Queue length measures the number of pending jobs waiting to be processed. A consistently growing queue length indicates that workers are overwhelmed or under-resourced. This metric helps identify bottlenecks and prompts scaling decisions.

Job Latency

Latency is the time a job spends waiting in the queue before being processed. High latency can cause delayed task execution, affecting user experience. Monitoring latency helps ensure critical tasks are processed promptly.

Processing Time

Processing time refers to how long it takes to execute a single job. Tracking this metric helps optimize worker performance by identifying slow processes or inefficient code.

Failure Rates

Failure rates indicate the percentage of jobs that fail during execution. A high failure rate can signal issues like bad input data, unstable external dependencies, or misconfigured workers. Regularly reviewing failure rates ensures system reliability.

Security Considerations in Queue Processing

Queues often handle sensitive data, making security a critical aspect of their implementation. Here are key considerations to ensure your queue system is secure:

Protect Redis Connections

Redis, as the backbone of most queue systems, must be secured to prevent unauthorized access. Use strong passwords and enable SSL/TLS to encrypt data in transit.

Validate Job Data

Job payloads should be validated to meet expected formats and contain no malicious content. For example, if your job expects an email address, validate its structure before processing.

Implement Role-Based Access Control (RBAC)

Restrict who can create, modify, or delete jobs in the queue. This prevents unauthorized access and accidental disruptions to critical processes.

Limit Job Payload Sizes

Large payloads can increase processing time and expose sensitive information. Keep job data minimal and use secure storage solutions for larger files.

Monitor for Anomalies

Monitor queue activity regularly for unusual patterns, such as unexpected spikes in job volume or repeated failures. These could indicate an attempted attack or misconfiguration.

Real-World Applications of Queue Processing

Queue processing is widely used across industries to enhance application performance and scalability. Here are some practical examples:

E-Commerce Applications

Queues are indispensable in e-commerce platforms, where they handle tasks like processing payments, sending order confirmations, and updating inventory. When a user places an order, the platform can add a job to the queue to generate an invoice and send it via email. This ensures the user doesn’t have to wait while these tasks are completed.

During peak shopping periods like Black Friday, queues also help manage spikes in traffic by distributing tasks across multiple workers.

Media Processing

In media and entertainment platforms, queue processing is used for tasks like video encoding, thumbnail generation, and content delivery. For example, if a user uploads a video, the system can queue the task for encoding into different formats. This allows the user to continue interacting with the platform while the video is processed in the background.

Finance and Banking

Financial systems rely on queues to ensure secure and reliable transaction processing. Banks sometimes queue credit card transactions for fraud analysis before final approval. Queues also handle recurring tasks like sending monthly account statements or reconciling data with third-party services.

Healthcare Applications

Healthcare platforms use queues for processing patient data, scheduling appointments, and managing health records. For example, when a doctor uploads a large medical report, the system queues the file for processing and encryption before making it available to the patient.

Conclusion

This step-by-step guide explored queue processing in a NestJS application using Bull and Redis. By offloading time-consuming tasks to background queues, you can enhance your app’s responsiveness and scalability.

With a clear understanding of how to set up queues, handle common challenges, and follow best practices, you are now equipped to manage background tasks efficiently. Implementing queues not only optimizes performance but also ensures a smooth user experience. Start applying these techniques to build faster and more reliable applications with NestJS.

The Future of Transit Technology: Innovations Shaping Public Transport

Slaknoah — Wed, 20 Nov 2024 09:13:00 +0000

Technology is transforming nearly every aspect of our lives, and public transportation is no exception. As cities grow and populations increase, the need for smarter, more efficient transit systems has become more pressing. The future of transit technology lies in innovations that not only improve operational efficiency but also enhance the overall passenger experience. In this article, I’ll explore the key innovations shaping public transport, drawing from my own experience building TOST, a transit management system for US-based operators.

The Current Landscape of Public Transit

Before diving into the innovations, it’s important to understand the challenges that public transportation faces today. From outdated infrastructure to inefficient route planning, many cities struggle to keep up with growing commuter demands. The result? Increased traffic congestion, unreliable schedules, and a poor user experience.

Traditional transit management systems have long relied on manual processes, leaving little room for optimization. However, with advancements in technology, we’re now seeing a shift towards more intelligent systems that leverage real-time data, automation, and even artificial intelligence (AI) to optimize every aspect of the journey.

The Rise of Intelligent Transit Systems

1. Real-Time Data and Predictive Analytics

One of the biggest advancements in transit technology is the use of real-time data to make decisions on the fly. In the past, passengers relied on static schedules, and any delays or changes would leave them stranded without alternatives. Today, real-time data allows transit operators to dynamically adjust routes, notify passengers of delays, and even reroute vehicles to optimize traffic flow.

With TOST, one of the key challenges was ensuring that transit operators had access to real-time data on routes, vehicle locations, and driver statuses. By integrating GPS tracking and using General Transit Feed Specification (GTFS) data, we were able to give operators live updates on their fleets. This information enabled more efficient route management, reduced delays, and allowed passengers to receive up-to-the-minute updates on their journeys.

Predictive analytics takes this a step further by using historical data to anticipate and solve problems before they occur. For example, during peak hours or bad weather, the system can predict potential delays and proactively adjust schedules or dispatch additional vehicles to avoid disruptions.

2. AI-Powered Route Optimization

Public transportation has long been burdened by inefficient routing. Traditionally, routes were static, designed based on outdated traffic patterns and passenger demand. But with advancements in AI and machine learning, transit systems are now capable of dynamically optimizing routes based on real-time conditions.

During the development of TOST, we implemented algorithms that continuously analyzed traffic patterns, passenger flow, and other variables. This enabled transit operators to adjust routes on the fly, avoiding traffic bottlenecks and reducing overall travel times. The result? Faster, more reliable service that passengers could count on.

In the future, we can expect AI to play an even bigger role, with systems that learn from passenger behavior and adjust routes accordingly. This could lead to on-demand public transportation, where routes are not fixed but instead adapt in real-time based on where passengers need to go. Imagine a bus that changes its route throughout the day to meet demand more efficiently — that’s the direction we’re heading.

3. Smart Ticketing and Payments

Another major innovation in transit technology is smart ticketing. Gone are the days of paper tickets and cash payments. Instead, we’re seeing the rise of contactless payments, mobile apps, and even QR code-based ticketing systems that make it easier for passengers to access and pay for transit services.

When working on TOST, one of the first things we looked at was integrating digital payments into the system. We worked with several payment providers to introduce mobile wallets and contactless cards. This not only sped up the boarding process but also reduced the operational burden of managing cash payments.

Looking ahead, blockchain technology could further revolutionize the way payments are handled in transit systems. Imagine an entirely decentralized system where users pay for rides using cryptocurrency, and all transactions are securely recorded on a blockchain ledger, eliminating fraud and reducing transaction fees.

4. Autonomous and Electric Vehicles

It’s impossible to talk about the future of transit technology without mentioning autonomous vehicles (AVs) and electric vehicles (EVs). AVs promise to reduce human error, improve safety, and provide more consistent service, while EVs are paving the way for greener, more sustainable public transportation.

Many cities are already experimenting with autonomous buses and shuttles. While we’re still some time away from fully autonomous public transit systems, early trials have shown that AVs can dramatically reduce operating costs and improve reliability. In TOST, we began laying the groundwork for future AV integrations by building a flexible platform capable of interfacing with autonomous vehicle fleets as they become more mainstream.

At the same time, EVs are quickly becoming the standard for transit fleets, thanks to their lower operating costs and reduced environmental impact. Many transit agencies are transitioning to electric buses, which not only cut down on emissions but also reduce maintenance costs compared to traditional fuel-powered vehicles. The future of transit will undoubtedly be powered by electricity.

5. The Role of IoT in Transit Systems

Another exciting development in transit technology is the integration of Internet of Things (IoT) devices. IoT allows for the seamless connection of vehicles, infrastructure, and users, enabling a more connected transit ecosystem. For example, smart sensors on buses and trains can monitor everything from vehicle health to passenger occupancy, providing valuable data that can be used to optimize operations in real-time.

In TOST, we incorporated IoT devices to monitor vehicle conditions and track driver performance. These sensors provided valuable insights into fuel efficiency, engine health, and even driver behavior, allowing operators to proactively address maintenance issues and improve safety.

In the future, IoT devices will likely play a larger role in creating smart cities. Imagine traffic lights that communicate with buses to extend green lights during peak hours or smart bus stops that provide real-time information to passengers based on vehicle locations and expected arrivals. These advancements will lead to a more connected and efficient urban transportation system.

Final Thoughts: The Road Ahead

The future of transit technology is bright. With innovations like real-time data, AI-powered route optimization, smart payments, and the rise of autonomous and electric vehicles, public transportation is on the cusp of a major transformation. The work we did with TOST showed me firsthand how impactful these technologies can be when implemented thoughtfully.

For transit systems to truly evolve, it’s essential to adopt a holistic approach — one that integrates technology into every aspect of operations, from vehicle management to passenger experience. It’s not just about solving today’s problems; it’s about building the foundation for a smarter, more sustainable transit future.

Public transportation plays a critical role in shaping the cities of tomorrow, and the innovations we implement today will define how we move in the future. Whether you're a developer, a transit operator, or simply a commuter, we all stand to benefit from these advancements.

======Human

Technology touches every corner of our life, and public transportation doesn't stand out as an exception to this. With growing cities and their populations, the demand for smarter and more efficient transit systems has gone from urgent to very urgent. The future of transit technology needs to rest in innovations improving not only operational efficiency but the overall experience of the passengers. I'll outline the main developments that shape public transport in this article by drawing from my own experience in building TOST—a transit management system for US-based operators.

The Current Landscape of Public Transit

Before discussing the innovations, let's briefly understand what challenges are faced today by public transportation. From aged infrastructure to inefficient route planning, many cities struggle to keep up with the demands brought on by growing commuters. The result? Increased traffic congestion, unreliable schedules, and a poor user experience.

Until now, the traditional type of transit management systems is usually done manually, with hardly any room for optimization, but now with the development of technology, the shift to an intelligent approach through the use of real-time data, automation, and sometimes even artificial intelligence has become more common in optimizing every last detail of the journey.

The Rise of Intelligent Transit Systems

Real-time Data and Predictive Analytics

Probably one of the biggest evolutionary changes in transit technologies is using real-time data to make decisions on the fly. In contrast, previous passengers relied on static schedules where delays or changes meant they were left with no alternatives. Today, with real-time data, operations can dynamically shift routes, warn passengers of any delays, and even reroute vehicles to optimize the flow of traffic.

With TOST, one of the very important challenges was how to provide real-time data to transit operators on routes, vehicle locations, and the status of drivers. We integrated GPS tracking and utilized data from the General Transit Feed Specification so operators could receive live updates about their fleets. This allowed for better route management, reduced delays, and the ability for passengers to be informed right up to the minute about their journeys.

Predictive analytics goes a step beyond by anticipating the problems before they happen by making use of historical data to solve them. For example, predictive analytics could predict possible delays during peak hours or bad weather so that the system may proactively adjust the schedules or deploy more vehicles to avoid disruption.
AI-Powered Route Optimization

Public transportation has always been plagued with inefficiency in routing. Conventionally, the routes were static, devised based on outdated patterns of traffic and passenger demand. However, with the rise of AI and machine learning, transit systems are now capable of dynamic re-optimization of routes depending on real-time conditions.

During the development of TOST, we implemented algorithms that were in constant observation of traffic patterns and passenger flow, among many variables. What that did was allow for quick changes to routes in order to avoid traffic bottlenecks, decreasing overall travel times. The result? Quicker, more reliable service that passengers can rely on.

In the future, AI will play an even more important role: systems will learn from passenger behavior and adjust routes accordingly. This may even lead to demand-based public transportation, where routes change in real time depending on passenger needs. Imagine a bus changing its route during the day to meet demand better—that's where we're heading.
Smart Ticketing and Payments

Another major innovation in transit technology allows for smart transit ticketing. Gone are the days of paper tickets or cash payments, as contactless payment systems, mobile apps, and even QR code-based ticketing systems make it easier for passengers to access and pay for services.

One of the early considerations when working on TOST was integrating digital payments. We worked with various payment providers to introduce mobile wallets and contactless cards. This not only accelerated the boarding process but also reduced the operational headache of managing cash payments.

In the future, blockchain might further disrupt how transit systems handle payments. Imagine a decentralized system where users pay for rides via cryptocurrency, with every transaction recorded on a blockchain ledger, securing it from fraud and reducing transaction fees.
Autonomous and Electric Vehicles

It is quite impossible to talk about the future of transit technology without referencing AVs and EVs. AVs promise a reduction in human error, improved safety, and more consistent service, while EVs are leading the charge toward a greener, more sustainable public transportation environment.

Many cities are already experimenting with autonomous buses and shuttles. While we remain a long way off from fully autonomous public transit systems, early trials have shown that AVs can dramatically reduce operating costs while improving reliability. Starting with TOST, we laid the groundwork for future AV integrations by providing a flexible platform to interface with autonomous vehicle fleets as they become mainstream.

Meanwhile, EVs are fast becoming the norm in transit fleets because of lower operating costs and environmental benefits. A host of transit agencies are converting their fleets into electric buses, which not only reduce emissions but also have lower maintenance costs compared to traditionally fueled vehicles. The future of transit will undoubtedly be electric.
Role of IoT in Transit Systems

Another exciting development in transit technology is the integration of Internet of Things (IoT) devices. IoT allows for wireless connectivity among vehicles, infrastructure, and users, creating a more integrated transit ecosystem. For instance, smart sensors on buses and trains can monitor vehicle health and passenger occupancy, providing valuable data to optimize operations in real time.

With TOST, we installed IoT devices on vehicles to monitor conditions and track driver performance. These sensors can track data such as fuel efficiency, engine health, and driver behavior, helping operators address maintenance issues proactively and improve safety.

In the future, IoT devices will make up larger components of smart cities: traffic lights could extend green lights for buses during peak hours, or smart bus stops could provide passengers with real-time arrival information based on vehicle locations. These developments will lead to more connected and efficient urban transit systems.

Final Thoughts: The Road Ahead

The future of transit technology is bright. Innovations like real-time data, AI-powered route optimization, smart payments, and the rise of autonomous and electric vehicles will usher in a new era of public transportation. The work we did with TOST showed me firsthand just how impactful these technologies can be when implemented thoughtfully.

Transit systems need to adopt a holistic approach, incorporating technology into every facet of their operations—from vehicle management to passenger experience. It's about solving today's problems while building for a smarter, more sustainable transit future.

Public transportation will shape the cities of tomorrow, and the innovations we set in motion today will determine how we move around in the future. Whether you're a developer, transit operator, or commuter, we all stand to benefit from these advancements.

Building Scalable Applications: Lessons from Nexaloom and Rareboots

Slaknoah — Mon, 18 Nov 2024 08:52:26 +0000

Scalability is the magic word every developer and startup founder talks about but often struggle to achieve. The process of building scalable applications requires making the right architectural and technological decisions from the outset. In my journey building Nexaloom and Rareboots Marketplace, I’ve had my fair share of learning experiences—both successes and challenges. In this article, I’ll walk you through key lessons learned in building scalable applications, covering architectural decisions, tech choices, and practical steps to ensure your application can handle growth.

Lesson 1: Start with the Right Tech Stack

When you're setting out to build a scalable application, the choice of tech stack can make or break your project’s ability to handle future growth.

Choosing Node.js for Nexaloom

For Nexaloom, an appointment scheduling and business management software, the need for rapid development and scalability was obvious from the start. I decided to go with Node.js for the backend, primarily because of its non-blocking, event-driven architecture. Node.js has a reputation for handling a high number of concurrent connections, which is exactly what Nexaloom needed as it scaled to serve multiple businesses and users simultaneously.

Another win for Node.js is the JavaScript ecosystem. With JavaScript running on both the front-end and back-end, it allowed for a unified development process, reducing context switching between languages. This also meant that I could easily onboard new developers, as JavaScript remains a common denominator among web developers.

Choosing Medusa.js for Rareboots

On the other hand, when building Rareboots Marketplace, I had a different set of challenges. Rareboots needed a robust e-commerce platform that could handle multiple sellers, extensive product listings, and high traffic. Instead of building everything from scratch, I chose to use Medusa.js, an open-source e-commerce framework built on Node.js.

Medusa provided a solid foundation for the marketplace with built-in support for core features like cart management, order handling, and even Stripe integration. But here’s the important part: it was extensible. Medusa allowed me to extend functionality, particularly in areas like custom product filtering, handling marketplace-specific logic, and advanced payment flows. This saved a lot of time in development while still keeping the codebase lean and modular.

Lesson 2: Database Architecture Matters

It’s tempting to throw everything into a single database and call it a day, but that’s a surefire way to run into scaling issues later. For both Nexaloom and Rareboots, I had to carefully consider how data was stored and accessed.

Choosing PostgreSQL for Structured Data

In both applications, I chose PostgreSQL as the primary database. PostgreSQL shines when it comes to handling relational data at scale. It’s also highly versatile, allowing for complex queries, transactions, and data integrity, which was critical for both Nexaloom’s appointment data and Rareboots’ marketplace transactions.

In Nexaloom, for example, we deal with a lot of user-generated data—appointment schedules, client profiles, and payments. The data relationships are clear, so relational databases made the most sense. Moreover, PostgreSQL’s ability to handle large datasets with proper indexing ensured that as the platform grew, the performance didn’t degrade.

Caching with Redis

Scaling isn't just about the database itself but how data is accessed. For frequently accessed data, caching can significantly reduce load on the database. In Rareboots, where certain product searches and recommendations were repeatedly requested, I implemented Redis for caching. This made a huge difference in reducing latency and improving user experience as the marketplace scaled up.

Lesson 3: Modular, Microservice-Based Architecture

Monolith vs. Microservices

When Nexaloom first started, the application was monolithic—a single codebase handling everything from user authentication to appointment scheduling. This was fine for the initial stages, but as we grew, it became clear that managing this single large application was going to be a nightmare.

Enter microservices.

Moving to a microservices architecture allowed us to decouple features and services into independent modules. For example, we separated the appointment scheduling logic from user management. This approach not only improved scalability but also made it easier to deploy updates independently without affecting other parts of the system.

In Rareboots, I adopted a hybrid approach. Some core services, like the checkout and payment processes, were split into microservices, while the core product listing and management were kept together in a more modular monolithic style. This allowed for flexibility without overcomplicating things too early.

Communication Between Services

One important lesson in microservices is that you need an efficient way for services to communicate. For both Nexaloom and Rareboots, I used message queues for inter-service communication, specifically RabbitMQ. This ensures that as traffic scales and services need to communicate asynchronously (e.g., handling payment confirmation or sending out email notifications), the system can handle the load without bottlenecking.

Lesson 4: Scaling the Frontend

Component-Based UI Development with React

Both Nexaloom and Rareboots needed responsive, scalable UIs. React was my go-to choice because of its component-based structure. Components are reusable, easy to maintain, and can be rendered dynamically based on the data flowing in from the backend.

For Nexaloom, this meant creating reusable components for things like the appointment calendar and client profile forms. As the platform grew and more features were added, React’s component reusability allowed for quick iterations and updates without rewriting the entire UI.

Server-Side Rendering for Performance

For Rareboots, scalability was not just about handling traffic but also about performance. As an e-commerce platform, every second counts when it comes to load time, and a slow experience can mean lost sales. To ensure optimal performance, I implemented server-side rendering (SSR) using Next.js. SSR allowed us to serve pre-rendered pages to users, reducing the initial load time and improving SEO, both crucial for marketplace success.

Lesson 5: Don’t Ignore DevOps and CI/CD

When you’re building an application that needs to scale, having a solid DevOps pipeline in place is crucial.

Automating Deployments

Both Nexaloom and Rareboots required frequent updates, and manual deployments simply weren’t feasible. By setting up a CI/CD pipeline with Jenkins and GitLab CI, we were able to automate builds, run tests, and deploy code to production with minimal downtime. This allowed us to push updates faster without breaking the application or creating unnecessary bottlenecks.

Monitoring and Scaling with Kubernetes

As both applications started handling more traffic, I introduced Kubernetes to manage containerized services. Kubernetes allowed us to scale individual services up or down based on demand, making sure we never over-provision resources. It also helped in managing deployment rollbacks in case any issues arose in production.

Final Thoughts

Building scalable applications requires careful planning, the right tech stack, and a deep understanding of how your application will grow. With Nexaloom, I learned the importance of modular architecture, and with Rareboots, I gained firsthand experience in building marketplace features at scale.

Here’s the takeaway: Start small, but design your application with growth in mind. Whether it’s choosing the right database, splitting your services into manageable parts, or caching frequently accessed data, every decision you make early on will impact your ability to scale later. Keep learning, keep iterating, and don’t be afraid to make adjustments as you go.

Full-Stack Development Evolution: Trends and Best Practices

Slaknoah — Mon, 28 Oct 2024 22:15:25 +0000

In the last ten years, full-stack development has changed a lot. From an out-of-the-ordinary responsibility for developers who could handle both backend and frontend tasks to becoming an industry standard, it has indeed been a strange journey.

Today, being a full-stack developer means more than juggling two sides of an application. It is about mastering many technologies in an array and adapting to the recent trends by following best practices. Let's dive into how the evolution of full-stack development came to be, what recent trends are setting the pace for it, and what is considered best practices for every full-stack developer to have in their toolkit.

From Specialization to Generalization: A Journey

Full-stack development began to gain pace when the tech industry especially startups in the early phases realized they needed versatile developers who could do it all: build the frontend, manage the backend, and even take care of infrastructure. Earlier, developers were either backend-focused, being masters at languages like PHP or Python, or frontend-centric, working with HTML, CSS, and JavaScript. But as web applications continued to get increasingly complex and interactive, strong was the need for developers to be able to switch between both worlds comfortably.

It meant that the shift in demands required developers to do more than know multiple languages; they had to understand the entire ecosystem, from databases to deployment pipelines.

Full Stack Development Trends

Following are some of the key trends to shape the future of full-stack development:

JavaScript Dominance

JavaScript strongly positions itself as a core language for a full-stack developer. With Node.js handling the back end and systems like React, Vue, and Angular dominating the front, JavaScript is the glue that stitches modern web applications together. As I found out during my transition from PHP to Node.js, if you know your JavaScript, you are ready for nearly everything on both client-side to server-side development. Other great languages include: JavaScript, Python, Ruby, Java, PHP etc.
API Driven Architecture

API-first development is more on the rise. RESTful APIs and GraphQL have become common go-to solutions for creating scalable and flexible web services. APIs can power interactions between the frontend and backend, affording developers even more ways to craft modular applications. This leads to a more decoupled architecture in which the evolution of both frontends and backends becomes independent of each other, a critical feature for scalability.
Cloud-Native Applications

Cloud-native development, with AWS, Azure, and Google Cloud leading from the front, becomes one of the essential skills that every developer should master. The developers need to know a thing or two about microservices, containerization-hence Docker and Kubernetes and serverless computing to create scalable and cost-efficient applications.
Jamstack and Static Site Generators

Jamstack has redefined the way we think about web applications. Allowing for a complete decoupling of frontends and backends into static files, APIs, and JavaScript empowers better performance and security. Static site generators like Next.js and Gatsby are on the rise, allowing developers the ability to deliver super-fast websites with very light server management.
Performance Optimization

As the web grows increasingly complex, performance optimizations have become a prime concern. A full-stack developer today needs to be efficient at optimizing either backend performance-thinking about query optimization or caching strategies, for example, with Redis-, or frontend assets, thinking about lazy loading or minification. Smoothening the load time and escalating the application so that it could scale efficiently are no longer optional; it's expected.

Best Practices Every Full-Stack Developer Should Follow

With the rapid evolution of the art of full-stack development, a focus on best practices really pays off in terms of staying productive and pitfall-free. Here are some core practices:

Modularity of Code

Well, working modularly and reusing code is very critical for full-stack development. A well-organized project with clear separations between different components reduces complexity and enhances the ability to collaborate. In back-end frameworks like Express.js, for example, a need exists to structure your routes and middleware well to keep things organized.
Version Control and Continuous Integration

A very important thing would be to integrate version control into your workflow, which in this day and age is basically the use of an industry standard like Git. That can be coupled with CI/CD pipelines so that testing, integration, and deployment are automated. Tools that would support such a process include Jenkins, Travis CI, or GitHub Actions, which simplify the process or at least ensure that every single change you push is as vetted as possible before hitting production.
Focus on Security

Security is not an option; always secure your application from every level that can be done, from HTTPS to proper authentication mechanisms, OAuth, JWT, and proper sanitizing of user input to avoid XSS or SQL injection. Knowing the most common security pitfalls saves you from building weak applications.
Scaling Considerations

In full-stack development, scalability should be a concern for every developer. It basically means building an application that would handle increased traffic, data, and features imposed on it. If it's about building APIs or managing databases, think about load balancing, database replication, and efficient query handling.
Documentation and Communication

You are not writing software in a vacuum. Documentation helps other people understand your code, and it maintains it for the long run. Comment on your code, have meaningful commit messages, and document your APIs. Tools like Swagger can ensure that API documentation is clear.

Conclusion

The role of a full-stack developer keeps changing with the changes in the technological world. It's all about understanding new trends that come up, such as API-driven architecture, cloud computing, and performance optimization. At the same time, however, one should not be able to forget that best practices include code modularity, security, and scalability to make a person battle-ready for modern web development challenges. Adaptability and learning in themselves are at the core of full-stack development. The sharper you keep these skills, the more successful you will be in this ever-evolving field.

Creating a global production ready marketplace with Medusa JS

Slaknoah — Fri, 25 Oct 2024 12:36:27 +0000

The Problem

Recently, at the company where I work, we faced the challenge of building a marketplace solution that could scale and support vast traffic. It’s for a niche market, but the specifications and requirements are quite similar to already available solutions like Vinted, eBay, or Depop. These platforms have established reputations, and the goal for the company was to bring all sellers in their industry together to trade in one place.

The Goal

For the Minimum Viable Product (MVP), there was quite an extensive list of features and capabilities expected, including:

Onboarding customers and sellers with full payment provider setup
Product listings
Product approval flow via the super admin
Fast search and product discovery
Cart-to-checkout flow
Buy now option
Offers/product negotiation flow
Messaging between seller and buyer
Order management and shipping
Return flow
Notifications throughout the system
Opt-in or opt-out of certain features like offers and returns

Selecting the Tech Stack

As this blog focuses primarily on backend technology, I will keep the discussion limited to backend stacks.

Based on project needs and our proficiency in Node.js (Typescript), we selected it for backend API development. The main question was whether to write the API from scratch or extend an existing solution.

Weighing the two options:

Writing the system from scratch (using NestJS) provides full control, allowing us to add features like messaging directly into the system while avoiding unnecessary bloat.
Using an existing solution, like Medusa JS, a highly extensible e-commerce platform, allows us to leverage a battle-tested e-commerce flow and only extend it for custom logic and additional marketplace-specific features. However, this comes with the tradeoff of unused features that are not needed by the marketplace.

Decision to Use Medusa

In the end, we decided to pick Medusa for the following reasons:

Written in Typescript
Uses ORMs we're familiar with (TypeORM)
Highly extensible and customizable
Well-documented
Modular architecture in V2 (coming soon), which helps address the issue of unused features/modules
Great developer adoption and support

The rest of the tech stack was decided based on Medusa, as it integrates seamlessly with many existing adapters.

Selecting the Database – PostgreSQL

PostgreSQL is the primary supported database for Medusa. We initially tried CockroachDB but encountered issues with certain data types, so we opted for the recommended PostgreSQL, which we set up using Google Cloud SQL.

Implementing Authentication

Medusa comes with built-in authentication, but we needed social authentication (at least Google). Fortunately, there is a Medusa plugin for this, which made it easy to set up. We just needed to configure Google credentials and environment variables.

Writing `medusa-plugin-marketplace`

This being the third marketplace I’ve built with Medusa, I’ve come across things I wished were essentially plug-and-play to transform a fresh Medusa installation into a basic marketplace. So, I decided to consolidate these elements and publish them as a public plugin: medusa-plugin-marketplace.

It includes the following features:

Multiple store support in the Medusa store model
Store-specific scoping for:
- Products
- Orders
- Shipping options
- Invites
- Users
Store roles and permissions
Middleware to prevent cross-store actions
Advanced Stripe module with marketplace logic:
- Application fee support
- Payment splitting (e.g., shipping fee goes to the marketplace Stripe account if it handles label printing)
- Payment split handling on returns
- Customizable payment intent and capture behavior
- Stripe account onboarding widget on super admin
Order cart splitting for multiple stores
Support for filtering public products by store ID

The plugin is still a work-in-progress, and more work is needed to make it customizable to user needs. Please give it a star to support the work.

Cache Service – Redis

Medusa supports both in-memory and Redis caching via plugins. For production use, Redis is recommended, so we opted for Redis.

Event Listeners and Storage

We implemented the Redis event module for Medusa, which is essential for listening to certain events and triggering actions, such as:

Deleting products or variants when the quantity is updated
Sending notifications (more on that later)
Removing products from the search index upon deletion
Syncing product categories upon option updates
Updating offers when orders are completed, etc.

Fast Search/Product Indexing with Algolia

We implemented Algolia search but found that the existing module wasn’t customizable enough for our business needs. Since our products are volatile and constantly deleted once out of stock, I wrote a custom Algolia search module for Medusa. This reduced the payload saved in the index. Although this approach comes at a cost with increased search traffic, it allowed us to get the MVP to market quickly. However, we are considering switching to Meilisearch in the future due to the costs associated with Algolia.

Choosing Google Cloud for Hosting and Solving Issues with Storage

We opted for Google Cloud App Engine to deploy the backend because of its scalability and integration with other dependencies like the database and storage. Though documentation for deploying Medusa to App Engine is scarce, setting it up was straightforward with the right IAM configurations and a continuous deployment setup using Cloud Build.

For file storage, we used Google Cloud Storage and wrote a custom Medusa file module to access public and private buckets. There was an issue with Medusa’s hardcoded /uploads folder (App Engine only allows writing to /tmp), so we patched it to resolve the issue.

Notification Orchestration System – Novu & Bird

Notifications are a key part of the project. Our requirements included:

Email notifications
Push notifications
In-app notifications
Digest or delayed notifications
Phone OTP SMS

To meet these needs, we opted for a notification orchestration system. We chose Novu due to its open-source availability and affordable pricing (compared to Knock). For push notifications, we used Expo Notifications, but we encountered issues integrating Bird (formerly MessageBird) with Novu, so we wrote a custom module to handle authentication codes while Novu handles the rest.

Implementing Offers

Another requirement was enabling price negotiation between buyers and sellers. This involved actions like:

Accepting an offer
Rejecting an offer
Making a counteroffer

We stored offer histories and used custom carts to redirect buyers once offers were approved. Notifications were crucial here since offers expire within a set timeframe.

Messaging with Censorship

For messaging, we initially planned a simple email-like system, but the scope expanded to include:

Messaging between buyers and sellers
Instant message delivery
Read receipts
Message notifications
Hashing of sensitive information like phone numbers, email addresses, and credit cards
Censorship of banned words

Error Reporting with Sentry

We used the Medusa Sentry plugin for error reporting. More details can be found in the documentation.

Shipping Provider for Printing Labels: Shippo

The requirements for shipping included:

Basic support for fulfillment by sellers
Automatic generation of shipping labels
Support for multiple providers
Customizable shipping prices via the super admin

Although Medusa has a plugin for Webshipper, we opted for Shippo due to its free entry and great documentation. Since there wasn’t a good Shippo plugin for Medusa, we wrote a custom integration, allowing us to customize workflows like setting fixed shipping prices.

Product Approval

During the early stages, we needed a product approval flow where the marketplace could review products before they went live. Fortunately, Medusa offers these statuses out-of-the-box, so we just needed to implement filtering with Algolia and build a custom UI for super admins to review products.

NB

This is just a surface level look at how the system was created and the reasoning behind certain decisions without going into detail of how to implement certain features, if you require more information regarding building out these systems or questions feel free to reach out via email.