Forem: Nic Raboy

Build a Movie Watchlist with Node.js, TypeScript, and MongoDB

Nic Raboy — Mon, 13 Apr 2026 15:19:10 +0000

Almost every modern web application will need a REST API for a client to talk to, and in almost every scenario, that client is going to expect JSON. The best developer experience is a stack where you can stay in JSON-shaped data end to end, without awkward transformations in the middle.

Take MongoDB, Express Framework, and Node.js as an example.

Express receives HTTP requests and sends responses. MongoDB sits in the middle and stores documents. The client can send JSON to your routes, your routes can send documents to MongoDB, and MongoDB can hand BSON back that maps naturally to what you serialize in the response. That works well because MongoDB is a document database. When you also want text search over fields like title and plot, MongoDB Search gives you a $search stage in an aggregation pipeline on the same cluster, so you are not bolting on a separate search system just to power a search box.

In this tutorial, we'll see how to build a small movie watchlist API using TypeScript and MongoDB. We'll explore a few different schema design opportunities and make use of MongoDB Search for full-text search.

The Prerequisites

Prior to starting this tutorial, you'll need a few things in place:

A MongoDB Atlas instance (M0 is fine) with the sample_mflix sample dataset loaded and available.
Node.js 22+

The expectation is that your cluster is already provisioned with credentials and network access rules, and that the sample dataset is loaded. If you need help deploying or configuring MongoDB Atlas, check out the MongoDB documentation for getting started.

We'll be working with an empty project, so to kick things off, you might want to run the following commands:

mkdir movie-watchlist-api
cd movie-watchlist-api
npm init -y

The above commands will create a new project directory and initialize it for Node.js by creating a package.json file.

There are a few dependencies that we need to install. They can be installed by executing the following commands:

npm install express mongodb dotenv

In the above, we're installing Express Framework and the MongoDB Node.js Driver, plus a package that will allow us to read from a .env file for configuration variables.

For TypeScript, the tsx runner (useful in development with watch mode), and the Express and Node type packages, run:

npm install --save-dev typescript tsx @types/express @types/node

To run this project in development mode, you'd execute tsx watch src/app.ts from your command line. However, that won't be possible until later in the tutorial.

Add a TypeScript Configuration File

Because we're using TypeScript, we need a tsconfig.json at the root of the project. Create the file with something along the following lines:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "CommonJS",
    "moduleResolution": "node",
    "outDir": "dist",
    "rootDir": "src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true
  },
  "include": ["src/**/*"]
}

The goal here is to emit CommonJS into dist/ from everything under src/, which keeps production as simple as node dist/app.js after tsc.

Understanding the Source Tree of the Project

With the tooling in place, it helps to know where things will live. This project will use the following foundation:

src/config.ts
src/db.ts
src/types/watchlist.ts
src/routes/movies.ts
src/routes/watchlist.ts
src/app.ts

Each file will serve a different purpose that we'll explore throughout the remainder of the tutorial. Go ahead and create the src, src/routes, and src/types directories so you're ready to paste code as we go.

Define and Configure the Environment Variables for the Project

We're going to need connection details before the Express Framework application can talk to MongoDB Atlas. Add the following to a .env file at the project's root:

MONGODB_URI=mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/
PORT=3000
SEARCH_INDEX_NAME=default

Make sure to add this file to your project's .gitignore if you're using version control, since the MONGODB_URI is considered sensitive information.

Rather than read process.env in every route file of the project, we'll define a small configuration module that validates what is required and exports plain values.

Open src/config.ts and include the following code:

import "dotenv/config";

function requiredEnv(name: string): string {
    const value = process.env[name];
    if (!value?.trim()) {
        throw new Error(`Missing required environment variable: ${name}`);
    }
    return value.trim();
}

export const config = {
    mongoUri: requiredEnv("MONGODB_URI"),
    port: Number(process.env.PORT) || 3000,
    dbName: "sample_mflix",
    moviesCollection: "movies",
    watchlistCollection: "mflix_watchlist",
    searchIndexName: process.env.SEARCH_INDEX_NAME?.trim() || "default",
};

The import "dotenv/config" line means that as soon as something imports config.ts, your .env values are loaded. Database and collection names are hard-coded to sample_mflix, movies, and mflix_watchlist, so this tutorial stays aligned with the sample dataset.

Connecting to MongoDB with a Singleton Database Class

You could open a new MongoClient on every request, but that usually is not what you want. It adds latency, wastes resources, and makes graceful shutdown harder. The pattern we'll use instead is a singleton where we'll connect once when the application boots, reuse that client for every request, and close it when the process exits.

With the environment variables already in config.ts, open src/db.ts and include the following code:

import { MongoClient, type Db } from "mongodb";
import { config } from "./config";

let singleton: Database | null = null;

export class Database {
    private constructor(
        private readonly mongoClient: MongoClient,
        readonly db: Db
    ) {}

    static async connect(): Promise<Database> {
        if (singleton) return singleton;
        const mongoClient = new MongoClient(config.mongoUri, {
            appName: "devrel-tutorial-typescript-moviewatchlist",
        });
        await mongoClient.connect();
        const db = mongoClient.db(config.dbName);
        singleton = new Database(mongoClient, db);
        await singleton.ensureWatchlistIndexes();
        return singleton;
    }

    static get instance(): Database {
        if (!singleton) {
            throw new Error("Database not connected");
        }
        return singleton;
    }

    async close(): Promise<void> {
        await this.mongoClient.close();
        singleton = null;
    }

    private async ensureWatchlistIndexes(): Promise<void> {
        const watchlistCollection = this.db.collection(
            config.watchlistCollection
        );
        await watchlistCollection.createIndex({ movieId: 1 }, { unique: true });
    }
}

In the connect method, we have the following logic:

static async connect(): Promise<Database> {
    if (singleton) return singleton;
    const mongoClient = new MongoClient(config.mongoUri, {
        appName: "devrel-tutorial-typescript-moviewatchlist",
    });
    await mongoClient.connect();
    const db = mongoClient.db(config.dbName);
    singleton = new Database(mongoClient, db);
    await singleton.ensureWatchlistIndexes();
    return singleton;
}

If singleton is already set, we return immediately and avoid opening another connection. Otherwise we create MongoClient with config.mongoUri and an appName that shows up in MongoDB Atlas, await connect(), take a Db reference for config.dbName (here sample_mflix), assign singleton, and call ensureWatchlistIndexes before returning. Doing the index step before the HTTP server starts listening means every route can assume the watchlist collection already has a unique index on movieId.

The ensureWatchlistIndexes function creates a unique index on movieId in mflix_watchlist, so MongoDB rejects a second insert for the same movie and the watchlist POST handler can reason about "no row", "inactive row", and "active row" without duplicate documents.

Define the Watchlist Document Type for TypeScript

Before we get too far into the routes, it's a good idea to describe the document shape we plan to store in mflix_watchlist. TypeScript will not infer that for you.

In src/types/watchlist.ts, add the following:

import type { ObjectId } from "mongodb";

export interface WatchlistDoc {
    _id: ObjectId;
    movieId: ObjectId;
    title: string;
    poster: string | null;
    rating?: number;
    active: boolean;
    createdAt: Date;
    updatedAt: Date;
}

We're denormalizing title and poster onto the watchlist document when the user adds or reactivates a movie, so a client can render the list without joining back to movies on every GET. The advantage of this approach is that we're making our queries faster by not doing a $lookup (JOIN) operation unless we absolutely have to. When listing all items in a watchlist, it is probably not common to need details such as the actor cast list, so we're only including the fields we absolutely need alongside the id value in case we want to expand the details later with a single fetch operation.

Searching for Movies within the Catalog with MongoDB Search

To search for movies in our application, it makes sense to leverage MongoDB Search for its full-text search capabilities. Before we can do that, we need to apply a search index on the collection and fields we want to search through.

Take the following MongoDB Search index:

{
    "mappings": {
        "dynamic": false,
        "fields": {
            "title": {
                "type": "string"
            },
            "plot": {
                "type": "string"
            }
        }
    }
}

In the above index, which is different from the unique index we created earlier, we are saying we only want to be able to use full-text search on the title and plot fields of a document.

To apply the index, navigate to MongoDB Atlas and make use of the Search & Vector Search navigation item. When prompted in the index creation process, make sure to choose sample_mflix as the database and movies as the collection. You can name the index "default" to match what we have in our configuration, or name it whatever makes sense to you. Just make sure to update the environment variables configuration if you change the name.

Open the project's src/routes/movies.ts file and include the following:

import { Router, type IRouter } from "express";
import { config } from "../config";
import { Database } from "../db";

export function moviesRouter(): IRouter {
    const router = Router();

    router.get("/search", async (req, res, next) => {
        try {
            const searchQuery = String(req.query.q ?? "").trim();
            if (!searchQuery) {
                throw new Error(
                    'Query parameter "q" is required and cannot be empty'
                );
            }

            const pipeline = [
                {
                    $search: {
                        index: config.searchIndexName,
                        text: {
                            query: searchQuery,
                            path: ["title", "plot"],
                        },
                    },
                },
                { $limit: 10 },
                {
                    $project: {
                        _id: 1,
                        title: 1,
                        poster: 1,
                        year: 1,
                    },
                },
            ];

            const database = Database.instance.db;
            const moviesCollection = database.collection(
                config.moviesCollection
            );
            const results = await moviesCollection
                .aggregate(pipeline)
                .toArray();

            res.json({ results });
        } catch (err) {
            next(err);
        }
    });

    return router;
}

The router above has a single GET endpoint that we'll use for searching. It expects a q query parameter to exist in our requests, where this parameter represents a search query.

With a MongoDB aggregation pipeline, we can use $search as our first stage, specifying the index name, the search query, and the fields we wish to search against. For the sake of this example, the second pipeline stage will limit our results to no more than 10 results. Finally, we end our pipeline with a $project stage that allows us to define what fields are actually important to us for returning to the client. In this example, we plan to return the _id, title, poster, and year fields for every document matched.

We are only using MongoDB Search for finding movies we want to add to our watchlist. We need to create a route and a series of endpoints for managing our watchlist next.

Building a Movie Watchlist with Ratings

The watchlist will make up most of our application logic for this tutorial. We want to accomplish the following:

Add movies based on id to the watchlist, with certain fields, as long as the movie doesn't already exist in the list.
Add ratings to movies in the watch list.
Soft-delete (deactivate) movies in the watchlist, but not actually removing the document from the collection.
Remove ratings from movies in the watchlist.

In the src/routes/watchlist.ts file, let's use the following as a starting point:

import { Router, type IRouter } from "express";
import { ObjectId } from "mongodb";
import { config } from "../config";
import { Database } from "../db";
import type { WatchlistDoc } from "../types/watchlist";

export function watchlistRouter(): IRouter {
    const router = Router();

    return router;
}

There aren't any endpoints for now, but we're going to add and make sense of each one at a time.

Our most complex endpoint will be the POST endpoint responsible for adding movies to the watchlist. You can find this endpoint below.

router.post("/", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(String(req.body?.movieId));
        const watchlistCollection = database.collection(config.watchlistCollection);
        const moviesCollection = database.collection(config.moviesCollection);

        const existingWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });

        if (existingWatchlistDoc) {
            const activeEntry = existingWatchlistDoc as WatchlistDoc;
            if (activeEntry.active) {
                res.status(200).json({
                    document: activeEntry,
                    alreadyActive: true,
                    reactivated: false,
                });
                return;
            }

            const now = new Date();
            await watchlistCollection.updateOne(
                { movieId },
                { $set: { active: true, updatedAt: now } }
            );
            const reactivated: WatchlistDoc = {
                ...activeEntry,
                active: true,
                updatedAt: now,
            };
            res.status(200).json({
                document: reactivated,
                alreadyActive: false,
                reactivated: true,
            });
            return;
        }

        const movieFromCatalog = await moviesCollection.findOne(
            { _id: movieId },
            { projection: { title: 1, poster: 1 } }
        );
        if (!movieFromCatalog) {
            throw new Error("Movie not found in sample_mflix");
        }
        const { title, poster } = movieFromCatalog;
        const now = new Date();
        const newWatchlistDocument = {
            movieId,
            title,
            poster,
            active: true,
            createdAt: now,
            updatedAt: now,
        };
        const insertResult = await watchlistCollection.insertOne(
            newWatchlistDocument
        );
        const document: WatchlistDoc = {
            _id: insertResult.insertedId,
            ...newWatchlistDocument,
        };
        res.status(201).json({
            document,
            alreadyActive: false,
            reactivated: false,
        });
    } catch (err) {
        next(err);
    }
});

To break down each stage of the above endpoint, we're going to start with the initial query:

const existingWatchlistDoc = await watchlistCollection.findOne({
    movieId,
});

The movieId should have been passed in with the request payload. It should represent a valid movie id from the movies collection. After executing the query, we check to see if results came back, and if they did, it means this particular movie is already in the watchlist. If the movie is already in the watchlist and it is in the active state, we can just return that information now. If it is in the watchlist, but inactive, we want to reactivate it and update the timestamp before returning it.

Finally, we have this code within the endpoint:

const movieFromCatalog = await moviesCollection.findOne(
    { _id: movieId },
    { projection: { title: 1, poster: 1 } }
);
if (!movieFromCatalog) {
    throw new Error("Movie not found in sample_mflix");
}
const { title, poster } = movieFromCatalog;
const now = new Date();
const newWatchlistDocument = {
    movieId,
    title,
    poster,
    active: true,
    createdAt: now,
    updatedAt: now,
};
const insertResult = await watchlistCollection.insertOne(
    newWatchlistDocument
);
const document: WatchlistDoc = {
    _id: insertResult.insertedId,
    ...newWatchlistDocument,
};
res.status(201).json({
    document,
    alreadyActive: false,
    reactivated: false,
});

If the movie is not in the watchlist already, we first need to query the movies collection for information about the movie based on the provided movieId value. With the title and poster information, we can construct a new watchlist document and then insert it into our collection.

For this particular project, we won't worry about keeping our movie posters up to date. We're just going to assume they never change.

The next endpoint we'll look at is a GET endpoint for listing all the movies in our watchlist:

router.get("/", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const raw = req.query.active;
        const activeOnly =
            raw === undefined
                ? true
                : !["false", "0"].includes(String(raw).toLowerCase());

        const watchlistCollection = database.collection(
            config.watchlistCollection
        );
        const items = await watchlistCollection
            .find({ active: activeOnly })
            .sort({ updatedAt: -1 })
            .toArray();
        res.json({ active: activeOnly, items });
    } catch (err) {
        next(err);
    }
});

This particular endpoint will accept an active query parameter, allowing the client to request watchlist items that are either active or have been soft-deleted (inactive).

When we run our MongoDB query, we are sorting by the updatedAt timestamp so the newest additions to the watchlist appear first. We aren't doing a $project here because our documents are small, but you could easily define which fields are important to you for returning.

The next endpoint is our PATCH endpoint, which is responsible for assigning a user rating to a movie in the watchlist:

router.patch("/:movieId/rating", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(req.params.movieId);
        const stars = Number(req.body?.stars);
        if (!Number.isInteger(stars) || stars < 1 || stars > 5) {
            throw new Error("Rating must be an integer from 1 to 5");
        }

        const watchlistCollection = database.collection(
            config.watchlistCollection
        );
        const now = new Date();
        const updateResult = await watchlistCollection.updateOne(
            { movieId },
            { $set: { rating: stars, updatedAt: now } }
        );
        if (updateResult.matchedCount === 0) {
            throw new Error(
                "Movie is not on the watchlist; only watchlist movies can be rated"
            );
        }
        const updatedWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });
        if (!updatedWatchlistDoc) {
            throw new Error(
                "Failed to load watchlist item after rating update"
            );
        }
        res.json({ document: updatedWatchlistDoc });
    } catch (err) {
        next(err);
    }
});

Here, we accept a movie id as a route parameter and a star rating within the request payload.

Using the updateOne operator, we can attempt to update the document. If the matchCount after the operation is zero, it means that there wasn't a document to update. This is likely because the movie id was incorrect, or it wasn't in our watchlist. When adding a rating, we could just respond with a simple message stating the update was complete, but in this example, we are querying for the watchlist item and returning what appears in the document.

The two final endpoints for our watchlist include removing a rating and soft-deleting a watchlist item.

Starting with the rating removal, we have the following DELETE request:

router.delete("/:movieId/rating", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(req.params.movieId);
        const watchlistCollection = database.collection(config.watchlistCollection);
        const now = new Date();
        const updateResult = await watchlistCollection.updateOne(
            { movieId },
            { $unset: { rating: "" }, $set: { updatedAt: now } }
        );
        if (updateResult.matchedCount === 0) {
            throw new Error(
                "Movie is not on the watchlist; only watchlist movies can be rated"
            );
        }
        const updatedWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });
        if (!updatedWatchlistDoc) {
            throw new Error(
                "Failed to load watchlist item after rating removal"
            );
        }
        res.json({ document: updatedWatchlistDoc });
    } catch (err) {
        next(err);
    }
});

The above code will take a movieId and make use of the $unset operator within an updateOne operation. This will remove the rating. Just like with the previous endpoint, we could have opted to return a message about the operation’s success, but here we are querying for the full document and returning it to the client instead.

The final endpoint in our route looks like the following:

router.delete("/:movieId", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(req.params.movieId);
        const watchlistCollection = database.collection(config.watchlistCollection);
        const existingWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });
        if (!existingWatchlistDoc) {
            throw new Error("Watchlist item not found");
        }
        if (!(existingWatchlistDoc as WatchlistDoc).active) {
            res.status(204).send();
            return;
        }
        const now = new Date();
        await watchlistCollection.updateOne(
            { movieId },
            { $set: { active: false, updatedAt: now } }
        );
        res.status(204).send();
    } catch (err) {
        next(err);
    }
});

Similar logic applies here. If the watchlist item exists and it is not already inactive, we do an updateOne operation and $set the active status to inactive. In this example, we are just doing an empty return with an appropriate status to show it was deactivated.

Configuring Express Framework to Connect to MongoDB and Serve the Application

At this point, our two routes are in place, and the appropriate indexes have been configured. We need to be able to serve our application so clients of any shape can interact with it.

Open the project's src/app.ts file and include the following:

import express, { type ErrorRequestHandler } from "express";
import { moviesRouter } from "./routes/movies";
import { watchlistRouter } from "./routes/watchlist";
import { config } from "./config";
import { Database } from "./db";

async function main() {
    await Database.connect();

    const app = express();
    app.use(express.json());

    app.get("/health", (_req, res) => {
        res.json({ ok: true });
    });

    app.use("/movies", moviesRouter());
    app.use("/watchlist", watchlistRouter());

    const onError: ErrorRequestHandler = (err, _req, res, next) => {
        if (res.headersSent) {
            next(err);
            return;
        }
        console.error(err);
        res.status(500).json({
            error: err instanceof Error ? err.message : String(err),
        });
    };
    app.use(onError);

    const server = app.listen(config.port, () => {
        console.log(`Listening on http://localhost:${config.port}`);
    });

    const shutdown = async () => {
        server.close();
        await Database.instance.close();
        process.exit(0);
    };
    process.on("SIGINT", () => void shutdown());
    process.on("SIGTERM", () => void shutdown());
}

main().catch((err) => {
    console.error(err);
    process.exit(1);
});

In the above code, we are configuring Express Framework to accept JSON payloads and to recognize our two route files. Skipping beyond the error handling and listening of our backend, you'll find the shutdown logic.

When a shutdown is triggered, we close the server from listening for connections, and we also close the connection to our MongoDB database. This is good practice for making sure everything is gracefully terminated.

Running the API to Test the Endpoints

For this particular tutorial, we are not exploring a frontend. However, we'll run and test our API endpoints so you can get an idea of what it looks like.

From a command line, execute the following to run the application in development mode:

tsx watch src/app.ts

With the application running, you can first test the health check endpoint:

curl -s http://localhost:3000/health

If that looks good, try searching for a movie:

curl -s "http://localhost:3000/movies/search?q=Terminator"

Replace the q with whatever value you'd like to search for. Make note of one of the movie id values in the response because you'll need it for populating your watchlist with the following command:

curl -s -X POST http://localhost:3000/watchlist \
  -H "Content-Type: application/json" \
  -d '{"movieId":"<PASTE_OBJECT_ID_HERE>"}'

Go ahead and give the other endpoints a test!

It's worth mentioning that what we built is good for a demo, but it will need some work for production. We didn't do much data validation, our error handling needs work, and we have no cross-origin resource sharing (CORS) policy in place to define who is allowed to access our application. However, it should get you started when working with MongoDB in unique ways.

Conclusion

You just saw how to use various features of MongoDB, such as MongoDB Search and CRUD, to build a movie watchlist application with ratings. We made use of MongoDB Search for full-text search functionality, and we used CRUD to process documents in our watchlist. We also saw that it is alright to duplicate data between the movies collection and the mflix_watchlist collection to boost performance and avoid JOIN operations.

To take this project to the next level, try adding a frontend or a few more watchlist features. Maybe you can include MongoDB Vector Search to come up with movie recommendations based on your watchlist items and ratings.

Want to try this project for yourself? Clone the repository on GitHub!

Migrating Your Content Management System (CMS) Assets With MongoDB and Node.js

Nic Raboy — Thu, 01 May 2025 14:40:00 +0000

Content platforms evolve as business strategies shift. At MongoDB, we embraced external publishing platforms like Dev.to, Medium, The Polyglot Developer, etc. to better engage developer communities, requiring us to redistribute content while maintaining our existing CMS data in MongoDB.

To support our multi-platform publishing strategy, we created a system to publish content between our MongoDB CMS and external platforms. As a result, we needed to migrate the content we had in our CMS to its new home. The migration process included exporting the written content stored in MongoDB and downloading a copy of the media assets that were stored on third-party servers.

In this tutorial, we'll explore the export process to get the job done with as little friction as possible.

The requirements

If you plan to reproduce this project or follow along, you'll want to make sure you have everything available and properly configured. In particular, we want the following:

A MongoDB database, either locally hosted or on MongoDB Atlas
Node.js v22+

I am using Node.js v22.14.0, but earlier and later versions might work as well. You may have to refer to the API documentation to be sure. For MongoDB, our CMS was hosted on MongoDB Atlas, but most variants should work fine for the purpose of the script. Just make sure the MongoDB instance is properly configured to allow for connections from your script.

It is important to note that our CMS was very custom and it's not likely that your setup will be the same. However, the concepts used here should put you on the right track.

Creating a Node.js project with the proper dependencies

To kick things off, we're going to need a project created with a few environment variables set. Pick a working directory with your terminal and execute the following:

npm init -y
npm install mongodb axios dotenv --save

The above commands will initialize a new Node.js project and download our three project dependencies. The MongoDB driver will allow us to query our database for our content, Axios will allow us to download our media assets through HTTP requests, and Dotenv will allow us to easily make use of environment variables within our project.

Next, execute the following commands:

touch main.js
touch .env

The touch command just creates files. If you’re on Windows, go ahead and create those files however it makes sense.

This is just a small script so all our code can go in the main.js file. Before we get there, open the .env file and add the following:

MONGODB_ATLAS_URI=
MONGODB_DATABASE=
MONGODB_COLLECTION=

EXPORT_URLS=./export_list.txt
OUTPUT_DIR=./output/

For safety, if you plan to track this project in Git, make sure you add the .env file to your .gitignore file.

Your CMS project probably won't be the same as mine and that is fine. It is just to give you an idea of how everything is working together. Feel free to follow along with the MongoDB side of things, or experiment with your own connection information. From a data perspective, everything we do will be read-only.

In my situation, I had a list of URLs that needed to be migrated out of MongoDB. This was not the entire content portfolio. The export_list.txt file in this example contains a list of URL slugs, one slug per line. For example, it might look something like this:

/products/mongodb/tensorflow-mongodb-charts/
/products/mongodb/trader-joes-llamaindex-vector-search/

The output directory is where we plan to save our content, which is Markdown in our system, along with any media assets.

Let's add some boilerplate code to our main.js file before we get into the specifics:

const { MongoClient } = require("mongodb");
const fs = require("node:fs/promises");
const { createWriteStream } = require("fs");
const axios = require("axios");

require("dotenv").config();

const MONGODB_URI = process.env.MONGODB_ATLAS_URI;
const MONGODB_DATABASE = process.env.MONGODB_DATABASE;
const MONGODB_COLLECTION = process.env.MONGODB_COLLECTION;
const EXPORT_URLS = process.env.EXPORT_URLS;
const OUTPUT_DIR = process.env.OUTPUT_DIR;

const mongoClient = new MongoClient(MONGODB_URI);
var database, collection;

(async() => {
    try {
        await mongoClient.connect();
        database = await mongoClient.db(MONGODB_DATABASE);
        collection = await database.collection(MONGODB_COLLECTION);
        // Code here soon ...
        await mongoClient.close();
    } catch (e) {
        console.error("ERROR: ", e.message);
    }
})();

In the above code, we've extracted our environment variables and established a connection to our database. What comes next will be two parts: downloading the Markdown, and downloading the media.

Downloading the content from MongoDB

Like previously mentioned, the CMS data we had stored in MongoDB included the raw Markdown of our tutorial content, but it also had various metadata information like author, tags, URL slug, etc.

In the main.js file, add the following function:

async function loadUrlFileData(removeTrailingSlash = false) {
    let data = await fs.readFile(EXPORT_URLS, { encoding: "utf8" });
    let slugs = data.split('\n');
    if(removeTrailingSlash == true) {
        slugs = slugs.map(slug => slug.replace(/\/$/, ""));
    }
    return slugs;
}

Remember, in this example, the URLs we want to download were specified in a file. Your use-case might differ, but it shouldn't be difficult to pivot.

With the file scenario, the loadUrlFileData function will take each line of the file and add it to an array. In this example, we can specify if we want to remove the trailing slash from any of our URL slugs.

With a file loader in place, we can revisit our main asynchronous function:

(async() => {
    try {
        await mongoClient.connect();
        database = await mongoClient.db(MONGODB_DATABASE);
        collection = await database.collection(MONGODB_COLLECTION);
        let slugs = await loadUrlFileData(true);
        let cursor = collection.aggregate([
            {
                "$match": {
                    "calculated_slug": {
                        "$in": slugs
                    }
                }
            },
            {
                "$project": {
                    title: "$name",
                    description: 1,
                    slug: "$calculated_slug",
                    authors: 1,
                    tags: 1,
                    content: 1
                }
            },
        ]);
        let matchedContent = await cursor.toArray();
        await mongoClient.close();
    } catch (e) {
        console.error("ERROR: ", e.message);
    }
})();

Notice the revisions we've made in the above code.

First, we load our URL slugs into an array. Then, we make use of an aggregation pipeline in MongoDB to find content matches based on each of the slugs. The calculated_slug, name, description, authors, tags, and content are all fields within my data so no need to feel like you missed a step. For all the matches in the first pipeline stage, we can specify the fields that are valuable to us in the $project stage.

Depending on the scope of your results, you can use the toArray() function or plan to iterate over the cursor. My data was less than 200 content entries, so the toArray() was fine.

So at this point, we queried for the data we wanted, but we haven't actually done anything with the results. We need to save the results to a file on our filesystem.

Let's create two more functions in our main.js file:

function saveMarkdownToFile(slug, markdown) {
    let parts = slug.split("/").filter(part => part != "");
    let entryName = parts[parts.length - 1];
    return fs.mkdir(`${OUTPUT_DIR}${entryName}/assets`, { recursive: true })
        .then(() => fs.writeFile(`${OUTPUT_DIR}${entryName}/${entryName}.md`, markdown));
}

function saveMetadataToFile(slug, data) {
    let parts = slug.split("/").filter(part => part != "");
    let entryName = parts[parts.length - 1];
    return fs.mkdir(`${OUTPUT_DIR}${entryName}/assets`, { recursive: true })
        .then(() => fs.writeFile(`${OUTPUT_DIR}${entryName}/meta.json`, data));
}

The two functions above, more or less, do the same thing. You provide the slug, which will be used for file naming purposes, and the data. In this case, the data is either going to be the raw Markdown or the metadata for the entry. We want to save both, but as two separate files.

Looking back at our main logic-driving asynchronous function, we can add the following:

// Other code above ...
let matchedContent = await cursor.toArray();
for (const match of matchedContent) {
    await saveMarkdownToFile(match.slug, match.content);
    delete match.content;
    await saveMetadataToFile(match.slug, JSON.stringify(match, undefined, 4));
}

The code above is looping through each CMS entry matched by our aggregation pipeline. For every entry, we save the Markdown and the metadata. We make sure to remove the content from the metadata prior to saving it because it isn't necessary to store our massive amounts of Markdown in that file as well.

At this point, we've exported our Markdown entries from MongoDB. The problem is that none of our media assets were exported. We want to export them in case they are ever removed from the original source.

Saving the media assets hosted on a third-party platform

In our old CMS, the media assets, such as images, were not stored separately within our database. The assets themselves existed on S3 and similar, and the references to those assets were stored directly within the Markdown. This means we had to do some basic web scraping.

Within the main.js file, add the following function:

function extractAssetUrls(text) {
    let assetUrlRegex = /(https?:\/\/.*\.(?:png|jpg|jpeg|gif|webp|pdf))/gi;
    let assetUrls = text.match(assetUrlRegex) || [];
    assetUrls = assetUrls.map(asset => {
        let urlObject = new URL(asset);
        let urlParams = new URLSearchParams(urlObject.search);
        if(urlParams.has("url")) {
            asset = urlParams.get("url");
        }
        return asset;
    });
    return assetUrls;
}

The above function will take text—in this case, our entire Markdown content—and look for a pre-defined list of assets with a regular expression. Some of those assets had funky URL problems, so for every asset match, we had to further parse it with a URL object in JavaScript.

The assets discovered in the extractAssetUrls function get added to an array and returned. To be clear, only the URLs are added to this array. We still need to download them.

Include the following function for downloading:

function downloadAsset(assetUrl, slug) {
    let urlParts = assetUrl.split("/");
    let slugParts = slug.split("/").filter(part => part != "");
    let fileName = urlParts[urlParts.length - 1];
    let filePath = `${OUTPUT_DIR}${slugParts[slugParts.length - 1]}/assets/`;
    return axios({
        url: assetUrl,
        method: "GET",
        responseType: "stream"
    }).then(response => 
            new Promise((resolve, reject) => {
                response.data
                    .pipe(createWriteStream(filePath + fileName))
                    .on("error", reject)
                    .once("close", () => resolve(filePath + fileName));
            })
    ).catch(error => {
        if(error.response && error.response.status >= 400 && error.response.status <= 500) {
            console.warn(`Skipping asset due to client error (status: ${error.response.status}): ${assetUrl}`);
        }
    })  
}

The above downloadAsset function will take a URL and a slug, the slug being used just for naming purposes when saving. The asset is downloaded with an HTTP request and saved directly to the file system.

To make these functions work, let’s revisit our main asynchronous function:

// Previous code up here ...
let matchedContent = await cursor.toArray();
for (const match of matchedContent) {
    await saveMarkdownToFile(match.slug, match.content);
    let assetUrls = extractAssetUrls(match.content);
    await Promise.all(assetUrls.map(asset => downloadAsset(asset, match.slug + "/")));
    delete match.content;
    await saveMetadataToFile(match.slug, JSON.stringify(match, undefined, 4));
}

Notice that before we save our metadata to a file, we are extracting the asset URLs from the Markdown content and then downloading them.

For clarity, the Markdown and each asset will live in a separate directory within our output path. The separate directory in our scenario is the slug name.

Seeing the migration in action

If you made it to this point and you either want to try to run the code or see all the code together, no problem. The main.js file should look like the following:

const { MongoClient } = require("mongodb");
const fs = require("node:fs/promises");
const { createWriteStream } = require("fs");
const axios = require("axios");

require("dotenv").config();

const MONGODB_URI = process.env.MONGODB_ATLAS_URI;
const MONGODB_DATABASE = process.env.MONGODB_DATABASE;
const MONGODB_COLLECTION = process.env.MONGODB_COLLECTION;
const EXPORT_URLS = process.env.EXPORT_URLS;
const OUTPUT_DIR = process.env.OUTPUT_DIR;

const mongoClient = new MongoClient(MONGODB_URI);
var database, collection;

(async() => {
    try {
        await mongoClient.connect();
        database = await mongoClient.db(MONGODB_DATABASE);
        collection = await database.collection(MONGODB_COLLECTION);
        let slugs = await loadUrlFileData(true);
        let cursor = collection.aggregate([
            {
                "$match": {
                    "calculated_slug": {
                        "$in": slugs
                    }
                }
            },
            {
                "$project": {
                    title: "$name",
                    description: 1,
                    slug: "$calculated_slug",
                    authors: 1,
                    tags: 1,
                    content: 1
                }
            },
        ]);
        let matchedContent = await cursor.toArray();
        for (const match of matchedContent) {
            await saveMarkdownToFile(match.slug, match.content);
            let assetUrls = extractAssetUrls(match.content);
            await Promise.all(assetUrls.map(asset => downloadAsset(asset, match.slug + "/")));
            delete match.content;
            await saveMetadataToFile(match.slug, JSON.stringify(match, undefined, 4));
        }
        await mongoClient.close();
    } catch (e) {
        console.error("ERROR: ", e.message);
    }
})();

// Load the URLs to be used in the query from a file
// Optionally remove the trailing slash from the URLs if necessary
async function loadUrlFileData(removeTrailingSlash = false) {
    let data = await fs.readFile(EXPORT_URLS, { encoding: "utf8" });
    let slugs = data.split('\n');
    if(removeTrailingSlash == true) {
        slugs = slugs.map(slug => slug.replace(/\/$/, ""));
    }
    return slugs;
}

// Save the Markdown content at the $OUTPUT/$SLUG/$SLUG.md path
function saveMarkdownToFile(slug, markdown) {
    let parts = slug.split("/").filter(part => part != "");
    let entryName = parts[parts.length - 1];
    return fs.mkdir(`${OUTPUT_DIR}${entryName}/assets`, { recursive: true })
        .then(() => fs.writeFile(`${OUTPUT_DIR}${entryName}/${entryName}.md`, markdown));
}

// Save the metadata at the $OUTPUT/$SLUG/meta.json path
function saveMetadataToFile(slug, data) {
    let parts = slug.split("/").filter(part => part != "");
    let entryName = parts[parts.length - 1];
    return fs.mkdir(`${OUTPUT_DIR}${entryName}/assets`, { recursive: true })
        .then(() => fs.writeFile(`${OUTPUT_DIR}${entryName}/meta.json`, data));
}

// Extract any RegEx match for an asset in a file and return it
// Note, some assets were parameterized weird using URL parameters
function extractAssetUrls(text) {
    let assetUrlRegex = /(https?:\/\/.*\.(?:png|jpg|jpeg|gif|webp|pdf))/gi;
    let assetUrls = text.match(assetUrlRegex) || [];
    assetUrls = assetUrls.map(asset => {
        let urlObject = new URL(asset);
        let urlParams = new URLSearchParams(urlObject.search);
        if(urlParams.has("url")) {
            asset = urlParams.get("url");
        }
        return asset;
    });
    return assetUrls;
}

// Download any asset file and store it in the $OUTPUT/$SLUG/assets directory
function downloadAsset(assetUrl, slug) {
    let urlParts = assetUrl.split("/");
    let slugParts = slug.split("/").filter(part => part != "");
    let fileName = urlParts[urlParts.length - 1];
    let filePath = `${OUTPUT_DIR}${slugParts[slugParts.length - 1]}/assets/`;
    return axios({
        url: assetUrl,
        method: "GET",
        responseType: "stream"
    }).then(response => 
            new Promise((resolve, reject) => {
                response.data
                    .pipe(createWriteStream(filePath + fileName))
                    .on("error", reject)
                    .once("close", () => resolve(filePath + fileName));
            })
    ).catch(error => {
        if(error.response && error.response.status >= 400 && error.response.status <= 500) {
            console.warn(`Skipping asset due to client error (status: ${error.response.status}): ${assetUrl}`);
        }
    })  
}

Remember, your CMS probably doesn't look like the CMS we were using at MongoDB. Make the changes where you need to, or feel free to pull bits and pieces from the code to cover your use-case.

With the code in place, you can run the project with the following:

node main.js

Depending on how many URLs you are exporting and how many image assets there are, it should take a minute to run. Images that cannot be accessed or are not found will be skipped and reported.

Conclusion

You just saw how to pull data out of MongoDB and save it to a file. There are countless ways of doing this, but this code worked for our needs at MongoDB. To be clear, our needs were to switch from publishing on our internal platform and instead start publishing on external platforms. This is why we needed to download our Markdown data and each asset associated with it.

Send Messages with Nexmo through Golang and AWS Lambda

Nic Raboy — Wed, 04 Dec 2019 17:24:34 +0000

I recently saw a tutorial by Tom Morris on the Nexmo blog titled, Sending SMS from Python with Google Cloud Functions, and thought it was an interesting example and use of SMS messaging, even though Python and Google Cloud Functions are not my normal.

After reading the blog, I thought it would be interesting to expand it to AWS Lambda as the functions provider and switch the language to Golang, something that I'm very passionate about.

In this tutorial, we're going to create an API using AWS Lambda and AWS API Gateway that sends an SMS message with the Nexmo Messaging API, but with the Go programming language.

The Requirements

There are a few requirements that must be satisfied prior to working through this tutorial:

You need to have Go installed and be at least somewhat familiar with the language.
You need to have a Nexmo developer account.
You need to have an Amazon Web Services account.

Since Go is a compiled language, we need to be able to develop and build locally. Our builds will be uploaded to AWS, hence the need for an AWS account. These applications will make use of the Nexmo Messaging API.

Building a Messaging Function with Go and the AWS Lambda SDK

When it comes to the development of the Go application, there isn't too much going on. In terms of the flow of events, they'll look something like this:

The user will make a request to the function.
The function will send an HTTP request to the Nexmo Messages API.
The function will respond with the error or message id from the API.

While the core logic will be around an HTTP request, there will be a bit of setup in regards to the request and response data model.

Within the $GOPATH, create a new project with a main.go file that contains the following:

package main

var NEXMO_API_KEY string = "API_KEY_HERE"
var NEXMO_API_SECRET string = "API_SECRET_HERE"

func main() {}

Both the API key and the API secret can be obtained from within the Nexmo developer portal.

Since this is going to be an AWS Lambda project, the Go SDK for Lambda must be installed. From the command line, execute the following:

go get github.com/aws/aws-lambda-go/lambda

With the SDK installed through the Go package manager, it can be used within the project. To use it within the project, we can make the following changes:

package main

import "github.com/aws/aws-lambda-go/lambda"

var NEXMO_API_KEY string = "API_KEY_HERE"
var NEXMO_API_SECRET string = "API_SECRET_HERE"

func Handler() {}

func main() {
    lambda.Start(Handler)
}

As you can probably suspect, all of our messaging logic will appear in the Handler function since that is what will be executed when the client makes a Lambda request.

Before we get into the logic, we need to define some data models for the various requests and responses. Go requires that we explicitly define our data models through struct variables so that JSON can be properly marshaled between requests.

Add the following to the main.go file:

type NexmoMessageContent struct {
    Type string `json:"type"`
    Text string `json:"text"`
}

type NexmoMessage struct {
    Content NexmoMessageContent `json:"content"`
}

type NexmoMessageService struct {
    Type   string `json:"type"`
    Number string `json:"number"`
}

type NexmoMessagesRequest struct {
    From    NexmoMessageService `json:"from"`
    To      NexmoMessageService `json:"to"`
    Message NexmoMessage        `json:"message"`
}

type NexmoMessagesResponse struct {
    Id string `json:"message_uuid"`
}

If you've ever used the Nexmo Messages API, you'll be familiar with the JSON that is expected to be sent with every request to the API. The above data structures represent the data that will be sent. With the JSON annotations in place, the JSON would look something like this:

{
    "from": { "type": "sms", "number": "SENDER_NUMBER_HERE" },
    "to": { "type": "sms", "number": "RECIPIENT_NUMBER_HERE" },
    "message": {
        "content": {
            "type": "text",
            "text": "Hello World"
        }
    }
}

When sending a request to the Nexmo Messages API, a response with a UUID will be returned if it was successful. We've written a NexmoMessagesResponse struct which holds this data and plan to send it back to the client.

The only thing we're missing now is the request that the client will be making to the Lambda function. If our example is trying to mimic the Google Cloud Functions example, we are expecting a recipient phone number and the mobile platform they are using.

With this information in mind, we can create a data structure that looks like the following:

type LambdaRequest struct {
    Platform  string `json:"platform"`
    Recipient string `json:"recipient"`
}

With all the data models crafted, we can focus the rest of our effort on making the messaging request.

Making HTTP requests in Go isn't a difficult process. I explained a few of the approaches in-depth, in a previous tutorial that I had written titled, Consume RESTful API Endpoints within a Golang Application, although my personal choice would be to use a Go http.Client structure. In this particular example, we're going to need to make a POST request as outlined in the Nexmo documentation.

Let's start by creating a SendMessage function which will essentially build a request like that of Nexmo's many cURL examples:

func SendMessage(body map[string]string) (NexmoMessagesResponse, error) {
    nexmoMessagesRequest := &NexmoMessagesRequest{
        From: NexmoMessageService{
            Type:   "sms",
            Number: body["from"],
        },
        To: NexmoMessageService{
            Type:   "sms",
            Number: body["to"],
        },
        Message: NexmoMessage{
            Content: NexmoMessageContent{
                Type: "text",
                Text: body["message"],
            },
        },
    }
    bodyData, _ := json.Marshal(nexmoMessagesRequest)
    request, _ := http.NewRequest("POST", "https://api.nexmo.com/v0.1/messages", bytes.NewBuffer(bodyData))
    request.Header.Set("Content-Type", "application/json")
    request.Header.Set("Accept", "application/json")
    request.Header.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(NEXMO_API_KEY+":"+NEXMO_API_SECRET)))
    client := &http.Client{}
    response, err := client.Do(request)
    if err != nil {
        return NexmoMessagesResponse{}, err
    } else {
        data, _ := ioutil.ReadAll(response.Body)
        var result NexmoMessagesResponse
        json.Unmarshal(data, &result)
        return result, nil
    }
}

The above function will take a map[string]string and return a NexmoMessagesResponse or an error. The first part of the function takes data from the map and constructs an object that will be marshaled into JSON. When it comes to constructing the HTTP request, the object is used as the body and various headers are defined, one of which is an authorization header.

With Nexmo a JSON Web Token (JWT) can be used or the API key and API secret. In this example, the API key and API secret will be used.

The SendMessage function is most of the work done.

Since we're using the AWS Lambda SDK for Go and we have a Handler function, the next step is to define what's in the Handler function. It should look something like this:

func Handler(request LambdaRequest) (NexmoMessagesResponse, error) {
    return SendMessage(
        map[string]string{
            "from":    "15404161937",
            "to":      request.Recipient,
            "message": request.Platform,
        },
    )
}

Remember, the client accessing our API should be providing the platform and recipient information. Remember, they need to define what mobile platform they are using and the phone number the SMS message should be sent to.

If you can believe it, the AWS Lambda function is complete. When triggered with the appropriate recipient and platform information, the SMS message will be sent and the id of the message will be returned.

Next, we need to get the function into AWS Lambda and make the function accessible as part of standard HTTP requests, typical in an API.

Configuring AWS Lambda and AWS API Gateway as a Scalable Web API

Before we even get into the AWS portal, we should probably build our Go project. From the command line, while in the project path, execute the following:

GOOS=linux go build

AWS requires our application to be compatible with Linux. Lucky for us, Go ships with all the correct tool-chains to cross-compile. For more information on cross-compiling with Go, check out my previous tutorial on the subject.

As part of a Lambda requirement, the binary must be zipped. Take note of the file name and add it to a ZIP archive at the root of the archive. The filename of the binary represents the handler’s name.

Next, jump into the AWS Developer Console and choose to create a new Lambda function. The name isn't too important, just make sure it is using the Go runtime.

Inside the dashboard for the particular function, choose to upload the ZIP archive that you have just just created. Remember to supply the correct filename for the handler information.

We’re also going to want to add API Gateway as a trigger for the function.

Enter the API Gateway part of the AWS Developer Console. This is where a public endpoint will be created to point to the function. Within the dashboard for the API Gateway, choose Create Method within the Resources tab.

The method created should be a POST request and it should reference the ARN value of the AWS Lambda function when asked. When finished, don't forget to deploy the API through the same menu as creating a new method.

At this point, the Lambda function should be reachable through the API Gateway that was configured. We can obtain the URL through the Stages tab if a custom domain isn’t configured.

Conclusion

We just saw how to send messages to mobile devices using AWS Lambda and the Go programming language. To sum it up, if a user were to make a request to the API, managed with API Gateway, the request would flow to AWS Lambda, be sent to Nexmo, and then be sent to the user’s device. A use case, as previously mentioned, might be to send the app store link, whether it be Google Play or iTunes, to the user's phone through SMS.

While not explored in this example, we can easily add to this project by creating a front-end for your users to engage with rather than having strictly a RESTful API.