Forem: Nic Raboy

Test Password Strength and Password History with TypeScript and MongoDB

Nic Raboy — Fri, 15 May 2026 13:55:05 +0000

Just about every application you build is going to need some form of user authentication, and the moment you have user accounts, you have passwords to manage. Storing them safely is only part of the job. You also need to make sure those passwords are strong enough to be worth protecting in the first place, and in many cases, you need to make sure the same password isn't recycled every time it’s rotated.

Password reuse is a bigger deal than it sounds. Compliance frameworks frequently require that the last several passwords cannot be used again, and even outside of compliance, it is good hygiene. If a password leaks today and a user just rotates back to it six months from now, your rotation policy did not actually protect anyone.

MongoDB happens to be a really good fit for this kind of feature. A user document can store the current password hash at the root of the document, along with a growing array of previous hashes, all in the same record. There is no separate join table to set up and no migration to run when a user changes their password for the tenth time.

In this tutorial, we'll see how to build a small TypeScript API using Express Framework, Zod, and the MongoDB Node.js driver. This application registers users with strong password rules, authenticates them, and rejects any password change that has been seen before.

The Prerequisites

Prior to starting this tutorial, you'll need a few things in place:

A MongoDB instance, either a local server or an Atlas cluster on the free tier
Node.js 22+

The expectation is that you can already connect to MongoDB with a connection string. If you need help getting an Atlas cluster running, check out the MongoDB documentation for getting started.

We're working from an empty project, so the first thing to do is create the directory and initialize it:

mkdir typescript-password-history-example
cd typescript-password-history-example
npm init -y

The above commands create the project directory and a starter package.json file.

Next, we'll install the runtime dependencies:

npm install express mongodb bcrypt zod dotenv

We're pulling in Express Framework as the HTTP layer, the official MongoDB Node.js driver, bcrypt for hashing passwords, zod for schema validation, and dotenv for loading configuration from an environment file.

For TypeScript and the type definitions, run:

npm install --save-dev typescript ts-node @types/express @types/node @types/bcrypt

The ts-node package lets us run TypeScript directly during development without a separate compile step, keeping the feedback loop tight.

Because this is a TypeScript project, we need a tsconfig.json at the root of the project:

{
    "compilerOptions": {
        "target": "ES2020",
        "module": "commonjs",
        "lib": ["ES2020"],
        "outDir": "./dist",
        "rootDir": "./",
        "strict": true,
        "esModuleInterop": true,
        "skipLibCheck": true,
        "forceConsistentCasingInFileNames": true,
        "resolveJsonModule": true
    },
    "include": ["./**/*.ts"],
    "exclude": ["node_modules", "dist"]
}

The configuration emits CommonJS to a dist/ directory when we run tsc, and as strict mode is turned on, the compiler actually enforces type safety rather than letting things slide.

It also helps to update the scripts block of package.json, so we have a development command and a production command:

"scripts": {
    "build": "tsc",
    "start": "node dist/main.js",
    "dev": "ts-node main.ts"
}

The dev script will run the TypeScript entry point directly. The build script compiles to dist/, and start runs the compiled output.

By the time we're done, the project will have the following shape:

.
├── .env
├── main.ts
├── tsconfig.json
├── package.json
├── libs/
│   ├── mongodb.ts
│   └── validate.ts
└── routes/
    └── passwords.ts

Each of those files will get built up as we work through the tutorial. Go ahead and create the libs and routes directories now so you have somewhere to drop code as we go.

The last piece of setup is the environment file. Create a .env file at the root of the project with the following:

MONGODB_URI=ADD_YOUR_ATLAS_URI_HERE
MONGODB_DB_NAME=password_history_demo
PORT=3000

Replace the MONGODB_URI with the connection string for whatever MongoDB instance you're using. The password_history_demo database specified in MONGODB_DB_NAME does not need to exist ahead of time. MongoDB will create it lazily the first time we write data. If you're using version control, make sure .env is listed in your .gitignore so the connection string does not get committed.

Establish an API Foundation with Express and TypeScript

Before we get into password logic, we need an Express application that listens for HTTP requests and a clean way to start and stop it. The entry point lives in main.ts at the root of the project.

Add the following to main.ts:

import * as dotenv from "dotenv";
dotenv.config();

import express from "express";
import { getDb, closeConnection } from "./libs/mongodb";
import passwordRouter from "./routes/passwords";

const app = express();
const PORT = process.env.PORT ?? 3000;

app.use(express.json());

app.use("/users", passwordRouter);

async function start() {
    const db = await getDb();
    await db.collection("users").createIndex({ username: 1 }, { unique: true });

    const server = app.listen(PORT, () => {
        console.log(`Server running on http://localhost:${PORT}`);
    });

    process.on("SIGTERM", async () => {
        console.log("SIGTERM received, shutting down gracefully...");
        server.close(async () => {
            await closeConnection();
            process.exit(0);
        });
    });

    process.on("SIGINT", async () => {
        console.log("SIGINT received, shutting down gracefully...");
        server.close(async () => {
            await closeConnection();
            process.exit(0);
        });
    });
}

start();

That file is doing a lot, so let's pull it apart.

At the top, we have two dotenv lines followed by the rest of the imports:

import * as dotenv from "dotenv";
dotenv.config();

import express from "express";
import { getDb, closeConnection } from "./libs/mongodb";
import passwordRouter from "./routes/passwords";

The order matters here. dotenv.config() runs before the MongoDB module is imported, which means the connection string and database name will already be present in process.env by the time libs/mongodb.ts is evaluated.

Next, we have the Express application setup:

const app = express();
const PORT = process.env.PORT ?? 3000;

app.use(express.json());

app.use("/users", passwordRouter);

The express.json() middleware parses incoming JSON bodies so we can read req.body as a plain object later. The router is mounted under /users, so every endpoint in routes/passwords.ts will be reachable beneath that prefix. We'll write that router shortly.

Then we have the start function, which is the part that actually brings the server online:

async function start() {
    const db = await getDb();
    await db.collection("users").createIndex({ username: 1 }, { unique: true });

    const server = app.listen(PORT, () => {
        console.log(`Server running on http://localhost:${PORT}`);
    });

    // ... signal handlers shown below
}

The first thing it does is grab a database reference and create a unique index on the username field of the users collection. Doing it at startup means we don't have to check for duplicate usernames before every insert. MongoDB will raise an error with code 11000 if anyone tries to register a username that already exists, and we'll catch that error in the route handler. It is a much cleaner pattern than running a findOne followed by an insertOne, which is also racy.

After the index is ready, we call app.listen and hold onto the returned server reference so we can close it later.

The final piece is the shutdown handling, which lives inside the same start function:

async function start() {
    // ... database, index, and app.listen above

    process.on("SIGTERM", async () => {
        console.log("SIGTERM received, shutting down gracefully...");
        server.close(async () => {
            await closeConnection();
            process.exit(0);
        });
    });

    process.on("SIGINT", async () => {
        console.log("SIGINT received, shutting down gracefully...");
        server.close(async () => {
            await closeConnection();
            process.exit(0);
        });
    });
}

When the process receives either SIGTERM (which most container orchestrators send) or SIGINT (which is what Ctrl+C sends), we stop accepting new connections, close the MongoDB client, and exit. Without this, you'd leave an open client connection behind every time you restart the server.

Of course, none of this will compile yet because libs/mongodb.ts does not exist. That's our next step.

Create a Singleton MongoDB Class to Manage Database Connections

You could open a new MongoClient on every incoming request, but it is almost never what you want. Connecting is expensive, it adds latency, and it makes graceful shutdown much harder than it needs to be. A long-running API like ours should connect once when the process boots and reuse the same client for every request after that.

Open libs/mongodb.ts and add the following:

import { MongoClient, Db } from "mongodb";

const uri = process.env.MONGODB_URI as string;
const dbName = process.env.MONGODB_DB_NAME as string;

if (!uri) {
    throw new Error("MONGODB_URI is not defined in the environment variables");
}

if (!dbName) {
    throw new Error("MONGODB_DB_NAME is not defined in the environment variables");
}

let client: MongoClient | null = null;

export async function getClient(): Promise<MongoClient> {
    if (!client) {
        client = new MongoClient(uri, {
            connectTimeoutMS: 5000,
            socketTimeoutMS: 30000,
            serverSelectionTimeoutMS: 5000,
            appName: "devrel-tutorial-typescript-passwordhistory"
        });
        await client.connect();
    }
    return client;
}

export async function getDb(): Promise<Db> {
    const mongoClient = await getClient();
    return mongoClient.db(dbName);
}

export async function closeConnection(): Promise<void> {
    if (client) {
        await client.close();
        client = null;
    }
}

The top of the file reads the two environment variables we set earlier and fails fast if either of them is missing. There is no point in starting the server if it cannot reach the database.

The interesting bit is getClient:

export async function getClient(): Promise<MongoClient> {
    if (!client) {
        client = new MongoClient(uri, {
            connectTimeoutMS: 5000,
            socketTimeoutMS: 30000,
            serverSelectionTimeoutMS: 5000,
            appName: "devrel-github-typescript-passwordhistory"
        });
        await client.connect();
    }
    return client;
}

The module-level client variable starts as null. The first call to getClient instantiates the MongoClient, awaits connect, and caches it. Every subsequent call returns the cached instance immediately. That is the entire singleton pattern. We did not bother setting a maxPoolSize because this is a traditional long-running OLTP server, and the default value of 100 is appropriate for that workload.

The three timeout options are worth a quick mention. connectTimeoutMS caps how long a single socket connection attempt can take, socketTimeoutMS caps how long a socket can sit idle, and serverSelectionTimeoutMS caps how long the driver will spend looking for a suitable server before giving up. The appName value is cosmetic but useful. It shows up in MongoDB Atlas logs and metrics, so you can tell which application is generating which queries.

Then we have the two helpers that the rest of the application actually uses:

export async function getDb(): Promise<Db> {
    const mongoClient = await getClient();
    return mongoClient.db(dbName);
}

export async function closeConnection(): Promise<void> {
    if (client) {
        await client.close();
        client = null;
    }
}

The getDb function returns a Db reference scoped to the database name from the environment file. Anywhere in the codebase that needs to talk to MongoDB will go through this function. The closeConnection function shuts the client down and resets the module-level variable. This is what gets called by the SIGTERM and SIGINT handlers back in main.ts.

Test Password Strength with Regular Expressions and Zod

A password is only as good as the rules you enforce around it. For this tutorial, we're going to require a minimum of eight characters, at least one uppercase letter, at least one lowercase letter, at least one digit, and at least one character that is not a letter or a number. None of these rules is particularly exotic, but together they keep users from registering with something like password or 12345678.

We're going to enforce them with Zod schemas wired into Express as middleware. That way, the route handlers themselves stay focused on database work and never have to inspect raw request bodies.

Open libs/validate.ts and add the following:

import { Request, Response, NextFunction } from "express";
import { ZodSchema } from "zod";

export function validate(schema: ZodSchema) {
    return (req: Request, res: Response, next: NextFunction): void => {
        const result = schema.safeParse(req.body);
        if (!result.success) {
            res.status(400).json({ errors: result.error.flatten().fieldErrors });
            return;
        }
        req.body = result.data;
        next();
    };
}

The validate function is a middleware factory. You hand it a Zod schema, and it hands you back an Express middleware function that runs that schema against the incoming req.body. If validation fails, the middleware sends back a 400 with a flattened map of field errors, and the route handler is never called. If validation succeeds, it replaces req.body with the parsed output before calling next().

Replacing req.body with result.data is intentional. Zod will strip unknown keys and apply any transformations defined on the schema, so by the time the route handler runs, the body is guaranteed to look exactly like the schema declared. There is no risk of a stray field sneaking through.

Now we can define the actual schemas. These live alongside the route handlers in routes/passwords.ts, but we'll write that file in pieces throughout the rest of the tutorial. For now, here's the schema that encodes the password strength rules:

import { z } from "zod";

const strongPasswordSchema = z
    .string()
    .min(8, "Password must be at least 8 characters")
    .regex(/[A-Z]/, "Password must contain at least one uppercase letter")
    .regex(/[a-z]/, "Password must contain at least one lowercase letter")
    .regex(/[0-9]/, "Password must contain at least one digit")
    .regex(/[^A-Za-z0-9]/, "Password must contain at least one special character");

Each rule is its own .regex call with a custom error message. If a user submits something that breaks two rules, Zod will report both errors at once, which makes for a much better registration experience than trickling them out one at a time.

The character class [^A-Za-z0-9] is doing the work for the special character rule. It matches anything that is not an uppercase letter, a lowercase letter, or a digit. We're not picky about which special character. Punctuation, symbols, and even whitespace will satisfy it.

With the strong password schema in place, we can compose the three request schemas the API needs:

const registerSchema = z.object({
    username: z.string().min(1, "Username is required").trim(),
    password: strongPasswordSchema,
});

const loginSchema = z.object({
    username: z.string().min(1, "Username is required").trim(),
    password: z.string().min(1, "Password is required"),
});

const changePasswordSchema = z.object({
    current_password: z.string().min(1, "Current password is required"),
    new_password: strongPasswordSchema,
});

Registration uses the strict rules on password because that's a brand new password being committed to the database. The change-password schema uses the strict rules on new_password for the same reason.

Notice that loginSchema and the current_password field in changePasswordSchema do not use the strict rules. That's deliberate. The strength rules are about what we are willing to store, not what we are willing to compare against. A user who registered before you tightened your password policy still needs to be able to log in with their old, weak password so they can rotate to a stronger one. If the strong rules applied to the login flow, you would lock those users out permanently.

Store Passwords Safely in MongoDB with Password Change History

Now we get to the part this whole tutorial is named after. The shape of a user document is going to look something like this:

{
    "_id": "ObjectId(...)",
    "username": "nraboy",
    "password": "BCRYPT_HASH",
    "password_history": [
        { "password": "BCRYPT_HASH", "created_at": "ISODate(...)" }
    ],
    "created_at": "ISODate(...)",
    "updated_at": "ISODate(...)"
}

The current password hash lives at the root of the document. That's the value we compare against during login, and it is the one we update during a password change. Alongside it, we keep a password_history array where each entry is a previous hash with the timestamp it was set. When a user first registers, the history starts with a single entry that mirrors the root password. Every password change appends another entry. To check if a candidate password has been used before, we walk the history and compare against each entry.

Let's start building routes/passwords.ts. The top of the file pulls in the dependencies, defines the data shapes, and declares a salt rounds constant:

import { Router, Request, Response } from "express";
import { z } from "zod";
import bcrypt from "bcrypt";
import { getDb } from "../libs/mongodb";
import { MongoServerError } from "mongodb";
import { validate } from "../libs/validate";

const SALT_ROUNDS = 12;

type PasswordHistoryEntry = {
    password: string;
    created_at: Date;
};

type UserDocument = {
    username: string;
    password: string;
    password_history: PasswordHistoryEntry[];
    created_at: Date;
    updated_at: Date;
};

The two types describe the user document and a single history entry. We're using them as generic parameters on db.collection<UserDocument>("users") calls below so TypeScript can give us reasonable autocomplete when we read fields from the results.

Twelve salt rounds is a sensible default for bcrypt. It is slow enough to make brute-force attacks expensive and fast enough that a legitimate login does not feel sluggish on modern hardware.

Next, define three small helper functions for working with bcrypt:

async function hashPassword(plaintext: string): Promise<string> {
    return bcrypt.hash(plaintext, SALT_ROUNDS);
}

async function verifyPassword(plaintext: string, hash: string): Promise<boolean> {
    return bcrypt.compare(plaintext, hash);
}

async function isPasswordInHistory(
    plaintext: string,
    history: PasswordHistoryEntry[]
): Promise<boolean> {
    for (const entry of history) {
        const match = await bcrypt.compare(plaintext, entry.password);
        if (match) return true;
    }
    return false;
}

The first two are thin wrappers around the bcrypt API. They exist mostly to keep the route handlers readable.

The third one, isPasswordInHistory, is where the history check actually happens. It loops through every entry in the array and calls bcrypt.compare against each one. You cannot just hash the candidate password and look for an exact string match because bcrypt mixes a fresh random salt into every hash. Two calls to bcrypt.hash("Nic12345$", 12) will produce two different hash strings. The only way to know if a plaintext matches a stored hash is to feed both into bcrypt.compare.

The loop short-circuits the moment it finds a match, so in the common case where the user picks something genuinely new, the cost is proportional to the size of the history. For typical history sizes of five to ten entries, that overhead is negligible.

With the schemas from the previous section and the helpers above, we can start defining routes. Initialize the router:

const router = Router();

And start with the demo endpoint that lists every user document. This one only exists so you can poke at the database during development:

router.get("/", async (_req: Request, res: Response): Promise<void> => {
    try {
        const db = await getDb();
        const users = await db.collection<UserDocument>("users").find({}).toArray();
        res.status(200).json(users);
    } catch (err) {
        console.error(err);
        res.status(500).json({ error: "Internal server error" });
    }
});

This endpoint returns the full user document, including the bcrypt hash and the entire history array. You would never ship something like this in production. It is genuinely useful while you're developing, though, because it lets you see exactly what the password history looks like as it grows.

The registration endpoint is where new users get created:

router.post("/", validate(registerSchema), async (req: Request, res: Response): Promise<void> => {
    const { username, password } = req.body;

    try {
        const db = await getDb();
        const hashed = await hashPassword(password);
        const now = new Date();

        await db.collection("users").insertOne({
            username,
            password: hashed,
            password_history: [{ password: hashed, created_at: now }],
            created_at: now,
            updated_at: now,
        });

        res.status(201).json({ message: "User created successfully" });
    } catch (err) {
        if (err instanceof MongoServerError && err.code === 11000) {
            res.status(409).json({ error: "Username already exists" });
            return;
        }
        console.error(err);
        res.status(500).json({ error: "Internal server error" });
    }
});

There are a couple of things worth pointing out here. First, validate(registerSchema) runs before the handler. If validation fails, the handler never executes, so by the time we destructure username and password from req.body, we already know they're well-formed.

Second, the new document seeds password_history with a single entry that contains the same hash we just stored at the root. That way, the very first password is part of the history from day one. If we did not do that, the user could "rotate" their password back to the original on their first password change, which is exactly the behavior we're trying to prevent.

Third, instead of running a findOne to see if the username is taken and then doing an insertOne, we just attempt the insert and catch MongoServerError with code 11000. That's the duplicate-key error code, and it is the cleanest way to enforce uniqueness because MongoDB itself is the only thing that needs to coordinate. There is no window between the check and the insert during which two requests could both decide the username is free.

The login endpoint is much simpler:

router.post("/login", validate(loginSchema), async (req: Request, res: Response): Promise<void> => {
    const { username, password } = req.body;

    try {
        const db = await getDb();
        const users = db.collection<UserDocument>("users");

        const user = await users.findOne({ username });
        if (!user) {
            res.status(401).json({ error: "Invalid credentials" });
            return;
        }

        const valid = await verifyPassword(password, user.password);
        if (!valid) {
            res.status(401).json({ error: "Invalid credentials" });
            return;
        }

        res.status(200).json({ message: "Login successful" });
    } catch (err) {
        console.error(err);
        res.status(500).json({ error: "Internal server error" });
    }
});

Notice that both failure cases return the same 401 with the same generic message. There is a temptation to send back "User not found" when the lookup misses and "Wrong password" when the comparison fails, but doing so leaks information. An attacker who can tell those two states apart can enumerate which usernames exist in your system, which is the first half of a credential stuffing attack. Returning a single opaque error keeps both branches indistinguishable.

This is also why a real production version would want some kind of rate limiting on this endpoint. We're not adding that here, but it is worth keeping in mind.

The change-password endpoint is the one where the history check actually pays off:

router.put("/:username/password", validate(changePasswordSchema), async (req: Request, res: Response): Promise<void> => {
    const { username } = req.params;
    const { current_password, new_password } = req.body;

    try {
        const db = await getDb();
        const users = db.collection<UserDocument>("users");

        const user = await users.findOne({ username });
        if (!user) {
            res.status(404).json({ error: "User not found" });
            return;
        }

        const currentValid = await verifyPassword(current_password, user.password);
        if (!currentValid) {
            res.status(401).json({ error: "Current password is incorrect" });
            return;
        }

        const history = user.password_history ?? [];

        const alreadyUsed = await isPasswordInHistory(new_password, history);
        if (alreadyUsed) {
            res.status(409).json({ error: "New password has been used before. Please choose a different password." });
            return;
        }

        const hashed = await hashPassword(new_password);
        const now = new Date();

        await users.updateOne(
            { username },
            {
                $set: {
                    password: hashed,
                    updated_at: now,
                },
                $push: {
                    password_history: { password: hashed, created_at: now },
                },
            }
        );

        res.status(200).json({ message: "Password updated successfully" });
    } catch (err) {
        console.error(err);
        res.status(500).json({ error: "Internal server error" });
    }
});

The handler runs through four checks in order. First, we look up the user by the username from the route parameter, returning a 404 if the user does not exist. Then we verify the supplied current_password against the stored hash. A mismatch returns a 401, which is appropriate because the caller is, in effect, failing to authenticate.

After that, we run the history check:

router.put("/:username/password", validate(changePasswordSchema), async (req: Request, res: Response): Promise<void> => {
    // ... user lookup and current password verification above

    const history = user.password_history ?? [];

    const alreadyUsed = await isPasswordInHistory(new_password, history);
    if (alreadyUsed) {
        res.status(409).json({ error: "New password has been used before. Please choose a different password." });
        return;
    }

    // ... update logic below
});

The nullish coalescing operator gives us a safe default for any users that might somehow be missing the password_history field, though in practice, the registration handler always sets it. If the candidate password matches any entry in the history, we send back a 409 and explain the situation to the caller.

If we make it past all four checks, the update itself uses two operators in one call:

router.put("/:username/password", validate(changePasswordSchema), async (req: Request, res: Response): Promise<void> => {
    // ... lookup, verification, and history check above

    const hashed = await hashPassword(new_password);
    const now = new Date();

    await users.updateOne(
        { username },
        {
            $set: {
                password: hashed,
                updated_at: now,
            },
            $push: {
                password_history: { password: hashed, created_at: now },
            },
        }
    );

    res.status(200).json({ message: "Password updated successfully" });
});

The $set operator overwrites the root password field with the new hash and bumps updated_at. At the same time, $push appends the new hash to the password_history array. Because both operators are part of the same update, the change is atomic from MongoDB's perspective. The current password and the history can never disagree about the user's most recent password.

Finally, the bottom of the file exports the router so main.ts can mount it:

export default router;

That completes the route layer. Time to try it out.

Testing the API Endpoints with cURL

We'll exercise the API from the command line with curl. The repository also includes a Bruno collection in the bruno/ directory if you'd rather click through requests in a GUI, but curl is faster to demonstrate in a tutorial.

Start the server in development mode:

npm run dev

You should see Server running on http://localhost:3000 in the terminal. Leave it running and open a second terminal for the requests.

First, register a new user:

curl -s -X POST http://localhost:3000/users \
    -H "Content-Type: application/json" \
    -d '{"username":"nraboy","password":"Nic12345$"}'

A successful registration responds with a 201 and {"message":"User created successfully"}. If you try the exact same request again, you'll get a 409 because the unique index on username rejects the duplicate.

To prove the validation middleware is doing its job, try sending a weak password:

curl -s -X POST http://localhost:3000/users \
    -H "Content-Type: application/json" \
    -d '{"username":"weakuser","password":"password"}'

The response is a 400 with a per-field map of the rules that were broken. No insert ever reaches MongoDB.

Now try logging in with the user we just created:

curl -s -X POST http://localhost:3000/users/login \
    -H "Content-Type: application/json" \
    -d '{"username":"nraboy","password":"Nic12345$"}'

A correct password returns {"message":"Login successful"}. Get one character wrong, and you'll see {"error":"Invalid credentials"} with a 401.

The interesting endpoint is the password change. Rotate to a brand new password:

curl -s -X PUT http://localhost:3000/users/nraboy/password \
    -H "Content-Type: application/json" \
    -d '{"current_password":"Nic12345$","new_password":"Nic67890$"}'

That responds with a 200 and {"message":"Password updated successfully"}. Now try to rotate back to the original password:

curl -s -X PUT http://localhost:3000/users/nraboy/password \
    -H "Content-Type: application/json" \
    -d '{"current_password":"Nic67890$","new_password":"Nic12345$"}'

This time the response is a 409 and {"error":"New password has been used before. Please choose a different password."}. The history check did exactly what we wanted.

You can verify what's actually in the database by hitting the demo list endpoint:

curl -s http://localhost:3000/users

The response includes the password_history array with two entries, one for the original password and one for the rotation, each with its own timestamp and bcrypt hash. Two different hashes for the same logical password value, which is the whole point of using bcrypt.compare rather than string equality during the history check.

Conclusion

You just saw how to combine TypeScript, Express Framework, Zod, and the MongoDB Node.js driver to enforce password strength and prevent password reuse. Zod schemas plugged into Express as middleware handle the strength rules, bcrypt handles the hashing, and a password_history array on each user document keeps a running record of every hash that user has ever used. The singleton MongoDB client keeps the connection layer simple, and a unique index on username lets MongoDB itself enforce uniqueness without us having to coordinate it in application code.

What we built is good for a demo, but a few things would need to change before this could be a production system. There is no rate limiting on the login or password change endpoints, both of which are attractive targets for brute-force attempts. There is no session or token mechanism, so right now a successful login just sends back a message. The GET /users endpoint exposes every hash in the database and would need to be removed entirely. And depending on your compliance requirements, you may want to cap the size of password_history so it does not grow without bound.

As a next step, try adding a password expiration field that forces a rotation after some number of days, or capping password_history so only the last ten entries are retained.

The full source for this project is available on GitHub.

Build a Movie Watchlist with Node.js, TypeScript, and MongoDB

Nic Raboy — Mon, 13 Apr 2026 15:19:10 +0000

Almost every modern web application will need a REST API for a client to talk to, and in almost every scenario, that client is going to expect JSON. The best developer experience is a stack where you can stay in JSON-shaped data end to end, without awkward transformations in the middle.

Take MongoDB, Express Framework, and Node.js as an example.

Express receives HTTP requests and sends responses. MongoDB sits in the middle and stores documents. The client can send JSON to your routes, your routes can send documents to MongoDB, and MongoDB can hand BSON back that maps naturally to what you serialize in the response. That works well because MongoDB is a document database. When you also want text search over fields like title and plot, MongoDB Search gives you a $search stage in an aggregation pipeline on the same cluster, so you are not bolting on a separate search system just to power a search box.

In this tutorial, we'll see how to build a small movie watchlist API using TypeScript and MongoDB. We'll explore a few different schema design opportunities and make use of MongoDB Search for full-text search.

The Prerequisites

Prior to starting this tutorial, you'll need a few things in place:

A MongoDB Atlas instance (M0 is fine) with the sample_mflix sample dataset loaded and available.
Node.js 22+

The expectation is that your cluster is already provisioned with credentials and network access rules, and that the sample dataset is loaded. If you need help deploying or configuring MongoDB Atlas, check out the MongoDB documentation for getting started.

We'll be working with an empty project, so to kick things off, you might want to run the following commands:

mkdir movie-watchlist-api
cd movie-watchlist-api
npm init -y

The above commands will create a new project directory and initialize it for Node.js by creating a package.json file.

There are a few dependencies that we need to install. They can be installed by executing the following commands:

npm install express mongodb dotenv

In the above, we're installing Express Framework and the MongoDB Node.js Driver, plus a package that will allow us to read from a .env file for configuration variables.

For TypeScript, the tsx runner (useful in development with watch mode), and the Express and Node type packages, run:

npm install --save-dev typescript tsx @types/express @types/node

To run this project in development mode, you'd execute tsx watch src/app.ts from your command line. However, that won't be possible until later in the tutorial.

Add a TypeScript Configuration File

Because we're using TypeScript, we need a tsconfig.json at the root of the project. Create the file with something along the following lines:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "CommonJS",
    "moduleResolution": "node",
    "outDir": "dist",
    "rootDir": "src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true
  },
  "include": ["src/**/*"]
}

The goal here is to emit CommonJS into dist/ from everything under src/, which keeps production as simple as node dist/app.js after tsc.

Understanding the Source Tree of the Project

With the tooling in place, it helps to know where things will live. This project will use the following foundation:

src/config.ts
src/db.ts
src/types/watchlist.ts
src/routes/movies.ts
src/routes/watchlist.ts
src/app.ts

Each file will serve a different purpose that we'll explore throughout the remainder of the tutorial. Go ahead and create the src, src/routes, and src/types directories so you're ready to paste code as we go.

Define and Configure the Environment Variables for the Project

We're going to need connection details before the Express Framework application can talk to MongoDB Atlas. Add the following to a .env file at the project's root:

MONGODB_URI=mongodb+srv://<USERNAME>:<PASSWORD>@<HOST>/
PORT=3000
SEARCH_INDEX_NAME=default

Make sure to add this file to your project's .gitignore if you're using version control, since the MONGODB_URI is considered sensitive information.

Rather than read process.env in every route file of the project, we'll define a small configuration module that validates what is required and exports plain values.

Open src/config.ts and include the following code:

import "dotenv/config";

function requiredEnv(name: string): string {
    const value = process.env[name];
    if (!value?.trim()) {
        throw new Error(`Missing required environment variable: ${name}`);
    }
    return value.trim();
}

export const config = {
    mongoUri: requiredEnv("MONGODB_URI"),
    port: Number(process.env.PORT) || 3000,
    dbName: "sample_mflix",
    moviesCollection: "movies",
    watchlistCollection: "mflix_watchlist",
    searchIndexName: process.env.SEARCH_INDEX_NAME?.trim() || "default",
};

The import "dotenv/config" line means that as soon as something imports config.ts, your .env values are loaded. Database and collection names are hard-coded to sample_mflix, movies, and mflix_watchlist, so this tutorial stays aligned with the sample dataset.

Connecting to MongoDB with a Singleton Database Class

You could open a new MongoClient on every request, but that usually is not what you want. It adds latency, wastes resources, and makes graceful shutdown harder. The pattern we'll use instead is a singleton where we'll connect once when the application boots, reuse that client for every request, and close it when the process exits.

With the environment variables already in config.ts, open src/db.ts and include the following code:

import { MongoClient, type Db } from "mongodb";
import { config } from "./config";

let singleton: Database | null = null;

export class Database {
    private constructor(
        private readonly mongoClient: MongoClient,
        readonly db: Db
    ) {}

    static async connect(): Promise<Database> {
        if (singleton) return singleton;
        const mongoClient = new MongoClient(config.mongoUri, {
            appName: "devrel-tutorial-typescript-moviewatchlist",
        });
        await mongoClient.connect();
        const db = mongoClient.db(config.dbName);
        singleton = new Database(mongoClient, db);
        await singleton.ensureWatchlistIndexes();
        return singleton;
    }

    static get instance(): Database {
        if (!singleton) {
            throw new Error("Database not connected");
        }
        return singleton;
    }

    async close(): Promise<void> {
        await this.mongoClient.close();
        singleton = null;
    }

    private async ensureWatchlistIndexes(): Promise<void> {
        const watchlistCollection = this.db.collection(
            config.watchlistCollection
        );
        await watchlistCollection.createIndex({ movieId: 1 }, { unique: true });
    }
}

In the connect method, we have the following logic:

static async connect(): Promise<Database> {
    if (singleton) return singleton;
    const mongoClient = new MongoClient(config.mongoUri, {
        appName: "devrel-tutorial-typescript-moviewatchlist",
    });
    await mongoClient.connect();
    const db = mongoClient.db(config.dbName);
    singleton = new Database(mongoClient, db);
    await singleton.ensureWatchlistIndexes();
    return singleton;
}

If singleton is already set, we return immediately and avoid opening another connection. Otherwise we create MongoClient with config.mongoUri and an appName that shows up in MongoDB Atlas, await connect(), take a Db reference for config.dbName (here sample_mflix), assign singleton, and call ensureWatchlistIndexes before returning. Doing the index step before the HTTP server starts listening means every route can assume the watchlist collection already has a unique index on movieId.

The ensureWatchlistIndexes function creates a unique index on movieId in mflix_watchlist, so MongoDB rejects a second insert for the same movie and the watchlist POST handler can reason about "no row", "inactive row", and "active row" without duplicate documents.

Define the Watchlist Document Type for TypeScript

Before we get too far into the routes, it's a good idea to describe the document shape we plan to store in mflix_watchlist. TypeScript will not infer that for you.

In src/types/watchlist.ts, add the following:

import type { ObjectId } from "mongodb";

export interface WatchlistDoc {
    _id: ObjectId;
    movieId: ObjectId;
    title: string;
    poster: string | null;
    rating?: number;
    active: boolean;
    createdAt: Date;
    updatedAt: Date;
}

We're denormalizing title and poster onto the watchlist document when the user adds or reactivates a movie, so a client can render the list without joining back to movies on every GET. The advantage of this approach is that we're making our queries faster by not doing a $lookup (JOIN) operation unless we absolutely have to. When listing all items in a watchlist, it is probably not common to need details such as the actor cast list, so we're only including the fields we absolutely need alongside the id value in case we want to expand the details later with a single fetch operation.

Searching for Movies within the Catalog with MongoDB Search

To search for movies in our application, it makes sense to leverage MongoDB Search for its full-text search capabilities. Before we can do that, we need to apply a search index on the collection and fields we want to search through.

Take the following MongoDB Search index:

{
    "mappings": {
        "dynamic": false,
        "fields": {
            "title": {
                "type": "string"
            },
            "plot": {
                "type": "string"
            }
        }
    }
}

In the above index, which is different from the unique index we created earlier, we are saying we only want to be able to use full-text search on the title and plot fields of a document.

To apply the index, navigate to MongoDB Atlas and make use of the Search & Vector Search navigation item. When prompted in the index creation process, make sure to choose sample_mflix as the database and movies as the collection. You can name the index "default" to match what we have in our configuration, or name it whatever makes sense to you. Just make sure to update the environment variables configuration if you change the name.

Open the project's src/routes/movies.ts file and include the following:

import { Router, type IRouter } from "express";
import { config } from "../config";
import { Database } from "../db";

export function moviesRouter(): IRouter {
    const router = Router();

    router.get("/search", async (req, res, next) => {
        try {
            const searchQuery = String(req.query.q ?? "").trim();
            if (!searchQuery) {
                throw new Error(
                    'Query parameter "q" is required and cannot be empty'
                );
            }

            const pipeline = [
                {
                    $search: {
                        index: config.searchIndexName,
                        text: {
                            query: searchQuery,
                            path: ["title", "plot"],
                        },
                    },
                },
                { $limit: 10 },
                {
                    $project: {
                        _id: 1,
                        title: 1,
                        poster: 1,
                        year: 1,
                    },
                },
            ];

            const database = Database.instance.db;
            const moviesCollection = database.collection(
                config.moviesCollection
            );
            const results = await moviesCollection
                .aggregate(pipeline)
                .toArray();

            res.json({ results });
        } catch (err) {
            next(err);
        }
    });

    return router;
}

The router above has a single GET endpoint that we'll use for searching. It expects a q query parameter to exist in our requests, where this parameter represents a search query.

With a MongoDB aggregation pipeline, we can use $search as our first stage, specifying the index name, the search query, and the fields we wish to search against. For the sake of this example, the second pipeline stage will limit our results to no more than 10 results. Finally, we end our pipeline with a $project stage that allows us to define what fields are actually important to us for returning to the client. In this example, we plan to return the _id, title, poster, and year fields for every document matched.

We are only using MongoDB Search for finding movies we want to add to our watchlist. We need to create a route and a series of endpoints for managing our watchlist next.

Building a Movie Watchlist with Ratings

The watchlist will make up most of our application logic for this tutorial. We want to accomplish the following:

Add movies based on id to the watchlist, with certain fields, as long as the movie doesn't already exist in the list.
Add ratings to movies in the watch list.
Soft-delete (deactivate) movies in the watchlist, but not actually removing the document from the collection.
Remove ratings from movies in the watchlist.

In the src/routes/watchlist.ts file, let's use the following as a starting point:

import { Router, type IRouter } from "express";
import { ObjectId } from "mongodb";
import { config } from "../config";
import { Database } from "../db";
import type { WatchlistDoc } from "../types/watchlist";

export function watchlistRouter(): IRouter {
    const router = Router();

    return router;
}

There aren't any endpoints for now, but we're going to add and make sense of each one at a time.

Our most complex endpoint will be the POST endpoint responsible for adding movies to the watchlist. You can find this endpoint below.

router.post("/", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(String(req.body?.movieId));
        const watchlistCollection = database.collection(config.watchlistCollection);
        const moviesCollection = database.collection(config.moviesCollection);

        const existingWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });

        if (existingWatchlistDoc) {
            const activeEntry = existingWatchlistDoc as WatchlistDoc;
            if (activeEntry.active) {
                res.status(200).json({
                    document: activeEntry,
                    alreadyActive: true,
                    reactivated: false,
                });
                return;
            }

            const now = new Date();
            await watchlistCollection.updateOne(
                { movieId },
                { $set: { active: true, updatedAt: now } }
            );
            const reactivated: WatchlistDoc = {
                ...activeEntry,
                active: true,
                updatedAt: now,
            };
            res.status(200).json({
                document: reactivated,
                alreadyActive: false,
                reactivated: true,
            });
            return;
        }

        const movieFromCatalog = await moviesCollection.findOne(
            { _id: movieId },
            { projection: { title: 1, poster: 1 } }
        );
        if (!movieFromCatalog) {
            throw new Error("Movie not found in sample_mflix");
        }
        const { title, poster } = movieFromCatalog;
        const now = new Date();
        const newWatchlistDocument = {
            movieId,
            title,
            poster,
            active: true,
            createdAt: now,
            updatedAt: now,
        };
        const insertResult = await watchlistCollection.insertOne(
            newWatchlistDocument
        );
        const document: WatchlistDoc = {
            _id: insertResult.insertedId,
            ...newWatchlistDocument,
        };
        res.status(201).json({
            document,
            alreadyActive: false,
            reactivated: false,
        });
    } catch (err) {
        next(err);
    }
});

To break down each stage of the above endpoint, we're going to start with the initial query:

const existingWatchlistDoc = await watchlistCollection.findOne({
    movieId,
});

The movieId should have been passed in with the request payload. It should represent a valid movie id from the movies collection. After executing the query, we check to see if results came back, and if they did, it means this particular movie is already in the watchlist. If the movie is already in the watchlist and it is in the active state, we can just return that information now. If it is in the watchlist, but inactive, we want to reactivate it and update the timestamp before returning it.

Finally, we have this code within the endpoint:

const movieFromCatalog = await moviesCollection.findOne(
    { _id: movieId },
    { projection: { title: 1, poster: 1 } }
);
if (!movieFromCatalog) {
    throw new Error("Movie not found in sample_mflix");
}
const { title, poster } = movieFromCatalog;
const now = new Date();
const newWatchlistDocument = {
    movieId,
    title,
    poster,
    active: true,
    createdAt: now,
    updatedAt: now,
};
const insertResult = await watchlistCollection.insertOne(
    newWatchlistDocument
);
const document: WatchlistDoc = {
    _id: insertResult.insertedId,
    ...newWatchlistDocument,
};
res.status(201).json({
    document,
    alreadyActive: false,
    reactivated: false,
});

If the movie is not in the watchlist already, we first need to query the movies collection for information about the movie based on the provided movieId value. With the title and poster information, we can construct a new watchlist document and then insert it into our collection.

For this particular project, we won't worry about keeping our movie posters up to date. We're just going to assume they never change.

The next endpoint we'll look at is a GET endpoint for listing all the movies in our watchlist:

router.get("/", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const raw = req.query.active;
        const activeOnly =
            raw === undefined
                ? true
                : !["false", "0"].includes(String(raw).toLowerCase());

        const watchlistCollection = database.collection(
            config.watchlistCollection
        );
        const items = await watchlistCollection
            .find({ active: activeOnly })
            .sort({ updatedAt: -1 })
            .toArray();
        res.json({ active: activeOnly, items });
    } catch (err) {
        next(err);
    }
});

This particular endpoint will accept an active query parameter, allowing the client to request watchlist items that are either active or have been soft-deleted (inactive).

When we run our MongoDB query, we are sorting by the updatedAt timestamp so the newest additions to the watchlist appear first. We aren't doing a $project here because our documents are small, but you could easily define which fields are important to you for returning.

The next endpoint is our PATCH endpoint, which is responsible for assigning a user rating to a movie in the watchlist:

router.patch("/:movieId/rating", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(req.params.movieId);
        const stars = Number(req.body?.stars);
        if (!Number.isInteger(stars) || stars < 1 || stars > 5) {
            throw new Error("Rating must be an integer from 1 to 5");
        }

        const watchlistCollection = database.collection(
            config.watchlistCollection
        );
        const now = new Date();
        const updateResult = await watchlistCollection.updateOne(
            { movieId },
            { $set: { rating: stars, updatedAt: now } }
        );
        if (updateResult.matchedCount === 0) {
            throw new Error(
                "Movie is not on the watchlist; only watchlist movies can be rated"
            );
        }
        const updatedWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });
        if (!updatedWatchlistDoc) {
            throw new Error(
                "Failed to load watchlist item after rating update"
            );
        }
        res.json({ document: updatedWatchlistDoc });
    } catch (err) {
        next(err);
    }
});

Here, we accept a movie id as a route parameter and a star rating within the request payload.

Using the updateOne operator, we can attempt to update the document. If the matchCount after the operation is zero, it means that there wasn't a document to update. This is likely because the movie id was incorrect, or it wasn't in our watchlist. When adding a rating, we could just respond with a simple message stating the update was complete, but in this example, we are querying for the watchlist item and returning what appears in the document.

The two final endpoints for our watchlist include removing a rating and soft-deleting a watchlist item.

Starting with the rating removal, we have the following DELETE request:

router.delete("/:movieId/rating", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(req.params.movieId);
        const watchlistCollection = database.collection(config.watchlistCollection);
        const now = new Date();
        const updateResult = await watchlistCollection.updateOne(
            { movieId },
            { $unset: { rating: "" }, $set: { updatedAt: now } }
        );
        if (updateResult.matchedCount === 0) {
            throw new Error(
                "Movie is not on the watchlist; only watchlist movies can be rated"
            );
        }
        const updatedWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });
        if (!updatedWatchlistDoc) {
            throw new Error(
                "Failed to load watchlist item after rating removal"
            );
        }
        res.json({ document: updatedWatchlistDoc });
    } catch (err) {
        next(err);
    }
});

The above code will take a movieId and make use of the $unset operator within an updateOne operation. This will remove the rating. Just like with the previous endpoint, we could have opted to return a message about the operation’s success, but here we are querying for the full document and returning it to the client instead.

The final endpoint in our route looks like the following:

router.delete("/:movieId", async (req, res, next) => {
    try {
        const database = Database.instance.db;
        const movieId = new ObjectId(req.params.movieId);
        const watchlistCollection = database.collection(config.watchlistCollection);
        const existingWatchlistDoc = await watchlistCollection.findOne({
            movieId,
        });
        if (!existingWatchlistDoc) {
            throw new Error("Watchlist item not found");
        }
        if (!(existingWatchlistDoc as WatchlistDoc).active) {
            res.status(204).send();
            return;
        }
        const now = new Date();
        await watchlistCollection.updateOne(
            { movieId },
            { $set: { active: false, updatedAt: now } }
        );
        res.status(204).send();
    } catch (err) {
        next(err);
    }
});

Similar logic applies here. If the watchlist item exists and it is not already inactive, we do an updateOne operation and $set the active status to inactive. In this example, we are just doing an empty return with an appropriate status to show it was deactivated.

Configuring Express Framework to Connect to MongoDB and Serve the Application

At this point, our two routes are in place, and the appropriate indexes have been configured. We need to be able to serve our application so clients of any shape can interact with it.

Open the project's src/app.ts file and include the following:

import express, { type ErrorRequestHandler } from "express";
import { moviesRouter } from "./routes/movies";
import { watchlistRouter } from "./routes/watchlist";
import { config } from "./config";
import { Database } from "./db";

async function main() {
    await Database.connect();

    const app = express();
    app.use(express.json());

    app.get("/health", (_req, res) => {
        res.json({ ok: true });
    });

    app.use("/movies", moviesRouter());
    app.use("/watchlist", watchlistRouter());

    const onError: ErrorRequestHandler = (err, _req, res, next) => {
        if (res.headersSent) {
            next(err);
            return;
        }
        console.error(err);
        res.status(500).json({
            error: err instanceof Error ? err.message : String(err),
        });
    };
    app.use(onError);

    const server = app.listen(config.port, () => {
        console.log(`Listening on http://localhost:${config.port}`);
    });

    const shutdown = async () => {
        server.close();
        await Database.instance.close();
        process.exit(0);
    };
    process.on("SIGINT", () => void shutdown());
    process.on("SIGTERM", () => void shutdown());
}

main().catch((err) => {
    console.error(err);
    process.exit(1);
});

In the above code, we are configuring Express Framework to accept JSON payloads and to recognize our two route files. Skipping beyond the error handling and listening of our backend, you'll find the shutdown logic.

When a shutdown is triggered, we close the server from listening for connections, and we also close the connection to our MongoDB database. This is good practice for making sure everything is gracefully terminated.

Running the API to Test the Endpoints

For this particular tutorial, we are not exploring a frontend. However, we'll run and test our API endpoints so you can get an idea of what it looks like.

From a command line, execute the following to run the application in development mode:

tsx watch src/app.ts

With the application running, you can first test the health check endpoint:

curl -s http://localhost:3000/health

If that looks good, try searching for a movie:

curl -s "http://localhost:3000/movies/search?q=Terminator"

Replace the q with whatever value you'd like to search for. Make note of one of the movie id values in the response because you'll need it for populating your watchlist with the following command:

curl -s -X POST http://localhost:3000/watchlist \
  -H "Content-Type: application/json" \
  -d '{"movieId":"<PASTE_OBJECT_ID_HERE>"}'

Go ahead and give the other endpoints a test!

It's worth mentioning that what we built is good for a demo, but it will need some work for production. We didn't do much data validation, our error handling needs work, and we have no cross-origin resource sharing (CORS) policy in place to define who is allowed to access our application. However, it should get you started when working with MongoDB in unique ways.

Conclusion

You just saw how to use various features of MongoDB, such as MongoDB Search and CRUD, to build a movie watchlist application with ratings. We made use of MongoDB Search for full-text search functionality, and we used CRUD to process documents in our watchlist. We also saw that it is alright to duplicate data between the movies collection and the mflix_watchlist collection to boost performance and avoid JOIN operations.

To take this project to the next level, try adding a frontend or a few more watchlist features. Maybe you can include MongoDB Vector Search to come up with movie recommendations based on your watchlist items and ratings.

Want to try this project for yourself? Clone the repository on GitHub!

Migrating Your Content Management System (CMS) Assets With MongoDB and Node.js

Nic Raboy — Thu, 01 May 2025 14:40:00 +0000

Content platforms evolve as business strategies shift. At MongoDB, we embraced external publishing platforms like Dev.to, Medium, The Polyglot Developer, etc. to better engage developer communities, requiring us to redistribute content while maintaining our existing CMS data in MongoDB.

To support our multi-platform publishing strategy, we created a system to publish content between our MongoDB CMS and external platforms. As a result, we needed to migrate the content we had in our CMS to its new home. The migration process included exporting the written content stored in MongoDB and downloading a copy of the media assets that were stored on third-party servers.

In this tutorial, we'll explore the export process to get the job done with as little friction as possible.

The requirements

If you plan to reproduce this project or follow along, you'll want to make sure you have everything available and properly configured. In particular, we want the following:

A MongoDB database, either locally hosted or on MongoDB Atlas
Node.js v22+

I am using Node.js v22.14.0, but earlier and later versions might work as well. You may have to refer to the API documentation to be sure. For MongoDB, our CMS was hosted on MongoDB Atlas, but most variants should work fine for the purpose of the script. Just make sure the MongoDB instance is properly configured to allow for connections from your script.

It is important to note that our CMS was very custom and it's not likely that your setup will be the same. However, the concepts used here should put you on the right track.

Creating a Node.js project with the proper dependencies

To kick things off, we're going to need a project created with a few environment variables set. Pick a working directory with your terminal and execute the following:

npm init -y
npm install mongodb axios dotenv --save

The above commands will initialize a new Node.js project and download our three project dependencies. The MongoDB driver will allow us to query our database for our content, Axios will allow us to download our media assets through HTTP requests, and Dotenv will allow us to easily make use of environment variables within our project.

Next, execute the following commands:

touch main.js
touch .env

The touch command just creates files. If you’re on Windows, go ahead and create those files however it makes sense.

This is just a small script so all our code can go in the main.js file. Before we get there, open the .env file and add the following:

MONGODB_ATLAS_URI=
MONGODB_DATABASE=
MONGODB_COLLECTION=

EXPORT_URLS=./export_list.txt
OUTPUT_DIR=./output/

For safety, if you plan to track this project in Git, make sure you add the .env file to your .gitignore file.

Your CMS project probably won't be the same as mine and that is fine. It is just to give you an idea of how everything is working together. Feel free to follow along with the MongoDB side of things, or experiment with your own connection information. From a data perspective, everything we do will be read-only.

In my situation, I had a list of URLs that needed to be migrated out of MongoDB. This was not the entire content portfolio. The export_list.txt file in this example contains a list of URL slugs, one slug per line. For example, it might look something like this:

/products/mongodb/tensorflow-mongodb-charts/
/products/mongodb/trader-joes-llamaindex-vector-search/

The output directory is where we plan to save our content, which is Markdown in our system, along with any media assets.

Let's add some boilerplate code to our main.js file before we get into the specifics:

const { MongoClient } = require("mongodb");
const fs = require("node:fs/promises");
const { createWriteStream } = require("fs");
const axios = require("axios");

require("dotenv").config();

const MONGODB_URI = process.env.MONGODB_ATLAS_URI;
const MONGODB_DATABASE = process.env.MONGODB_DATABASE;
const MONGODB_COLLECTION = process.env.MONGODB_COLLECTION;
const EXPORT_URLS = process.env.EXPORT_URLS;
const OUTPUT_DIR = process.env.OUTPUT_DIR;

const mongoClient = new MongoClient(MONGODB_URI);
var database, collection;

(async() => {
    try {
        await mongoClient.connect();
        database = await mongoClient.db(MONGODB_DATABASE);
        collection = await database.collection(MONGODB_COLLECTION);
        // Code here soon ...
        await mongoClient.close();
    } catch (e) {
        console.error("ERROR: ", e.message);
    }
})();

In the above code, we've extracted our environment variables and established a connection to our database. What comes next will be two parts: downloading the Markdown, and downloading the media.

Downloading the content from MongoDB

Like previously mentioned, the CMS data we had stored in MongoDB included the raw Markdown of our tutorial content, but it also had various metadata information like author, tags, URL slug, etc.

In the main.js file, add the following function:

async function loadUrlFileData(removeTrailingSlash = false) {
    let data = await fs.readFile(EXPORT_URLS, { encoding: "utf8" });
    let slugs = data.split('\n');
    if(removeTrailingSlash == true) {
        slugs = slugs.map(slug => slug.replace(/\/$/, ""));
    }
    return slugs;
}

Remember, in this example, the URLs we want to download were specified in a file. Your use-case might differ, but it shouldn't be difficult to pivot.

With the file scenario, the loadUrlFileData function will take each line of the file and add it to an array. In this example, we can specify if we want to remove the trailing slash from any of our URL slugs.

With a file loader in place, we can revisit our main asynchronous function:

(async() => {
    try {
        await mongoClient.connect();
        database = await mongoClient.db(MONGODB_DATABASE);
        collection = await database.collection(MONGODB_COLLECTION);
        let slugs = await loadUrlFileData(true);
        let cursor = collection.aggregate([
            {
                "$match": {
                    "calculated_slug": {
                        "$in": slugs
                    }
                }
            },
            {
                "$project": {
                    title: "$name",
                    description: 1,
                    slug: "$calculated_slug",
                    authors: 1,
                    tags: 1,
                    content: 1
                }
            },
        ]);
        let matchedContent = await cursor.toArray();
        await mongoClient.close();
    } catch (e) {
        console.error("ERROR: ", e.message);
    }
})();

Notice the revisions we've made in the above code.

First, we load our URL slugs into an array. Then, we make use of an aggregation pipeline in MongoDB to find content matches based on each of the slugs. The calculated_slug, name, description, authors, tags, and content are all fields within my data so no need to feel like you missed a step. For all the matches in the first pipeline stage, we can specify the fields that are valuable to us in the $project stage.

Depending on the scope of your results, you can use the toArray() function or plan to iterate over the cursor. My data was less than 200 content entries, so the toArray() was fine.

So at this point, we queried for the data we wanted, but we haven't actually done anything with the results. We need to save the results to a file on our filesystem.

Let's create two more functions in our main.js file:

function saveMarkdownToFile(slug, markdown) {
    let parts = slug.split("/").filter(part => part != "");
    let entryName = parts[parts.length - 1];
    return fs.mkdir(`${OUTPUT_DIR}${entryName}/assets`, { recursive: true })
        .then(() => fs.writeFile(`${OUTPUT_DIR}${entryName}/${entryName}.md`, markdown));
}

function saveMetadataToFile(slug, data) {
    let parts = slug.split("/").filter(part => part != "");
    let entryName = parts[parts.length - 1];
    return fs.mkdir(`${OUTPUT_DIR}${entryName}/assets`, { recursive: true })
        .then(() => fs.writeFile(`${OUTPUT_DIR}${entryName}/meta.json`, data));
}

The two functions above, more or less, do the same thing. You provide the slug, which will be used for file naming purposes, and the data. In this case, the data is either going to be the raw Markdown or the metadata for the entry. We want to save both, but as two separate files.

Looking back at our main logic-driving asynchronous function, we can add the following:

// Other code above ...
let matchedContent = await cursor.toArray();
for (const match of matchedContent) {
    await saveMarkdownToFile(match.slug, match.content);
    delete match.content;
    await saveMetadataToFile(match.slug, JSON.stringify(match, undefined, 4));
}

The code above is looping through each CMS entry matched by our aggregation pipeline. For every entry, we save the Markdown and the metadata. We make sure to remove the content from the metadata prior to saving it because it isn't necessary to store our massive amounts of Markdown in that file as well.

At this point, we've exported our Markdown entries from MongoDB. The problem is that none of our media assets were exported. We want to export them in case they are ever removed from the original source.

Saving the media assets hosted on a third-party platform

In our old CMS, the media assets, such as images, were not stored separately within our database. The assets themselves existed on S3 and similar, and the references to those assets were stored directly within the Markdown. This means we had to do some basic web scraping.

Within the main.js file, add the following function:

function extractAssetUrls(text) {
    let assetUrlRegex = /(https?:\/\/.*\.(?:png|jpg|jpeg|gif|webp|pdf))/gi;
    let assetUrls = text.match(assetUrlRegex) || [];
    assetUrls = assetUrls.map(asset => {
        let urlObject = new URL(asset);
        let urlParams = new URLSearchParams(urlObject.search);
        if(urlParams.has("url")) {
            asset = urlParams.get("url");
        }
        return asset;
    });
    return assetUrls;
}

The above function will take text—in this case, our entire Markdown content—and look for a pre-defined list of assets with a regular expression. Some of those assets had funky URL problems, so for every asset match, we had to further parse it with a URL object in JavaScript.

The assets discovered in the extractAssetUrls function get added to an array and returned. To be clear, only the URLs are added to this array. We still need to download them.

Include the following function for downloading:

function downloadAsset(assetUrl, slug) {
    let urlParts = assetUrl.split("/");
    let slugParts = slug.split("/").filter(part => part != "");
    let fileName = urlParts[urlParts.length - 1];
    let filePath = `${OUTPUT_DIR}${slugParts[slugParts.length - 1]}/assets/`;
    return axios({
        url: assetUrl,
        method: "GET",
        responseType: "stream"
    }).then(response => 
            new Promise((resolve, reject) => {
                response.data
                    .pipe(createWriteStream(filePath + fileName))
                    .on("error", reject)
                    .once("close", () => resolve(filePath + fileName));
            })
    ).catch(error => {
        if(error.response && error.response.status >= 400 && error.response.status <= 500) {
            console.warn(`Skipping asset due to client error (status: ${error.response.status}): ${assetUrl}`);
        }
    })  
}

The above downloadAsset function will take a URL and a slug, the slug being used just for naming purposes when saving. The asset is downloaded with an HTTP request and saved directly to the file system.

To make these functions work, let’s revisit our main asynchronous function:

// Previous code up here ...
let matchedContent = await cursor.toArray();
for (const match of matchedContent) {
    await saveMarkdownToFile(match.slug, match.content);
    let assetUrls = extractAssetUrls(match.content);
    await Promise.all(assetUrls.map(asset => downloadAsset(asset, match.slug + "/")));
    delete match.content;
    await saveMetadataToFile(match.slug, JSON.stringify(match, undefined, 4));
}

Notice that before we save our metadata to a file, we are extracting the asset URLs from the Markdown content and then downloading them.

For clarity, the Markdown and each asset will live in a separate directory within our output path. The separate directory in our scenario is the slug name.

Seeing the migration in action

If you made it to this point and you either want to try to run the code or see all the code together, no problem. The main.js file should look like the following:

const { MongoClient } = require("mongodb");
const fs = require("node:fs/promises");
const { createWriteStream } = require("fs");
const axios = require("axios");

require("dotenv").config();

const MONGODB_URI = process.env.MONGODB_ATLAS_URI;
const MONGODB_DATABASE = process.env.MONGODB_DATABASE;
const MONGODB_COLLECTION = process.env.MONGODB_COLLECTION;
const EXPORT_URLS = process.env.EXPORT_URLS;
const OUTPUT_DIR = process.env.OUTPUT_DIR;

const mongoClient = new MongoClient(MONGODB_URI);
var database, collection;

(async() => {
    try {
        await mongoClient.connect();
        database = await mongoClient.db(MONGODB_DATABASE);
        collection = await database.collection(MONGODB_COLLECTION);
        let slugs = await loadUrlFileData(true);
        let cursor = collection.aggregate([
            {
                "$match": {
                    "calculated_slug": {
                        "$in": slugs
                    }
                }
            },
            {
                "$project": {
                    title: "$name",
                    description: 1,
                    slug: "$calculated_slug",
                    authors: 1,
                    tags: 1,
                    content: 1
                }
            },
        ]);
        let matchedContent = await cursor.toArray();
        for (const match of matchedContent) {
            await saveMarkdownToFile(match.slug, match.content);
            let assetUrls = extractAssetUrls(match.content);
            await Promise.all(assetUrls.map(asset => downloadAsset(asset, match.slug + "/")));
            delete match.content;
            await saveMetadataToFile(match.slug, JSON.stringify(match, undefined, 4));
        }
        await mongoClient.close();
    } catch (e) {
        console.error("ERROR: ", e.message);
    }
})();

// Load the URLs to be used in the query from a file
// Optionally remove the trailing slash from the URLs if necessary
async function loadUrlFileData(removeTrailingSlash = false) {
    let data = await fs.readFile(EXPORT_URLS, { encoding: "utf8" });
    let slugs = data.split('\n');
    if(removeTrailingSlash == true) {
        slugs = slugs.map(slug => slug.replace(/\/$/, ""));
    }
    return slugs;
}

// Save the Markdown content at the $OUTPUT/$SLUG/$SLUG.md path
function saveMarkdownToFile(slug, markdown) {
    let parts = slug.split("/").filter(part => part != "");
    let entryName = parts[parts.length - 1];
    return fs.mkdir(`${OUTPUT_DIR}${entryName}/assets`, { recursive: true })
        .then(() => fs.writeFile(`${OUTPUT_DIR}${entryName}/${entryName}.md`, markdown));
}

// Save the metadata at the $OUTPUT/$SLUG/meta.json path
function saveMetadataToFile(slug, data) {
    let parts = slug.split("/").filter(part => part != "");
    let entryName = parts[parts.length - 1];
    return fs.mkdir(`${OUTPUT_DIR}${entryName}/assets`, { recursive: true })
        .then(() => fs.writeFile(`${OUTPUT_DIR}${entryName}/meta.json`, data));
}

// Extract any RegEx match for an asset in a file and return it
// Note, some assets were parameterized weird using URL parameters
function extractAssetUrls(text) {
    let assetUrlRegex = /(https?:\/\/.*\.(?:png|jpg|jpeg|gif|webp|pdf))/gi;
    let assetUrls = text.match(assetUrlRegex) || [];
    assetUrls = assetUrls.map(asset => {
        let urlObject = new URL(asset);
        let urlParams = new URLSearchParams(urlObject.search);
        if(urlParams.has("url")) {
            asset = urlParams.get("url");
        }
        return asset;
    });
    return assetUrls;
}

// Download any asset file and store it in the $OUTPUT/$SLUG/assets directory
function downloadAsset(assetUrl, slug) {
    let urlParts = assetUrl.split("/");
    let slugParts = slug.split("/").filter(part => part != "");
    let fileName = urlParts[urlParts.length - 1];
    let filePath = `${OUTPUT_DIR}${slugParts[slugParts.length - 1]}/assets/`;
    return axios({
        url: assetUrl,
        method: "GET",
        responseType: "stream"
    }).then(response => 
            new Promise((resolve, reject) => {
                response.data
                    .pipe(createWriteStream(filePath + fileName))
                    .on("error", reject)
                    .once("close", () => resolve(filePath + fileName));
            })
    ).catch(error => {
        if(error.response && error.response.status >= 400 && error.response.status <= 500) {
            console.warn(`Skipping asset due to client error (status: ${error.response.status}): ${assetUrl}`);
        }
    })  
}

Remember, your CMS probably doesn't look like the CMS we were using at MongoDB. Make the changes where you need to, or feel free to pull bits and pieces from the code to cover your use-case.

With the code in place, you can run the project with the following:

node main.js

Depending on how many URLs you are exporting and how many image assets there are, it should take a minute to run. Images that cannot be accessed or are not found will be skipped and reported.

Conclusion

You just saw how to pull data out of MongoDB and save it to a file. There are countless ways of doing this, but this code worked for our needs at MongoDB. To be clear, our needs were to switch from publishing on our internal platform and instead start publishing on external platforms. This is why we needed to download our Markdown data and each asset associated with it.

Send Messages with Nexmo through Golang and AWS Lambda

Nic Raboy — Wed, 04 Dec 2019 17:24:34 +0000

I recently saw a tutorial by Tom Morris on the Nexmo blog titled, Sending SMS from Python with Google Cloud Functions, and thought it was an interesting example and use of SMS messaging, even though Python and Google Cloud Functions are not my normal.

After reading the blog, I thought it would be interesting to expand it to AWS Lambda as the functions provider and switch the language to Golang, something that I'm very passionate about.

In this tutorial, we're going to create an API using AWS Lambda and AWS API Gateway that sends an SMS message with the Nexmo Messaging API, but with the Go programming language.

The Requirements

There are a few requirements that must be satisfied prior to working through this tutorial:

You need to have Go installed and be at least somewhat familiar with the language.
You need to have a Nexmo developer account.
You need to have an Amazon Web Services account.

Since Go is a compiled language, we need to be able to develop and build locally. Our builds will be uploaded to AWS, hence the need for an AWS account. These applications will make use of the Nexmo Messaging API.

Building a Messaging Function with Go and the AWS Lambda SDK

When it comes to the development of the Go application, there isn't too much going on. In terms of the flow of events, they'll look something like this:

The user will make a request to the function.
The function will send an HTTP request to the Nexmo Messages API.
The function will respond with the error or message id from the API.

While the core logic will be around an HTTP request, there will be a bit of setup in regards to the request and response data model.

Within the $GOPATH, create a new project with a main.go file that contains the following:

package main

var NEXMO_API_KEY string = "API_KEY_HERE"
var NEXMO_API_SECRET string = "API_SECRET_HERE"

func main() {}

Both the API key and the API secret can be obtained from within the Nexmo developer portal.

Since this is going to be an AWS Lambda project, the Go SDK for Lambda must be installed. From the command line, execute the following:

go get github.com/aws/aws-lambda-go/lambda

With the SDK installed through the Go package manager, it can be used within the project. To use it within the project, we can make the following changes:

package main

import "github.com/aws/aws-lambda-go/lambda"

var NEXMO_API_KEY string = "API_KEY_HERE"
var NEXMO_API_SECRET string = "API_SECRET_HERE"

func Handler() {}

func main() {
    lambda.Start(Handler)
}

As you can probably suspect, all of our messaging logic will appear in the Handler function since that is what will be executed when the client makes a Lambda request.

Before we get into the logic, we need to define some data models for the various requests and responses. Go requires that we explicitly define our data models through struct variables so that JSON can be properly marshaled between requests.

Add the following to the main.go file:

type NexmoMessageContent struct {
    Type string `json:"type"`
    Text string `json:"text"`
}

type NexmoMessage struct {
    Content NexmoMessageContent `json:"content"`
}

type NexmoMessageService struct {
    Type   string `json:"type"`
    Number string `json:"number"`
}

type NexmoMessagesRequest struct {
    From    NexmoMessageService `json:"from"`
    To      NexmoMessageService `json:"to"`
    Message NexmoMessage        `json:"message"`
}

type NexmoMessagesResponse struct {
    Id string `json:"message_uuid"`
}

If you've ever used the Nexmo Messages API, you'll be familiar with the JSON that is expected to be sent with every request to the API. The above data structures represent the data that will be sent. With the JSON annotations in place, the JSON would look something like this:

{
    "from": { "type": "sms", "number": "SENDER_NUMBER_HERE" },
    "to": { "type": "sms", "number": "RECIPIENT_NUMBER_HERE" },
    "message": {
        "content": {
            "type": "text",
            "text": "Hello World"
        }
    }
}

When sending a request to the Nexmo Messages API, a response with a UUID will be returned if it was successful. We've written a NexmoMessagesResponse struct which holds this data and plan to send it back to the client.

The only thing we're missing now is the request that the client will be making to the Lambda function. If our example is trying to mimic the Google Cloud Functions example, we are expecting a recipient phone number and the mobile platform they are using.

With this information in mind, we can create a data structure that looks like the following:

type LambdaRequest struct {
    Platform  string `json:"platform"`
    Recipient string `json:"recipient"`
}

With all the data models crafted, we can focus the rest of our effort on making the messaging request.

Making HTTP requests in Go isn't a difficult process. I explained a few of the approaches in-depth, in a previous tutorial that I had written titled, Consume RESTful API Endpoints within a Golang Application, although my personal choice would be to use a Go http.Client structure. In this particular example, we're going to need to make a POST request as outlined in the Nexmo documentation.

Let's start by creating a SendMessage function which will essentially build a request like that of Nexmo's many cURL examples:

func SendMessage(body map[string]string) (NexmoMessagesResponse, error) {
    nexmoMessagesRequest := &NexmoMessagesRequest{
        From: NexmoMessageService{
            Type:   "sms",
            Number: body["from"],
        },
        To: NexmoMessageService{
            Type:   "sms",
            Number: body["to"],
        },
        Message: NexmoMessage{
            Content: NexmoMessageContent{
                Type: "text",
                Text: body["message"],
            },
        },
    }
    bodyData, _ := json.Marshal(nexmoMessagesRequest)
    request, _ := http.NewRequest("POST", "https://api.nexmo.com/v0.1/messages", bytes.NewBuffer(bodyData))
    request.Header.Set("Content-Type", "application/json")
    request.Header.Set("Accept", "application/json")
    request.Header.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(NEXMO_API_KEY+":"+NEXMO_API_SECRET)))
    client := &http.Client{}
    response, err := client.Do(request)
    if err != nil {
        return NexmoMessagesResponse{}, err
    } else {
        data, _ := ioutil.ReadAll(response.Body)
        var result NexmoMessagesResponse
        json.Unmarshal(data, &result)
        return result, nil
    }
}

The above function will take a map[string]string and return a NexmoMessagesResponse or an error. The first part of the function takes data from the map and constructs an object that will be marshaled into JSON. When it comes to constructing the HTTP request, the object is used as the body and various headers are defined, one of which is an authorization header.

With Nexmo a JSON Web Token (JWT) can be used or the API key and API secret. In this example, the API key and API secret will be used.

The SendMessage function is most of the work done.

Since we're using the AWS Lambda SDK for Go and we have a Handler function, the next step is to define what's in the Handler function. It should look something like this:

func Handler(request LambdaRequest) (NexmoMessagesResponse, error) {
    return SendMessage(
        map[string]string{
            "from":    "15404161937",
            "to":      request.Recipient,
            "message": request.Platform,
        },
    )
}

Remember, the client accessing our API should be providing the platform and recipient information. Remember, they need to define what mobile platform they are using and the phone number the SMS message should be sent to.

If you can believe it, the AWS Lambda function is complete. When triggered with the appropriate recipient and platform information, the SMS message will be sent and the id of the message will be returned.

Next, we need to get the function into AWS Lambda and make the function accessible as part of standard HTTP requests, typical in an API.

Configuring AWS Lambda and AWS API Gateway as a Scalable Web API

Before we even get into the AWS portal, we should probably build our Go project. From the command line, while in the project path, execute the following:

GOOS=linux go build

AWS requires our application to be compatible with Linux. Lucky for us, Go ships with all the correct tool-chains to cross-compile. For more information on cross-compiling with Go, check out my previous tutorial on the subject.

As part of a Lambda requirement, the binary must be zipped. Take note of the file name and add it to a ZIP archive at the root of the archive. The filename of the binary represents the handler’s name.

Next, jump into the AWS Developer Console and choose to create a new Lambda function. The name isn't too important, just make sure it is using the Go runtime.

Inside the dashboard for the particular function, choose to upload the ZIP archive that you have just just created. Remember to supply the correct filename for the handler information.

We’re also going to want to add API Gateway as a trigger for the function.

Enter the API Gateway part of the AWS Developer Console. This is where a public endpoint will be created to point to the function. Within the dashboard for the API Gateway, choose Create Method within the Resources tab.

The method created should be a POST request and it should reference the ARN value of the AWS Lambda function when asked. When finished, don't forget to deploy the API through the same menu as creating a new method.

At this point, the Lambda function should be reachable through the API Gateway that was configured. We can obtain the URL through the Stages tab if a custom domain isn’t configured.

Conclusion

We just saw how to send messages to mobile devices using AWS Lambda and the Go programming language. To sum it up, if a user were to make a request to the API, managed with API Gateway, the request would flow to AWS Lambda, be sent to Nexmo, and then be sent to the user’s device. A use case, as previously mentioned, might be to send the app store link, whether it be Google Play or iTunes, to the user's phone through SMS.

While not explored in this example, we can easily add to this project by creating a front-end for your users to engage with rather than having strictly a RESTful API.