Forem: jashwanth0712

How to connect Convex to RunPod for serverless GPU workloads

jashwanth0712 — Tue, 10 Feb 2026 19:40:57 +0000

So I was building a fun project where users upload video files...

The problem?

Each file was around 50–100 MB, and storing + processing them at full resolution was going to blow up my bandwidth and storage costs.

I needed a way to:

Compress the video instantly after upload
Store both high-res and low-res versions
And keep everything neatly synced with my database

But that wasn't the only issue.

I also had workflows like AI background removal where no affordable API existed, but there were open-source models that needed GPUs.

This is where RunPod comes in.

RunPod lets you deploy GPU models as serverless endpoints.

Meaning:

GPU spins up only when needed
Runs your AI task
Shuts down immediately after
And you pay only for the usage minutes

Now combine that with Convex and suddenly your backend can orchestrate GPU tasks just like normal database operations. This was helpful for a very wide range of use cases that I built, so thought I bring it as a blog so that anyone who is planning to try out a GPU intensive operation can get a kickstart.

For this walkthrough, we'll implement background removal.

Here's the flow: the moment a user uploads a video, a Convex action fires off a GPU job for background removal. That GPU task runs completely in the background — it processes the video, removes the background, and then automatically uploads the final output and saves it straight into the database. All without blocking the backend or making the user wait.

Building a GPU task

Let's build a sample GPU task.

For this walkthrough, I'm using Convex with Next.js and Convex Auth.

I'm also using the following open-source repository for video background removal:

https://github.com/srinivastls/Fg-bg-Separator

Setting up the project

Convex provides a great CLI that lets you choose your frontend framework and authentication method, and it instantly generates all the boilerplate you need.

To start a new project, run:

npm create convex@latest

The first thing I usually do after generating the boilerplate is hand it over to Cursor and ask it to clean out all the default placeholder UI.

Here's the exact prompt I used to transform the starter template into a real app with authentication, an upload interface, and a simple image gallery to view uploaded files.

Update the entire code base of @onupload-compressor to replace the place holder website after login with file uploader, store the uploaded files (can be images, videos, audio) and store them as well as show them as a masonry grid

We then run the application using the command.

npx convex dev

Our file upload website is running well, now lets try to work on adding background removal feature as soon as file is uploaded.

Before continuing, make sure to create this table in your schema, to track the background removal status for each media file.

removeBgJobs: defineTable({
  fileId: v.id("_storage"),
  status: v.optional(v.union(
    v.literal("pending"),
    v.literal("processing"),
    v.literal("completed"),
    v.literal("failed")
  )),
  resultId: v.optional(v.id("_storage"))
})

This table stores the lifecycle of a background-removal job for a media file. It links the original uploaded file (fileId) to its processing state (status) and, once completed, to the processed output (resultId).

Building background removal serverless API

I have created the runpod handler which does three things

step 1 : accepts a URL of the image/video of which the background is to be removed

step 2: process the background removal task

step 3 : upload the background removed video into convex directly from the runpod instance

Let's start with writing required convex functions that invokes the Runpod serverless API.

Internal mutation: updateRemoveBgInternal

This internal mutation updates the status and result of a background-removal job. It's designed to be called only from trusted server-side code (like actions), ensuring controlled updates to job state without exposing this mutation to clients.

Action: startRemoveBgWorker

This action starts the background-removal pipeline. It fetches a public URL for the uploaded video, marks the job as pending, and triggers the Runpod serverless worker with the required inputs. Because it's an action, it can safely call external APIs while still coordinating state changes through Convex mutations.

export const updateRemoveBgInternal = internalMutation({
  args: {
    removeBgJobId: v.id("removeBgJobs"),
    status: v.optional(
      v.union(
        v.literal("pending"),
        v.literal("processing"),
        v.literal("completed"),
        v.literal("failed")
      )
    ),
    resultId: v.optional(v.id("_storage")),
  },
  returns: v.null(),
  handler: async (ctx, args) => {
    const updateData: {
      status?: "pending" | "processing" | "completed" | "failed";
      resultId?: Id<"_storage">;
    } = {};

    if (args.status !== undefined) {
      updateData.status = args.status;
    }

    if (args.resultId !== undefined) {
      updateData.resultId = args.resultId;
    }

    await ctx.db.patch(args.removeBgJobId, updateData);
    return null;
  },
});


export const startRemoveBgWorker = action({
  args: {
    removeBgJobId: v.id("removeBgJobs"),
    videoFileId: v.id("_storage"),
  },
  returns: v.object({
    jobId: v.string(),
  }),
  handler: async (ctx, args) => {
    const videoUrl = await ctx.storage.getUrl(args.videoFileId);
    if (!videoUrl) {
      throw new Error("Video file not found");
    }

    // Mark job as pending
    await ctx.runMutation(updateRemoveBgInternal, {
      removeBgJobId: args.removeBgJobId,
      status: "pending",
    });

    const runpodApiKey = process.env.RUNPOD_API_KEY!;
    const runpodEndpointId = process.env.RUNPOD_REMOVE_BG_ENDPOINT_ID!;

    const response = await fetch(
      `https://api.runpod.ai/v2/${runpodEndpointId}/run`,
      {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `Bearer ${runpodApiKey}`,
        },
        body: JSON.stringify({
          input: {
            remove_bg_job_id: args.removeBgJobId,
            file_url: videoUrl,
          },
        }),
      }
    );

    if (!response.ok) {
      throw new Error(`RunPod API error: ${response.status}`);
    }

    const result = await response.json();
    return { jobId: result.id };
  },
});

Next, we'll write the mutations that the RunPod worker will use externally, to modify the database:

Writing the external mutations

External mutation: workerUpdateRemoveBg

This mutation is used by the RunPod worker to report progress back to Convex. It allows the worker to update the job's processing status and attach the resulting file once background removal is complete. Any UI subscribed to this job automatically reacts to these updates.

Mutation: workerGenerateUploadUrl

This mutation generates a secure, time-limited upload URL for Convex Storage. It allows external workers to upload processed files directly to Convex without needing database access or credentials.

export const workerUpdateRemoveBg = mutation({
  args: {
    removeBgJobId: v.id("removeBgJobs"),
    status: v.optional(
      v.union(
        v.literal("pending"),
        v.literal("processing"),
        v.literal("completed"),
        v.literal("failed")
      )
    ),
    resultId: v.optional(v.id("_storage")),
  },
  returns: v.null(),
  handler: async (ctx, args) => {
    const updateData: {
      status?: "pending" | "processing" | "completed" | "failed";
      resultId?: Id<"_storage">;
    } = {};

    if (args.status !== undefined) {
      updateData.status = args.status;
    }

    if (args.resultId !== undefined) {
      updateData.resultId = args.resultId;
    }

    await ctx.db.patch(args.removeBgJobId, updateData);
    return null;
  },
});

export const workerGenerateUploadUrl = mutation({
  args: {},
  returns: v.string(),
  handler: async (ctx) => {
    return await ctx.storage.generateUploadUrl();
  },
});

Great, our Convex functions are ready. Let's start writing the RunPod handler in Python to utilize these convex functions.

If you're new to RunPod, take a look at this tutorial from Runpod on creating a serverless worker.

Once you've done with setting up the worker, we can start adding our code to handle the Convex function calls.

RunPod handler (Python worker)

This handler implements the actual background-removal workflow.

It downloads the input media, performs background removal, uploads the processed output to Convex Storage, and updates job status at each stage.

By calling Convex mutations from the worker, job progress is reflected in the UI in real time.

Let's start by installing the convex Python package (https://pypi.org/project/convex/), and add it to requirements.txt

pip install convex

import os
import requests
import runpod
from convex import ConvexClient

CONVEX_URL = os.environ["CONVEX_URL"]
client = ConvexClient(CONVEX_URL)

def download_video(url: str, save_path: str):
    response = requests.get(url, timeout=(5, 60), stream=True)
    response.raise_for_status()

    with open(save_path, "wb") as f:
        for chunk in response.iter_content(chunk_size=8192):
            f.write(chunk)

def upload_video_to_convex(upload_url: str, file_path: str) -> str:
    with open(file_path, "rb") as f:
        response = requests.post(
            upload_url,
            headers={"Content-Type": "application/octet-stream"},
            data=f,
        )
        response.raise_for_status()
    return response.json()["storageId"]

def generate_upload_url() -> str:
    return client.mutation("workerGenerateUploadUrl", {})

def update_remove_bg_status(
    remove_bg_job_id: str,
    status: str,
    result_id: str | None = None,
):
    client.mutation(
        "workerUpdateRemoveBg",
        {
            "removeBgJobId": remove_bg_job_id,
            "status": status,
            "resultId": result_id,
        },
    )

def handler(event):
    remove_bg_job_id = event["input"]["remove_bg_job_id"]
    file_url = event["input"]["file_url"]

    input_path = "input.mp4"

    update_remove_bg_status(remove_bg_job_id, "processing")

    download_video(file_url, input_path)

    # Your background removal logic here
    result_path = remove_background(input_path)

    upload_url = generate_upload_url()
    result_id = upload_video_to_convex(upload_url, result_path)

    update_remove_bg_status(remove_bg_job_id, "completed", result_id)

    return {
        "storageId": result_id,
        "status": "completed",
    }

if __name__ == "__main__":
    runpod.serverless.start({"handler": handler})

Let's break down all the functions used in the handler

Initializing the Convex Python client

from convex import ConvexClient

CONVEX_URL = os.environ["CONVEX_URL"]
client = ConvexClient(CONVEX_URL)

The convex Python package gives us a ConvexClient that connects directly to your Convex deployment. You pass it your deployment URL (stored as an environment variable on RunPod), and from that point on, you can call any Convex mutation or query straight from Python — no REST wrappers, no custom HTTP calls. This single client instance is reused across all the helper functions below.

download_video()

def download_video(url: str, save_path: str):
    response = requests.get(url, timeout=(5, 60), stream=True)
    response.raise_for_status()
    with open(save_path, "wb") as f:
        for chunk in response.iter_content(chunk_size=8192):
            f.write(chunk)

This function takes the public URL that Convex generated for the uploaded file and streams it down to the GPU instance. We use chunked streaming so we don't blow up memory on large video files. Nothing Convex-specific here — it's just downloading the file that Convex Storage is serving.

generate_upload_url()

def generate_upload_url() -> str:
    return client.mutation("workerGenerateUploadUrl", {})

This is where the Convex Python client shines. Instead of building an HTTP request manually, we just call client.mutation() with the name of the mutation we defined earlier in our Convex backend. This calls workerGenerateUploadUrl, which returns a secure, time-limited upload URL from Convex Storage. The worker can then POST the processed file directly to that URL — no database credentials needed, no complex auth flow.

upload_video_to_convex()

def upload_video_to_convex(upload_url: str, file_path: str) -> str:
    with open(file_path, "rb") as f:
        response = requests.post(
            upload_url,
            headers={"Content-Type": "application/octet-stream"},
            data=f,
        )
        response.raise_for_status()
    return response.json()["storageId"]

Once we have the upload URL from generate_upload_url(), we POST the processed video file to it. Convex responds with a storageId — this is the ID we'll use to link the result back to the job in the database. The upload URL handles all the storage plumbing; the worker just sends raw bytes.

update_remove_bg_status()

def update_remove_bg_status(
    remove_bg_job_id: str,
    status: str,
    result_id: str | None = None,
):
    client.mutation(
        "workerUpdateRemoveBg",
        {
            "removeBgJobId": remove_bg_job_id,
            "status": status,
            "resultId": result_id,
        },
    )

This is the callback function that keeps the UI in sync. Every time the worker hits a new stage, it calls client.mutation("workerUpdateRemoveBg", ...) to update the job's status in the Convex database. Because Convex queries are live, the moment this mutation runs, any UI subscribed to that job's data gets the update pushed to it automatically. The worker calls this twice: once to set "processing" when it starts, and once to set "completed" with the resultId when it's done.

handler(event) — Putting it all together

def handler(event):
    remove_bg_job_id = event["input"]["remove_bg_job_id"]
    file_url = event["input"]["file_url"]
    input_path = "input.mp4"

    update_remove_bg_status(remove_bg_job_id, "processing")
    download_video(file_url, input_path)
    result_path = remove_background(input_path)
    upload_url = generate_upload_url()
    result_id = upload_video_to_convex(upload_url, result_path)
    update_remove_bg_status(remove_bg_job_id, "completed", result_id)

    return {"storageId": result_id, "status": "completed"}

The main handler ties everything together. RunPod passes in the event payload (which contains the job ID and file URL that Convex sent). The handler then:

Marks the job as "processing" via Convex mutation
Downloads the video from Convex Storage
Runs the background removal model
Requests an upload URL from Convex
Uploads the processed result back to Convex Storage
Marks the job as "completed" with the new storageId

The key takeaway: the Convex Python client lets the GPU worker participate in the same data flow as your frontend. Mutations called from Python behave exactly like mutations called from your Next.js app — they update the database and trigger live query refreshes for all connected clients.

With this, our background removal workflow implementation is now complete!

Deploying the serverless API on RunPod

Now that our handler code is ready, let's deploy it as a serverless endpoint on RunPod.

Step 1: Create a serverless endpoint

Go to RunPod > Serverless > Import from Github

Select the branch and the location of docker file and then configure the environment variables, most importantly CONVEX_URL

Step 2: Configure the serverless endpoint

Pick your GPU type, set the max/min workers, and configure the idle timeout. The idle timeout controls how quickly workers scale down to zero when there are no jobs.

Step 3: Grab your endpoint ID and API key

Once the endpoint is created, copy the Endpoint ID from the dashboard. You'll also need your RunPod API Key from your account settings.

Add both to your Convex environment variables:

RUNPOD_API_KEY — your RunPod API key
RUNPOD_REMOVE_BG_ENDPOINT_ID — the endpoint ID you just copied

Step 4: Test the endpoint

You can test the endpoint directly from the RunPod dashboard using the "Run" tab, or trigger it from your app by uploading a file.

Once triggered, you should see the job appear in your RunPod logs, and simultaneously see the status update from pending > processing > completed in your Convex dashboard as the worker calls back into your mutations.

With this, our background removal workflow implementation is now complete!

Wrapping up

The pattern we built here — Convex action triggers GPU job, GPU worker calls back into Convex mutations, UI updates in real time — isn't limited to background removal. You can swap in any GPU-intensive task: video compression, style transfer, transcription, object detection, or whatever open-source model fits your use case.

What makes this work well is that the GPU worker isn't a black box. It participates in the same reactive data flow as your frontend. Every status update, every uploaded result, flows through Convex mutations and gets pushed to connected clients instantly. No polling, no webhooks to manage, no separate status-checking infrastructure.

If you're building something that needs heavy compute but doesn't justify a dedicated GPU server, this Convex + RunPod combo is worth trying. Fork the repo, swap in your own model, and you're up and running.

Build in minutes, scale forever.

Convex is the backend platform with everything you need to build your full-stack AI project. Cloud functions, a database, file storage, scheduling, workflow, vector search, and realtime updates fit together seamlessly.

Matrix: Building a real-time RPG game with Convex

jashwanth0712 — Sun, 23 Feb 2025 18:30:00 +0000

Problem I was trying to solve

This year, I moved to a new place where people speak a different language than my mother tongue. Although English can be a common language for communication, I really wanted to learn the local language as a fun experiment.

I found language learning apps too boring and soon realized that the only effective way to learn a new language is by speaking it frequently. Through conversations, we gradually build vocabulary and eventually become fluent. However, this process can be particularly challenging for introverts like me. That’s when I got the idea to create a real-time game where everyone can join a map augmenting real-world situations.

Game idea

Similar to Minecraft, it’s a real-time RPG game where you can talk to people you come close to on the map. This would help users talk or chat with new people or AI NPC bots in the map and, in the process, learn the language.

I wanted a backend infrastructure to store the real-time locations of players and update their coordinates on all devices with the game open. Additionally, whenever two players came close, the chat interface would open quickly to enable communication between them.

Features I wanted:

A real-time way to store the coordinates of players
No need to handle all synchronization issues manually
A free tier without requiring a credit card to start
Good integration with Next.js

Convex for reactive DB

Convex had all the features I wanted for the application, so I proceeded with Convex as the backend and Next.js for the frontend. The templates in Convex made it very easy to get started. I wanted to use Next.js, Convex, and Clerk, and the Informal template was a perfect starting point for me.

Building the Frontend for the game

With a basic UI idea and user flow in mind, I started with v0, using prompts to generate the frontend for the game scene and map selection scene. To my surprise, the outputs from v0 were so good that I had to write very little code for the frontend myself. For those who don’t know v0 is a frontend code generator website that works really well with shadcn , tailwindcss and Nextjs which are the exact techstack that we are currently using.

Building the Backend Logic

ChatGPT + Convex = 🔥. I observed that by providing a sample mutation and query, ChatGPT can generate the entire database plan, schema.ts, and all the mutations and queries required for the backend logic. There are three main tables that i have used

Map Table
Chat Table
Player Table

Schemas Used

Map Table
It’s just a table configuration to store the dimensions and name, but in the future, it can store map tile images, nature, weather, and much more.

I added an index for map_name so that anyone searching for a particular map name can quickly retrieve the map details.

maps: defineTable({
    dimensions: v.object({
      height: v.float64(),
      width: v.float64(),
    }),
    map_name: v.string(),
  }).index("map_name", ["map_name"]),

Player Table
Any player should be present in only one map at a time. To fix this, instead of storing players in the map table, I have added x_coordinate, y_coordinate, and map_id to the player table itself. To make the search faster, I have added an index to the player's email ID

players: defineTable({
    img_url: v.string(),
    player_mail: v.string(),
    present_map_id: v.optional(v.id("maps")),
    x_coordinate: v.float64(),
    y_coordinate: v.float64(),
  })
    .index("player_mail", ["player_mail"])
    .index("present_map_id", ["present_map_id"]),
});

Chat Table

The chat table stored the details of each message and its corresponding sender-receiver pair. Encryption should have been added, but for the current implementation, I have left it as it is.

chat: defineTable({
    messages: v.array(
      v.object({
        message: v.string(),
        receiver: v.id("players"),
        sender: v.id("players"),
        timestamp: v.float64(),
      })
    ),
    receiver: v.id("players"),
    sender: v.id("players"),
  }).index("sender_receiver", ["sender", "receiver"]),

One thing I really like about Convex is that it offers amazing analytics and the ability to manage both production and development environments as easily as changing environment variables.

After the schema, the next most important aspects are the mutations and queries. I personally love the syntax of their mutations and queries. Unlike ORMs, they have a simple, code-like syntax that makes the developer experience truly great.

Whenever two players come close to each other, we enable the chat option between them. If this were any other traditional database, it would have taken a lot of effort to create this real-time functionality. But with Convex, it’s as simple as writing a better query.

Fetching Realtime player proximity

The following code is the query I used to present the chat options for players who are close to the current player:

    export const nearPlayer = query({
  args: {
    player_mail: v.string(),
    map_name: v.string(),
  },
  handler: async ({ db }, { player_mail, map_name }) => {
    const map = await db.query("maps")
      .withIndex("map_name", q => q.eq("map_name", map_name))
      .first();
    if (!map) {
      throw new Error("Map not found");
    }

    const player = await db.query("players")
      .withIndex("player_mail", q => q.eq("player_mail", player_mail))
      .first();
    if (!player) {
      throw new Error("Player not found");
    }

    const playersOnMap = await db.query("players")
      .withIndex("present_map_id", (q) => q.eq("present_map_id", map._id))
      .collect();

    const { x_coordinate, y_coordinate } = player;

    const nearbyPositions = [
      { dx: 0, dy: 0 }, 
      { dx: 1, dy: 0 },  
      { dx: -1, dy: 0 }, 
      { dx: 0, dy: 1 },  
      { dx: 0, dy: -1 }, 
    ];

    const nearbyPlayers = playersOnMap.filter(otherPlayer => {
      if (otherPlayer._id === player._id) return false;

      return nearbyPositions.some(pos => {
        const targetX = x_coordinate + pos.dx;
        const targetY = y_coordinate + pos.dy;
        return otherPlayer.x_coordinate === targetX && otherPlayer.y_coordinate === targetY;
      });
    });

    return nearbyPlayers.map(player => ({
      _id: player._id,
      player_mail: player.player_mail,
      img_url: player.img_url
    }));
  },
});

Writing similar database interactions with real-time features in other databases could have been a daunting task. Thankfully, Convex is here to save us with its intuitive syntax. For any player, I searched for all the players who are on the same map and are just one block away from the player.

Bringing Magic of AI into the application

Now that the basic implementation of the map, movements, and chat is done, we have added AI functionality to convert the message text into the language specified by the user. OpenAI perfectly handles the translation and provides accurate results.

Do you wanna build a RPG game? use this template

Following is the github repo of the Game and this has the entire boiler plate code to maintain maps, skins , movements and authentication. its easy to install it as well, looking forward for more fun games to be built with this template.