Forem: Bobby Alv

Bringing Convex Dashboard into my dev environment

Bobby Alv — Mon, 17 Mar 2025 18:43:13 +0000

Why I built convexpanel.dev and what it means for my workflow

When I first started using convex, I realized how awesome and easy it was to use the dashboard. It offers an amazing overview of logs, the ability to edit table data, and view your app health. However, as a developer who always has an insane amount of browser tabs open, I found myself constantly switching back and forth just to check on things, and with the massive amount of tabs, I always spent more time looking for the dashboard tab than I did looking at it. Even though I could use the Convex CLI to open the dashboard directly, there was still that nagging need for a more streamlined, integrated experience.

The Spark Behind Convex Panel

After countless times of having to hunt down my convex dashboard tab, I thought: Why not build something that brings the dashboard directly into my development environment? I wanted something that allowed me to do numerous things.

View logs without leaving my dev environment
Being able to edit tables directly within my dev environment
View my data analytics / function rates quickly

The desire to reduce wasted time looking for a browser tab (having a cluttered browser doesn’t help) led me to build this awesome tool. A tool designed to integrate the key features I love from the Convex Dashboard directly into my development environment.

The Initial Approach: A Chrome Extension

My first thought was to build this as a Chrome extension, I figured it would be nice to have something accessible directly in my browser. The idea was simple: have a side panel that could display logs, table data, and function analytics while I worked in the browser. This seemed like the perfect solution—until I ran into security issues and the realization that maybe not everyone uses Chrome.

Convex requires authentication and access tokens to gather data, and Chrome extensions are highly restrictive when it comes to handling sensitive credentials. Storing API keys in local storage or extension settings felt like a security risk, and I didn’t want to force users to manually authenticate each time they opened the extension. Additionally, interacting with the Convex API directly from a browser extension posed CORS limitations, making it more complex than I initially anticipated.

Pivoting to a React Component

After running into these roadblocks, I realized that the best approach would be to integrate directly into the developer’s existing workflow—right in the dev environment. Instead of a browser-based solution, I decided to build ConvexPanel as a React component, making it easy to embed with any Convex project.

This approach had several advantages:

Direct API Integration: No need to deal with CORS restrictions or browser security policies.
Better Authentication Handling: Credentials could be managed securely within the dev environment by accessing .env files.
Seamless Dev Experience: No more switching between browser tabs—just open the panel in your editor by clicking the convex icon!

How Convex Panel Works

At its core, Convex Panel taps into the same Convex APIs that power the dashboard. By leveraging modern practices, it allows you to interact with your Convex backend without ever leaving your dev environment.

Quick Start Code Snippet

Install the package via npm:

npm install convex-panel

Then you can easily integrate this into your project

import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App";
import "./index.css";
import { ConvexProvider, ConvexReactClient } from "convex/react";
import { ConvexPanel } from 'convex-panel';

const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string);

ReactDOM.createRoot(document.getElementById("root")!).render(
  <React.StrictMode>
    <ConvexProvider client={convex}>
      <App />
      <ConvexPanel
        accessToken="YOUR_ACCESS_TOKEN"
        deployKey={process.env.CONVEX_DEPLOYMENT}
        convex={convex}
      />
    </ConvexProvider>
  </React.StrictMode>,
);

You can easily get your access token from your convex config file:

cat ~/.convex/config.json

This will give you a response similar to this:

{
  "accessToken": "eyJ2BeI6A....."
}%

Your deploy key can be created directly in your dashboard

Displayed Logs

The logs displayed show information needed for any application:

Timestamp: Each log entry is tied to a specific time, allowing you to track when each request was made.
Function Calls: The logs display calls to functions: Queries, Mutations, Actions, etc., helping to monitor backend operations.
Performance Metrics: Each request includes an execution time (in milliseconds or microseconds), which is useful for debugging performance issues.
Filtering & Debugging Tools: The UI provides filters for log types, success statuses, and request IDs, making it easier to find relevant logs.
Detailed Log Output: The expanded log view reveals function details, execution times, and logs related to filters, calls, and tasks, which can assist in diagnosing errors or understanding data flow.

All of this kind of logging is essential for real-time debugging, performance monitoring, and troubleshooting any interaction directly from your development environment.

Table View

Edit Table Cells: Easily update data directly within the table.
Browse All Tables: View a complete list of your tables in the left-hand panel.
Filter Tables & Columns: Quickly find relevant data by filtering tables and individual columns.

This streamlined experience makes it easy to manage and inspect your Convex data.

Health View

Data Analysis: Performance metrics for cache hit rates and failure rates in graphs
Scheduler Lag: A quick scheduler status and lag time metrics shown in a graph.
Convex Version: An indication of your current Convex version and an update button if your convex is out of date.

Under the Hood

Convex Panel is built with simplicity and efficiency in mind:

Code Practice: The tool leverages modern Typescript practices to keep the codebase clean, modular, and easy to extend.
Direct API Integration: By tapping directly into Convex’s APIs, it ensures that the data you see is always in sync with your backend.
Standard Styling: Instead of using Tailwind, this project uses regular CSS Styling.

Conclusion

The ambition to build this component was born out of a personal need to reduce wasted time and improve efficiency and development. If you're like me and are always juggling too many browser tabs, even when amazing tools like Convex dashboard and Convex CLI are available, convex panel might just help you too!

Try it out, contribute, or provide feedback!

Resources

Convex
Convex Panel
NPM Package
GitHub
Convex Discord

How I Wrote a Script to Migrate Data from MongoDB to Convex (and Lived to Tell the Tale)

Bobby Alv — Fri, 21 Feb 2025 18:22:31 +0000

Ah, data migration. The unsung hero of the developer’s life. It’s like moving houses, but instead of furniture, you’re hauling bits and bytes from one database to another. And just like moving, it’s always more complicated than you think. In this article, I’ll walk you through how I wrote a script to migrate data from MongoDB to Convex (shoutout to convex.dev), and hopefully, you’ll learn something — or at least get a good laugh.

The Setup: MongoDB and Convex Sitting in a Tree
First, let’s set the stage. I had a MongoDB database with over 30 tables and over 2M documents that relate to one another. But MongoDB, while great for many things, wasn’t cutting it for our new app’s real-time needs. Enter Convex, a backend-as-a-service that promised to make my life easier with its real-time capabilities and serverless architecture. The only problem? Getting all that data from MongoDB into Convex.

So, I rolled up my sleeves, brewed a pot of coffee, and got to work.

The Code: A Tale of Two Databases
Here’s the script I wrote to migrate the data. It’s a bit of a beast, but I’ll break it down for you. (You can find the full code at the end of this article, but let’s walk through the highlights.)

| Step 1: Connecting to MongoDB and Convex
First things first: I needed to connect to both databases. MongoDB was easy — I just used the MongoClient from the mongodb package. Convex, on the other hand, required a bit more setup, including an auth token.

const mongoClient = new MongoClient(MONGODB_URI);
const convexClient = new ConvexHttpClient(CONVEX_URL);

await mongoClient.connect();
convexClient.setAuth(authToken);

Pro tip: If your auth token is invalid, Convex will throw an error faster than you can say “unauthorized.” So, make sure you’ve got the right token.

| Step 2: Filtering and Counting Documents
Next, I needed to filter the team/organization/workspace object in MongoDB that I wanted to migrate. I created a filter only include documents from specific teams and to make sure that the object was not deleted.

const filter = {
  team: { $in: [
    new ObjectId('628...'),
    new ObjectId('628...')
  ]},
  isDeleted: { $ne: true }
};

const contactsCount = await contactsCollection.countDocuments(filter);
const taskCount = await tasksCollection.countDocuments(filter);
// ...and so on

I console logged This gave me a sense of how much data I was dealing with. Spoiler: It was a lot.

| Step 3: The Migration Loop
Now, the fun part: the migration loop. I processed the data in batches of 100 to avoid overwhelming the system (Convex only allows for a little over 16k). For each contact, I checked if it had already been merged (using a Set to track processed IDs) and skipped it if it had.

for (let skip = skipCount; ; skip += batchSize) {
  const batch = await contactsCollection.find(filter).skip(skip).limit(batchSize).toArray();
  if (batch.length === 0) break;

  for (const oldContact of batch) {
    if (processedMongoIds.has(oldContact._id.toString())) {
      console.log(`Contact already merged: ${oldContact.firstName} ${oldContact.lastName}`);
      continue;
    }
    // ... process the contact ...
  }
}

This loop is the heart of the script. It’s where the magic (and the headaches) happen.

| Step 4: Formatting Data for Convex
I had changed my schema in Convex, so I had to reformat the MongoDB data to fit. This included mapping phone numbers, emails, and addresses into my new structure.

const phoneNumbers = [];
if (oldContact.phoneNumbers?.length > 0) {
  phoneNumbers.push(...oldContact.phoneNumbers.map(p => ({
    label: p.phoneLabel || "Other",
    number: p.phone,
    isBad: p.isBadNumber || false,
    isPrimary: p.isPrimary || false
  })));
}

I also had to handle edge cases, like contacts with no first or last name. (Yes, those exist. No, I don’t know why.)

| Step 5: Creating Contacts and Tasks in Convex
Once the data was formatted, I used Convex mutations to create new records, tasks, tags, activities… and so on.

const result = await convexClient.mutation('contacts:create', newContact);

If a contact had associated tasks, tags or anything that was related to it, I created those too. This part of the script was like assembling IKEA furniture — tedious, but satisfying when it worked. (Also I never buy from IKEA anymore).

| Step 6: Error Handling and Retries
Of course, nothing ever goes perfectly. I added error handling and a retry mechanism to deal with hiccups like network issues or cursor timeouts.

async function retryOperation(operation, maxRetries = 3, delay = 1000) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error) {
      if (attempt === maxRetries) throw error;
      console.log(`Attempt ${attempt} failed, retrying in ${delay}ms...`);
      await new Promise(resolve => setTimeout(resolve, delay));
      delay *= 2; // Exponential backoff
    }
  }
}

The Aftermath: Lessons Learned
After hours of debugging, coffee, and the occasional existential crisis, the migration was complete. Here’s what I learned:

Batch processing is your friend. It keeps things manageable and prevents your script from crashing under its own weight.
Error handling is non-negotiable. Things will go wrong, so plan for it.
Log everything. When something breaks (and it will), you’ll want to know where and why.
Convex is awesome. Once the data was in, Convex made real-time updates a breeze.

The Code
Here’s the full script for your reading pleasure. Feel free to adapt it for your own migrations — just don’t forget the coffee.

import { MongoClient, ObjectId } from 'mongodb';
import { ConvexHttpClient } from 'convex/browser';

const MONGODB_URI = 'YOUR MONGO URI';
const CONVEX_URL = 'CONVEX CLOUD URL';
const createdContacts = new Map();
async function migrateProperties() {
  const mongoClient = new MongoClient(MONGODB_URI);
  const convexClient = new ConvexHttpClient(CONVEX_URL);
  try {
    console.log("Connecting to MongoDB...");
    await mongoClient.connect();
    const db = mongoClient.db('propbear');
    console.log("Connected to MongoDB.");
    console.log("Authenticating with Convex...");

    const authToken = "YOUR AUTH TOKEN";
    convexClient.setAuth(authToken);
    const user = await convexClient.query('users:viewer');
    if (!user) {
      throw new Error("Authentication failed");
    }
    console.log("Authenticated with Convex.");
    const propertiesCollection = db.collection('propertydetails');
    const contactsCollection = db.collection('contactdetails');
    const tasksCollection = db.collection('tasks');
    const linkedPropertyDetailsCollection = db.collection('linkedpropertydetails');
    const filter = {
      team: { $in: [
        new ObjectId('628...'),
        new ObjectId('627...')
      ]}
    };
    const propertyCount = await propertiesCollection.countDocuments(filter);
    const contactCount = await contactsCollection.countDocuments(filter);
    const taskCount = await tasksCollection.countDocuments(filter);
    const linkedPropertyDetailsCount = await linkedPropertyDetailsCollection.countDocuments(filter);

    console.log("Mongo Properties:", propertyCount);
    console.log("Mongo Contacts:", contactCount);
    console.log("Mongo Tasks:", taskCount);
    console.log("Mongo Linked Property Details:", linkedPropertyDetailsCount);
    console.log("Starting migration...");
    let migratedCount = 0;
    let errorCount = 0;
    let skippedCount = 0;
    const batchSize = 100;
    let lastId = null;
    const skipCount = 6800;
    const skipCursor = propertiesCollection.find(filter)
      .sort({ _id: 1 })
      .skip(skipCount)
      .limit(1);

    const lastSkippedDoc = await skipCursor.next();
    if (lastSkippedDoc) {
      lastId = lastSkippedDoc._id;
    }
    while (true) {
      const query = lastId ? { ...filter, _id: { $gt: lastId } } : filter;
      const cursor = propertiesCollection.find(query).sort({ _id: 1 }).limit(batchSize);
      let batch = await retryOperation(async () => await cursor.toArray());
      if (batch.length === 0) {
        console.log("No more properties to process. Migration completed.");
        break;
      }
      let allSkipped = true;
      for (const oldProperty of batch) {
        try {
          async function recordExistsByMongoId(client, table, mongoId) {
            try {
              const result = await client.query(`${table}:getByMongoId`, { mongoId });
              return result ? result._id : null;
            } catch (error) {
              console.error(`Error checking if ${table} record exists:`, error);
              return null;
            }
          }
          try {
            const existingPropertyId = await recordExistsByMongoId(convexClient, 'properties', oldProperty._id.toString());

            if (existingPropertyId) {
              console.log(`Property already exists: ${oldProperty.name}`, existingPropertyId);
              skippedCount++;
              continue;
            }
            allSkipped = false;
            let contactId = null;
            if (oldProperty.owner) {
              const oldContact = await contactsCollection.findOne({ _id: oldProperty.owner });
              if (oldContact) {
                contactId = await recordExistsByMongoId(convexClient, 'contacts', oldContact._id.toString());
                if (!contactId) {
                  const newContact = {
                    mongoId: oldContact._id.toString(),
                    recordType: "contacts",
                    firstName: oldContact.firstName || "",
                    lastName: oldContact.lastName || "",
                    fullName: `${oldContact.firstName || ""} ${oldContact.lastName || ""}`.trim(),
                    orgId: mapOrgId(oldContact.team?.toString()) || "nd70p3ngxkh8n7chaddb81tjpx7166mr",
                    phone: (() => {
                      let phoneNumbers = [];
                      if (oldContact.phoneNumbers && oldContact.phoneNumbers.length > 0) {
                        phoneNumbers = oldContact.phoneNumbers.map(p => ({
                          label: p.phoneLabel || "Other",
                          number: p.phone,
                          isPrimary: p.isPrimary || false,
                          isBad: oldContact.phoneDetails?.isBadNumber || false
                        }));
                      }
                      if (oldContact.phone && !phoneNumbers.some(p => p.number === oldContact.phone)) {
                        phoneNumbers.push({
                          label: oldContact.phoneDetails?.phoneLabel || "Other",
                          number: oldContact.phone,
                          isPrimary: oldContact.phoneDetails?.isPrimary || phoneNumbers.length === 0,
                          isBad: oldContact.phoneDetails?.isBadNumber || false
                        });
                      }
                      return phoneNumbers.length > 0 ? phoneNumbers : undefined;
                    })(),
                    email: (() => {
                      let emails = [];
                      if (oldContact.emails && oldContact.emails.length > 0) {
                        emails = oldContact.emails.map(e => ({
                          label: e.emailLabel || "Other",
                          address: e.email,
                          isPrimary: e.isPrimary || false,
                          isBad: oldContact.emailDetails?.isBadEmail || false
                        }));
                      }
                      if (oldContact.email && !emails.some(e => e.address === oldContact.email)) {
                        emails.push({
                          label: oldContact.emailDetails?.emailLabel || "Other",
                          address: oldContact.email,
                          isPrimary: oldContact.emailDetails?.isPrimary || emails.length === 0,
                          isBad: oldContact.emailDetails?.isBadEmail || false
                        });
                      }
                      return emails.length > 0 ? emails : [{ label: "Other", address: "", isPrimary: true, isBad: false }];
                    })(),
                    address: oldContact.address ? [{
                      label: "Home",
                      street: oldContact.address,
                      city: oldContact.city,
                      state: oldContact.state,
                      zip: oldContact.zipCode ? parseFloat(oldContact.zipCode) : undefined,
                    }] : undefined,
                    summary: oldContact.stickyNote,
                  };
                  Object.keys(newContact).forEach(key => newContact[key] === undefined && delete newContact[key]);
                  try {
                    let contactId;
                    if (createdContacts.has(oldContact._id.toString())) {
                      contactId = createdContacts.get(oldContact._id.toString());
                      await convexClient.mutation('contacts:updateContact', {
                        id: contactId,
                      });
                    } else {
                      contactId = await convexClient.mutation('contacts:createContactWithoutEnrichment', newContact);
                      createdContacts.set(oldContact._id.toString(), contactId);
                    }
                    if (oldContact.tags && oldContact.tags.length > 0) {
                      for (const tag of oldContact.tags) {
                        if (tag !== "Import") {
                          try {
                            const tagId = await createOrGetTag(
                              convexClient, 
                              tag.name || tag,
                              "contacts", 
                              contactId, 
                              newContact.orgId, 
                              await mapUserId(oldContact.createdBy?.toString()),
                              tag._id ? tag._id.toString() : undefined
                            );
                            if (tagId) {
                              console.log(`Created or got tag: ${tag.name || tag}`, tagId);
                            } else {
                              console.log(`Failed to create or get tag: ${tag.name || tag}`);
                            }
                          } catch (error) {
                            console.error(`Error processing tag ${tag.name || tag} for contact:`, error);
                          }
                        }
                      }
                    }
                    console.log(`Created contact: ${newContact.firstName} ${newContact.lastName}`, contactId);
                    // Create tasks for the contact
                    if (oldContact.tasks && oldContact.tasks.length > 0) {
                      for (const taskId of oldContact.tasks) {
                        const oldTask = await tasksCollection.findOne({ _id: taskId });
                        if (oldTask) {
                          const existingTaskId = await recordExistsByMongoId(convexClient, 'tasks', oldTask._id.toString());
                          if (!existingTaskId) {
                            const newTask = {
                              mongoId: oldTask._id.toString(),
                              orgId: newContact.orgId,
                              title: oldTask.task,
                              description: oldTask.description,
                              dueDate: oldTask.endDate ? new Date(oldTask.endDate).toISOString() : undefined,
                              linkedRecord: contactId,
                              assignedTo: await mapUserId(oldTask.createdForUserId?.toString()),
                              priority: mapPriority(oldTask.taskPriority),
                              column: mapStatus(oldTask.status),
                            };
                            Object.keys(newTask).forEach(key => newTask[key] === undefined && delete newTask[key]);
                            try {
                              const taskResult = await convexClient.mutation('tasks:createTask', newTask);
                              console.log(`Created contact task: ${newTask.title}`, taskResult);
                            } catch (error) {
                              console.error(`Error creating contact task ${newTask.title}:`, error);
                            }
                          } else {
                            console.log(`Task already exists: ${oldTask.task}`, existingTaskId);
                          }
                        }
                      }
                    }
                    // Create related contacts
                    if (oldContact.relatedContacts && oldContact.relatedContacts.length > 0) {
                      for (const relatedContact of oldContact.relatedContacts) {
                        const newRelatedContact = {
                          label: "Related Contact",
                          firstName: relatedContact.name.split(' ')[0] || "",
                          lastName: relatedContact.name.split(' ').slice(1).join(' ') || "",
                          email: [{
                            label: "Other",
                            address: relatedContact.email,
                            isBad: false,
                            isPrimary: true
                          }],
                          phone: [{
                            label: "Other",
                            number: relatedContact.phone,
                            isBad: false,
                            isPrimary: true
                          }],
                          orgId: newContact.orgId,
                          recordId: contactId
                        };
                        try {
                          const relatedContactResult = await convexClient.mutation('relatedContacts:createRelatedContact', newRelatedContact);
                          console.log(`Created related contact: ${newRelatedContact.firstName} ${newRelatedContact.lastName}`, relatedContactResult);
                        } catch (error) {
                          console.error(`Error creating related contact ${newRelatedContact.firstName} ${newRelatedContact.lastName}:`, error);
                        }
                      }
                    }
                  } catch (error) {
                    console.error(`Error creating contact ${newContact.firstName} ${newContact.lastName}:`, error);
                  }
                } else {
                  console.log(`Contact already exists: ${oldContact.firstName} ${oldContact.lastName}`, contactId);
                }
              }
            }
            // Create the property
            const newProperty = {
              mongoId: oldProperty._id.toString(),
              name: oldProperty.name,
              recordType: "properties",
              image: oldProperty.propertyImages && oldProperty.propertyImages.length > 0 ? oldProperty.propertyImages[0] : undefined,
              orgId: mapOrgId(oldProperty.team?.toString()) || "nd70p3ngxkh8n7chaddb81tjpx7166mr",
              propertyType: oldProperty.type || undefined,
              address: {
                street: oldProperty.address ? toProperCase(oldProperty.address) : toProperCase(oldProperty.name), 
                city: oldProperty.city ? toProperCase(oldProperty.city) : undefined,
                state: oldProperty.state ? oldProperty.state.toUpperCase() : undefined,
                zip: oldProperty.zipCode ? parseFloat(oldProperty.zipCode) : undefined,
              },
              location: oldProperty.coordinates && oldProperty.coordinates.lng && oldProperty.coordinates.lat ? {
                type: "Point",
                coordinates: [
                  parseFloat(oldProperty.coordinates.lng),
                  parseFloat(oldProperty.coordinates.lat)
                ],
              } : { type: "Point", coordinates: [0, 0] }, // Default coordinates if not available
              isDeleted: false,
              tags: oldProperty.tags ? oldProperty.tags.map(tag => tag._id?.toString()) : [],
              yearBuilt: oldProperty.propertyData?.yearBuilt ? parseFloat(oldProperty.propertyData.yearBuilt) : undefined,
              squareFootage: oldProperty.propertyData?.acres ? oldProperty.propertyData.acres * 43560 : undefined,
              units: oldProperty.units ? parseFloat(oldProperty.units) : undefined,
              price: oldProperty.propertyData?.forsale?.price ? parseFloat(oldProperty.propertyData.forsale.price) : undefined,
              parcelNumber: oldProperty.parcelNo || oldProperty.propertyData?.parcelNo || undefined,
              saleDate: oldProperty.lastSold ? new Date(parseFloat(oldProperty.lastSold)).toISOString() : undefined,
              salePrice: oldProperty.propertyData?.soldPrice ? parseFloat(oldProperty.propertyData.soldPrice) : undefined,
              landValue: oldProperty.landValue ? parseFloat(oldProperty.landValue) : undefined,
              buildingValue: oldProperty.propertyData?.bldgValue ? parseFloat(oldProperty.propertyData.bldgValue) : undefined,
              status: oldProperty.status,
              primaryUse: oldProperty.propertyData?.occupancy || undefined,
              construction: oldProperty.propertyData?.construction || undefined,
              lotSize: oldProperty.propertyData?.acres ? parseFloat(oldProperty.propertyData.acres) : undefined,
              zoning: oldProperty.propertyData?.zoning || undefined,
              meterType: oldProperty.propertyData?.meterType || undefined,
              class: oldProperty.propertyData?.class || undefined,
              structures: oldProperty.propertyData?.structures ? parseInt(oldProperty.propertyData.structures) : undefined,
              parking: oldProperty.propertyData?.parking || undefined,
            };
            // Remove createdBy field as it's not allowed
            delete newProperty.createdBy;
            Object.keys(newProperty).forEach(key => newProperty[key] === undefined && delete newProperty[key]);
            // Ensure location is always present
            if (!newProperty.location || isNaN(newProperty.location.coordinates[0]) || isNaN(newProperty.location.coordinates[1])) {
              newProperty.location = { type: "Point", coordinates: [0, 0] };
            }
            try {
              const result = await convexClient.mutation('properties:createPropertyWithoutEnrichment', newProperty);
              console.log(`Created property: ${newProperty.name}`, result);
              // Process tags for the property
              if (oldProperty.tags && oldProperty.tags.length > 0) {
                for (const tag of oldProperty.tags) {
                  if (tag !== "Import") {
                    try {
                      const tagId = await createOrGetTag(
                        convexClient, 
                        tag.name || tag,
                        "properties", 
                        result, 
                        newProperty.orgId, 
                        await mapUserId(oldProperty.creator?.toString()),
                        tag._id ? tag._id.toString() : undefined
                      );
                      if (tagId) {
                        console.log(`Created or got tag: ${tag.name || tag}`, tagId);
                      } else {
                        console.log(`Failed to create or get tag: ${tag.name || tag}`);
                      }
                    } catch (error) {
                      console.error(`Error processing tag ${tag.name || tag} for property:`, error);
                    }
                  }
                }
              }
              // Create tasks for the property
              if (oldProperty.tasks && oldProperty.tasks.length > 0) {
                for (const taskId of oldProperty.tasks) {
                  const oldTask = await tasksCollection.findOne({ _id: taskId });
                  if (oldTask) {
                    const existingTaskId = await recordExistsByMongoId(convexClient, 'tasks', oldTask._id.toString());
                    if (!existingTaskId) {
                      const newTask = {
                        mongoId: oldTask._id.toString(),
                        orgId: newProperty.orgId,
                        title: oldTask.task,
                        description: oldTask.description,
                        dueDate: oldTask.endDate ? new Date(oldTask.endDate).toISOString() : undefined,
                        linkedRecord: result,
                        assignedTo: await mapUserId(oldTask.createdForUserId?.toString()),
                        priority: mapPriority(oldTask.taskPriority),
                        column: mapStatus(oldTask.status),
                      };
                      Object.keys(newTask).forEach(key => newTask[key] === undefined && delete newTask[key]);
                      try {
                        const taskResult = await convexClient.mutation('tasks:createTask', newTask);
                        console.log(`Created property task: ${newTask.title}`, taskResult);
                      } catch (error) {
                        console.error(`Error creating property task ${newTask.title}:`, error);
                      }
                    } else {
                      console.log(`Task already exists: ${oldTask.task}`, existingTaskId);
                    }
                  }
                }
              }
              if (contactId) {
                await convexClient.mutation('contactLinkedProperties:linkPropertyFromMongo', {
                  contactId: contactId,
                  propertyId: result,
                  relation: 'Owner',
                  orgId: newProperty.orgId,
                });
                console.log(`Linked contact ${contactId} as Owner to property ${result}`);
              }
              // After creating the main property and owner contact
              await processLinkedProperties(convexClient, oldProperty, result, newProperty.orgId, {
                linkedPropertyDetailsCollection,
                contactsCollection
              }, createdContacts);
              migratedCount++;
            } catch (error) {
              console.error(`Error migrating property ${newProperty.name}:`, error);
              errorCount++;
            }
          } catch (error) {
            console.error(`Error migrating property ${oldProperty.name}:`, error);
            errorCount++;
          }
        } catch (error) {
          if (error.code === 43 && error.codeName === 'CursorNotFound') {
            console.error('Cursor not found, restarting migration from the last successful point');
            // You might want to implement a way to track the last successfully migrated property
            // and restart from there instead of from the beginning
            break;
          }
          throw error; // Rethrow other errors
        }
        // Update lastId after processing each property
        lastId = oldProperty._id;
      }
      if (allSkipped) {
        console.log(`All ${batchSize} properties in this batch were already migrated. Moving to next batch.`);
      }
      console.log(`Processed batch. Total migrated: ${migratedCount}, Skipped: ${skippedCount}, Errors: ${errorCount}`);
    }
    console.log(`Migration completed. Total migrated: ${migratedCount}, Skipped: ${skippedCount}, Errors: ${errorCount}`);
  } catch (error) {
    console.error('Error during migration:', error);
  } finally {
    await mongoClient.close();
  }
}
async function processLinkedProperties(convexClient, oldProperty, propertyId, orgId, collections, createdContacts) {
  const { linkedPropertyDetailsCollection, contactsCollection } = collections;
  const batchSize = 10; // Reduced batch size
  let skip = 0;
  while (true) {
    const linkedPropertyDetails = await linkedPropertyDetailsCollection.find({ property: oldProperty._id })
      .skip(skip)
      .limit(batchSize)
      .toArray();
    if (linkedPropertyDetails.length === 0) break;
    for (const link of linkedPropertyDetails) {
      await processLinkedProperty(convexClient, link, propertyId, orgId, contactsCollection, createdContacts);
      await new Promise(resolve => setTimeout(resolve, 100)); // 100ms delay between each link
    }
    skip += batchSize;
    await new Promise(resolve => setTimeout(resolve, 1000)); // 1 second delay between batches
  }
}
async function processLinkedProperty(convexClient, link, propertyId, orgId, contactsCollection, createdContacts) {
  try {
    const oldContact = await contactsCollection.findOne({ _id: link.contact });
    if (oldContact && oldContact._id) {
      let linkedContactId;
      if (createdContacts.has(oldContact._id.toString())) {
        linkedContactId = createdContacts.get(oldContact._id.toString());
      } else {
        // Create the linked contact
        const newLinkedContact = {
          mongoId: oldContact._id.toString(),
          firstName: oldContact.firstName || "",
          lastName: oldContact.lastName || "",
          fullName: `${oldContact.firstName || ""} ${oldContact.lastName || ""}`.trim(),
          email: oldContact.email ? [{
            label: "Other",
            address: oldContact.email,
            isBad: false,
            isPrimary: true
          }] : [{ label: "Other", address: "", isPrimary: true, isBad: false }],
          orgId: orgId,
          recordType: "contacts",
        };
        // Add company name if it exists
        if (oldContact.company) {
          delete oldContact.company;
        }
        // Use company name as fullName if no individual name is provided
        if (!newLinkedContact.fullName && newLinkedContact.company) {
          newLinkedContact.fullName = newLinkedContact.company;
        }
        // Skip truly empty contacts
        if (!newLinkedContact.fullName.trim() && !newLinkedContact.email[0].address.trim() && !newLinkedContact.company) {
          console.log(`Skipping empty linked contact for property ${propertyId}`);
          return;
        }
        try {
          linkedContactId = await convexClient.mutation('contacts:createContactWithoutEnrichment', newLinkedContact);
          console.log(`Created linked contact: ${newLinkedContact.fullName || newLinkedContact.company}`, linkedContactId);
          await convexClient.mutation('contactLinkedProperties:linkPropertyFromMongo', {
            contactId: linkedContactId,
            propertyId: propertyId,
            relation: link.relation || "Other",
            orgId: orgId,
          });
          console.log(`Linked contact ${linkedContactId} as ${link.relation || "Other"} to property ${propertyId}`);
          createdContacts.set(oldContact._id.toString(), linkedContactId);
        } catch (error) {
          console.error(`Error creating linked contact ${newLinkedContact.fullName || newLinkedContact.company}:`, error);
          return;
        }
      }
    } else {
      console.error(`Invalid linked contact found for property ${propertyId}:`, link);
    }
  } catch (error) {
    console.error(`Error processing linked property for property ${propertyId}:`, error);
  }
}
/**
 * @param {string | undefined} oldId
 * @returns {string | null}
 */
function mapOrgId(oldId) {
  if (!oldId) return "nd70p3ngxkh8n7chaddb81tjpx7166mr";
  const orgIdMap = { 
    '628..': 'nd7b...', // first is mongoId second is convex _id
    '627...': 'nd7d...'
  };
  return orgIdMap[oldId] || null;
}
function mapUserId(oldId) {
  if (!oldId) return null;

  const userIdMap = {
    '628c...': 'jx7...', // first is mongoId second is convex _id
    ...
  };
  return userIdMap[oldId] || null;
}
function mapPriority(oldPriority) {
  const priorityMap = {
    'Low': 'low',
    'Medium': 'medium',
    'High': 'high'
  };
  return priorityMap[oldPriority] || 'no-priority';
}
function mapStatus(oldStatus) {
  const statusMap = {
    'In Progress': 'in-progress',
    'Completed': 'done',
    'Not Started': 'todo'
  };
  return statusMap[oldStatus] || 'todo';
}
function toProperCase(str) {
  return str.replace(/\w\S*/g, function(txt) {
    if (isNaN(txt)) {
      return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();
    } else {
      return txt;
    }
  });
}
async function retryOperation(operation, maxRetries = 3, delay = 1000) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error) {
      if (attempt === maxRetries) throw error;
      console.log(`Attempt ${attempt} failed, retrying in ${delay}ms...`);
      await new Promise(resolve => setTimeout(resolve, delay));
      delay *= 2; // Exponential backoff
    }
  }
}
async function createOrGetTag(convexClient, tagName, recordType, recordId, orgId, createdBy, mongoId = undefined) {
  try {
    let existingTag;

    // Try to get the tag by mongoId if it exists
    if (mongoId) {
      existingTag = await convexClient.query('tags:getTagByMongoId', { mongoId });
    }
    if (existingTag) {
      // If the tag exists, return its ID
      return existingTag._id;
    } else {
      // If the tag doesn't exist, create a new one
      const newTag = {
        name: tagName,
        recordType,
        orgId,
        createdBy,
        mongoId
      };
      const tagId = await convexClient.mutation('tags:createTag', newTag);
      // Link the tag to the record
      await convexClient.mutation('tags:linkTagToRecord', {
        tagId,
        recordId
      });
      return tagId;
    }
  } catch (error) {
    console.error(`Error creating or getting tag ${tagName}:`, error);
    return null;
  }
}

migrateProperties().catch(console.error);

Final Thoughts
Data migration isn’t glamorous, but it’s a necessary evil. With the right tools and a bit of patience, you can move mountains of data without losing your sanity. And if all else fails, remember: there’s always more coffee.

More Examples: https://github.com/robertalv/merge-mongodb-convex

Happy coding! 🚀