Forem: Anjana Vakil

Help, my app is overreacting!

Anjana Vakil — Tue, 21 Feb 2023 23:55:04 +0000

The apps we build don’t exist in a vacuum; they’re used by real users, and we want to give them a really good experience!

Let’s say I’m building a simple task manager app where I want to show the user a list of tasks, each of which has a certain status like “New” or “In Progress”, and possibly an owner assigned to the task:

For the best user experience, I want the app to always show the latest data, even if that data is actively changing while I’m viewing the page. For example, if another user somewhere adds a new task, or changes the title of a task, I want to see the app update immediately without having to manually reload the page, hit a “refresh” button, or the like.

In this post, we’ll explore:

How a reactive backend like Convex helps build live-updating apps that show users the fresh data they deserve
The default behavior of reactive data updates from Convex’s useQuery and usePaginatedQuery hooks, and how that might affect UX in different contexts
How I can customize the way my app reacts to data updates to more easily deliver the intended user experience

Let’s dig in!

A visit from the reactive-query fairy

With traditional backends, to achieve the desired behavior I’d have to go out of my way to keep the data updated, for example by polling (actively re-fetching the data every so often). That works to a certain extent, but means:

more code for me to write/maintain (more work! more bugs!)
more request-response cycles that might slow down my app
some lag time in when the user sees the new data if it changes between polling cycles

I also might run the risk of inconsistencies in what the user sees, if I’m making multiple queries of the same data (e.g. one query that fetches the total number of tasks, and other that fetches the detailed task list); with no way to guarantee their polling cycles will stay in sync, one query might pick up new data before the other.

Luckily, we live in the futuristic-sounding year of 2023, and we now have not only reactive frontend frameworks like React, but also reactive backends like Convex that work hand-in-hand with my reactive frontend to automatically keep my app’s data fresh!

For example, Convex’s useQuery hook returns a reactive value that gives me the up-to-date result of running a particular database query. Say I have a listAllTasks Convex query function that queries the tasks table in my database:

// convex/listAllTasks.ts
import { query } from './_generated/server'

export default query(async ({ db }) => {
  // Grab all the rows in `tasks` table and collect into an array
  return await db.query('tasks').collect()
})

I can pull the reactive results of running that query into my frontend with useQuery like so:

// pages/index.tsx
import React from 'react'
import { useQuery } from '../convex/_generated/react'
import { TaskList } from '../components/tasks'
import { LoginHeader, NewTaskButton } from '../components/util'

export default function App() {
  const tasks = useQuery('listAllTasks')

  return (
    <main>
      <LoginHeader />
      <div id="controls">
        <NewTaskButton />
      </div>
      <TaskList tasks={tasks} />
    </main>
  )
}

Thanks to the useQuery hook, the tasks value updates instantly any time the data changes, and the component re-renders. So in the case where another user adds a task while I’m viewing the list, I see it show up instantly:

And if I have multiple queries referencing the same data (e.g. say I have another function countTasks that also reads from the tasks table, which I invoke in another component with useQuery('countTasks')), I don’t have to worry about the kind of race condition possible with polling that could lead to the count of tasks being inconsistent with what’s shown in the task list. Convex ensures all of my useQuery calls stay in sync, consistently pushing out the exact same data to all of my queries whenever that data changes. One less thing to worry about? Music to my ears!

But what happens while the data is loading? The value returned by useQuery is initially undefined until the data has loaded, so I can check for that to display some kind of loading state to my users (here I just show a simple ‘loading’ message, but in a real app I might display e.g. a spinner icon or ghost component):

// in App() function
{tasks === undefined ? <p>Loading tasks...</p> : <TaskList tasks={tasks} />}

Fantastic! My app auto-updates with the latest data without the user having to do anything, and I can show a loading state while initially fetching the data. My users always see the freshest data, the app doesn’t have to constantly poll for data updates, and I didn’t even have to write that much code to make it happen!

In other words, with this kind of pattern for reactive data, it feels like the answer to all my wishes fell right into my lap, er, app!

Overreacting can be distracting

However, this convenient out-of-the-box reactivity might be more than I need in certain situations. For example, say I want to let users check boxes to specify the particular task status(es) they’re interested in, e.g. only New or In Progress tasks:

To achieve this, I can make a listTasksWithStatus query function that looks similar to listAllTasks, but with an additional taskStatuses parameter that accepts an array of status values used to filter the query results:

// convex/listTasksWithStatus.ts
import { query } from './_generated/server'

export default query(async ({ db }, taskStatuses: string[]) => {
  // Grab rows in `tasks` table matching the given filter
  return await db
    .query('tasks')
    .filter((q) =>
      q.or(
        // Match any of the given status values
        ...taskStatuses.map((status) => q.eq(q.field('status'), status))
      )
    )
    .collect() // collect all results into an array
}

Then in my frontend I can wire up some checkbox inputs so that whenever the user changes the checked values, their selections are captured as state and passed along to useQuery:

// in pages/index.tsx
import React, { useState, type ChangeEventHandler } from 'react'
import { useQuery } from '../convex/_generated/react'
import { TaskList } from '../components/taskList'
import { LoginHeader, NewTaskButton, Checkboxes } from '../components/util'

const allStatuses = ['New', 'In Progress', 'Done', 'Cancelled']

export default function App() {
  const user = useQuery('getCurrentUser')

  const [checkedValues, setCheckedValues] = useState(['New', 'In Progress'])

  const handleChangeChecked = ((event) => {
    // Process a checkbox change event affecting the status filter
    const target = event.target as HTMLInputElement
    if (target.checked) {
      // A formerly unchecked status filter is now checked; add value to array
      setCheckedValues([...checkedValues, target.value])
    } else {
      // A formerly checked status filter is now unchecked; remove value from array
      setCheckedValues(checkedValues.filter((s) => s !== target.value))
    }
  }) as ChangeEventHandler

  const tasks = useQuery('listTasksWithStatus', checkedValues)

  return (
    <main>
      <LoginHeader />
      <div id="controls">
        <NewTaskButton />
                <Checkboxes // simple component creating a checkbox input for each status
          values={allStatuses}
          checkedValues={checkedValues}
          onChange={handleChangeChecked}
        />
      </div>
      {tasks === undefined ?  <p>Loading tasks...</p> : <TaskList tasks={tasks} />}
    </main>
  )
}

This basically works, updating the list reactively based on the user’s input, but unfortunately whenever checkedValues updates, something annoying happens - do you see it?

Whenever the user updates their selection, there’s a brief, distracting flash of the loading state. This is because whenever checkedValues changes:

the component re-renders, making a new call to useQuery
useQuery does its intended job of returning undefined while the updated query is initially running
the component sees tasks is undefined and renders the loading state, until
the new results come back, tasks updates, and the component finally re-renders with the new data

That behavior might be what I want in some contexts, but in this case I don’t want my users to see that distracting flash; instead, during that brief loading period after they’ve checked a box I’d rather keep showing them the old, stale data from the previous selection, and wait to re-render until the new, fresh data has finished loading.

In other words, you might say my app is “overreacting” to updates from useQuery, not all of which I want to translate into UI updates! I don’t want to give up the convenient reactivity of useQuery, but I want to customize its behavior to smash the flash.

Impacting how the query’s reacting

Essentially, for this use case what I’d like is a version of useQuery that’s a little bit less reactive, skipping those intermediate undefined states when the query changes, and instead keeping the data more “stable” by continuing to give me the stale data from the previous query until the fresh data has finished loading.

Refs to the rescue! To customize the behavior of useQuery to fit my use case, I can implement a custom React hook that I’ll call useStableQuery, which functions similarly to useQuery but keeps track of the resulting data with React’s builtin useRef hook, which gives me a Ref object whose identity remains stable between re-renders, and which does not trigger a re-render when its value (accessed via the object’s .current property) changes.

By using a ref to capture the reactive useQuery return value, I can decide to only update the value returned from useStableQuery once the query result is no longer undefined:

// hooks/useStableQuery.ts

import { useRef } from 'react'
import { useQuery } from '../convex/_generated/react'

export const useStableQuery = ((name, ...args) => {
  const result = useQuery(name, ...args)

    // useRef() creates an object that does not change between re-renders
  // stored.current will be result (undefined) on the first render
  const stored = useRef(result) 

    // After the first render, stored.current only changes if I change it
  // if result is undefined, fresh data is loading and we should do nothing
  if (result !== undefined) {
    // if a freshly loaded result is available, use the ref to store it
    stored.current = result
  }

  // undefined on first load, stale data while reloading, fresh data after loading
  return stored.current
}) as typeof useQuery // make sure we match the useQuery signature & return type

(Note: I could also implement this pattern directly in the component that calls useQuery, without writing a custom hook, but putting it in a hook lets me more easily reuse this logic across multiple components/queries.)

In my component, I can now swap the original useQuery out for my custom useStableQuery, capturing the resulting tasks just like before:

// in pages/index.tsx
import { useStableQuery } from '../hooks/useStableQuery'

// ...

export default function App() {
  // ...
  const tasks = useStableQuery('listTasks', checkedValues)
  // ...
}

Now, tasks is only undefined on the very first load, and whenever checkedValues updates in reaction to user input and its new value is passed in to useStableQuery, tasks does not update until the fresh new data is ready, skipping the intermediate undefined state that was causing the loading flash before. Success!

What about pagination, is that a complication?

If the app I’m building is for a big organization likely to have a ton of tasks, I probably want to use a paginated query instead. Initially, I’ll only show users the first page of results, then load additional pages as needed (e.g. when the user clicks a button, or scrolls to the bottom).

I can update my listTasksWithStatus function to return paginated results like so, accepting a paginationOptions object as the second parameter and replacing .collect() with .paginate(paginationOptions):

// convex/listTasksWithStatus.ts
import { query } from './_generated/server'

export default query(
  async ({ db }, paginationOptions, taskStatuses: string[]) => {
    // Grab rows in `tasks` table matching the given filter
    return await db
      .query('tasks')
      .filter((q) =>
        q.or(
          // Match any of the given status values
          ...taskStatuses.map((status) => q.eq(q.field('status'), status))
        )
      )
      // paginate the results instead of collecting into an array
      .paginate(paginationOptions)
  }
)

In my component, I can now replace useQuery with Convex’s analogous usePaginatedQuery hook, which accepts the additional paginationOptions argument that lets me specify the initial number of items I want in the first page. In addition to the results data for the loaded page(s), usePaginatedQuery also returns a status value indicating pagination status (either 'CanLoadMore', 'LoadingMore' or 'Exhausted') and a loadMore function I can call to load additional pages when the user clicks a button.

I can use this hook in my component like so, checking status to know when to display the loading state and adding a simple button to load the next page, if any:

// in pages/index.tsx
import { usePaginatedQuery } from 'convex/react'

export default function App() {
    // ...set up checkedValues & handler same as before

  const {results, status, loadMore} = usePaginatedQuery(
    'listTasks',
    { initialNumItems: 10 },
    checkedValues
  )

  return (
    <main>
      {/* ...header & controls same as before */}      
      {status === 'LoadingMore' 
        ?  <p>Loading tasks...</p> 
        : <TaskList tasks={results} />}
      {loadMore && <button onClick={() => loadMore(10)}>Load more</button>}
    </main>
  )
}

But once again, the user sees an empty flash whenever they change their checkbox selections, since the status switches back to LoadingMore while the new page is being fetched.

Ugh, there goes my app overreacting again, what a drama queen! How do I rein it in while still using a paginated query?

To get the stable behavior I want and ignore the intermediate loading states as before, I can make a paginated version of my custom query hook called useStablePaginatedQuery. It follows the same pattern as useStableQuery, but checks for the LoadingMore status rather than undefined to determine when not to update the results:

// in hooks/useStableQuery.ts
import { useRef } from 'react'
import { usePaginatedQuery } from '../convex/_generated/react'

export const useStablePaginatedQuery = ((name, options, ...args) => {
  const result = usePaginatedQuery(name, options, ...args)
  const stored = useRef(result)

  // If new data is still loading, wait and do nothing
  // If data has finished loading, use the ref to store it
  if (result.status !== 'LoadingMore') {
    stored.current = result
  }

  return stored.current
}) as typeof usePaginatedQuery

Now, when I replace usePaginatedQuery with useStablePaginatedQuery in my component, I get the slightly-less-reactive behavior I’m looking for; no flash, no drama!

// in pages/index.tsx
import { useStablePaginatedQuery } from '../hooks/useStableQuery'

// ...

export default function App() {
  // ...
  const {results, status, loadMore} = useStablePaginatedQuery(
    'listTasks',
    { initialNumItems: 10 },
    checkedValues
  )
  // ...
}

Let's recap this (less-)reactive app

To recap, in a use case like this task manager app, where I want to reactively query data based on user input while still giving users a smooth & stable experience:

Using a reactive backend like Convex with a reactive frontend framework like React lets me easily build live-updating apps, without having to constantly poll for updates in the background or make users manually refresh the page
The reactive value returned by the Convex useQuery hook (which is undefined while data is loading) is exactly what I want in some cases (e.g. listAllTasks), as Convex will automatically update it whenever the data changes
In other cases (like listTasksWithStatus), the undefined returned by useQuery while loading might not be ideal, e.g. causing an undesirable reloading flash if I’m dynamically updating the query arguments based on user input/app state
If the default behavior of useQuery doesn't quite fit my use case, I can customize it by writing my own version, e.g. useStableQuery, which ‘skips’ intermediate undefined states with the help of React’s useRef hook
If I want to paginate the query results, I can write an analogous useStablePaginatedQuery which uses the same useRef pattern in conjunction with [usePaginatedQuery](https://docs.convex.dev/generated-api/react#usepaginatedquery)

If you have a use case similar to mine, feel free to use these hooks in your own apps! You can find the code in the get-convex/convex-helpers repo on Github.

And if your use case is slightly different and you want to customize the reactive behavior of useQuery some other way, I hope this has provided a useful example of how to implement your own version with exactly the behavior you want! For another example of tweaking an app’s reactive dataflow with a custom React hook, check out Jamie Turner’s video on Managing Reactivity with useBufferedState.

Have you run into other issues with reactive data updates? Do you have other favorite patterns for managing reactive query results? Feel free to jump into the Convex community Discord to share & discuss!

Cover image: Roy Lichtenstein, “Crying Girl" (1963), via WikiArt

Searching for retro games with Xata & Next.js 13

Anjana Vakil — Wed, 02 Nov 2022 10:40:51 +0000

Long ago, in the beautiful kingdom of JavaScript surrounded by cascades and DOM trees...
legends told of an omnipotent and omniscient Gaming Power that resided in a hidden app.

It's hidden because you haven't built it yet.
And the time of destiny for Princess ~~Zelda~~ You is drawing near.

An epic quest

In this tutorial we'll build a fullstack web app that allows us to search a collection of retro games data to find the old-school console games of our dreams. And I hope you like adventure, Princess, for this will be a treacherous journey through the bleeding edge of recently released web technologies!

We'll learn how to:

Use Xata's serverless platform to store & retrieve data without needing a database
Build an app with React Server & Client Components using Next.js 13 and its app/ directory
Implement full-text search with boosting via the Xata SDK

Before we begin, make sure you have Node/npm installed, and download the games.csv data we'll be working with. You can optionally check out the completed tutorial code for reference.

The games may be old, but the tech is brand spankin' new - so let's play!

Backstory

My friend & avid retro gamer Sara Vieira had compiled a great collection of retro games data for her awesome site letsplayretro.games, a Next app which already had a ton of great features. But one feature Sara still wished for was full-text search over not just games' names, but all their metadata, with the ability to prioritize (aka "boost") the highest-rated games in the search results.

Since every minute spent coding is one less minute spent playing NES, neither of us wanted to waste a lot of time on making this happen. That's when it dawned on me that I knew of a way to easily implement full-text, boosted search over her tabular data: Xata provides it out of the box!

It's-a me, Xata!

Xata is a serverless data platform that's new on the Jamstack scene, and aims to make life easier for developers to build fullstack apps need without worrying about the details of a scalable, performant backend database. Let's give it a whirl.

Set up Xata CLI

Then, from the command line install the CLI and authenticate with your Xata account:

npm i --location=global @xata.io/cli
xata auth login

Import CSV data

Now we need to get our games data into Xata. Sara gave us a file games.csv with information about 7.3K games, and conveniently Xata offers a CSV importer that lets us create a database table from that file, with an automatically-inferred schema.

Import the file with the following command. Xata will ask you to set up a new workspace & database (I called my database retrogames but you can name yours as you please), then create a new table in that database named games:

xata import csv games.csv --table=games

Now let's take a look at our data in Xata's web interface.

Browse the dashboard

Open your newly-created database in your browser with the command:

xata browse

This brings us to a spreadsheet-like view of our games data in the Xata dashboard. We can use the dashboard to browse, query, edit, and search our data, view and edit our databases, tables and schemas, and more.

For our app's purposes, these columns of the data are going to be particularly useful:

id: the game's unique ID in the database
name: the game's name
totalRating: the game's average rating on IGDB
cover: URL of the game's cover image

Search all the things!

The "Search" tab lets us try out Xata's built-in full text search capabilities. Whatever search term we enter will be matched against any of the text-based columns in our database; each result is assigned a relevance score, with most relevant results appearing first.

Using Xata's "Boost" feature, we can increase the relevance score of certain results by telling Xata which information we care most about. For example, we can add a numeric booster based on totalRating to have the highest-rated games appear higher in the search results.

To deliver this awesome, customized search experience to our users, we'll use Xata's SDK to interact with our database from our app. On the Search page in the dashboard, click "Get code snippet" to reveal the SDK code for performing that search, including any boosters applied:

// Generated with CLI
import { getXataClient } from "./xata";
const xata = getXataClient();

const records = await xata.search.all("demon", {
  tables: [
    {
      table: "games",
      boosters: [{ numericBooster: { column: "totalRating", factor: 2 } }],
    },
  ],
  fuzziness: 0,
  prefix: "phrase",
});

console.log(records);

We'll use this code later to search our Xata data from our web app. But that app doesn't exist yet, so first we'll need to create it!

Next 13, fresh on the scene

Because life is full of contradictions, we're going to serve up our super vintage games data with the newest, most cutting-edge fullstack web framework we know: Next.js 13, which is only about a week old at the time of writing.

Next.js is beloved by React developers for its server-side rendering capabilities, which make apps more performant by letting us pre-load data and render React components on the server when we can, rather sending all that JS over the wire and making the client to do the heavy lifting on each request.

With v13 and its experimental app/ directory, Next is now changing up the way we think about architecting & fetching data in Next apps, prioritizing server-side rendering as default via React Server Components, while still allowing us to write hook-based Client Components when we need them.

The Next and React features this new architecture is based on are still experimental and may change, but it's worth wrapping our heads around the new direction in which the React/Next community is headed - so let's dive in and try them out!

Bootstrapping Next with `app/` directory

The handy tool create-next-app helps us bootstrap a new Next app, and in the latest version we can try out the experimental new app/ directory.

Back on the command line, create an app with the following command, walking through the prompts to name the project as you please and use TypeScript (or JavaScript if you prefer):

npx create-next-app@latest --experimental-app

Exploring our newly-created directory, we notice the following structure:

app/
  layout.tsx (or .jsx if using JS)
  page.tsx
pages/
  api/
    hello.ts (or .js)

This represents a partial migration to the new app/ directory architecture, in which app/page.tsx will be rendered at our app's / route, using the default layout app/layout.tsx. However, the new app/ directory doesn't yet support API routes, so we're still using the old structure pages/api/hello.ts to create the API route /api/hello. See the Next 13 documentation for more details.

To build our search app, we'll need to write all three of these:

Server Components
Client Components
API routes

In order to fetch the data we'll need for these, we first need to set up the Xata SDK so that our app can talk to the database we created earlier in Xata.

Connect to Xata

In the new directory created by create-next-app, initialize a Xata project with the command:

xata init

In the prompts that follow, select the database you created earlier, and choose to generate TypeScript code (or JS code with ESM if you chose not to use TS) for the Xata client. This will create a xata.ts file at the path you specify; you can use the default path src/xata.ts or change it to e.g. util/xata.ts as I did.

After running the init script, you'll have a .env file with the environment variables the Xata client needs to run, and a xata.ts file which exports a getXataClient() function we can now use to create a client object that can read our database on Xata.

Create a Server Component

By default, components in the app/ directory (like app/page.jsx) are Server Components. Server components are great for parts of our app that don't require client-side interactivity.

Let's try writing our first server component to load & display data about a certain game. We'll make the route for this page /games/[gameID], where [gameID] is a dynamic route segment matching the ID of a game in our dataset, e.g. /games/123 will show info for "Clash at Demonhead".

To make the new route, create a new file at the path app/games/[gameID]/page.tsx. Now, we need to flesh out that file to load data from Xata into a Server Component.

In the new page.tsx file, import & call getXataClient() to instantiate a new Xata client:

// app/games/[gameID]/page.tsx

import { getXataClient } from "../../../util/xata"; // or the path you chose during 'xata init'

const xata = getXataClient();

Now, we can query our data with the Xata SDK by calling methods on xata.db[tableName], in our case xata.db.games.

To retrieve a game by its ID, for example, we can use .read():

const clashAtDemonhead = await xata.db.games.read('123')

Unlike with previous versions of Next, when fetching data in app/ Server Components we don't need anything fancy tricks like getServerSideProps. We can await our data-fetching functions like we would anywhere else!

In page.tsx, export a Page() function that captures the ID from our dynamic gameID route segment from the params object passed in by Next, and passes that ID to the call to .read(). We can then use properties of the game record, e.g. its name, cover image, summary text and console, to render a basic page:

// app/games/[gameID]/page.tsx

import { getXataClient } from "../../../util/xata"; // or the path you chose during 'xata init'

const xata = getXataClient();

export default async function Page({ params }: { params: { gameID: string } }) {
  const game = await xata.db.games.read(params.gameID);

  if (!game) return <h1>Game not found</h1>;

  return (
    <main style={{ padding: "0px 2rem" }}>
      <h1>{game.name}</h1>
      {game.cover && <img src={game.cover} />}
      <p>{game.summary}</p>
      <p>{game.console}</p>
    </main>
  );
}

To see the new page in action, start up a local Next server with the command:

npm run dev

Then navigate to localhost:3000/games/123 and behold your beautiful new server-rendered component, which automatically uses the default layout in app/layout.tsx.

Server Components are great for pages that don't require interactivity, like our game-info page, because they can be more performantly rendered server-side. But what we really wanted in this app was a search page that dynamically loads data based on user input; for that, we're going to need a Client Component.

Create a Client Component

Client Components are what we might think of as "traditional" React components that allow us to power client-side interactions with hooks like useState(). Now that the app/ directory makes Server Components the default, we have to explicitly designate our Client Components with the 'use client'; directive.

Create a new Search Client Component by creating a new file app/search.tsx with 'use client'; on the first line and exporting a function called Search() that renders a basic search input:

// app/search.tsx
"use client";

export default function Search() {
  return (
    <div style={{ marginTop: "2rem" }}>
      <input type="search" />
    </div>
  );
}

We want this component to show up on our app's home page at the / route, so in app/page.tsx import the Search component and use it to replace the default content in the Home component generated by create-next-app. While you're at it, why not change the page heading to something more fun?

// app/page.tsx
import styles from "./page.module.css";
import Search from "./search";

export default function Home() {
  return (
    <div className={styles.container}>
      <main className={styles.main}>
        <h1 className={styles.title}>Search Retro Games!</h1>

        <Search />
      </main>
    </div>
  );
}

Navigate to localhost:3000/ and you'll see the Search Client Component successfully rendered within the Home Server Component and default layout!

Now let's capture what the user types into the search input as state, so that we can later use it to dynamically search our Xata database. Editing app/search.tsx, use the useState() React hook to connect the value of the search input to a piece of state called something like searchTerm:

// app/search.tsx

"use client";

import { useState } from "react";

export default function Search() {
  const [searchTerm, setSearchTerm] = useState("");

  return (
    <div style={{ marginTop: "2rem" }}>
      <input
        type="search"
        value={searchTerm}
        onChange={e => setSearchTerm(e.target.value)}
      />
    </div>
  );
}

Now that we've got a functioning search input connect to our app state, we need to actually send the search terms to Xata to find relevant games!

However, we can't directly call Xata from the client side, because that would potentially expose the secret API key Xata uses to access our database (which we set up during xata auth login and added to our app's .env file during xata init). So we'll need to set up an API route on our server to fetch the data from Xata, and then fetch from that API endpoint from our client component.

Create an API route to return search results

The new app/ directory will eventually support API routes, but as of the time of writing it doesn't yet do so. In the meantime, we can still use the old pages/api/ path structure to create API routes that will seamlessly mesh with the routes in our app/ directory. This is showcased by create-next-app with the automatically generated pages/api/hello.ts, which demonstrates how to handle an API request and return a response with JSON data:

// pages/api/hello.ts (generated by create-next-app)

// Next.js API route support: https://nextjs.org/docs/api-routes/introduction
import type { NextApiRequest, NextApiResponse } from 'next'

type Data = {
  name: string
}

export default function handler(
  req: NextApiRequest,
  res: NextApiResponse<Data>
) {
  res.status(200).json({ name: 'John Doe' })
}

Let's repurpose this file for the /api/search endpoint we need. Rename the file to pages/api/search.ts and navigate to localhost:3000/api/search to see it returning the dummy "John Doe" data.

Now, we can finally use the xata.search.all(...) code snippet we copied from the Xata dashboard earlier to retrieve data based on a given search term, boosting by the game's rating.

We can use the Next request helper req.query to capture the query parameters of the request, so that we can pass the search term to our endpoint via a term parameter like so: /api/search?term=mario

Each object in the records array returned by xata.search.all() has the shape { table: "games", record: { id, name, console, ... }}, but since we don't need the table name, extract just the record of each object by mapping over the array, and send the resulting array as JSON data in the response.

Your finished search.ts file should look something like this:

// pages/api/search.ts
import type { NextApiRequest, NextApiResponse } from "next";
import { getXataClient } from "../../util/xata";

// Instantiate the Xata client
const xata = getXataClient();

// Make the function async to await the search data
export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  // Parse the 'term' query parameter from the request 
  const term = req.query.term as string;

  // Paste the code snippet copied from the Xata search dashboard
  // passing in the search term from the request
  const records = await xata.search.all(term, { 
    tables: [
      {
        table: "games", // Search the 'games' data
        // Prioritize games with a higher 'totalRating'
        boosters: [{ numericBooster: { column: "totalRating", factor: 2 } }],
      },
    ],
    fuzziness: 0,
    prefix: "phrase",
  });

  // Extract the `record` property from the { table, record } objects returned by the search
  res.status(200).json(records.map(r => r.record));
}

Navigate to your new & improved search endpoint to try out different search terms and verify it's working! For example http://localhost:3000/api/search?term=batman.

Now all that remains is to connect our client side code to the new search API route, so that our users can search games from the UI.

Fetch data from the client

The React and Next teams are working on some big changes to our current patterns for fetching data on the server & client. At the time of writing, these new patterns are quite experimental and not quite at the stage where developers can fully take advantage of them.

As we saw earlier in our games/[gameID]/page.tsx component, we can use await to fetch data in React Server Components, but it isn't supported in Client Components. Instead, React's new use() hook is intended to bring await-like functionality to Client Components, so in the future we should be able to asynchronously fetch data with use() in a Client Component as described in the Next docs and React's promises RFC.

However, at the time of writing, the caching functionality that's a necessary complement to use() has not been fully implemented in React/Next yet, and the intended use() pattern seems to create an infinite loop of rerenders and network requests in our Client Component.

So while we eagerly await (ha!) full use (ha!!) of use(), we still need a way to fetch search results from our API endpoint and use it to render a basic list of games (each linking to its corresponding /games/[gameID] page). For now, we can do this by fetching data within a useEffect() and capturing the results with useState(), like so:

// app/search.tsx

"use client";

import { useEffect, useState } from "react";
import Link from "next/link";

// fetch() is not yet supported inside Client Components
// so we wrap it in a utility function
async function getData(term: string) {
  const res = await fetch(`/api/search?term=${term}`);
  return res.json();
}

export default function Search() {
  const [searchTerm, setSearchTerm] = useState(""); // Track the search term
  const [games, setGames] = useState([]); // Track the search results

  useEffect(() => {
    if (searchTerm) {
      // Update the games array once data has loaded
      getData(searchTerm).then(results => setGames(results));
    } else {
      // Reset games if the search term has been cleared
      setGames([]);
    }
  }, [searchTerm]);

  return (
    <>
      <div style={{ marginTop: "2rem" }}>
        <input
          type="search"
          value={searchTerm}
          onChange={e => setSearchTerm(e.target.value)}
        />
      </div>
      {games.map(({ id, name }) => {
        // Render a basic link to the info page for each game
        return (
          <div key={id} style={{ marginTop: "1rem" }}>
            <Link href={`/games/${id}`}>
              <h2>{name}</h2>
            </Link>
          </div>
        );
      })}
    </>
  );
}

Now our search app, while basic, is complete! Reload the home page at localhost:3000/ and enjoy searching through thousands of retro games.

Recap & next steps

In the process of building our app to search retro games data, we've covered a lot of ground! We learned how to:

Import CSV data into Xata
Set up the Xata SDK in a Next project
Use the new app/ directory in Next 13
Build React Server and Client components in app/
Perform full-text search with boosting via the Xata SDK
(Almost) use the new use() hook to fetch data from Client Components

There is still a lot more Xata functionality we haven't had time to explore. Check out search-retro-games.vercel.app for an enhanced version of this project featuring:

Filtering search results by console
Aggregating to count the total number of games
Debounced search input to avoid over-fetching
More details in search results & game pages

Now let's take a break from all this coding, and go play some retro games!

Thanks very much to Xata for sponsoring this work and to retro gaming queen Sara Vieira for making it possible!