Forem: RJ Franco

A simple key-value store with SQLite

RJ Franco — Sat, 15 Feb 2025 17:58:24 +0000

A little context first

Hello folks. One day, when working on a side project, I was on the database design step. I knew from the beginning that I did not want to store my data in a relational format but rather as a simple key-value store (even though the data have relations). The thing is, in my opinion, the best service out there to store data is Turso database which is a SQLite database. Also, my project was about a mobile app, local first but with online syncing, so SQLite too.

I know Expo provides a simple key-value storage (based on SQLite by the way), but the inconsistency with the local and the online storage would bother me. But hey, since Expo made that key-value storage with SQLite, why not create one that works for both local and online data storage ?

Well… let's do it!

Step 1 : Data Schema

What? What do you mean "data schema"? I thought we were going to create a simple key-value storage?

While users of our tool will interact with a simple key-value, schemaless storage, but for us, we will be working with a SQLite database behind the scenes, which requires a strict data schema.

To keep things simple, this is our database schema:

CREATE TABLE IF NOT EXISTS store_name (
    key TEXT PRIMARY KEY,
    value TEXT NOT NULL,
    updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

Things are pretty clear here but let’s get it clearer 😉

CREATE TABLE IF NOT EXISTS is our initialization query. Since there is no need for something like migration, but we still need to make sure our store is created each time our app starts, we need to create the table ourselves (if it does not exist yet).
store_name well, yes 😅 that’s our store name. But a little precision here, each store is in one table. Which means if the user wants to create multiple stores (maybe a multi domain app), the SQLite database will have multiple tables.
key TEXT PRIMARY KEY a key has to be unique and indexed by the database since that’s our unique way to retrieve elements.
value TEXT NOT NULL the value, a string that will be parsed by javascript to json when we get elements, and we stringify any data that will be stored.
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP do you remember when I said that our app will be local first but with online data sync ? This is to prevent conflict (we will use the last data always wins policy)

And that’s pretty it for the schema, let’s continue to the next step.

Step 2 : Queries

We are going to use the exact same queries again and again every time because the user will only call and use our javascript functions. So all the queries that are going to be used should be prepared and optimized from the beginning.

So… CRUD they say

Upsert item

This is something we’ve all seen when working with database system: create a resource if not exists, update it otherwise. Fortunately, SQL let us do this exact action with a simple query:

INSERT INTO store_name (key, value)
VALUES ('some_key', 'some_value')
ON CONFLICT(key) DO UPDATE SET value = excluded.value;

Simple explanation here:

INSERT INTO store_name (key, value) VALUES ('some_key', 'some_value') Create the resource in the database
ON CONFLICT(key) Ensures that if the key already exists, the value is updated instead of throwing an error.
excluded.value Refers to the new value provided in the query.

Get item

This might be the easiest part, a simple SELECT query, the most basic way:

SELECT value FROM store_name WHERE key = 'some_key';

You nailed it, you big SQL developer 👏🏾.

Delete item

I bet you already guessed this one also:

DELETE FROM store_name WHERE key = 'some_key';

Exactly, you 10+ SQL specialist 🫡.

Some extras

Those actions are great and will surely cover 80% of the use cases our users will face, but let’s give them more possibilities:

Multiple upsert

BEGIN TRANSACTION;
INSERT INTO store_name (key, value)
VALUES ('key1', 'value1')
ON CONFLICT(key) DO UPDATE SET value = excluded.value;
INSERT INTO store_name (key, value)
VALUES ('key2', 'value2')
ON CONFLICT(key) DO UPDATE SET value = excluded.value;
COMMIT;

Get all keys in the store

SELECT key FROM store_name LIMIT 50 OFFSET 0;

Get key-values from some specific keys

SELECT key, value FROM store_name WHERE key LIKE 'user:%';

Get all key-value pair (for what reason ? I don’t know, I don’t make the rules)
```
SELECT key, value FROM store_name LIMIT 50 OFFSET 0;
```

Step 3 : SQLite Executor

Now, we can start writing some Typescript code (what you mean Javascript ? We are civilized here).

I will let you configure your project as you want.

As we said, this tool is meant to be used with SQLite but not a specific one, which means the user will have to provide and configure the SQLite adapter. But we will only need the query executor, so the user should provide a function that follows this structure :

export type SQLiteExecutor = <T=unknown>(query:string,params?:unknown[]) => Promise<T>

That’s all, the rest will be handled by us. So let’s write the core of our project.

Step 4 : Store object

As we said earlier, the first thing we should do is to make sure our table exist, but at the same time, we need the user to provide the store name first for our table name. So how do we do that ?

The constructor and the factory

We create the class that takes the executor and the store name.

import { SQLiteExecutor } from './types'

export default class Store {
  private executor: SQLiteExecutor
  private storeName: string

  constructor(executor: SQLiteExecutor, storeName: string) {
    this.executor = executor
    this.storeName = storeName
  }

  async initStore() {
    const init = `
    CREATE TABLE IF NOT EXISTS ${this.storeName} (
      key TEXT PRIMARY KEY,
      value TEXT NOT NULL,
      updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
    );
    `

    await this.executor(init)
  }
}

But the user will never instantiate the class directly, instead, the user will use a factory that returns the store object that will be used all over the app (yes, the signleton pattern you OOP master jedi):

import Store from './store'
import { SQLiteExecutor } from './types'

export async function createStore(executor: SQLiteExecutor, storeName: string): Promise<Store> {
  const store = new Store(executor, storeName)
  await store.initStore()

  return store
}

This way, we make sure everything is fine when the user starts using the store.

The functions

Well, everything is ready for us to write the core functions, so here we go:

import { SQLiteExecutor } from './types'

export default class Store {
  ...
  async set(key: string, value: unknown) {
    const stringValue = JSON.stringify(value)
    const insert = `
    INSERT INTO ${this.storeName} (key, value)
    VALUES (?, ?)
    ON CONFLICT(key) DO UPDATE SET value = excluded.value;
    `

    await this.executor(insert, [key, stringValue])
  }

  async get<T>(key: string): Promise<T | undefined> {
    const select = `SELECT value FROM ${this.storeName} WHERE key = ?;`
    const result = await this.executor<Array<{ value: string }>>(select, [key])
    if (result.length === 0) {
      return undefined
    }
    return JSON.parse(result[0].value) as T
  }

  async delete<T>(key: string) {
    const del = `DELETE FROM ${this.storeName} WHERE key = ?`
    await this.executor<T>(del, [key])
  }

  async instertMultiple(items: Array<{ key: string; value: unknown }>) {
    const insert = `
    INSERT INTO ${this.storeName} (key, value)
    VALUES (?, ?)
    ON CONFLICT(key) DO UPDATE SET value = excluded.value;
    `

    await this.executor('BEGIN TRANSACTION;')

    try {
      for (const { key, value } of items) {
        await this.executor(insert, [key, JSON.stringify(value)])
      }
      await this.executor('COMMIT;')
    } catch (error) {
      await this.executor('ROLLBACK;') // Ensure rollback on failure
      throw error
    }
  }

  async getKeys(options?: { limit: number; offset: number }): Promise<string[]> {
    const optionsString = options ? `LIMIT ${options.limit} OFFSET ${options.offset}` : ''
    const select = `SELECT key FROM ${this.storeName} ${optionsString};`
    const result: Array<{ key: string }> = await this.executor(select)
    return result.map(row => row.key)
  }

  async getAll(options?: { limit: number; offset: number }): Promise<Record<string, unknown>> {
    const optionsString = options ? `LIMIT ${options.limit} OFFSET ${options.offset}` : ''
    const select = `SELECT key, value FROM ${this.storeName} ${optionsString};`
    const result: Array<{ key: string; value: string }> = await this.executor(select)
    return result.reduce(
      (acc, row) => {
        acc[row.key] = JSON.parse(row.value)
        return acc
      },
      {} as Record<string, unknown>
    )
  }

  async getSubFields<T>(prefix: string): Promise<Array<{ key: string; value: T }>> {
    const select = `SELECT key, value FROM ${this.storeName} WHERE key LIKE ?;`
    const result: Array<{ key: string; value: T }> = await this.executor(select, [`${prefix}%`])
    return result
  }
}

You can read the code, or just copy past it, it’all yours anyway now.

Ending up

Thank you for following me throughout this journey of madness, breaking the sacred rule of « don’t re-invent the wheel ». If you want to play with the code, you can find it here on Github.

Now, I am going to continue what I was doing, but feel free to give me feed back if you are planning to use the same kv system based on any SQL database too, it would be great to share knowledges like this.
’Till this moment, see you folks.

Bonus part

Of course we can do more, but the main part is already done so you can work with what we did. But if you want more, we can be more generous by giving our users a more complete solution to work with. So let’s create the adapter for the main usage our user may face : expo-sqlite adapter, libsql (Turso) adapter and better-sqlite3 adapter for node.js.

Adapter Factory

A simple type to generate any executor:

export type ExecutorFactory<T> = (db: T) => SQLiteExecutor

And this is the type we are going to use for every adapters:

Adapters

expo-sqlite

import * as ExpoSQLite from 'expo-sqlite'

import { ExecutorFactory } from '../types'

export const createExpoExecutor: ExecutorFactory<ExpoSQLite.SQLiteDatabase> = db => {
  return <T>(query: string, params: unknown[] = []) => {
    if (query.trim().toUpperCase().startsWith('SELECT')) {
      return db.getAllAsync(query, params as ExpoSQLite.SQLiteBindValue[]) as T
    } else {
      return db.runAsync(query, params as ExpoSQLite.SQLiteBindValue[]) as T
    }
  }
}

libsql (turso)

Note that for libsql, we can have an adapter for Node and for edge runtimes (whatever that means):

import * as LibSQLNode from '@libsql/client'
import * as LibSQLWeb from '@libsql/client/web'

import { ExecutorFactory } from '../types'

const createExecutor = <DB extends LibSQLNode.Client>(db: DB) => {
  return async <T>(query: string, params: unknown[] = []) => {
    const result = await db.execute({ sql: query, args: params as LibSQLNode.InArgs })

    return result.rows.map(row => {
      const res: Record<string, unknown> = {}
      Object.keys(row).forEach(key => {
        if (key !== 'length' && typeof key !== 'number') res[key] = row[key]
      })
      return res
    }) as T
  }
}

export const createLibSQLNodeExecutor: ExecutorFactory<LibSQLNode.Client> = createExecutor
export const createLibSQLEdgeExecutor: ExecutorFactory<LibSQLWeb.Client> = createExecutor

better-sqlite3

import BetterSQLite3 from 'better-sqlite3'

import { ExecutorFactory } from '../types'

export const createBetterSqliteExecutor: ExecutorFactory<BetterSQLite3.Database> = db => {
  return <T>(query: string, params: unknown[] = []) => {
    return new Promise<T>((resolve, reject) => {
      try {
        const stmt = db.prepare(query)
        let result
        if (query.trim().toUpperCase().startsWith('SELECT')) {
          result = stmt.all(params)
        } else {
          result = stmt.run(params)
        }
        resolve(result as T)
      } catch (error) {
        reject(error)
      }
    })
  }
}

Conclusion ?

Well, I already did this part. Note that I’m still active in evolving this project because it’s a tool that I (’m planning to) personaly use so you can check the repo here on Github(You can even offer to contribue 😉)

Re-inventing the wheel - Use list

RJ Franco — Sat, 11 May 2024 12:03:35 +0000

This is the first of a series where we'll try to re-invent the wheel. And yes, that's exactly what people told you not to do but we are rebels 😉.

For the project scafolding, I just made a simple pnpm monorepos, but you can use any configuration you want. The only thing that matter is that you have a fully functional react project. This serie is not about that, but if your're interested, you can follow the same tuto I followed: https://dev.to/vinomanick/create-a-monorepo-using-pnpm-workspace-1ebn
But if you absolutely want to follow my config, you can clone the repos here : https://github.com/FrancoRATOVOSON/use-wheel

Introduction

In the vast landscape of the React ecosystem, we're spoiled for choice with a plethora of libraries and frameworks promising to streamline our development workflows. Yet, as projects grow in complexity, we often find ourselves wrestling with bloated dependencies and intricate systems that exceed our needs.

This dilemma was all too familiar to me when I embarked on a recent side project. As I tinkered with various tools and libraries, I realized that many of my requirements could be fulfilled with simpler, more focused solutions. And so, the idea for this series was born - a journey back to basics, where we strip away the excess and embrace the elegance of simplicity.

But this isn't just about reinventing the wheel for the sake of it. It's about rediscovering the joy of creation and learning through hands-on exploration. By distilling complex concepts into their fundamental components, we gain a deeper understanding of how things work and uncover new possibilities for innovation.

What are we cooking today ?

So, let's embark on this journey together, starting with something as ubiquitous as it is indispensable: lists. From displaying users to showcasing products, lists form the backbone of countless applications. While existing solutions like useList and @tastack/table (which is more related to table, I know), offer compelling features, what if we could craft something tailored precisely to our needs?

Join me as we delve into the art of list management in React, with the aim of creating a lightweight, versatile utility that puts you in control. By the end of this series, you'll have a newfound appreciation for the power of simplicity - and a practical tool to enhance your development workflow.

The requirements

Alright, let's get down to business! When it comes to taming our wild lists, we've got a checklist of essential features:

Ordering (Sorting): Ever tried navigating a messy stack of data? Ain't nobody got time for that! We need a way to sort our lists and bring some order to the chaos.
Filtering: Say goodbye to information overload! We want the power to filter out the noise and focus on what truly matters.
Displaying (Pagination & Page Size): Paging through endless pages of results? No thanks! Let's break it down into bite-sized chunks and make navigation a breeze.
Selection: Sometimes, you gotta pick and choose! Whether it's selecting multiple items or zeroing in on that one special nugget of data, we need tools to get the job done.

Now that we've got our wish list laid out, it's time to roll up our sleeves and get to work. Behold, the birth of our trusty "useList" utility!

// useList.ts

export function useList<T>(data: Array<T>) {
  return {
    list: data
      .filter(() => true) // Placeholder for filtering logic
      .sort(() => 0)      // Placeholder for sorting logic
      .slice()            // Placeholder for pagination logic
  }
}

What are we seeing here ? Well, we've whipped up a generic function that takes an array of data and returns a neat little package with a "list" property. This property holds a filtered, sorted, and sliced version of our original data array.

But hold your horses, we're not done just yet! Sure, we've got the basics in place, but now it's time to sprinkle in some real logic. We're talking sorting, filtering, pagination and selection . So grab your thinking caps and let's sprinkle some coding magic on this. It's gonna be a wild ride! 🎩✨

Let's make the easier first.

Selection

Alright folks, it's time to talk selection! When it comes to choosing items from our list, we've got a couple of options up our sleeves:

User-Specified Unique IDs: One option is to put the ball in the user's court and ask them to provide unique IDs for each element. This gives them full control over how selections are identified.
Flexible Selection Mechanism:: Alternatively, we could let users decide how the selection process works. That way, they can tailor it to fit their specific needs and preferences.

But there's another option, we'll take charge of the selection mechanism, but we'll let users choose how they want to identify each element. How? By providing a nifty little function that does the trick:

type UseListOptions<T, U> = {
  getId: (element: T) => U
}

This way, users can specify their own way of identifying elements. If the data they provide already has a unique ID, great! If not, they can customize the function to generate one.

Now, let's dive into the implementation! All we need to do is add a selection list and a function to toggle selections on and off:

export function useList<T, U>(data: Array<T>, { getId }: UseListOptions<T, U>) {
  const [selection, setSelection] = React.useState<Set<U>>(new Set([]))

  const toogleSelection = React.useCallback(
    (item: T, state?: boolean) =>
      setSelection(currentSelection => {
        const itemId = getId(item)
        const selectionList = new Set(currentSelection)
        if (state || !selectionList.has(itemId)) selectionList.add(itemId)
        else selectionList.delete(itemId)
        return new Set(selectionList)
      }),
    [getId]
  )

  return {
    list: data
      .filter(() => true)
      .sort(() => 0)
      .slice(),
    selection,
    toogleSelection
  }
}

Did we forget something ? Mmh yeah, a too usefull feature to not offer :

const toogleSelectionAll = React.useCallback(
  (state?: boolean) =>
    setSelection(currentSelection => {
      if (state || currentSelection.size === 0)
        return new Set(data.map(item => getId(item)))
      const selectionList = new Set(currentSelection)
      selectionList.clear()
      return selectionList
    }),
  [data, getId]
)

And that's a wrap, we've covered all the bases. So, what's next on the agenda?

Sorting & Filtering

Alright, folks, time to tackle sorting and filtering! And guess what? It's even easier than selection because the user provides the functions, and we just work our magic.

Let's update our code :

type UseListOptions<T, U> = {
  filterFn: (element: T) => boolean
  getId: (element: T) => U
  sortFn: (elementA: T, elementB: T) => number
}

export function useList<T, U>(
  data: Array<T>,
  { filterFn, getId, sortFn }: UseListOptions<T, U>
) {

  ...

  return {
    list: data.filter(filterFn).sort(sortFn).slice(),
    selection,
    toggleSelection
  }
}

Now you may ask « Does the order of filtering and sorting matter? » And the answer is yes - mainly for performance reasons. We filter the data first to potentially reduce the array size before sorting kicks in. However, the slice operation comes last because it's all about pagination. We filter and sort the entire dataset, not just what's displayed. This ensures consistency even if the displayed items change based on sorting and filtering parameters.

But hey, let's not stop there! We'll delve deeper into optimization and code cleanliness later in the article.

With these tweaks in mind, our sorting and filtering feature is ready to shine! Stay tuned as we dive deeper into the wonderful world of list management in React.

Pagination

Now, let's get the party started 😎.

Pagination may sound simple, but trust me, there's more to it than meets the eye. First off, we need to keep track of our current spot (aka the current page), figure out how many items to display at once (page size), and have some handy functions to switch between pages (first, previous, next, last, and even go-to!).

Let's start simple :

type UseListOptions<T, U> = {
  defaultPageSize: number
  ...
}

export function useList<T, U>(
  data: Array<T>,
  { defaultPageSize, filterFn, getId, sortFn }: UseListOptions<T, U>
) {
  const [pageSize, setPageSize] = React.useState<number>(defaultPageSize)
  const [index, setIndex] = React.useState<number>(0)

  ...

  return {
    list: data
      .filter(filterFn)
      .sort(sortFn)
      .slice(index, index + pageSize),
    ...
  }
}

Pretty simple, right? Now, let's tackle determining the current page:

export function useList<T, U>(
  data: Array<T>,
  { defaultPageSize, filterFn, getId, sortFn }: UseListOptions<T, U>
) {
  ...

  const currentPage = React.useMemo(
    () => index / pageSize + 1,
    [index, pageSize]
  )

  return {
    ...
    currentPage,
  }
}

That's pretty easy, the current page the current index divided by the page size and we add 1 because we start from 1. Now, this statement assume that the index is allways a multiple of the page size. But that makes sense, everytime we change a page, we add to index the pageSize.

Now, let's ensure smooth navigation between pages:

const nextPage = React.useCallback(() => {
  if (pageSize < data.length && index + pageSize < data.length)
    setIndex(currentIndex => currentIndex + pageSize)
}, [data.length, index, pageSize])

First, we can go to nextPage only if there's still data to show. To make that sure, the page size must be smaller to the length of the data, and when we increment the index, it's still in the data array.
With such logic, we know that the index will allways be a multiple of the page size since it can be 0, pageSize, pageSize + pageSize, …
But what if we suddenly change the page size ? Well, as long as we're in the first page,, it doesn't matter. But if we're somewhere in the list, not in the first page, we'll going to need to re-calculate what should be the current page and what should be displayed.

Or, we can choose to ignore the problem and just step back to the first page when we change the page size 😜

Mmh yeah, let's first complete our navigation and we can go back here later.
So, previous, first and last page :

const previousPage = React.useCallback(() => {
  if (pageSize < data.length && index - pageSize >= 0)
    setIndex(currentIndex => currentIndex - pageSize)
}, [data.length, index, pageSize])

const firstPage = React.useCallback(() => setIndex(0), [])

const lastPage = React.useCallback(() => {
  if (pageSize < data.length && index + pageSize < data.length) {
    const pageCount = Math.ceil(data.length / pageSize)
    setIndex(pageSize * pageCount)
  }
}, [data.length, index, pageSize])

Did you notice it ? We created another crutial information : the pageCount. We should return it also.

Let's change the code:

const pageCount = React.useMemo(() => {
  if (pageSize >= data.length) return 1
  return Math.ceil(data.length / pageSize)
}, [data.length, pageSize])

...

const lastPage = React.useCallback(() => {
  if (pageSize < data.length && index + pageSize < data.length)
    setIndex(pageSize * (pageCount - 1))
}, [data.length, index, pageCount, pageSize])

Let's add a bonus feature before the setPageSize implementation :

const goToPage = React.useCallback(
  (destinationPage: number) => {
    if (destinationPage < pageCount)
      setIndex(pageSize * (destinationPage - 1))
  },
  [pageCount, pageSize]
)

As you can see, it's almost the same as the lastPage. We'll clean that up later.

Set page size

Ah, here's where things get interesting! When a user decides to change the size of the page, it's usually because they want to see more information at once. However, if they're already browsing through multiple pages, what should take priority: maintaining their current page or displaying more elements? Well, that's entirely up to us, but from a user's perspective, losing track of previously found information just because they changed the view size isn't ideal. Plus, they might not even care about which page they're on at the moment.

The currently diplayed informations are between the current index and the current page size. So when the user updates the page size, we also need to adjust the current index to a value that, when adding the page size to it, include the last index.

This is the easiest way to do that :

const setPageSize = React.useCallback(
  (size: number) => {
    setPageSizeState(size)
    const newIndex = Math.floor(index / size) * size
    setIndex(newIndex)
  },
  [index]
)

I know, it seems easier than I make it feels 😛. Because the simplicity of a solution doesn't hide the complexity of the problem to solve. Let's explain the code now:
First, we set the page size to the new page size. Simple enough, right? But now comes the tricky part: why use Math.floor instead of Math.ceil? And what exactly are we calculating here?

Consider this: any element displayed lies between index and index + pageSize. In other words:

Elements in page 1 are between 0 and pageSize
Elements in page 2 are between pageSize and pageSize * 2
Elements in page 3 are between pageSize * 2 and pageSize * 3
And so on…

In general, elements in page n are between (pageSize * (n-1)) and (pageSize * n).By reversing this calculation, we find that n is always less than or equal to (i / pageSize) + 1. But why not "greater than" or "equal"? Well, while this formulation might be mathematically correct, it's important to remember that i starts at 0 while page indices start at 1. And since the current index represents the current page multiplied by the page size, we multiply the result to find the correct index.

Phew! That's quite the mathematical journey, but understanding these intricacies helps ensure our pagination system runs smoothly, no matter how users interact with it.

The test

Setting up

Now that we've crafted our pagination system, it's time to put it to the test! We'll create a simple list of user orders and explore various functionalities such as sorting by date or amount, filtering by email, and toggling order states (delivered or not). To populate our data, we'll utilize faker.js, a handy tool for generating realistic test data.

First, let's take a look at how we generate our data:

// I created a list of users to hold all the orders we create.
// To prevent each order from belonging to an individual user, 
// this is for the search filter feature.
function fakeUserList() {
  const list: string[] = []
  for (let index = 0; index < 10; index++) {
    list.push(faker.internet.email())
  }
  return list
}

export function fakeOrdersList() {
  const userList = fakeUserList()
  const list: Order[] = []
  const dataSize = faker.number.int({ max: 100, min: 50 })
  for (let index = 0; index < dataSize; index++) {
    list.push({
      amount: faker.number.int({ max: 500, min: 150 }),
      date: faker.date.recent({ days: 100 }),
      id: faker.string.uuid(),
      isDelivered: faker.number.int({ max: 10, min: 0 }) % 2 === 0,
      user: userList[faker.number.int({ max: userList.length - 1, min: 0 })]
    })
  }

  return list
}

typescript

Now, let's set up our parent component:

import React from 'react'

import { OrderList } from '@/components/common'
import { fakeOrdersList } from '@/lib/faker'

export default function OrdersPage() {
  const [orders, setOrders] = React.useState(fakeOrdersList())

  const deleteOrders = React.useCallback(
    (ids: string[]) =>
      setOrders(list => list.filter(element => !ids.includes(element.id))),
    []
  )

  return <OrderList deleteOrders={deleteOrders} list={orders} />
}

And then, the list component :

// import statements...

interface OrderListProps {
  deleteOrders: (ids: string[]) => void
  list: Array<Order>
}

export default function OderList({ deleteOrders, list: data }: OrderListProps) {
  const [search, setSearch] = React.useState<string>('')
  const [sort, setSort] = React.useState<'asc' | 'desc' | null>(null)
  const {
    currentPage,
    firstPage,
    lastPage,
    list,
    nextPage,
    pageCount,
    pageSize,
    previousPage,
    selection,
    setPageSize,
    toogleSelection,
    toogleSelectionAll
  } = useList(data, {
    defaultPageSize: 5,
    filterFn: order => {
      if (search) return order.user.toLowerCase().includes(search.toLowerCase())
      return true
    },
    getId: elt => elt.id,
    sortFn: (a, b) => {
      if (!sort) return 0
      if (sort === 'asc') return a.amount - b.amount
      return b.amount - a.amount
    }
  })

  return (
    <div className="space-y-4">
      <div className="flex justify-between gap-2 items-center">
        <div className="flex justify-start items-center gap-2">
          <div className="relative ml-auto flex-1 md:grow-0">
            <Search className="absolute left-3 top-3 h-4 w-4 text-muted-foreground" />
            <Input
              className="w-full rounded-lg bg-background pl-9 md:w-[200px] lg:w-[336px]"
              placeholder="Search..."
              type="search"
              onChange={e => setSearch(e.target.value)}
            />
          </div>
          <div>
            <DropdownMenu>
              <DropdownMenuTrigger asChild>
                <Button variant="outline">
                  {sort === 'asc' ? (
                    <>
                      <ArrowUpNarrowWide className="size-4 mr-4" />
                      Asc
                    </>
                  ) : sort === 'desc' ? (
                    <>
                      <ArrowDownNarrowWide className="size-4 mr-4" />
                      Desc
                    </>
                  ) : (
                    <>
                      <ArrowUpDown className="size-4 mr-4" />
                      Sort
                    </>
                  )}
                </Button>
              </DropdownMenuTrigger>
              <DropdownMenuContent className="w-28 h-fit p-2 bg-background border rounded-md mt-2">
                <DropdownMenuRadioGroup
                  value={sort || undefined}
                  onValueChange={value => setSort(value as 'asc' | 'desc')}
                >
                  <DropdownMenuRadioItem className="cursor-pointer" value="asc">
                    Asc
                  </DropdownMenuRadioItem>
                  <DropdownMenuRadioItem
                    className="cursor-pointer"
                    value="desc"
                  >
                    Desc
                  </DropdownMenuRadioItem>
                </DropdownMenuRadioGroup>
              </DropdownMenuContent>
            </DropdownMenu>
          </div>
          {search && (
            <div>{`${data.filter(order => order.user.toLowerCase().includes(search.toLowerCase())).length} results`}</div>
          )}
        </div>
        <div className="flex justify-end gap-2 items-center">
          <div className="text-muted-foreground text-nowrap">Rows per page</div>
          <Select
            value={`${pageSize}`}
            onValueChange={value => setPageSize(Number(value))}
          >
            <SelectTrigger className="w-16">
              <SelectValue placeholder={`${pageSize}`} />
            </SelectTrigger>
            <SelectContent side="bottom">
              {[5, 10, 15, 20, 30, 50].map(size => (
                <SelectItem key={`pageSize-${size}`} value={`${size}`}>
                  {size}
                </SelectItem>
              ))}
            </SelectContent>
          </Select>
        </div>
      </div>
      <div className="space-y-4">
        {list.map(order => {
          const { amount, date, id, isDelivered, user } = order
          return (
            <Card
              className={cn('p-4', 'flex items-center justify-between')}
              key={id}
            >
              <div className="flex item-center gap-2">
                <div className="size-9 grid content-center ml-2">
                  <Checkbox
                    checked={selection.has(id)}
                    id={user}
                    onCheckedChange={state =>
                      toogleSelection(
                        order,
                        typeof state === 'string' ? false : state
                      )
                    }
                  />
                </div>
                <label
                  className={cn(
                    'flex flex-col gap-1',
                    'font-medium leading-none',
                    'cursor-pointer'
                  )}
                  htmlFor={user}
                >
                  <span className="font-semibold">{user}</span>
                  <span className="text-sm text-muted-foreground">
                    {format(date, 'PPP')}
                  </span>
                </label>
                <div>{isDelivered && <Badge>delivered</Badge>}</div>
              </div>
              <div>{`${amount} $`}</div>
            </Card>
          )
        })}
      </div>
      <div className="flex justify-between items-center">
        <div className="flex justify-start items-center gap-4">
          <div className="flex flex-col gap-0">
            <div className="flex gap-2 justify-start items-center">
              <Checkbox
                checked={selection.size > 0}
                id="all-users-selection"
                onCheckedChange={state =>
                  toogleSelectionAll(typeof state === 'string' ? false : state)
                }
              />
              <label
                className="text-muted-foreground cursor-pointer"
                htmlFor="all-users-selection"
              >
                {selection.size > 0 ? `Unselect all` : `Select all`}
              </label>
            </div>
            {data && (
              <p>{`${selection.size} of ${data.length} user(s) selected.`}</p>
            )}
          </div>
          {selection.size > 0 && (
            <Button
              variant="destructive"
              onClick={() => deleteOrders(Array.from(selection))}
            >
              Delete selected users
            </Button>
          )}
        </div>
        <div className={cn('flex justify-start items-center gap-2')}>
          <Button
            disabled={currentPage === 0}
            size="icon"
            variant="outline"
            onClick={firstPage}
          >
            <DoubleArrowLeftIcon className="size-4" />
          </Button>
          <Button
            disabled={currentPage === 0}
            size="icon"
            variant="outline"
            onClick={previousPage}
          >
            <ChevronLeftIcon className="size-4" />
          </Button>
          <div className="text-muted-foreground">
            {`Page ${currentPage} of ${pageCount}`}
          </div>
          <Button
            disabled={currentPage >= pageCount}
            size="icon"
            variant="outline"
            onClick={nextPage}
          >
            <ChevronRightIcon className="size-4" />
          </Button>
          <Button
            disabled={currentPage >= pageCount}
            size="icon"
            variant="outline"
            onClick={lastPage}
          >
            <DoubleArrowRightIcon className="size-4" />
          </Button>
        </div>
      </div>
    </div>
  )
}

I know, that's a lot, and I could've slpit the code but clean code is not the purpose of this article 😝.

Without waiting, let's try

Let's jump right into testing our example! Everything should work smoothly at first glance. Try selecting some elements, unselecting all, selecting all, sorting in ascending and descending order, changing the page size, and navigating through pages. Yes! everything goes well, untill you search.

Do you notice the issue? We're supposed to have only 9 elements displayed. This means that page 2 should contain only 4 elements (which it does), and it should be the last page. However, that's not the case. When we filter elements, the pagination logic doesn't follow the current state of the list. As a result, navigating to other pages reveals nothing.

Now, let's rectify these errors and tidy up our code:

Last sprint

First, fix the issues

The solution is surprisingly straightforward:

export function useList<T, U>(
  dataList: Array<T>,
  { defaultPageSize, filterFn, getId, sortFn }: UseListOptions<T, U>
) {
  const data = React.useMemo(
    () => dataList.filter(filterFn ?? (() => true)),
    [dataList, filterFn]
  )
  ...
}

Now, there's no need to rename a lot of things. We simply base our pagination and other functionalities, like selection, on the filtered list.
Now, rerun the example and observe the changes:

You can even try selecting all, and you'll notice that only the filtered results will be selected. However, if you delete all selected values, you'll notice that the "select all" checkbox remains checked. This is because we don't update the selection list when the data changes.
Let's address that:

export function useList<T, U>(
  dataList: Array<T>,
  { defaultPageSize, filterFn, getId, sortFn }: UseListOptions<T, U>
) {
  ...

  React.useEffect(() => {
    setSelection(new Set([]))
  }, [dataList.length])

  ...
}

On each render, if the length of the original data changes, we clear the selection list.

Yes, our hook is indeed becoming more extensive and complex. Now, let's proceed to clean up the code.
(Yes, I know, it's still not about clean code, but come on! 🙄)

Cleaning up

First, let's separate each feature into its own function. We'll start with the selection feature. Create a file for selection and paste the following code inside:

export function useListSelection<T, U>(
  data: Array<T>,
  getId: (element: T) => U
) {
  const [selection, setSelection] = React.useState<Set<U>>(new Set([]))

  React.useEffect(() => {
    setSelection(new Set([]))
  }, [data.length])

  const toogleSelection = React.useCallback(
    (item: T, state?: boolean) =>
      setSelection(currentSelection => {
        const itemId = getId(item)
        const selectionList = new Set(currentSelection)
        if (state || !selectionList.has(itemId)) selectionList.add(itemId)
        else selectionList.delete(itemId)
        return new Set(selectionList)
      }),
    [getId]
  )

  const toogleSelectionAll = React.useCallback(
    (state?: boolean) =>
      setSelection(currentSelection => {
        if (state || currentSelection.size === 0)
          return new Set(data.map(item => getId(item)))
        const selectionList = new Set(currentSelection)
        selectionList.clear()
        return selectionList
      }),
    [data, getId]
  )

  return { selection, toogleSelection, toogleSelectionAll }
}

Now, let's tackle pagination. Create the file and paste the code below:

export function useListPagination<T>(data: Array<T>, defaultPageSize: number) {
  const [pageSize, setPageSizeState] = React.useState<number>(defaultPageSize)
  const [index, setIndex] = React.useState<number>(0)

  const currentPage = React.useMemo(
    () => index / pageSize + 1,
    [index, pageSize]
  )

  const pageCount = React.useMemo(() => {
    if (pageSize >= data.length) return 1
    return Math.ceil(data.length / pageSize)
  }, [data.length, pageSize])

  const nextPage = React.useCallback(() => {
    if (pageSize < data.length && index + pageSize < data.length)
      setIndex(currentIndex => currentIndex + pageSize)
  }, [data.length, index, pageSize])

  const previousPage = React.useCallback(() => {
    if (pageSize < data.length && index - pageSize >= 0)
      setIndex(currentIndex => currentIndex - pageSize)
  }, [data.length, index, pageSize])

  const firstPage = React.useCallback(() => setIndex(0), [])

  const lastPage = React.useCallback(() => {
    if (pageSize < data.length && index + pageSize < data.length)
      setIndex(pageSize * (pageCount - 1))
  }, [data.length, index, pageCount, pageSize])

  const goToPage = React.useCallback(
    (destinationPage: number) => {
      if (destinationPage < pageCount)
        setIndex(pageSize * (destinationPage - 1))
    },
    [pageCount, pageSize]
  )

  const setPageSize = React.useCallback(
    (size: number) => {
      setPageSizeState(size)
      const newIndex = Math.floor(index / size) * size
      setIndex(newIndex)
    },
    [index]
  )

  return {
    currentPage,
    firstPage,
    goToPage,
    index,
    lastPage,
    nextPage,
    pageCount,
    pageSize,
    previousPage,
    setPageSize
  }
}

I know, it's still pretty big but that's life, you don't allways get what you want 🤷🏾‍♂️.
Now, let's update our main hook:

export function useList<T, U>(
  dataList: Array<T>,
  { defaultPageSize, filterFn, getId, sortFn }: UseListOptions<T, U>
) {
  const data = React.useMemo(
    () => dataList.filter(filterFn ?? (() => true)),
    [dataList, filterFn]
  )

  const selection = useListSelection(data, getId)
  const { index, pageSize, ...pagination } = useListPagination(
    data,
    defaultPageSize
  )

  const list = React.useMemo(
    () => [...data].sort(sortFn).slice(index, index + pageSize),
    [data, index, pageSize, sortFn]
  )

  return {
    list,
    pageSize,
    ...pagination,
    ...selection
  }
}

If you notice, the "select all" checkbox is unchecked when the size of the data changes (such as when deleting), but also when filtering. This is because the useEffect we use to clear the selection is based on the filtered data, not the original. We can keep this behavior, or keep our selection regardless of the search or any filter the user applies.

If we keep the current behavior, the selection will be based exclusively on the data returned by our hooks, but the old behavior can work on data that are not part of what the end user decided to see, which can lead to unpredictable behavior. For UX and security, our current implementation is better. However, if you're not convinced, remove the useEffect from the selection hook, put it back in our useList function, and replace the dependency with dataList.length.

Last feature

Now, the final consideration is offering users the flexibility to choose which features they want to utilize. Users may not always require sorting, filtering, or selection capabilities. Therefore, we should allow these features to be optional. Let's delve into the implementation.

type UseListOptions<T, U> = {
  defaultPageSize?: number
  filterFn?: (element: T) => boolean
  getId?: (element: T) => U
  sortFn?: (elementA: T, elementB: T) => number
}

export function useList<T, U>(
  dataList: Array<T>,
  { defaultPageSize, filterFn, getId, sortFn }: UseListOptions<T, U>
) {
  const data = React.useMemo(
    () => dataList.filter(filterFn ?? (() => true)),
    [dataList, filterFn]
  )

  const selection = useListSelection(data, getId ?? (() => ({}) as U))
  const { index, pageSize, ...pagination } = useListPagination(
    data,
    defaultPageSize || 5
  )

  const list = React.useMemo(() => {
    if (!sortFn && !defaultPageSize) return [...data]
    if (sortFn && !defaultPageSize) return [...data].sort(sortFn)
    if (!sortFn && defaultPageSize) return data.slice(index, index + pageSize)

    return [...data].sort(sortFn).slice(index, index + pageSize)
  }, [data, defaultPageSize, index, pageSize, sortFn])

  return {
    list,
    ...(defaultPageSize && { pageSize }),
    ...(defaultPageSize && pagination),
    ...(getId && selection)
  }
}

A quick insight: Users must provide a value for each feature they intend to use. For instance, defaultPageSize for pagination, filterFn for filtering, sortFn for sorting, and getId for selection. If a user does not provide a value for a parameter, the respective feature is disabled.

Now, you might wonder, "What if users provide an empty object?" Fear not! TypeScript comes to our rescue. To ensure that users provide at least one parameter, we've created a utility type called AtLeastOne. Here's how it looks:

type AtLeastOne<T, U = { [K in keyof T]: Pick<T, K> }> = Partial<T> & U[keyof U]

type UseListOptions<T, U> = {
  defaultPageSize: number
  filterFn: (element: T) => boolean
  getId: (element: T) => U
  sortFn: (elementA: T, elementB: T) => number
}

type UseListParamsType<T, U> = AtLeastOne<UseListOptions<T, U>>

By using the UseListParamsType as the type of the second parameter of our hook, users can easily pick and choose the functionalities they need. Voila! Customizable functionality achieved.

It's time to fix our errors in the example code (you can also try to remove some params to see what happens):

return (
  <div className="space-y-4">
    <div className="flex justify-between gap-2 items-center">
      <div className="flex justify-start items-center gap-2">
        <div className="relative ml-auto flex-1 md:grow-0">
          <Search className="absolute left-3 top-3 h-4 w-4 text-muted-foreground" />
          <Input
            className="w-full rounded-lg bg-background pl-9 md:w-[200px] lg:w-[336px]"
            placeholder="Search..."
            type="search"
            onChange={e => setSearch(e.target.value)}
          />
        </div>
        <div>
          <DropdownMenu>
            <DropdownMenuTrigger asChild>
              <Button variant="outline">
                {sort === 'asc' ? (
                  <>
                    <ArrowUpNarrowWide className="size-4 mr-4" />
                    Asc
                  </>
                ) : sort === 'desc' ? (
                  <>
                    <ArrowDownNarrowWide className="size-4 mr-4" />
                    Desc
                  </>
                ) : (
                  <>
                    <ArrowUpDown className="size-4 mr-4" />
                    Sort
                  </>
                )}
              </Button>
            </DropdownMenuTrigger>
            <DropdownMenuContent className="w-28 h-fit p-2 bg-background border rounded-md mt-2">
              <DropdownMenuRadioGroup
                value={sort || undefined}
                onValueChange={value => setSort(value as 'asc' | 'desc')}
              >
                <DropdownMenuRadioItem className="cursor-pointer" value="asc">
                  Asc
                </DropdownMenuRadioItem>
                <DropdownMenuRadioItem
                  className="cursor-pointer"
                  value="desc"
                >
                  Desc
                </DropdownMenuRadioItem>
              </DropdownMenuRadioGroup>
            </DropdownMenuContent>
          </DropdownMenu>
        </div>
        {search && (
          <div>{`${data.filter(order => order.user.toLowerCase().includes(search.toLowerCase())).length} results`}</div>
        )}
      </div>
      {pageSize && setPageSize && (
        <div className="flex justify-end gap-2 items-center">
          <div className="text-muted-foreground text-nowrap">
            Rows per page
          </div>
          <Select
            value={`${pageSize}`}
            onValueChange={value => setPageSize?.(Number(value))}
          >
            <SelectTrigger className="w-16">
              <SelectValue placeholder={`${pageSize}`} />
            </SelectTrigger>
            <SelectContent side="bottom">
              {[5, 10, 15, 20, 30, 50].map(size => (
                <SelectItem key={`pageSize-${size}`} value={`${size}`}>
                  {size}
                </SelectItem>
              ))}
            </SelectContent>
          </Select>
        </div>
      )}
    </div>
    <div className="space-y-4">
      {list.map(order => {
        const { amount, date, id, isDelivered, user } = order
        return (
          <Card
            className={cn('p-4', 'flex items-center justify-between')}
            key={id}
          >
            <div className="flex item-center gap-2">
              {selection && toogleSelection && (
                <div className="size-9 grid content-center ml-2">
                  <Checkbox
                    checked={selection.has(id)}
                    id={user}
                    onCheckedChange={state =>
                      toogleSelection(
                        order,
                        typeof state === 'string' ? false : state
                      )
                    }
                  />
                </div>
              )}
              <label
                className={cn(
                  'flex flex-col gap-1',
                  'font-medium leading-none',
                  'cursor-pointer'
                )}
                htmlFor={user}
              >
                <span className="font-semibold">{user}</span>
                <span className="text-sm text-muted-foreground">
                  {format(date, 'PPP')}
                </span>
              </label>
              <div>{isDelivered && <Badge>delivered</Badge>}</div>
            </div>
            <div>{`${amount} $`}</div>
          </Card>
        )
      })}
    </div>
    <div className="flex justify-between items-center">
      {selection && (
        <div className="flex justify-start items-center gap-4">
          <div className="flex flex-col gap-0">
            <div className="flex gap-2 justify-start items-center">
              {toogleSelectionAll && (
                <Checkbox
                  checked={selection.size > 0}
                  id="all-users-selection"
                  onCheckedChange={state =>
                    toogleSelectionAll(
                      typeof state === 'string' ? false : state
                    )
                  }
                />
              )}
              <label
                className="text-muted-foreground cursor-pointer"
                htmlFor="all-users-selection"
              >
                {selection.size > 0 ? `Unselect all` : `Select all`}
              </label>
            </div>
            {data && (
              <p>{`${selection.size} of ${data.length} user(s) selected.`}</p>
            )}
          </div>
          {selection.size > 0 && (
            <Button
              variant="destructive"
              onClick={() => deleteOrders(Array.from(selection))}
            >
              Delete selected users
            </Button>
          )}
        </div>
      )}
      {currentPage && pageCount && (
        <div className={cn('flex justify-start items-center gap-2')}>
          <Button
            disabled={currentPage === 0}
            size="icon"
            variant="outline"
            onClick={firstPage}
          >
            <DoubleArrowLeftIcon className="size-4" />
          </Button>
          <Button
            disabled={currentPage === 0}
            size="icon"
            variant="outline"
            onClick={previousPage}
          >
            <ChevronLeftIcon className="size-4" />
          </Button>
          <div className="text-muted-foreground">
            {`Page ${currentPage} of ${pageCount}`}
          </div>
          <Button
            disabled={currentPage >= pageCount}
            size="icon"
            variant="outline"
            onClick={nextPage}
          >
            <ChevronRightIcon className="size-4" />
          </Button>
          <Button
            disabled={currentPage >= pageCount}
            size="icon"
            variant="outline"
            onClick={lastPage}
          >
            <DoubleArrowRightIcon className="size-4" />
          </Button>
        </div>
      )}
    </div>
  </div>
)

Conclusion

Long run huh ? Take a deep breathe, we've just got it started. There are some other usefull tools we usually use in react applications that we're going to re-invent in this series. Until then, I'll be glad to have your feedback in the comment section. Also, you can give suggestion for the next one.

See you soon folks!!!