Forem: shallowdepth

Why useNavigate hook in react-router v6 triggers waste re-renders and how to solve it

shallowdepth — Thu, 28 Apr 2022 15:36:19 +0000

While optimizing performance in one of my React.js projects I stumbled upon components re-rendering for no apparent reason whatsoever. After some experiments the culprit was found:

import { useNavigate } from "react-router-dom"; // v6

...

const Component = () => {
    const navigate = useNavigate();
    ...
}

Turns out that if you use the useNavigate hook in a component, it will re-render on every call to navigate() or click on <Link />, even if the path has not changed. You cannot prevent it with the React.memo().

Here is a demonstration:

The first block does not call useNavigate and is rendered only once. The second uses the hook and is re-rendered twice on every path "change" (I am not clear on why twice, maybe useNavigate is to blame again 🤷). The third uses a "stable" version of useNavigate, more on that below.

I would say that this is unexpected behavior, especially since useHistory in react-router v5 did not cause re-renders. There is a long discussion on GitHub about this behavior. It boils down to the position that it is not a bug, but expected behavior.

Comment for #7634

timdorr commented on Sep 28, 2020

useNavigate changes when the current location changes. It depends on it for relative navigation. Wrapping it in memo only prevents re-renders from parent components. If hooks within the component cause re-renders, there is nothing memo can do.

View on GitHub

It happens because useNavigate subscribes to contexts that change when path change is triggered (even if it stays the same):

let { basename, navigator } = React.useContext(NavigationContext);
let { matches } = React.useContext(RouteContext);
let { pathname: locationPathname } = useLocation();

Usually, it is not a big problem, because changing the path means changing the view and you need to render a new set of components anyway. Re-rendering several menu elements is not a problem.

However, when you change parameters in the path without changing the view or there are a lot of constant components that are independent of the path change, it can become painful.

There are several ways to solve this problem:

Use the useNavigate hook in the smallest/lowest-level component possible. It will not save you from re-renders but makes them less costly.
Decouple use of the hook from the component, if possible. For example, some of my components can trigger popups and notifications passing to them navigate function. I could move the hook to the popup and notification components themselves, although it would unnecessarily complicate otherwise simple setup.
"Stabilize" the hook by putting it into a separate context and utilizing a mutable object from the useRef hook. This is a simplified version of this approach.

// StableNavigateContext.tsx

import { 
  createContext,
  useContext,
  useRef, 
  MutableRefObject 
} from "react";
import { 
  useNavigate, 
  NavigateFunction 
} from "react-router-dom";

const StableNavigateContext = createContext<MutableRefObject<
  NavigateFunction
> | null>(null);

const StableNavigateContextProvider = ({ children }) => {
  const navigate = useNavigate();
  const navigateRef = useRef(navigate);

  return (
    <StableNavigateContext.Provider value={navigateRef}>
      {children}
    </StableNavigateContext.Provider>
  );
};

const useStableNavigate = (): NavigateFunction => {
  const navigateRef = useContext(StableNavigateContext);
  if (navigateRef.current === null)
    throw new Error("StableNavigate context is not initialized");

  return navigateRef.current;
};

export {
  StableNavigateContext,
  StableNavigateContextProvider,
  useStableNavigate
};


// App.tsx

import { BrowserRouter } from "react-router-dom";
import { 
  StableNavigateContextProvider 
} from "./StableNavigateContext";

export default function App() {
  return (
    <BrowserRouter>
      <StableNavigateContextProvider>
        // ...
      </StableNavigateContextProvider>
    </BrowserRouter>
  );
}


// Component file

import { useStableNavigate } from "./StableNavigateContext";

const Component = () => {
  const navigate = useStableNavigate();
  // ...
};

You can use a similar approach for the useLocation hook or combine them in one context like in the original solution. However, since the components will not re-render on the path change anymore, their state may get stale.

TodoX: task manager and time tracker. Rationale, stack, mistakes, and challenges

shallowdepth — Thu, 14 Apr 2022 22:52:50 +0000

For the past 1.5 years, I've been developing TodoX in my free time -- a task manager and time tracker. Now the project is in the state I am not too ashamed to show to other people.

This post is not only a shameless plug 🥸, but a story about reasons for creating TodoX, choosing the stack, testing, DevOps, challenges, good and bad decisions.

Does the world need one more "productivity" tool?

The main question is why create one more task manager? There is already a gazillion of them. Three reasons.

First and foremost, for the past 10+ years (with some gaps) I track time spent on most projects, work, etc. Why I do this is a topic deserving a separate post. In short: we reach goals by (1) investing enough time and effort in (2) what matters. The results suffer if any of these two factors lags.

Tracking time increases focus, promotes self-accountability, and allows for better retrospection. In turn, all of this helps me invest time in what matters and not what feels the most urgent or easy right now.

Most productivity tools concentrate on either task management or time tracking, leaving out the second part, or having it only as an afterthought. In addition, larger tools were too bloated with unnecessary (for me) features.

Second, in my product I can introduce some quality of life features from smaller, like a time counter in the favicon or a global timer, to larger, like analytics and customizable widgets.

Finally, it was a rather ambitious real-life product I can learn a lot from, use myself, and add to my portfolio. Regardless of whether it turns into something more than a personal project.

Tech stack

I decided to go for a classic design: a frontend consuming a backend via REST API. It is flexible and allows for a good separation of concerns with less code coupling. In other words, no server-side rendering.

Backend: Python, FastAPI, MongoDB

For the backend, I chose Python, mostly due to personal preferences and available skills. It could have as easily been any other language.

The next question is which framework to choose: Django, Flask, or FastAPI. Django is ruled out immediately -- it is too opinionated and has too many "batteries included" for my taste and use case.

Both Flask and FastAPI would suffice. At that time, I had more experience with Flask but wanted to dive deeper into the new and shiny FastAPI. In addition, FastAPI offers a lot of convenience around type hints and data validation. Though it is not free: this convenience comes at a cost of some magic.

Database: both relational and document-based databases will work. After some prototyping and thought experiments, I decided to go with MongoDB: document structure fitted the use case better, namely storing almost all information about a task in one document, and not spreading it over several tables.

Frontend: JS/TS, React.js, SPA, PWA

Choosing the frontend stack was more complicated. A task manager (outside of certain circumstances) must be multi-platform: work on virtually any device and operating system. As I saw it, there were three options to achieve it:

Build separate web and native apps for all platforms. It's too resource-heavy and unfeasible for a small project.
Use a cross-platform framework that can be compiled for several platforms (React Native, Flutter, etc.). I did not have enough expertise in them to be sure of this solution. Moreover, I didn't want to stray too far from regular web technologies, vendor-locking myself with a particular multi-purpose framework.
Develop a regular Single-Page Application (SPA) accessible via any modern browser. SPA can do almost everything that a regular app can, plus it is possible to mimic the native app experience by employing service workers and making it a Progressive Web App (PWA), though with some limitations. Should the need arise, one can wrap it in a native app to overcome these limitations, or start developing a "real" native app.

After zeroing in on an SPA/PWA approach, choosing React.js as the main framework was no-brainier. It is the most popular (i.e. extensive code and knowledge base) and simple, especially after the introduction of hooks and functional components.

At first, I wrote everything in regular JavaScript, thinking that for this initially small project TypeScript was not worth the hassle. Well... It was a mistake. Types are super helpful: static type checking reduces the number of bugs/typos, autocomplete becomes crazy powerful, speeding up the development.

Now I am transitioning the project to TypeScript. It is a good thing that you can do it gradually. However, the transition was more difficult for an existing code base with a lot of ad hoc types and temporary any's at first. Although, the more TypeScript there is, the easier it is to convert other parts.

Component library: Material UI

The final frontend question: build all UI/UX from scratch or use an existing design system and component base. I decided to use Material UI for React.js. First, it allows building the frontend fast, focusing on features, rather than basic elements, styles, and layout. Second, I don't have a designer bone in my body, so Material UI with some tweaks is definitively better than what I could do in any reasonable amount of time.

There are two main drawbacks:

These general-purpose components tend to have too much functionality built-in, making the app bundle larger than it should be, and in some cases, performance leaves much to be desired (on very complex pages). Again, this is manageable, but you need to keep an eye on it. Spoiler alert: now I am gradually rewriting components from scratch to improve overall performance and experience.
Some people might say that the interface looks too generic. It may be true to a degree, but at the moment the project lives in the mode of "function over form".

Testing: End-to-End

When starting a new pet project, there is a great temptation to skip tests to develop faster. In my experience, it never works. Of course, you can compromise in some places, but tests allow you to develop faster since you are more sure of the quality when making changes.

Test-Driven Development is a different issue. I practice it more often than not, but it is not the goal in itself.

Backend: pytest, FastAPI TestClient

Unit tests only for complex logic that has many edge cases. All other functionality is tested via API end-to-end tests. Nothing fancy: pytest + TestClient provided by FastAPI.

Frontend: Cypress, Jest

The story of frontend tests is more complicated. For the longest time, I was skeptical about covering the frontend with tests. "Frontend is too dynamic and fluid to cover it with tests, they will be flaky and I will have to rewrite them often," told I myself. To some degree it is true: frontend tests differ from backend and you have to rewrite them more often.

After several WTFs breaking the frontend in unexpected places with innocent-looking changes, I decided to invest time in frontend testing. The approach is the same: more end-to-end tests (Cypress) and a few unit tests (Jest) for complex logic.

At first, it took a lot of time to write Cypress tests. However, after I've written some project-specific convenience functions, added data-test attributes to the most elements, and got a hang of it, writing Cypress tests became a breeze. Often writing the tests itself unearths bugs and rough edges.

DevOps: Docker, Nginx, Traefik, GitHub Actions

From the get-go, I wanted to build the architecture the "right way" with all the components running in separate Docker containers, orchestrated by docker-compose. Static content (frontend SPA) is served via Nginx. All the containers communicate with the outside world via Traefik reverse proxy. It all runs on a regular Ubuntu with a clear path for scaling.

Self-hosted MongoDB or managed? While the project is small I decided in favor of a self-hosted solution: it is simply cheaper. When the user base grows switching to a managed solution may be a viable path forward.

Deploy via GitHub Actions: pipelines automate everything and save a lot of time, allowing to release effortlessly.

Backups: of course, regular backups. Simple cron + bash script + mongodump. With some system snapshots sprinkled here and there.

Tech highlights

As with any project some parts were more interesting or challenging to work on:

React.js state management. Always fun, especially since I wanted to do without external libraries, only with contexts, reducers, and hooks. This may change in the future as state management grows more complex.
Background sync. It is based on Server-Sent Events which are simple enough with two caveats. First, I needed to avoid "echoes" when an event is pushed to the client that caused it. Second, changes must be reconciled before being pushed to the store, otherwise, the user will see jitters in the app state and other unwelcome effects.
Date and priority recognition in the taskbar. Regular regexp would have done the trick, but I wanted to have a simple and extensible solution, without resorting to all-powerful external libraries.
Text editor. It was the most painful part. I've tried many, but only one fit the bill. Then it tuned out that got discontinued. After all, I settled for plaintext editor + rendering markdown. It is definitely not the final solution, but for the time being, it should suffice.

Decisions I am happy about

There were many things I wasn't sure about. Now I see that some of them were right:

End-to-end testing, both for the backend and frontend. The best thing since sliced bread.
Using GitHub Actions (or any other CI/CD). Even for a small project it automates away the friction of testing and deploying. Saves much more time than requires setting up.
Betting SPA and PWA. It's not perfect by far, but has outstanding ROI.
Using Material UI component base. I guess this is the most controversial decision. At this stage of the project, the pros outweigh the cons.

Mistakes to avoid

If I were to start again I would do several things differently:

Use TypeScript from the very beginning. It would have saved me a lot of time.
Start frontend end-to-end testing with Cypress earlier. +10 to confidence in a release.
Invest less time in less important features with low ROI. The most prominent example is the text editor.

What to expect

TodoX is a fully functional task manager and time tracker:

usual task manager stuff: tasks, projects, deadlines, repeated tasks, notes
time tracking: time can be recorded separately on any task
customizable widgets panel: notes, time by projects, time by tasks, and general stats
analytics: aggregate time by tasks or projects for any period and other stats
background sync between devices
works on any platform, installable as PWA on iOS and Android

A complete description with screenshots is on the website.

The main limitation: at the moment there are no collaboration features, although it is something on the horizon.

Future

There is a long list of features to be implemented, from straightforward, like tags, notifications, and reminders, to more complex, like collaboration, better analytics, performance tracking, and predictive assistance.

Feedback

Feel free to leave feedback in any way: comments here, email at care@todox.online, comments on the contacts page.

End-to-end testing for Telegram bots

shallowdepth — Wed, 26 Jan 2022 22:15:33 +0000

I think there is no need to say that automatic testing is a must in software development. Tests improve the quality of the code, give confidence in it and make it more stable. However, not all tests are created equal: they differ both in what they test and how hard it is to set them up and maintain.

For instance, it is relatively easy to write unit tests. In turn, they tell you very little about how your system will behave in production. Do not get me wrong, they are very useful, especially in testing a complicated piece of business logic. However, they are too low-level to give much confidence that the code will run properly in production.

Here end-to-end tests come in: you test how your system responds to simulated user actions as a whole. Below we will look at the main advantages and disadvantages of this kind of tests and how to build them for a Telegram bot, using as an example listOK – bot for shopping and simple to-do lists.

Pros and cons of end-to-end tests

Main advantages of end-to-end tests:

They test the system as a whole. Your system likely consists of many moving parts: code itself, load balancer, reverse proxy, database, web-server, cache, queues, storage, etc., let alone microservices. If any of these parts break, the whole system may fail.
They test the code in an environment close to production. Sometimes an obscure difference in environment configuration changes the behavior of the system.
They can test interactions with third parties. If your product consumes other services (e.g. APIs), it is beneficial to test that it works for the user. Of course, you can mock them, but mocks tell you little about the real-life performance of the system.
They test real user scenarios. Users are not interested whether some getter or setter deep in your code is working properly, they need results. Some parts of your code may even fail, but it is still a success if the users get what they came for.

That is why end-to-end tests tell you much more about how your code behaves in the wild and give you more confidence that it will work as planned.

However, everything comes at a cost and end-to-end tests are no exception:

End-to-end tests can be a pain to set up and maintain. Plus it often requires a broader set of technologies, which are not needed in production.
They test a lot of things at once, so if they fail, it can be difficult to untangle what happened.
These tests often involve third-party products, which you have little to no control over. However, so does the production environment: users also use third-party tools to access your product.
They can easily break when you change something in the secondary details of the product since they rely on them for testing (for instance, whether a particular text or HTML element is present on a web page).
End-to-end tests can take much more time, than other types of tests. For instance, unit tests in the bot we show below take maybe 10-20 seconds, while end-to-end tests take around four minutes (you can scale it to your product). No surprise here, the scope of these tests is very different. This is more of a nuisance than a real problem.

I would argue, that despite their difficulties, end-to-end tests are worth every penny, especially, if the system is actively developed. Of course, the depth and breadth of tests will depend on the project and requirements to its reliability, cost of errors, etc., but it is definitely something to consider.

The architecture of chatbot end-to-end tests

It is relatively easy to make end-to-end tests for backend API – you spin up a test server and test requests. It is a bit more tricky for the web frontend – you have to use some kind of browser emulator, for instance, Selenium or Cypress. In addition, the frontend is less formalized, compared to an API, and these tests tend to break on layout changes which even do not affect functionality.

The approach to end-to-end tests of chatbots is similar to the web frontend. The closest to end-to-end tests for Telegram bots I found on the internet were How to Write Integration Tests for a Telegram Bot and some JS mocking libraries.

To start building tests we need to answer two design questions.

How to run end-to-end tests?

When building end-to-end tests we have two options: mock Telegram API used by the bot or use the real Telegram API. Using real API is much better:

creating a full-scale mock is cumbersome and can be a project of its own
it is too artificial, making the testing environment nothing like production. After running mocked tests, you cannot be sure that the bot would behave the same in production.

Therefore, we will use real Telegram integration to test as a user, not unlike Selenium frontend testing which automates real browsers.

How to run the bot for tests?

The bot and the tests can run together – the tests will spin up the project – or separately when the test instance is running independently. Both approaches have their pros and cons and will depend on the project nature, team size, development practices. It may even make sense to use both, one for day-to-day development and another to test releases.

I opted for running the bot and tests together – in my case, it was easier to set up and allowed for easier integration of these tests into CI/CD pipeline.

Setting up the testing environment

The examples below require at least Python 3.8 with the following libraries:

pytest
pytest-asyncio
python-telegram-bot or another library on which the bot is based
Telethon – for using Telegram API in the tests, not needed in production
Optional: python-dotenv – for loading env-files

ℹ️ The code below is not intended to be run as-is. I omitted some parts for clarity and brevity since they are too project-dependent and not required to understand the code.

Get credentials for Telegram API

To set up the testing environment we will need two sets of credentials:

for test bot via BotFather, and
for Telegram API to log in as a user. I would strongly recommend registering a separate testing Telegram account for safety reasons.

To get credentials for accessing Telegram API as a user you need first to log into the Telegram developer portal using the phone number used in the Telegram account and an access code sent to that account. After that you can get api_id and api_hash to access Telegram API in section API development tools.

On the first log-in, Telegram will ask for an access code (second factor) by sending the code to the account requesting access. To skip this step in the future we need to save the session and provide it for later log-ins:

# tests/e2e/get_session_string.py

from telethon import TelegramClient
from telethon.sessions import StringSession


api_id = "YOUR API ID"
api_hash = "YOUR API HASH"


with TelegramClient(
    StringSession(), api_id, api_hash
) as client:
    print("Session string:", client.session.save())

You need to run this code once, save the session string and use it in the tests.

⚠️ api_id, api_hash, and the session string must be kept secret. Treat then as your login and password.

Prepare bot for tests

First, we need to modify the bot. While it can be run in a separate thread as is, I found no clear way of programmatically stopping the bot after all tests have been completed. This is why we need to introduce a stop_event. It will not be sued in production, but when running a test it will allow stopping the bot from the main thread.

# list_bot/main.py

from threading import Event
from typing import Optional

from dotenv import load_dotenv
from telegram import Update
from telegram.ext import Updater

from list_bot.core import config, db
from list_bot.logic.conversations import (
    start_conversation_handler
)


def start_bot(stop_event: Optional[Event] = None) -> None:
    # Load config and connect to DB
    config.load_config()
    db.connect()

    # Set up bot
    updater = Updater(token=config.TELEGRAM_TOKEN)
    dispatcher = updater.dispatcher

    # Register handlers
    dispatcher.add_handler(start_conversation_handler)
    # Other handlers

    # Start listening for updates
    updater.start_polling()

    # Stop manually on event or use built-in idle()
    if stop_event is not None:
        if stop_event.wait():
            updater.stop()
    else:
        updater.idle()

Test fixture for running the bot

The next step is to set up the bot fixture that will be run automatically for the whole duration of the tests.

# tests/conftest.py

import asyncio
import os
import threading

import pytest
from list_bot.main import start_bot
from telethon import TelegramClient
from telethon.sessions import StringSession
from telethon.tl.custom.conversation import Conversation

from tests import config
from tests.integration.helpers import wait


@pytest.fixture(autouse=True, scope="session")
def bot():
    """Start bot to be tested."""
    os.environ["ENV_FILE"] = config.INTEGRATION_ENV_PATH
    stop_event = threading.Event()
    thread = threading.Thread(
        target=start_bot, kwargs={"stop_event": stop_event}
    )
    thread.start()
    yield
    stop_event.set()
    thread.join()


# Default event_loop fixture has "function" scope and will
# through ScopeMismatch exception since there are related
# fixtures with "session" scope. Need to override to set scope.
# https://github.com/pytest-dev/pytest-asyncio#event_loop
@pytest.fixture(scope="session")
def event_loop():
    loop = asyncio.get_event_loop()
    yield loop
    loop.close()

Telegram client fixtures

Now we need fixtures that will connect to Telegram API on the other side – as a user.

@pytest.fixture(scope="session")
async def telegram_client():
    """Connect to Telegram user for testing."""
    load_dotenv(dotenv_path=config.INTEGRATION_ENV_PATH)
    api_id = int(os.environ.get("TELEGRAM_APP_ID"))
    api_hash = os.environ.get("TELEGRAM_APP_HASH")
    session_str = os.environ.get("TELEGRAM_APP_SESSION")

    client = TelegramClient(
        StringSession(session_str),
        api_id,
        api_hash,
        sequential_updates=True
    )
    await client.connect()
    await client.get_me()
    await client.get_dialogs()

    yield client

    await client.disconnect()
    await client.disconnected


@pytest.fixture(scope="session")
async def conv(telegram_client) -> Conversation:
    """Open conversation with the bot."""
    async with telegram_client.conversation(
        config.BOT_NAME, timeout=10, max_messages=10000
    ) as conv:
        conv: Conversation

        # These are bot-specific preparation steps.
        # In listOK /start command registers the user and
        # sends a welcome message and a list of user's
        # lists with the main menu. These messages
        # must be awaited before yielding control to tests.
        await conv.send_message("/start")
        await conv.get_response()  # Welcome message
        await conv.get_response()  # User lists
        wait()
        yield conv

Notes about conv fixture:

Strictly speaking, it is not required: each test can do it on its own. However, doing it here avoids code repetition.
With scope="session" all tests will use the same conversation object. It is a tradeoff between tests speed and their isolation. In my case, it's worth it since isolation is already imperfect: I do not clean the database on each test.
timeout=10 seconds – needed to break test faster if bot becomes unresponsive. The default timeout is 60 seconds.
max_messages=10000 – by default telethon conversation can handle only 100 messages and throws ValueError after that (documentation). 10,000 messages for tests should be enough.

Helper functions

The code above should be enough to start writing end-to-end tests. However, many tests involve similar steps and actions which can be moved to helper functions to avoid repetition and make tests more versatile.

I wrote first tests without these functions and abstracted common actions later, when the code repetition became painful enough.

General-purpose helpers

# tests/helpers.py

import random
import string
from time import sleep
from typing import Optional

from telethon.tl.custom.message import Message, MessageButton

from tests import config


# Used to generate random unique names for lists and items. 
# This way you do not need to think about clashing names 
# when you do not clear DB after each test.
def random_string(length=16):
    """Return random string of ASCII letters
    in both registers."""
    return ''.join(
        random.choice(string.ascii_letters)
        for _ in range (length)
    )


# It is useful to pause from time to time, otherwise 
# tests may start failing due to weird latency effects.
# Experimentally I arrived at a delay of 0.5 seconds.
def wait():
    """Sleep of fixed duration (in tests.config)."""
    sleep(config.DELAY)


# Simplifies the most frequent action - look for a button
# with a given text either to check that it exists or click it.
def get_button_with_text(
    message: Message, text: str, strict: bool = False
) -> Optional[MessageButton]:
    """Return MessageButton from Message with text or None."""
    if message.buttons is None:
        return None

    for row in message.buttons:
        for button in row:
            if strict:
                is_match = text == button.text
            else:
                is_match = text in button.text
            if is_match:
                return button

    return None

Bot-specific helpers

In addition to general-purpose helpers I use bot-specific functions for frequent bot actions: get user lists, create a list, open a list, etc.

# tests/e2e/test_lists.py

from telethon.tl.custom.conversation import Conversation
from telethon.tl.custom.message import Message, MessageButton

from tests import config
from tests.helpers import random_string


async def get_user_lists(conv: Conversation) -> Message:
    """Return message with user's lists and main menu."""
    await conv.send_message("/my_lists")
    wait()
    return await conv.get_response()


async def create_list(conv: Conversation) -> str:
    """Create list with random name and return it."""
    user_lists = await get_user_lists(conv)

    await user_lists.buttons[-1][0].click()
    await conv.get_edit()

    list_name = random_string()
    await conv.send_message(list_name)

    # Wait for confirmation
    await conv.get_response()
    # Wait for user's lists
    await conv.get_response()

    return list_name


async def open_list(
    conv: Conversation, list_name: str
) -> Message:
    """Open list with a given name and
    return message with its contents."""
    await conv.send_message("/my_lists")
    wait()
    user_lists: Message = await conv.get_response()

    button = get_button_with_text(user_lists, list_name)
    await button.click()
    wait()
    message: Message = await conv.get_edit()
    wait()

    return message

Writing end-to-end tests

Finally, we have everything ready for our first test:

@pytest.mark.asyncio
async def test_command_my_lists(conv: Conversation):
    """Test /my_lists bot command."""  
    await conv.send_message("/my_lists")
    user_lists: Message = await conv.get_response()

    # Check that the message contains necessary text
    assert "Choose list or action" in user_lists.text
    # Check that there is a button inviting a user 
    # to create a list
    assert get_button_with_text(
        user_lists, "Create new list"
    ) is not None

This is a very basic test, it just checks that the command works and that in response the bot sends a message with user's lists.

Something more involved with several helpers:

@pytest.mark.asyncio
async def test_create_list_item(conv: Conversation):
    """Test creating an item in a list."""
    list_name = await create_list(conv)
    await open_list(conv, list_name)

    item_name = random_string()
    await conv.send_message(item_name)

    # Wait for confirmation
    await conv.get_response()
    # Wait for updated list contents
    list_contents: Message = await conv.get_response()

    assert list_contents.button_count == 3
    assert get_button_with_text(
        list_contents, item_name
    ) is not None

While listOK is a rather small bot, it is covered by about fifty end-to-end tests, that take up to four minutes to run. If I decreased wait() duration they would run faster, but some strange effects happen from time to time, especially in CI/DC pipeline. So I decided to keep them not too fast.

While it took me several hours to set up this testing environment and then some more to write tests it was worth it. Now it is easy to write new tests and they catch nearly all issues in covered user scenarios, allowing me to confidently refactor and add new functionality.

What is more, these tests are highly readable and I caught myself gravitating to test-driven development more than usual, first modeling the integration in tests and then implementing it.

5 ways to implement case-insensitive search in SQLite with full Unicode support

shallowdepth — Sun, 23 Jan 2022 03:23:29 +0000

Recently I needed a case-insensitive search in SQLite to check if an item with the same name already exists in one of my projects – listOK. At first, it looked like a simple task, but upon deeper dive, it turned out to be easy, but not simple at all, with many twists and turns.

Built-in SQLite capabilities and their drawbacks

In SQLite you can get a case-insensitive search in three ways:

-- 1. Use a NOCASE collation
-- (we will look at other ways for applying collations later):
SELECT * 
    FROM items 
    WHERE text = "String in AnY case" COLLATE NOCASE;

-- 2. Normalize all strings to the same case,
-- does not matter lower or upper:
SELECT * 
    FROM items 
    WHERE LOWER(text) = "string in lower case";

-- 3. Use LIKE operator which is case insensitive by default:
SELECT * 
    FROM items 
    WHERE text LIKE "String in AnY case";

If you use SQLAlchemy and its ORM these approaches will look as follows:

from sqlalchemy import func
from sqlalchemy.orm.query import Query

from package.models import YourModel


text_to_find = "Text in AnY case"

# NOCASE collation
Query(YourModel)
.filter(
    YourModel.field_name.collate("NOCASE") == text_to_find
)

# Normalizing text to the same case
Query(YourModel)
.filter(
    func.lower(YourModel.field_name) == text_to_find.lower()
).all()

# LIKE operator. No need to use SQLAlchemy's ilike
# since SQLite LIKE is already case-insensitive.
Query(YourModel)
.filter(YourModel.field_name.like(text_to_find))

All these approaches are not ideal. First, without special considerations they do not make use of indexes on the field they are working on, with LIKE being the worst offender: in most cases it is incapable of using indexes. More on the use of indexes for case-insensitive queries is below.

Second, and more importantly, they have a rather limited understanding of what case-insensitive mean:

SQLite only understands upper/lower case for ASCII characters by default. The LIKE operator is case sensitive by default for unicode characters that are beyond the ASCII range. For example, the expression 'a' LIKE 'A' is TRUE but 'æ' LIKE 'Æ' is FALSE.

It is not a problem if you plan to work with strings that contain only English alphabet letters, numbers, etc. I needed the full Unicode spectrum, so a better solution was in order.

Below I summarize five ways to achieve case insensitive search/comparison in SQLite for all Unicode symbols. Some of these solutions can be adapted to other databases and for implementing Unicode-aware LIKE, REGEXP, MATCH, and other functions, although these topics are out of the scope of this post.

We will look at the pros and cons of each approach, implementation details, and, finally, at indexes and performance considerations.

Solutions

1. ICU extension

Official SQLite documentation mentions the ICU extension as a way to add complete support for Unicode in SQLite. ICU stands for International Components for Unicode.

ICU solves the problems of both case-insensitive LIKE and comparison/search, plus adds support for different collations for a good measure. It may even be faster than some of the later solutions since it is written in C and is more tightly integrated with SQLite.

However, it comes with its challenges:

It is a new type of dependency: not a Python library, but an extension that should be distributed together with the application.
ICU needs to be compiled before use, potentially for different OS and platforms (not tested).
ICU does not itself implement Unicode conversions, but relies on the underline operating system – I have seen multiple mentions of OS-specific issues, especially with Windows and macOS.

All other solutions will depend on your Python code to perform the comparison, so it is important to choose the right approach to converting and comparing strings.

Choosing the right python function for case-insensitive comparison

To perform case-insensitive comparison and search we need to normalize strings to one case. My first instinct was to use str.lower() for this. It will work in most circumstances, but it is not the proper way. Better to use str.casefold() (docs):

Return a casefolded copy of the string. Casefolded strings may be used for caseless matching.

Casefolding is similar to lowercasing but more aggressive because it is intended to remove all case distinctions in a string. For example, the German lowercase letter 'ß' is equivalent to "ss". Since it is already lowercase, lower() would do nothing to 'ß'; casefold() converts it to "ss".

Therefore, below we will use the str.casefold() function for all conversions and comparisons.

2. Application-defined collation

To perform a case-insensitive search for all Unicode symbols we need to define a new collation in the application after connecting to the database (documentation). Here you have a choice – overload the built-in NOCASE or create your own – we will discuss the pros and cons below. For the sake of an example we will use a new name:

import sqlite3

# Custom collation, maybe it is more efficient
# to store strings
def unicode_nocase_collation(a: str, b: str):
    if a.casefold() == b.casefold():
        return 0
    if a.casefold() < b.casefold():
        return -1
    return 1

connection.create_collation(
    "UNICODE_NOCASE", unicode_nocase_collation
)

# Connect to the DB and register the function
connection = sqlite3.connect("your_db_path")
connection.create_collation(
    "UNICODE_NOCASE", unicode_nocase_collation
)

# Or, if you use SQLAlchemy you need to register
# the collation via an event
@sa.event.listens_for(sa.engine.Engine, 'connect')
def sqlite_engine_connect(connection, _):
    connection.create_collation(
    "UNICODE_NOCASE", unicode_nocase_collation
)

Collations have several advantages compared to the next solutions:

They are easy to use. You can specify collation in the table schema and it will be automatically applied to all queries and indexes on this field unless you specify otherwise:

CREATE TABLE test (text VARCHAR COLLATE UNICODE_NOCASE);

For the sake of completeness, let's look at two more ways to use collations:

-- In a particular query:
SELECT * FROM items
    WHERE text = "Text in AnY case" COLLATE UNICODE_NOCASE;

-- In an index:
CREATE INDEX IF NOT EXISTS idx1 
    ON test (text COLLATE UNICODE_NOCASE);

-- Word of caution: your query and index 
-- must match exactly,including collation, 
-- otherwise, SQLite will perform a full table scan.
-- More on indexes below.
EXPLAIN QUERY PLAN
    SELECT * FROM test WHERE text = 'something';
-- Output: SCAN TABLE test
EXPLAIN QUERY PLAN
    SELECT * FROM test WHERE text = 'something' COLLATE NOCASE;
-- Output: SEARCH TABLE test USING COVERING INDEX idx1 (text=?)

Collation provides case-insensitive sorting with ORDER BY out of the box. It is especially easy to get if you define the collation in the table schema.

Performance-wise collations have some peculiarities, which we will discuss further.

3. Application-defined SQL function

Another way to achieve case-insensitive search is to create an application-defined SQL function (documentation):

import sqlite3

# Custom function
def casefold(s: str):
    return s.casefold()

# Connect to the DB and register the function
connection = sqlite3.connect("your_db_path")
connection.create_function("CASEFOLD", 1, casefold)

# Or, if you use SQLAlchemy you need to register 
# the function via an event
@sa.event.listens_for(sa.engine.Engine, 'connect')
def sqlite_engine_connect(connection, _):
    connection.create_function("CASEFOLD", 1, casefold)

In both cases create_function accepts up to four arguments:

name of the function as it will be used in the SQL queries
number of arguments the function accepts
the function itself
optional bool deterministic, default False (added in Python 3.8) – it is important for indexes, which we will discuss below.

As with collations, you have a choice – overload built-in function (for instance, LOWER) or create new. We will look at it in more detail later.

4. Compare in the application

Another way of case-insensitive search would be comparing in the app itself, especially if you could narrow down the search by using an index on other fields. For instance, in listOK case-insensitive comparison is needed for items in a particular list. Therefore, I could select all items in the list, normalize them to one case and compare them with the normalized new item.

Depending on your circumstances it is not a bad solution, especially if the subset you will be comparing with is small. However, you will not be able to utilize database indexes on the text, only on other parameters you will be using to narrow down the scope.

The advantage of this approach is its flexibility: in the application you can check not only equality but, for instance, implement "fuzzy" comparison to take into account possible misprints, singular/plural forms, etc. This is the route I chose for listOK since the bot needed fuzzy comparison for the "smart" item creation.

In addition, it eliminates any coupling with the database – it is simple storage that does not know anything about the data.

5. Store normalized field separately

There is one more solution: create a separate column in the database and keep there normalized text you will be searching on. For instance, the table may have this structure (only relevant fields):

id	name	name_normalized
1	Sentence capitalization	sentence capitalization
2	CAPITAL LETTERS	capital letters
3	Non-ASCII symbols: Найди Меня	non-ascii symbols: найди меня

This may look excessive at first: you always need to keep the normalized version updated and effectively doubling the size of the name field. However, with ORMs or even manually it is easy to do and the disk space plus RAM is relativity cheap.

Advantages of this approach:

It completely decouples the application and database – you can easily switch.
You can pre-process normalized filed if your queries require it (trim, remove punctuation or spaces, etc.).

Should you overload built-in functions and collations?

When using application-defined SQL functions and collations you often have a choice: use a unique name or overload built-in functionality. Both approaches have their pros and cons in two main dimensions:

First, reliability/predictability when for some reason (a one-off mistake, bug, or intentionally) you do not register these functions or collations:

Overloading: the database will still work, but the results may not be correct:
- the built-in function/collation will behave differently than their custom counterparts;
- if you used now absent collation in an index it will appear to work, but results may be wrong even when reading;
- if the table with index and index using custom function/collation gets updated, the index may get corrupted (updated using built-in implementation), but continue working as if nothing happened.
Not overloading: the database will not work in any respects where the absent functions or collations are used:
- if you use an index on an absent function you will be able to use it for reading, but not for updates;
- indexes with application-defined collation will not work at all, since they use the collation while searching in the index.

Second, accessibility outside of the main application: migrations, analytics, etc.:

Overloading: you will be able to modify the database without problem, keeping in mind the risk of corrupting indexes.
Not overloading: in many cases you will need to register these functions or collations or take extra steps to avoid parts of the database that depend on it.

If you decide to overload, it may be a good idea to rebuild indexes based on custom functions or collations in case they get wrong data recorded there, for example:

-- Rebuild all indexes using this collation
REINDEX YOUR_COLLATION_NAME;

-- Rebuild particular index
REINDEX index_name;

-- Rebuild all indexes
REINDEX;

Performance of application-defined functions and collations

Custom functions or collation are much slower than built-in functions: SQLite "returns" to your application each time it calls the function. You can easily check it by adding a global counter to the function:

counter = 0

def casefold(a: str):
    global counter
    counter += 1
    return a.casefold()

# Work with the database

print(counter)
# Number of times the function has been called

If you are querying rarely or your database is small, you will not see any meaningful difference. However, if you do not use an index on this function/collation, the database may perform a complete table scan applying the function/collation on each row. Depending on the size of the table, hardware, and the number of requests, the low performance may be surprising. Later I will publish a review of application-defined functions and collations performance.

Strictly speaking, collations are a bit slower than SQL functions since for each comparison they need to casefold two strings, instead of one. Although this difference is very small: in my tests, the casefold function was faster than similar collation for around 25% which amounted to a difference of 10 seconds after 100 million iterations.

Indexes and case-insensitive search

Indexes and functions

Let's start with the basics: if you define an index on any field, it will not be used in queries on a function applied to this field:

CREATE TABLE table_name (id INTEGER, name VARCHAR);
CREATE INDEX idx1 ON table_name (name);
EXPLAIN QUERY PLAN
    SELECT id, name FROM table_name WHERE LOWER(name) = 'test';
-- Output: SCAN TABLE table_name

For such queries you need a separate index with the function itself:

CREATE INDEX idx1 ON table_name (LOWER(name));
EXPLAIN QUERY PLAN
    SELECT id, name 
        FROM table_name WHERE LOWER(name) = 'test';
-- Output: SEARCH TABLE table_name USING INDEX idx1 (<expr>=?)

In SQLite, it can be done on a custom function as well, but it must be marked as deterministic (meaning that with the same inputs it returns the same result):

connection.create_function(
    "CASEFOLD", 1, casefold, deterministic=True
)

After that you can create an index on a custom SQL function:

CREATE INDEX idx1 
    ON table_name (CASEFOLD(name));
EXPLAIN QUERY PLAN
    SELECT id, name 
        FROM table_name WHERE CASEFOLD(name) = 'test';
-- Output: SEARCH TABLE table_name USING INDEX idx1 (<expr>=?)

Indexes and collations

The situation with collations and indexes is similar: for a query to utilize an index they must use the same collation (implied or provided expressly), otherwise, it will not work.

-- Table without specified collation will use BINARY
CREATE TABLE test (id INTEGER, text VARCHAR);

-- Create an index with a different collation
CREATE INDEX IF NOT EXISTS idx1 ON test (text COLLATE NOCASE);


-- Query will use default column collation -- BINARY
-- and the index will not be used
EXPLAIN QUERY PLAN
    SELECT * FROM test WHERE text = 'test';
-- Output: SCAN TABLE test


-- Now collations match and index is used
EXPLAIN QUERY PLAN
    SELECT * FROM test WHERE text = 'test' COLLATE NOCASE;
-- Output: SEARCH TABLE test USING INDEX idx1 (text=?)

As noted above, collation can be specified for a column in the table schema. This is the most convenient way – it will be applied to all queries and indexes on the respective field automatically unless you specify otherwise:

-- Using application defined collation UNICODE_NOCASE from above
CREATE TABLE test (text VARCHAR COLLATE UNICODE_NOCASE);

-- Index will be built using the collation
CREATE INDEX idx1 ON test (text);

-- Query will utilize index and collation automatically
EXPLAIN QUERY PLAN
    SELECT * FROM test WHERE text = 'something';
-- Output: SEARCH TABLE test USING COVERING INDEX idx1 (text=?)

Which solution to choose?

To choose a solution we need some criteria for comparison:

Simplicity – how difficult it is to implement and maintain it
Performance – how fast your queries will be
Extra space – how much additional database space the solution requires
Coupling – how much your solution intertwines the code and storage

Solution	Simplicity	Performance (relative, without index)	Extra space	Coupling
ICU extension	Difficult: requires a new type of dependency and compiling	Medium to high	No	Yes
Custom collation	Simple: allows to set collation in the table schema and apply it automatically to any query on the field	Low	No	Yes
Custom SQL function	Medium: requires either building an index based on it or using in all relevant queries	Low	No	Yes
Comparing in the app	Simple	Depends on the use case	No	No
Storing normalized string	Medium: you need to keep the normalized string updated	Low to Medium	x2	No

As usual, the choice of the solution will depend on your use case and performance demands. Personally, I would go either with custom collation, comparing in the app, or storing a normalized string. For example, in listOK, I first used a collation and moved to comparing in the app when added fuzzy search.

Using Python decorators to process and authorize requests

shallowdepth — Thu, 20 Jan 2022 03:45:04 +0000

– One syntactic sugar pill to rule them all.

When building any system that interacts with users you always need to check that they are who they are claiming to be and whether they are allowed to do what they are trying to do. In other words, you need to authenticate and authorize users.

Chatbots are no exception. Strictly speaking, with chatbots you do not need to authenticate users yourself – the platform does it for you. However, on each request you still have to load the user model and authorize it.

Below I will show how I approached authorization and preprocessing requests in my Telegram bot for shopping and to-do lists listOK (project page to avoid code duplication and make it more explicit. This approach is not limited to Telegram or chatbots in general. It can be applied in any request-centered projects, which in our API date and age are the majority.

The code below is close to what I use with some simplifications and omissions for brevity.

Naive approach

To communicate with Telegram bot-API I use the python-telegram-bot library. It processes each message or other type of communication via separate handlers: functions that receive update details and overall bot context.

In listOK each user has lists and each list consists of items. This means that all handlers for creating, reading, updating, deleting lists and items should be able to do similar things: load required models from the database and authorize the user.

At the very beginning of the project I tried a direct approach – loading user models and authorizing them in the request handlers themselves:

def command_my_lists(
    update: Update, context: CallbackContext
) -> None:
    user = find_user_by_telegram_id(
        db.session, update.effective_user.id
    )
    if user is None:
        return
    # Display user's lists

However, very soon it became unviable: there are dozens of handlers, and repeating the same code everywhere was not DRY at all. What if I wanted to change the logic or react to an unauthorized user request?

Loading models with decorators

Here Python decorators came in very handy. They allow adding behavior and pre-processing to any function without modifying it:

def load_user_or_fail(fn):
    """Decorator: loads User model and passes it to the function
    or stops the request."""
    def wrapper(*args, **kwargs):
        # Expects that Update object is always the first arg
        update: Update = args[0]
        user = find_user_by_telegram_id(
            db.session, update.effective_user.id
        )

        # Ignore requests from unknown users
        if user is None:
            return
        return fn(*args, **kwargs, user=user)

    return wrapper

Now I can add user loading to any handler with only one line and additional argument in the handler.

@load_user_or_fail
def command_my_lists(
    update: Update,
    context: CallbackContext,
    user: User
) -> None:
    # Display user's lists

I use this decorator for all handlers except for the /start command, since new users need it to register. It handles the user registration/loading itself.

The same is true for loading other models. For instance, all handlers that work with lists need to load the respective models. It also can be moved to a decorator to reduce the boilerplate code:

def load_list_or_fail(source):
    def load_list_or_fail_outer_wrapper(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            # Expects the update and context objects 
            # to be first arguments
            update: Update = args[0]
            context: CallbackContext = args[1]
            list_id = None

            if source == "callback_data":
                list_id = int(re.findall(
                    r"[0-9]+", update.callback_query.data
                )[0])

            if source == "user_data":
                list_id = context.user_data.get(
                    "active_list_id", None
                )

            items_list = find_list_by_id(db.session, list_id)

            # Ignore request for non-existing list
            if items_list is None:
                logging.getLogger().error(
                    "Could not load list."
                )
                return

            return fn(*args, **kwargs, items_list=items_list)
        return wrapper
    return load_list_or_fail_outer_wrapper

This is a bit more involved than loading a user. A list_id of the list to load can arrive from either of two sources: a callback data (string returned after a user presses an inline keyboard button) or stored in user context (required for multi-step communications). To accommodate this I added a source argument to the decorator.

It adds complexity: I need to provide in advance what source to use. I thought about making source detection automatic but decided against it. First, sometimes both sources can be legitimately present. Second, this would have introduced unnecessary at that moment 'magic' and reduced code readability.

Authorize with decorators

Decorators help with authorization as well, although their implementation will vary greatly from project to project. For example, here is what I did to check permissions for updating a list name:

from enum import Enum, auto


class Permissions(Enum):
    """Permissions constants"""
    CREATE = auto()
    READ = auto()
    UPDATE = auto()
    DELETE = auto()


def permission_required(permission):
    """Decorator for checking permissions"""
    def outer_wrapper(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            is_authorized = False
            keys = kwargs.keys()

            if "items_list" in keys:
                target = kwargs["items_list"]
            elif "list_item" in keys:
                target = kwargs["list_item"]

            is_authorized = target.check_user_permission(
                kwargs["user"], permission
            )
            if not is_authorized:
                return

            return fn(*args, **kwargs)
        return wrapper
    return outer_wrapper


@load_user_or_fail
@load_list_or_fail(source="user_data")
@permission_required(Permissions.UPDATE)
def message_rename_list(
    update: Update,
    context: CallbackContext,
    user: User,
    items_list: ItemsList
) -> int:
    # Update list name

Here I chained three decorators: two that we saw before and a new one:

@load_user_or_fail loads the user model and passes it as a keyword argument.
@load_list_or_fail(source="user_data") loads a list model based on user_data context and passes it further as a keyword argument as well.
Finally, @permission_required(Permissions.UPDATE) checks whether items_list keyword argument is present, and if it is, calls list's method check_user_permission which checks if the loaded user has the permission required.

If instead of a list we loaded a list item, the permission_required decorator would have detected it and called the item's check_user_permission method.

Unlike in load_list_or_fail, here I opted to automatically detect the target. It does not introduce any magic: by controlling the decorator I am calling before checking the permission I control the target.

Note: @permission_required will fail if kwargs contain nether items_list nor list_item. This is by design: it means that I forgot a loading decorator and requires immediate attention. If it happens in production, the bot will send me a message with all the exception details.

These decorators made the code much more concise and DRY, saving me a lot of time. What I especially like about this approach is how modular and explicit it is.

Transform any Telegram bot into (almost) an app

shallowdepth — Thu, 13 Jan 2022 21:49:59 +0000

Telegram bots are powerful tools that can do a lot of things. However, they have at least one major drawback: to use a bot you have to find the Telegram app, open it, find and, finally, open the bot. It introduces significant overhead and friction, reducing the likelihood that the bot will be used.

Turns out, you can create a shortcut for any one on one chat with a bot – it will open the Telegram app and required chat automatically. My access time decreased from 5-10 seconds to 1.

How it works

When you open a bot link https://telegram.me/[bot_handle], for example, https://telegram.me/list_ok_bot, in most cases the browser will suggest opening the bot in the Telegram app. It does it by calling a link like tg://resolve?domain=[bot_handle] (you can look it up in the page's sources). This is a Telegram call to open a chat with this bot, either existing or a new one.

As far as I can judge, it should work on any modern platform. I have tested it on iOS, Windows, and Linux (Linux Mint to be precise). I see no reasons why it should not work elsewhere.

Therefore, to use this mechanism you can create a shortcut that opens one of the two links:

https://telegram.me/[bot_handle] aka web link – it will first open the default browser, which in turn should redirect to the bot. The exact behavior will depend on your browser and settings.
tg://resolve?domain=[bot_handle] aka app link – it will directly call the Telegram app, without the browser as an intermediary.

The second approach should be faster and I will use it in the examples below. It may take a couple of tries to find the best way for your device.

A word on styling the shortcut: usually you can provide your logo (e.g., the bot picture) or choose from existing.

iOS

In the Shortcuts app you need to create a new shortcut with one command:

After that you can add it to the Home Screen as an icon from the sharing or shortcut settings interface (third icon):

macOS

On macOS, you can use the Shortcuts app with the same settings as on iOS. You can also enable your iOS shortcuts on your Mac.

Android

I do not have an Android device at the moment, but I am sure that it can be done here as well – you just need to create a shortcut with the web or app link, or use similar a approach as on iOS.

Windows

On Windows, it is enough to create a shortcut with the app link tg://resolve?domain=[bot_handle].

Linux

Shortcut

Details in each Linux distribution may be different, but the idea should be the same: create a shortcut with a command that would open the web or app link.

For example, the following does the trick in Linux Mint Cinnamon: xdg-open tg://resolve?domain=[bot_handle]. To add this shortcut (they are called launchers in LM) to the panel you need first to agree to add it to the menu (you will be prompted after creating the launcher), then find it in the menu, right-click and choose 'Add to panel'.

Ulauncher

In Ulauncher – an application launcher for Linux – you need to open settings and create a shortcut with xdg-open tg://resolve?domain=[bot_handle] (as discussed, can differ in other distributions) and allow it to run without arguments.

After that you can call the shortcut as follows:

Browser

Add a bookmark with the web or app link – depending on the browser and settings one or the other may work better.