Forem: German Cyganov

Building a Voice and Eye-Controlled To-Do App - Part 3

German Cyganov — Fri, 12 Apr 2024 01:35:16 +0000

Some time has passed since the last article, during which time I slightly changed the layout of the app to this.

Added drag and drop and drag scroll. The changes can be seen here.

Now it's time to add voice control.

So, to begin with, there will be only two voice control commands, these are Add and Delete.

An example scenario

A user says Add and then describes the task with voice, after he finishes speaking, the task creates.
When a user says Delete and proceed with the task description, that task is deleted.
But it's quite tedious to pronounce the full description. So we will look for the task that most matches the description and if we find it, delete it. And if not, then display a notification? I don’t know yet, we’ll see.

First, let's look at WebSpeechApi. Specifically with its SpeechRecognition module.

It allows us to recognize the user's words quite easily. you just need to create an instance of SpeechRecognition something like this.

    const recognition = useRef()
    useEffect(() => {
        recognition.current = new (window.SpeechRecognition || window.webkitSpeechRecognition)()
    }, [])

Further, to start listening you just need to call the start method and to finish the end method.
There are also some callback functions we will need only two of them onend and onresult.

First try

Create a hook to separate the commands listening logic from the application logic.

src/hooks/useSpeechCommand.ts

import { useEffect, useRef, useState } from "react"

interface Parameters {
    listen: boolean;
    command: string;
    timeout?: number;
    onEnd?: (value: string) => void;
}
export const useSpeechCommand = ({ listen = false, command = '', timeout = 3000, onEnd = () => { } }: Parameters): [string, boolean] => {
    const recognition = useRef<SpeechRecognition>()
    const [commandTriggered, setCommandTriggered] = useState(false);
    const [value, setValue] = useState('');
    const timerID = useRef<NodeJS.Timeout>()
    const innerValue = useRef('')

    useEffect(() => {
        if (window.SpeechRecognition || window.webkitSpeechRecognition) {
            recognition.current = new (window.SpeechRecognition || window.webkitSpeechRecognition)()
        }

        if (!recognition.current) {
            return
        }

        recognition.current.continuous = true
        recognition.current.interimResults = true
        recognition.current.maxAlternatives = 1
        recognition.current.lang = 'en-EN'
    }, [])

    useEffect(() => {
        if (!recognition.current) {
            return
        }

        if (listen) {
            recognition.current.start()
        } else {
            recognition.current.stop()
        }

        recognition.current.onend = (event) => {
            setCommandTriggered(false)

            if (innerValue.current) {
                onEnd(innerValue.current)
            }

            if (timerID.current) {
                clearTimeout(timerID.current)
            }
            if (listen) {
                recognition.current?.start?.()
            }
        }

        recognition.current.onresult = function (event) {
            let result = '';
            for (let i = event.resultIndex; i < event.results.length; i++) {
                if (event.results[i].isFinal) {
                    result += event.results[i][0].transcript + ' ';
                } else {
                    result += event.results[i][0].transcript;
                }
            }

            if (!commandTriggered && result.toLowerCase().includes(command)) {
                setCommandTriggered(true)
                const commandRegExp = new RegExp(command, 'gi')
                result = result.replace(commandRegExp, '');
                setValue(result)
            }

            if (commandTriggered) {
                setValue(result)
            }
        };

        return () => {
            if (recognition.current) {
                recognition.current.stop()
                recognition.current.onend = null
                recognition.current.onresult = null
            }
        }
    }, [listen])

    useEffect(() => {
        innerValue.current = value
        if (timerID.current) {
            clearTimeout(timerID.current)
        }
        if (value) {
            timerID.current = setTimeout(() => {
                timerID.current = undefined
                recognition.current?.stop()
            }, timeout)
        }
    }, [value])

    return [value, commandTriggered]
}

It turns out to be quite long, but in reality nothing too complicated is going on here. We listen to what is said and try to find the command, if we find it we set its flag to true and listen further, when the user has finished we wait for timeout and call onEnd with recognized text. Chrome browser also finishes listening after some time, but it is usually too long.

The little bad things

Now if we put this hook in TaskCreator and in TaskList for adding and deleting respectively, nothing will work. It seems that browsers do not allow to use two instances of WebSpeechAPI at the same time. So to achieve the desired result you should use only one instance. For this purpose I will use a context that will contain a hook, and return methods for subscribing and unsubscribing to the commands.

Move further

First of all, let's delete the hook, as it will be inconvenient to work with it further and move its contents to a separate class.

src/libs/SpeechCommands.ts

export type CommandEvent = { command: string, value: string }

class SpeechCommands {
    private recognition: SpeechRecognition
    private commands: Record<string, boolean>
    private results: Record<string, string>
    private timers: Record<string, NodeJS.Timeout>
    private lang: string
    public recording = false
    public onEnd: (e: CommandEvent) => void
    public onResult: (e: CommandEvent) => void

    constructor(public continuous: boolean, public timeout: number) {
        this.recognition = new (window.SpeechRecognition || window.webkitSpeechRecognition)()
        this.commands = {}
        this.results = {}
        this.timers = {}

        this.lang = 'en-EN'
        this.onEnd = () => { }
        this.onResult = () => { }

        this.recognition.onend = this.onEndHandler.bind(this)
        this.recognition.onresult = this.onResultHandler.bind(this)
    }

    start(lang: string) {
        this.lang = lang ?? this.lang
        this.recognition.continuous = true
        this.recognition.interimResults = true
        this.recognition.maxAlternatives = 1
        this.recognition.lang = this.lang
        this.recognition.start()
        this.recording = true
    }

    stop() {
        this.recognition.stop()
        this.recording = false
    }

    private onEndHandler() {
        for (const [command, triggered] of Object.entries(this.commands)) {
            if(triggered) {
                this.onEnd({ command, value: this.results[command] })
            }

            this.commands[command] = false
            this.results[command] = ''
            if (this.timers[command]) {
                clearTimeout(this.timers[command])
                delete this.timers[command]
            }
        }

        if (this.continuous) {
            this.recognition.start()
        }
    }

    private onResultHandler(event: SpeechRecognitionEvent) {
        let result = '';
        for (let i = event.resultIndex; i < event.results.length; i++) {
            if (event.results[i].isFinal) {
                result += event.results[i][0].transcript + ' ';
            } else {
                result += event.results[i][0].transcript;
            }
        }

        for (const [command, triggered] of Object.entries(this.commands)) {
            if (result.toLowerCase().includes(command)) {
                this.commands[command] = true
                const commandRegExp = new RegExp(command, 'gi')
                this.results[command] = result.replace(commandRegExp, '');
                this.onResult({ command, value: this.results[command] })
            }

            if (triggered) {
                const commandRegExp = new RegExp(command, 'gi')
                this.results[command] = result.replace(commandRegExp, '');
                this.onResult({ command, value: this.results[command] })

                if (this.timers[command]) {
                    clearTimeout(this.timers[command])
                }
                this.timers[command] = setTimeout(() => {
                    this.onEnd({ command, value: this.results[command] })
                    delete this.timers[command]
                    this.commands[command] = false
                    this.results[command] = ''
                }, this.timeout)
            } else {
                this.results[command] = ''
            }
        }
    }

    addCommand(command: string) {
        this.commands[command] = this.commands[command] || false
    }
    removeCommand(command: string) {
        delete this.commands[command]
    }
}

export default SpeechCommands

Now let's create a context that will use this class and provide the listen method to anyone who wants to use it.

src/contexts/SpeechCommandContext.tsx

import SpeechCommands, { CommandEvent } from '@/libs/SpeechCommands';
import { createContext, useCallback, useContext, useEffect, useRef } from 'react';

type SpeechCommandListener = (value: string) => void
type SpeechEvents = 'result' | 'end'

const initialValue = {
    listen: (type: SpeechEvents, command: string, listener: SpeechCommandListener) => () => { },
}

const SpeechCommandContext = createContext(initialValue);
const SpeechCommandProvider: React.FC<React.PropsWithChildren> = ({ children }) => {
    const commander = useRef<SpeechCommands>()
    const listeners = useRef<Record<SpeechEvents, Record<string, SpeechCommandListener[]>>>({
        result: {},
        end: {}
    })

    const listen = useRef((type: SpeechEvents, command: string, listener: SpeechCommandListener) => {
        if (listeners.current[type][command]) {
            listeners.current[type][command].push(listener)
        } else {
            listeners.current[type][command] = [listener]
        }

        if (!commander.current?.recording) {
            commander.current?.start('en-EN')
        }
        commander.current?.addCommand?.(command)

        return () => {
            if (listeners.current[type][command]) {
                const idx = listeners.current[type][command].findIndex((l) => l === listener)
                if (idx !== -1) {
                    listeners.current[type][command].splice(idx, 1)
                }
            }

            if (listeners.current[type][command].length == 0) {
                commander.current?.removeCommand?.(command)
            }
        }
    })

    const onEndHandler = useCallback(({ command, value }: CommandEvent) => {
        listeners.current['end'][command].forEach((listener) => {
            listener(value)
        })
    }, [])
    const onResultHandler = useCallback(({ command, value }: CommandEvent) => {
            listeners.current['result'][command].forEach((listener) => {
            listener(value)
        })
    }, [])

    useEffect(() => {
        if (!commander.current) {
            commander.current = new SpeechCommands(true, 3000)
        }
    }, [])

    useEffect(() => {
        if (commander.current) {
            commander.current.onEnd = onEndHandler
        }
    }, [onEndHandler])

    useEffect(() => {
        if (commander.current) {
            commander.current.onResult = onResultHandler
        }
    }, [onResultHandler])

    return (
        <SpeechCommandContext.Provider value={{ listen: listen.current }}>
            {children}
        </SpeechCommandContext.Provider>
    )
}

const useSpeechCommandContext = () => useContext(SpeechCommandContext)
export { SpeechCommandContext, SpeechCommandProvider, useSpeechCommandContext }

And now we use it in the CreateTask component to create a task, like this.

src/components/TaskCreator.tsx

...
 const { listen } = useSpeechCommandContext()
    const onEnd = useRef((value: string) => {
        createTask(value).then((task) => {
            refresh([...tasks, task])
        }).catch(() => {
            toast({
                title: 'Error on creating a task',
                status: 'error'
            })
        })
        setTaskName('')
    })
    const onResult = useRef((value: string) => {
        setTaskName(value)
    })
    useEffect(() => {
        const removeEndListener = listen('end', 'add', onEnd.current)
        const removeResultListener = listen('result', 'add', onResult.current)

        return () => {
            removeEndListener()
            removeResultListener()
        }
    }, [])
...

Finally you can try to create tasks by voice, say Add and keep saying the task, 3 seconds after the end of the phrase the task will be created.

Try to create

Well, it's a little hard to say so that the recognition is correct. Maybe it's because English is not my native language and I speak with a terrible accent. But here I said "add - buy a milk." It's funny how it's recognized the other way around.

Delete what we've created

Let's add the ability to delete a task by its name. It is worth saying that for simplicity, it is arranged in such a way that if we say the delete command, something will be deleted for sure. To find the most similar name we use Levenshtein distance.

npm i --save levenshtein-edit-distance





```tsx
...
    const tasksRef = useRef<TaskListType>([]);
    const { listen } = useSpeechCommandContext()
    const onEnd = useRef((value: string) => {
        let minDstId
        let minDst
        for(let i = 0; i < tasksRef.current.length; i++) {
            const dst = levenshteinEditDistance(value, tasksRef.current[i].name)
            if(dst === 0) {
                minDstId = tasksRef.current[i].id
                break
            }
            if(!minDst || dst < minDst) {
                minDst = dst
                minDstId = tasksRef.current[i].id
            }
        }

        if(minDstId) {
            handleDeleteTask(minDstId)
        }
    })
    useEffect(() => {
        const removeEndListener = listen('end', 'delete', onEnd.current)

        return () => {
            removeEndListener()
        }
    }, [])
    useEffect(() => {
        tasksRef.current = tasks
    }, [tasks])
...

Let's try to remove an unnecessary task from the list.

Yes it seems to work, of course there are still some bugs and a lot of opportunities for optimization and improvements on user interaction. Maybe I will fix them later, maybe not. Most likely I'll just add a button to enable and disable voice control.

So it seems we have more or less figured out WebSpeechApi.
And in the next article we will try to track the look and do something about it.

If you have any questions or suggestions please leave them in the comments.

The full code is available at GitHub

Building a Voice and Eye-Controlled To-Do App - Part 2

German Cyganov — Thu, 21 Mar 2024 19:23:23 +0000

In the last part we created the basic functionality with adding and removing tasks, now let's make the TODO application a bit prettier. I sketched the layout in figma using ChakraUI Figma KIT, here are two themes light and dark which we will implement.

First set the dependencies for the icons.
npm i --save @chakra-ui/icons

Write the code

Create a new Menu component, which will have control buttons. Right now there is only a theme switch, but in the future we will add buttons for microphone, speakers, camera and possibly biometric authentication.

src/components/Menu/index.tsx

import { MoonIcon, SunIcon, } from "@chakra-ui/icons"
import { HStack, IconButton, StackProps, useColorMode } from "@chakra-ui/react"

export const Menu: React.FC<StackProps> = (props) => {
    const { colorMode, toggleColorMode } = useColorMode()

    return (
        <HStack justify='end' {...props}>
            <IconButton variant='ghost' aria-label="change theme"
                onClick={toggleColorMode}
                icon={colorMode === 'light' ? <SunIcon color='yellow.500' /> : <MoonIcon />} />
        </HStack>
    )
}

Stylize the existing TaskList and TaskCreator components.

src/components/TaskList/index.tsx

const TaskCard: React.FC<TaskCardProps> = ({ id, name, onDelete }) => {
    const handleDelete = () => {
        onDelete(id)
    }

    return (
        <Card
            direction={{ base: 'column', sm: 'row' }}
            overflow='hidden'
        >
            <CardBody>
                <Heading size='md'>{name}</Heading>
            </CardBody>
            <IconButton
                aria-label='remove task'
                colorScheme="red"
                height='auto'
                variant='insideCard'
                borderTopLeftRadius='0'
                borderBottomLeftRadius='0'
                icon={<DeleteIcon color="white" />}
                onClick={handleDelete} />
        </Card>
    )
}

export const TaskList = () => {
...
if (error) {
        return (
            <Center>
                <Text color='red' fontSize={'xl'}>
                    {'Network Error'}
                </Text>
            </Center>
        )
    }

    if (isLoadingDelayed) {
        return (
            <Center>
                <Spinner size='xl' />
            </Center>
        )
    }

    return (
        <Stack spacing={4} pr={'8'}>
            {...tasks.map((task, idx) => (
                <TaskCard key={task.id || `${task.name}_${idx}`} {...task} onDelete={handleDeleteTask} />
            ))}
        </Stack>
    )
}

src/components/TaskCreator/index.tsx

return (
        <form onSubmit={handleAddTask} {...props}>
            <Card
                direction={{ base: 'column', sm: 'row' }}
                overflow='hidden'
            >
                <CardBody pt='3' pb='3' pl='3' pr='8'>
                    <InputGroup>
                        <Input variant='outline' placeholder="Input a task" onChange={handleChange} />
                    </InputGroup>
                </CardBody>
                <IconButton
                    type="submit"
                    aria-label='add task'
                    colorScheme="green"
                    height='auto'
                    variant='insideCard'
                    borderTopLeftRadius='0'
                    borderBottomLeftRadius='0'
                    icon={<AddIcon color="white" />} />
            </Card>
        </form>
    )

Basic layout.

Let's make it so that the task input field was initially a little lower than the first quarter of the screen, and when adding tasks and appearing scroll, there is an opportunity to scroll it up after which it will stick to the top edge and the tasks will go as if under it.

src/pages/index.tsx

...
      <Container pt='25%' pb='5%' style={{ background: 'inherit' }}>
        <Stack spacing='8' style={{ background: 'inherit' }}>
          <Menu style={{
            position: 'sticky', top: '0rem', zIndex: '999',
            background: 'inherit',
            padding: '0.4rem 0',
            paddingLeft: '1rem',
            marginLeft: '-1rem',
          }} />
          <TaskCreator style={{
            position: 'sticky', top: '3.2rem', zIndex: '999',
            background: colorMode === 'light' ? 'inherit' : 'transparent',
            paddingLeft: '1rem',
            marginLeft: '-1rem',
        }}/>
          <TaskList />
        </Stack>
      </Container>
...

There appeared a little magic paddings and margins, they are necessary for that the shadow of cards did not appear when the card behind the input field and menu. Also need to propagate background: inherit through all components. A bit clumsy, but anyway.

src/pages/_app.tsx

...
      <style jsx global>
        {`
          #__next {
            background: inherit;
          }
        `}
      </style>
...

Theme switcher

In the Menu component was colorMode variable.

const { colorMode, toggleColorMode } = useColorMode()

Now add the theme information to the Chakra provider.

src/chakra.tsx

import {
    ChakraProvider,
    cookieStorageManagerSSR,
    localStorageManager,
} from '@chakra-ui/react'
import { GetServerSideProps } from 'next'
import { PropsWithChildren } from 'react'
import theme from './theme'

interface ChakraProps {
    cookies: string
}

export const Chakra: React.FC<PropsWithChildren<ChakraProps>> = ({ cookies, children }) => {
    const colorModeManager =
        typeof cookies === 'string'
            ? cookieStorageManagerSSR(cookies)
            : localStorageManager

    return (
        <ChakraProvider colorModeManager={colorModeManager} theme={theme}>
            {children}
        </ChakraProvider>
    )
}

export const getServerSideProps = (async ({ req }) => {
    return {
        props: {
            cookies: req.headers.cookie ?? '',
        },
    }
}) satisfies GetServerSideProps<ChakraProps>

src/theme.tsx

import { extendTheme, type ThemeConfig } from '@chakra-ui/react'

const config: ThemeConfig = {
    initialColorMode: 'dark',
    useSystemColorMode: false,
}

const theme = extendTheme({
    config,
})

export default theme

src/pages/_document.tsx

import theme from "@/theme";
import { ColorModeScript } from "@chakra-ui/react";
import { Html, Head, Main, NextScript } from "next/document";

export default function Document() {
  return (
    <Html lang="en">
      <Head />
      <body>
        <ColorModeScript initialColorMode={theme.config.initialColorMode} type="cookie"/>
        <Main />
        <NextScript />
      </body>
    </Html>
  );
}

src/pages/_app.tsx

import type { AppProps } from "next/app";
import { SWRConfig } from "swr";
import { Chakra } from "@/chakra";

export default function App({ Component, pageProps }: AppProps) {
  return (
    <>
      <SWRConfig
        value={{
          refreshInterval: 3000,
        }}
      >
        <Chakra cookies={pageProps.cookies}>
          <Component {...pageProps} />
        </Chakra>
      </SWRConfig>
      <style jsx global>
        {`
          #__next {
            background: inherit;
          }
        `}
      </style>
    </>
  )
}

src/pages/index.tsx

...
export { getServerSideProps } from '@/chakra';

In order to keep the theme from blinking on page refresh, information about theme stored in cookies, so that Next.js on the pre-render knows what color to use.

And make the color of the buttons in the task card a little bit darker.

src/theme.ts

const theme = extendTheme({
    config,
    components: {
        Button: {
            variants: {
                insideCard: (props: StyleFunctionProps) => {
                    const c = props.theme.colors[props.colorScheme]
                    return {
                        ...props.theme.components.Button.variants.solid(props),
                        background: c['500'],
                    }
                }
            }
        }
    },
})

Result

We are done with basic styling. Thank you for doing this with me. In the next part, we'll add animations for a task list and theme switching.

If you have any questions or suggestions please leave them in the comments.

The full code is available at GitHub

Building a Voice and Eye-Controlled To-Do App - Part 1

German Cyganov — Wed, 20 Mar 2024 19:14:39 +0000

Hi, I invite you to join me in creating another TODO app. I hope we can learn a lot during the creation.

Task list

Create basic functionality
Add styles and animations
Add voice control and responses
Add cursor control with gaze
Protect tasks with face/voice recognition and possibly fingerprint recognition

I will post an article each time after I finish a task, so if you have any questions or suggestions please speak up.

Create basic functionality

I don't want to overload the application with logic at the start. So let's leave only the basic actions of creating and deleting a task.

Create a project

npx create-next-app@latest

What is your project named? todo-ai
Would you like to use TypeScript? Yes
Would you like to use ESLint? Yes
Would you like to use Tailwind CSS? No
Would you like to use `src/` directory? Yes
Would you like to use App Router? (recommended) No
Would you like to customize the default import alias (@/*)? No

and install the dependencies npm i --save @chakra-ui/next-js @chakra-ui/react @emotion/react @emotion/styled framer-motion swr

At this stage things like chakra-ui or framer-motion will be unnecessary, but when we will style and animate the application they will be useful, so let's use them right away.

Creating the skeleton

src/pages/_app.tsx

import type { AppProps } from "next/app";
import { ChakraProvider } from '@chakra-ui/react'
import { SWRConfig } from "swr";

export default function App({ Component, pageProps }: AppProps) {
  return (
    <SWRConfig
      value={{
        refreshInterval: 3000,
      }}
    >
      <ChakraProvider>
        <Component {...pageProps} />
      </ChakraProvider>
    </SWRConfig>
  )
}

src/pages/index.tsx

import Head from "next/head";
import { TaskCreator } from "@/components/TaskCreator";
import { TaskList } from "@/components/TaskList";

export default function Home() {
  return (
    <>
      <Head>
        <title>Create Next App</title>
        <meta name="description" content="Generated by create next app" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
        <link rel="icon" href="/favicon.ico" />
      </Head>
      <main>
        <TaskCreator />
        <TaskList />
      </main>
    </>
  );
}

Create a components

src/components/TaskCreator/index.tsx

import { createTask } from "@/api"
import { useTasks } from "@/hooks/useTasks"
import { Button, FormControl, Input, InputGroup, InputRightElement, useToast } from "@chakra-ui/react"
import { ChangeEvent, ChangeEventHandler, FormEvent, FormEventHandler, ReactEventHandler, useState } from "react"

export const TaskCreator = () => {
    const toast = useToast()
    const { tasks, refresh } = useTasks()

    const [taskName, setTaskName] = useState('')
    const handleChange = (event: ChangeEvent<HTMLInputElement>) => setTaskName(event.target.value)

    const handleAddTask = async (e: FormEvent<HTMLFormElement>) => {
        e.preventDefault()
        createTask(taskName).then((task) => {
            refresh([...tasks, task])
        }).catch(() => {
            toast({
                title: 'Error on creating a task',
                status: 'error'
            })
        })
    }

    return (
        <form onSubmit={handleAddTask}>
            <InputGroup>
                <Input variant='outline' placeholder="Input a task" onChange={handleChange} />
                <InputRightElement>
                    <Button type='submit'>
                        Add
                    </Button>
                </InputRightElement>
            </InputGroup>
        </form>
    )
}

src/components/TaskList/index.tsx

import { Card, CardBody, Heading, Spinner, Stack, useToast } from "@chakra-ui/react"
import { removeTask } from "@/api"
import { Task } from "@/types"
import { useTasks } from "@/hooks/useTasks"
import { useDelay } from "@/hooks/useDelay"

type onDeleteHandler = (id: string) => void
type TaskCardProps = Task & {
    onDelete: onDeleteHandler
}
const TaskCard: React.FC<TaskCardProps> = ({ id, name, onDelete }) => {
    const handleDelete = () => {
        onDelete(id)
    }

    return (
        <Card
            direction={{ base: 'column', sm: 'row' }}
            overflow='hidden'
            variant='outline'
        >
            <CardBody>
                <Heading size='md'>{name}</Heading>
            </CardBody>
            <button onClick={handleDelete}>
                Delete task
            </button>
        </Card>
    )
}

export const TaskList = () => {
    const toast = useToast()
    const { tasks, error, isLoading, refresh } = useTasks()
    const isLoadingDelayed = useDelay(isLoading, {
        initialValue: false,
    })

    const handleDeleteTask = (id: string) => {
        removeTask(id).then(({ id }) => {
            refresh([...tasks.filter((task: Task) => task.id !== id)])
        }).catch(() => {
            toast({
                title: 'Error on delete task',
                status: 'error'
            })
        })
    }

    if (error) {
        return <div>
            {error.message}
        </div>
    }

    return isLoadingDelayed ? <Spinner size='xl' /> : (
        <Stack spacing={4}>
            {...tasks.map((task, idx) => (
                <TaskCard key={task.id || `${task.name}_${idx}`} {...task} onDelete={handleDeleteTask} />
            ))}
        </Stack>
    )
}

The hooks

useDelay - delays changes of variable state - it's almost like debounce hook but with different initial state. Used to not show the download spinner if the download was fast enough.

src/hooks/useDelay.ts

import { useEffect, useState } from "react";

interface UseDelayParams<T> {
    initialValue: T
    delay?: number
}
export const useDelay = <T>(value: T, { initialValue, delay = 300 }: UseDelayParams<T>) => {
    const [currentValue, setCurrentValue] = useState(initialValue)

    useEffect(() => {
        let timer: NodeJS.Timeout;

        timer = setTimeout(() => {
            setCurrentValue(value)
        }, delay)

        return () => {
            timer && clearTimeout(timer)
        }
    }, [value, delay])

    return currentValue
}

useTask - encapsulates the logic of task handling. Basically it is just a wrapper over useSWR.

src/hooks/useTasks.ts

import { FetchError, fetchTasks } from "@/api"
import useSWR from "swr"

export const useTasks = () => {
    const { data = [], error, isLoading, mutate } = useSWR('/api/tasks', fetchTasks)

    return {
        tasks: data,
        error: error as FetchError,
        isLoading: isLoading,
        refresh: mutate
    }
}

Since I don't want to complicate the application by using something like redux or mobx or even a small zustand. Tasks will be stored here on the server, and access to them will be through the api that we will write a bit later.

The missing parts

src/api/index.ts

import { Task, TaskList } from '@/types'

export class FetchError extends Error {
    constructor(message: string, public status: number) {
        super(message);
    }
}

const fetcher = async (url: string, init?: RequestInit | undefined) => {
    const res = await fetch(url, init)

    if (!res.ok) {
        const message = (await res.json())?.message
        const error = new FetchError(message, res.status)
        throw error
    }

    return res.json()
}
export const fetchTasks = async (): Promise<TaskList> => {
    return fetcher('/api/tasks')
}

export const createTask = async (name: string): Promise<Task> => {
    return fetcher('/api/tasks', {
        method: 'POST',
        body: JSON.stringify({
            name
        }, null, 0)
    })
}

export const removeTask = async (id: string): Promise<Task> => {
    return fetcher('/api/tasks', {
        method: 'DELETE',
        body: JSON.stringify({
            id
        }, null, 0)
    })
}

And not forget about types
src/types/index/ts

export interface Task {
    id: string
    name: string
}

export type TaskList = Task[]

API

We need three methods GET, POST and DELETE. To get, create and delete tasks.
For storage a relational database is used, with the following schema.

Authentication

As you can see there is a user here, but where can he come from?
I don't want to add a registration and login form for now. So we will create a new user ourselves when a request comes in without the necessary cookies. And if the request has the necessary cookies, we will take the user's id from them and give back the data associated with it.

Initialize database

First, let's install the dependencies for the server.
npm i --save @prisma/client cookie jose
npm i --save-dev prisma @types/cookie

Next, initialize prisma.
npx prisma init --datasource-provider sqlite

Create a schema.
prisma/schema.prisma

generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "sqlite"
  url      = env("DATABASE_URL")
}

model User {
  id    String @id @default(cuid())
  tasks Task[]
}

model Task {
  id      String @id @default(cuid())
  name    String
  owner   User   @relation(fields: [ownerId], references: [id])
  ownerId String
}

Run the migration to create the database and types.
npx prisma migrate dev --name init

Write functions for necessary CRUD operations.
db/tasks.ts

import prisma from "./prisma"

export const addTask = (userId: string, taskName: string) => {
    return prisma.task.create({
        data: {
            name: taskName,
            ownerId: userId
        }
    })
}
export const deleteTask = (taskId: string) => {
    return prisma.task.delete({
        where: {
            id: taskId
        }
    })
}
export const getTasks = (userId: string) => {
    return prisma.task.findMany({
        where: {
            ownerId: userId
        }
    })
}
export const getTask = (taskId: string) => {
    return prisma.task.findUnique({
        where: {
            id: taskId
        }
    })
}

db/users.ts

import prisma from "./prisma"

export const createUser = () => {
    return prisma.user.create({ data: {} })
}

export const getUser = (id: string) => {
    return prisma.user.findUnique({
        where: {
            id: id
        }
    })
}

And PrismaClient initialization code.
db/prisma.ts

import { PrismaClient } from '@prisma/client'

const prismaClientSingleton = () => {
    return new PrismaClient()
}

declare global {
    var prismaGlobal: undefined | ReturnType<typeof prismaClientSingleton>
}

const prisma = globalThis.prismaGlobal ?? prismaClientSingleton()

export default prisma

if (process.env.NODE_ENV !== 'production') globalThis.prismaGlobal = prisma

The API

src/pages/api/tasks.ts

import type { NextApiRequest, NextApiResponse } from 'next'
import { serialize } from 'cookie'
import { createUser, getUser } from '@/db/users'
import { createSession, decrypt, encrypt } from '@/session'
import { addTask, deleteTask, getTask, getTasks } from '@/db/tasks'
import { User } from '@prisma/client'

const taskHandlers = {
    'GET': async (req: NextApiRequest, res: NextApiResponse, userId: string) => {
        const tasks = await getTasks(userId)
        return res.status(200).json(tasks)
    },
    'POST': async (req: NextApiRequest, res: NextApiResponse, userId: string) => {
        const { name } = JSON.parse(req.body)
        if (name) {
            const task = await addTask(userId, name)
            return res.status(200).json(task)
        } else {
            return res.status(406).send({ message: 'Not Acceptable' })
        }
    },
    'DELETE': async (req: NextApiRequest, res: NextApiResponse, userId: string) => {
        const { id } = JSON.parse(req.body)
        if (id) {
            const processedTask = await getTask(id)
            if (processedTask?.ownerId !== userId) {
                return res.status(403).send({ message: 'Forbidden' })
            }
            const task = await deleteTask(id)
            return res.status(200).json(task)
        } else {
            return res.status(406).send({ message: 'Not Acceptable' })
        }
    },
}
type AllowedMethods = keyof typeof taskHandlers

const authorize = async (req: NextApiRequest, res: NextApiResponse) => {
    const initializeNewUser = async (req: NextApiRequest, res: NextApiResponse) => {
        const user = await createUser()
        const sessionData = createSession(user.id)
        const encryptedSessionData = await encrypt(sessionData)
        const cookie = serialize('session', encryptedSessionData, {
            httpOnly: true,
            secure: process.env.NODE_ENV === 'production',
            expires: sessionData.expires,
            path: '/'
        })
        res.setHeader('Set-Cookie', cookie)
        return user
    }

    const session = req.cookies['session']
    let user!: User | null
    if (!session) {
        user = await initializeNewUser(req, res)
    } else {
        try {
            const userId = (await decrypt(session)).payload.userId as string
            user = await getUser(userId)
        } catch (e) {
            user = await initializeNewUser(req, res)
        }
    }

    return user
}

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
    try {
        const user = await authorize(req, res)
        if (!user) {
            return res.status(401).send({ message: 'Unauthorized' })
        }
        if (req.method && req.method in taskHandlers) {
            return await taskHandlers[req.method as AllowedMethods](req, res, user.id)
        } else {
            return res.status(501).send({ message: 'Not Implemented' })
        }
    } catch (e) {
        return res.status(500).send({ message: 'Internal Error' })
    }
}

export const config = {
    api: {
        bodyParser: {
            sizeLimit: '1mb',
        },
    },
    maxDuration: 5,
}

Session managment
src/session.ts

import { SignJWT, jwtVerify } from "jose"

const secretKey = process.env.SECRET_KEY
const key = new TextEncoder().encode(secretKey)

interface Session {
    userId: String
    expires: Date
}

export const encrypt = async (payload: Session) => {
    return new SignJWT(payload as any).setProtectedHeader({ alg: 'HS256' }).setIssuedAt().setExpirationTime(payload.expires).sign(key)
}
export const decrypt = async (input: string) => {
    return jwtVerify<Session>(input, key, {
        algorithms: ['HS256']
    })
}
export const createSession = (userId: string): Session => {
    const expires = new Date(new Date().setDate(new Date().getDate() + 1));

    return {
        userId,
        expires
    }
}
export const updateSession = async (enryptedSession: string): Promise<Session> => {
    const parsed = await decrypt(enryptedSession)

    return {
        ...parsed.payload,
        expires: new Date(new Date().setDate(new Date().getDate() + 1))
    }
}

Our session will be considered valid one day after creation, let's make it so that the session expiration time is updated with each request and we can use the application every day without losing the task list too often.

To do this, let's write middleware that updates the session.
src/middleware.ts

import { NextResponse, type NextRequest } from 'next/server'
import { encrypt, updateSession } from './session'

export async function middleware(req: NextRequest) {
    const res = NextResponse.next()
    const session = req.cookies.get('session')?.value
    if (!session) {
        return res
    }

    try {
        const updatedSession = await updateSession(session)
        res.cookies.set({
            name: 'session',
            value: await encrypt(updatedSession),
            expires: updatedSession.expires,
            httpOnly: true
        })
    } finally {
        return res
    }
}

Result

That's all for the functional part. In the next part, we'll give it a little bit prettier look and add some animations too. Thank you for creating part one with me. If you have any questions or suggestions please leave them in the comments.

The full code is available at GitHub

RTCP stream in HTML throught WebSocket

German Cyganov — Sat, 31 Jul 2021 17:32:03 +0000

In this article we will create an application that will output the RTCP stream to an HTML page

To convert video we will use the node-rtsp-stream server, it will convert RTCP to MPEG1 and pass the result to WebSocket. Internally it uses ffmpeg so you will need to install it

We will use jsmpeg to display the video on the page

Download minified version of jsmpeg and put it in the libs folder next to the index.html file

Also to change the link to the stream we need Express

So we need to implement a server on Node.js which will, at the request of the client, start the transformation of the stream and response with a link as a result

Implementing the server

Create a folder where the server will be stored and install the dependencies

mkdir socket-server && cs socket-server
npm init -y
npm i node-rtsp-stream express cors

Let’s write code in which we will create an express application with the /start route, which when called will create a new instance of the streaming server with a given link to the stream

const express = require('express')
const cors = require('cors')
const Stream = require('node-rtsp-stream')

const app = express()

app.use(cors())
app.use(express.json())

const port = 3000
const wsPort = 9999

let stream = null

app.post('/start', function (req, res) {
  if (stream !== null) {
    stream.stop()
    stream = null
  }

  const { streamUrl } = req.body

  try {
    stream = new Stream({
      name: 'videoStream',
      streamUrl: streamUrl,
      wsPort: wsPort,
      ffmpegOptions: {
        '-stats': '',
        '-r': 30,
      },
    })
    res.json({ url: `ws://127.0.0.1:${wsPort}` })
  } catch (e) {
    console.error(e)
  }
})

app.listen(port, () => {
  console.log(
    `server listening commands at http://127.0.0.1:${port} and serve stream at ws://127.0.0.1:${wsPort}`
  )
})

Run the server

node index.js
server listening commands at http://127.0.0.1:3000

The server started successfully, to start the broadcast you need to send a POST request with a link to the stream to the address http://127.0.0.1:3000/start

Implementing the client

Create a simple index.html file where we’ll add a container for embedding the video stream. The entry field for the link to the RTCP stream and the button to start playing a new stream, as well as don’t forget to connect jsmpeg to play the video and index.js, which will contain all the logic

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <link rel="stylesheet" href="css/main.css" />
    <title>Video Stream</title>
  </head>
  <body>
    <div id="container" class="player">
      <div id="video-container" class="video"></div>
      <div class="stream-input">
        <input
          id="stream-url-input"
          type="text"
          name="stream-url-input"
          value="rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov"
          placeholder="stream url"
        />
        <input id="stream-play-btn" type="button" value="Play" />
      </div>
    </div>
  </body>
  <script type="text/javascript" src="libs/jsmpeg.min.js"></script>
  <script type="text/javascript" src="js/index.js"></script>
</html>

Write the logic

When the Play button pressed, send a request with the streamUrl parameter to /start route of our server, upon receiving a reply we create a player that plays the video with the link given by the server response

;(function () {
  let jsmpegPlayer
  const streamInput = document.getElementById('stream-url-input')
  const playBtn = document.getElementById('stream-play-btn')
  const videoContainer = document.getElementById('video-container')

  playBtn.onclick = () => {
    const streamUrl = streamInput.value
    fetch('http://127.0.0.1:3000/start', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json;charset=utf-8',
      },
      body: JSON.stringify({ streamUrl }),
    })
      .then((response) => response.json())
      .then((data) => {
        if (jsmpegPlayer) {
          jsmpegPlayer.stop()
          jsmpegPlayer.destroy()
        }

        const canvas = document.createElement('canvas')
        canvas.id = 'jsmpeg-player'
        videoContainer.app(canvas)

        jsmpegPlayer = new JSMpeg.Player(data.url, {
          canvas,
          autoplay: true,
        })
      })
  }
})()

Try!

As you can see the streaming video appears with a pretty long delay, about 12 seconds. This may be enough, but if you want to reduce the delay of the stream and get closer to real time read the next article in which we will try to reduce that delay

Complete code can be found on GitHub

Teaching the computers to write number names using JavaScript

German Cyganov — Sat, 24 Jul 2021 09:52:51 +0000

Today we will try to teach the computer to translate numbers into words

First Step

Let's use the number-word dictionary

const numberWords = {
  0: 'zero',
  1: 'one',
  2: 'two',
  3: 'three',
  4: 'four',
  5: 'five',
  6: 'six',
  7: 'seven',
  8: 'eight',
  9: 'nine',
  10: 'ten',
}

And then?

Let's write a function that will return a value if there is one in the dictionary, and false if no word for the number is given

function getNumberWord(number) {
  if (numberWords.hasOwnProperty(number)) {
    return numberWords[number]
  } else {
    return false
  }
}

Let's write the code to output the results of our work

const number = process.argv[2]
console.log(getNumberWord(number))

First result

node index.js 5
five

Great!

Moving forward

node index.js 101
false

Not bad

We learned how to convert simple numbers into words, but what if we want to work with numbers that are not in our dictionary? Possibly we could identify all the numbers in the dictionary. But

This is boring
And memory consuming

Learning to make words

So that you don't have to spend all day compiling dictionaries and leave your computer with a little more memory for more important tasks. And it has them, believe me!

Modify our program so that it adds the result of digit translation to the resulting string

const number = process.argv[2]

let result = ''
number.forEach((digit) => {
  result += getNumberWord(digit)
})

console.log(result)

node index.js 101
onezeroone

Good!

Add power

Define a dictionary for the powers, the key of which will be the number of zeros in the number

const numberZeros = {
  1: 'ten',
  2: 'hundred',
  3: 'thousand',
  6: 'million',
}

By analogy with the function of getting the number word value, let's write the function of getting the word for the power

function getPowerWord(power) {
  if (numberZeros.hasOwnProperty(power)) {
    return numberZeros[power]
  } else {
    return false
  }
}

Change the program so that it adds a power of a number

const number = process.argv[2]

let result = ''

const power = number.length - 1
const numberPowerWord = getPowerWord(power)
result += numberPowerWord + getNumberWord(number.substr(power))

console.log(result)

node index.js 101
hundredone

But the current solution has problems, firstly it stopped working with the our previous number, and secondly it knows how to work with only one power

Try this solution with a number whose power is not defined in the dictionary

node index.js 10000001
false1

Any problem?

Putting words together

To convert the power of a number, divide the number into two by adding the name of the digit between them. Also remember to check for power numbers such as "ten" in the dictionary, and use them instead of further dividing the number

function convertPoweredNumber(number) {
  let firstDigits = ''

  for (let i = 0; i < number.length; i++) {
    firstDigits += number[i]

    const power = number.length - firstDigits.length
    const powerWord = getPowerWord(power)

    if (powerWord) {
      const nextDigitPos = firstDigits.length
      const numberWord = getNumberWord(firstDigits[0] + Array(power).fill('0').join(''))

      return numberWord
        ? numberWord + convert(number.substr(nextDigitPos))
        : convert(firstDigits) + powerWord + convert(number.substr(nextDigitPos))
    }
  }

  return ''
}

In the second number we can get extra zeros in the beginning. Let's split the number 10001 without having a thousand power in the dictionary, we get 100 and 01, on the next iteration we will not find 01 in the number dictionary, and again send it for splitting for powers. As a result, instead of the correct output "one" we will get "tenone"

Moreover, the user can enter a number starting with zero himself

To avoid such situations, we will delete the leading zeros in the number

function removeLeadingZeros(number) {
  return number.replace(/^0+/, '')
}

Let's return the former functionality of the program and add the ability to work with power

function convert(number) {
  const numberWithoutZeros = removeLeadingZeros(number)
  return getNumberWord(numberWithoutZeros) || convertPoweredNumber(numberWithoutZeros)
}

const number = process.argv[2]
console.log(convert(number))

node index.js 101
onehundredone

Finish!

node index.js 1000000047
onethousandmillionfourtenseven

To separate words, let's add a space to the values in the dictionaries, at the same time adding the value of "forty" to the dictionary of numbers

node index.js 1000000047
one thousand million forty seven

Complete code can be found on GitHub

Finding 3.14 in JavaScript

German Cyganov — Sun, 18 Jul 2021 14:59:57 +0000

Chudnovsky algorithm

To date, one of the fastest and most efficient algorithms for calculating PI is considered to be the Chudnovsky algorithm

The principle of this algorithm is the basis of the 2019 record for the calculation of PI - 31.4 trillion digits

\frac{1}{\pi}=12\sum_{n=0}^{\propto}\frac{(-1)^n(6n)!(A+B_n)}{(n!)^3(3n)!C^{3_n+3/2}}

Skipping all mathematical transformations

We get

\pi \approx \frac{Q(0,N)}{12T(0,N)+12AQ(0,N)}C^{3/2}

To translate this formula into code, we need to know what Q and T are

Q and T - Mathematical functions which are expressed as

P(n_1, n_2) = P(n_1, m)P(m, n_2)

Q(n_1, n_2) = Q(n_1, m)P(m, n_2)

T(n_1, n_2) = T(n_1, m)Q(m, n_2)c^{n_2-m}+P(n_1,m)T(m,n_2)

This looks a little confusing, but let's go over it step by step

Define the constants

const A = 13591409
const B = 545140134
const C = 640320

Implementing the algorithm for calculating P, Q and T

function computePQT(n1, n2) {
  let m = 0
  let PQT = {
    P: 0,
    Q: 0,
    T: 0,
  }

  if (n1 + 1 === n2) {
    PQT.P = n2 * 2 - 1
    PQT.P = PQT.P * (n2 * 6 - 1)
    PQT.P = PQT.P * (n2 * 6 - 5)
    PQT.Q = Math.floor((C * C * C) / 24) * n2 * n2 * n2
    PQT.T = (A + B * n2) * PQT.P
    if (n2 % 2 === 1) {
      PQT.T = -PQT.T
    }
  } else {
    m = Math.floor((n1 + n2) / 2)
    let res1 = computePQT(n1, m)
    let res2 = computePQT(m, n2)
    PQT.P = res1.P * res2.P
    PQT.Q = res1.Q * res2.Q
    PQT.T = res1.T * res2.Q + res1.P * res2.T
  }

  return PQT
}

Find PI

We need to decide to what decimal place we will count to. This algorithm at each iteration allows us to find 14.1816474627... significant digits

You can try to calculate it yourself

\frac{1}{\pi}=\sum_{k=0}^{\propto}c_k

10^d=lim_{k \mapsto \propto }|c_k/ck+1|

d=log_{10}151931373056000

After calculating the value, let's put it in a constant

const DIGITS_PER_TERM = 14.1816474627

Write a function to calculate PI

function computePI(digits) {
  if (digits <= 0) {
    return '0'
  }

  const N = Math.floor(digits / DIGITS_PER_TERM) + 1
  const PQT = computePQT(0, N)
  const PI = (PQT.Q / (12 * PQT.T + 12 * A * PQT.Q)) * Math.pow(C, 3 / 2)

  return PI.toFixed(digits)
}

Finally, we are ready to count the decimal places

const hrstart = process.hrtime()
const PI = computePI(28)
const hrend = process.hrtime(hrstart)

console.log(PI.toString())
console.info(`Execution time (hr): ${hrend[0]}s ${hrend[1] / 1000000}ms`)

Checking the result

> node index.js
3.1415926535897935600871733186
Execution time (hr): 0s 0.139102ms

Yeah? Mistake!

We were able to find the number of characters we are interested in, now we can breathe easy and ~~apply the obtained value~~ in practice

But if you look closely, you can find an error

Compare

3.1415926535897935600871733186
3.1415926535897932384626433832

The first value was obtained by us, the second was taken from the Internet

The divergence starts after 15 characters. That is how many significant characters the double type has in JavaScript

Working on the bugs

To calculate more characters, we need to understand how to work with large numbers in JS

The BigNumber.js library for working with large numbers might be suitable for this purpose

But before that we need to simplify the formula a bit by removing the fractional degree from it

\pi=\frac{D\sqrt{E}Q}{AQ+T}

Rewrite the old constant definitions and add new ones. At the same time, we take unnecessary calculations out of the compute_PQT method

const A = new BigNumber('13591409')
const B = new BigNumber('545140134')
const C = new BigNumber('640320')

const D = new BigNumber('426880')
const E = new BigNumber('10005')

const DIGITS_PER_TERM = new BigNumber('14.1816474627254776555')

const C3_24 = C.multipliedBy(C).multipliedBy(C).dividedToIntegerBy(24)

Rewrite our calculation functions

function computePI(digits) {
  if (digits <= 0) {
    return '0'
  }

  const DIGITS = new BigNumber(digits)
  const N = DIGITS.dividedToIntegerBy(DIGITS_PER_TERM).plus(1)
  const PREC = DIGITS.multipliedBy(Math.log2(10))

  BigNumber.config({
    DECIMAL_PLACES: Math.ceil(PREC.toNumber()),
    POW_PRECISION: Math.ceil(PREC.toNumber()),
  })

  const PQT = computePQT(new BigNumber(0), N)

  let PI = D.multipliedBy(E.sqrt()).multipliedBy(PQT.Q)
  PI = PI.dividedBy(A.multipliedBy(PQT.Q).plus(PQT.T))

  return PI.toFixed(digits)
}

function computePQT(n1, n2) {
  let m = new BigNumber(0)
  let PQT = {
    P: new BigNumber(0),
    Q: new BigNumber(0),
    T: new BigNumber(0),
  }

  if (n1.plus(1).isEqualTo(n2)) {
    PQT.P = n2.multipliedBy(2).minus(1)
    PQT.P = PQT.P.multipliedBy(n2.multipliedBy(6).minus(1))
    PQT.P = PQT.P.multipliedBy(n2.multipliedBy(6).minus(5))
    PQT.Q = C3_24.multipliedBy(n2).multipliedBy(n2).multipliedBy(n2)
    PQT.T = A.plus(B.multipliedBy(n2)).multipliedBy(PQT.P)
    if (n2.modulo(2).isEqualTo(1)) {
      PQT.T = PQT.T.negated()
    }
  } else {
    m = n1.plus(n2).dividedToIntegerBy(2)

    let res1 = computePQT(n1, m)
    let res2 = computePQT(m, n2)

    PQT.P = res1.P.multipliedBy(res2.P)
    PQT.Q = res1.Q.multipliedBy(res2.Q)
    PQT.T = res1.T.multipliedBy(res2.Q).plus(res1.P.multipliedBy(res2.T))
  }

  return PQT
}

Second try

> node index.js
3.1415926535897932384626433833
Execution time (hr): 0s 3.432017ms

Note that the running time of the algorithm is longer, this is a consequence of storing numbers in strings

Compare

3.1415926535897935600871733186
3.1415926535897932384626433833
3.1415926535897932384626433832

Good!

Only the last digit is different, and that's because we use toFixed, which rounds up the number when converting it to a string

Another problem is

RangeError: Maximum call stack size exceeded

This error occurs when the node.js runtime has a call stack overflow

It can be avoided by giving the runtime the ability to clear the stack

let res1 = await new Promise((resolve) =>
  process.nextTick(async () => resolve(await computePQT(n1, m)))
)
let res2 = await new Promise((resolve) =>
  process.nextTick(async () => resolve(await computePQT(m, n2)))
)

Complete code can be found on GitHub