Forem: BrainRepo

A pseudo imperative approach for react confirmation dialogs

BrainRepo — Tue, 20 Feb 2024 16:09:08 +0000

Hello, this is the first technical article I am writing since we started developing fouriviere.io; for more info about the Fourier project, please visit fourviere.io.

The problem I want to discuss regards the confirmation modal; we have a few of them in our most complex flows (e.g., feed sync, feed/episode deletion).

Having a confirmation modal is often a good practice for managing un-revertable or destructive actions, and we adopted it in our critical paths for protecting the user from accidental actions.

Our frontend is built with React, and one of React's peculiarities is its very declarative approach, an approach that contrasts with the imperative approach of the confirmation modal. Considering this, our initial implementation bypassed the obstacle by effectively circumventing it; in fact, we used the tauri dialog function, which mimics the web api confirm method confirm method in a certain way.


//...do something
const confirmed = await confirm('This action cannot be reverted. Are you sure?', { title: 'Tauri', type: 'warning' });

if(!confirmed){
    //...exit
} 
//...continue the action

This is cool because it can be used in complex workflows without fighting with components and complex states; in fact, we don't need to track whether the modal is shown or the confirmation button is pressed.

However, there is a downside: the design of this confirmation modal comes from the operating system and does not fit our design styles at all.

How we solved the problem

First of all, we designed a confirmation modal, for laziness we based our component on the tailwindui dialog .

Here is an oversimplified version. If you want to see the implementation with the tailwind classes, please look at our ui lib

type Props = {
    ok: () => void;
    cancel: () => void;
    title: string;
    message: string;
    okButton: string;
    cancelButton: string;
    icon?: React.ElementType;
};

export default function Alert({ok, cancel, title, message, okButton, cancelButton, icon}: Props) {
    const Icon = icon as React.ElementType;
    return (
        <div>
            <h3>{icon} {title}</h3>
            <p>{message}</p>
            <div>
                <button onClick={ok}>{okButton}</button>
                <button onClick={cancel}>{cancelButton}</button>
            </div>
        </div>
    );
}

Now, we need to display this Alert modal in a portal in the most imperative way possible. To do that, we created a hook that exposes an askForConfirmation method that does all the dirty work under the hood.

interface Options {
    title: string;
    message: string;
    icon ?: ElementType;
}

const useConfirmationModal = () => {
    async function askForConfirmation({title, message, icon}:Options)
    {
        //Here we will put our implementation
    }
    return {askForConfirmation}
}
export default useConfirmationModal;

This hook will return an askForConfirmation method for being called by the component logic, this method takes a Options object for defining the modal title, message and icon.

Now we need to track when modal is displayed and eventually the title, message, icon, the okAction and the cancelAction, we define a state for the component, the state can be false or object of type ModalState, if false the modal is hidden.


interface Options {
    title: string;
    message: string;
    icon ?: ElementType;
}

interface ModalState 
    title: string;
    message: string;
    ok: () => void;
    cancel: () => void;
    icon?: ElementType;
}

const useConfirmationModal = () => {

    const [modal, setModal] = useState<false | ModalState>(false);

    async function askForConfirmation({title, message, icon}:Options)
    {
        //Here we will put our implementation
    }
    return {askForConfirmation}
}
export default useConfirmationModal;

Now the askForConfirmation method should set the modal state, let's implement. But we want that does it following an async approach using promises, like that we can call in this way


//inside the component//

const {askForConfirmation} = useConfirmationModal()
//...previous logic
if (!await askForConfirmation()) {
    return
}
continue

This means that askForConfirmation should return a promise that is resolved (with true or false) when the ok button is pressed or when the cancel button is pressed; before resolving the promise, the modal is hidden.


interface Options {
    title: string;
    message: string;
    icon ?: ElementType;
}

interface ModalState 
    title: string;
    message: string;
    ok: () => void;
    cancel: () => void;
    icon?: ElementType;
}

const useConfirmationModal = () => {
    const [modal, setModal] = useState<false | ModalState>(false);
    async function askForConfirmation({title, message, icon}:Options)
    {
        return new Promise<boolean>((resolve) => {
            setModal({
                title, 
                message, 
                icon,
                ok: () => {
                    setModal(false);
                    resolve(true);
                },
                cancel: () => {
                    setModal(false);
                    resolve(false);
                },
            });
        });
    }
    return {askForConfirmation}
}
export default useConfirmationModal;

Now stays to implement the display part. This is a hook, and it does not render directly jsx; then we need to find a "sabotage" for managing the render phase. What if the hook returns a function component for rendering it?

Let's try.


interface Options {
    title: string;
    message: string;
    icon ?: ElementType;
}

interface ModalState 
    title: string;
    message: string;
    ok: () => void;
    cancel: () => void;
    icon?: ElementType;
}

const useConfirmationModal = () => {
    const [modal, setModal] = useState<false | ModalState>(false);
    const modals = document.getElementById("modals") as HTMLElement;

    async function askForConfirmation({title, message, icon}:Options)
    {
        return new Promise<boolean>((resolve) => {
            setModal({
                title, 
                message, 
                icon,
                ok: () => {
                    setModal(false);
                    resolve(true);
                },
                cancel: () => {
                    setModal(false);
                    resolve(false);
                },
            });
        });
    }

    function renderConfirmationModal() {
        return (
            <>
                {modal && createPortal(
                    <Alert
                    icon={modal.icon ?? ExclamationTriangleIcon}
                    title={modal.title}
                    message={modal.message}
                    okButton="ok"
                    cancelButton="cancel"
                    ok={modal.ok}
                    cancel={modal.cancel}
                    />,
                    modals,
                )
                }
            </>
        );
    return {askForConfirmation, renderConfirmationModal}
}
export default useConfirmationModal;

Now, our hook returns aside the askForConfirmation, a function component renderConfirmationModal that displays the modal in the portal (in our case, inside the <div id="modal"> in the HTML page).

Now, let's try to use it in a simple component

export default function SimpleComponent() {

    const {askForConfirmation, renderConfirmationModal} = useConfirmationModal()

    async function doSomething() {
        if(!askForConfirmation({
            title: "Are you sure?",
            message: "This operation cannot be reverted",
        })) {
            return false
        }

        //do stuff...
    }

    return <>
        {renderConfirmationModal()}
        <button onClick={doSomething}>DO IT/button>
    </>
}

Conclusions

After this journey, we have a hook that helps us to have a confirmation modal with a simple api. It is essential to keep simple and reusable parts of the UI; this helps to keep the code readable, and we know how it can become messy our react components.

But keeping things simple needs complex effort.

Code or no-code, diaries of a half developer half entrepreneur using Directus

BrainRepo — Sun, 04 Jun 2023 12:01:56 +0000

Code or no-code

I have been an entrepreneur for more than one-third of my life; for half of that, I've also been a software engineer.

Two different identities for the same person can make some target achievements hard to be accomplished.

Creating side projects or digital products is nearly impossible in this case. My "Prof of concept" folder is full of unfinished projects, all stranded before I even started writing the business logic, and this demonstrates that theory.

Why does this happen?

I spent a long time figuring out the causes and ended up with a simple answer.

When I start a new project, I want to reach two goals from my two personalities.
The first is to create something that resembles the product I imagined (for fulfilling my entrepreneurial soul)
the second is to learn a new language, framework or technology.

These two targets rarely blend harmoniously. And it usually happens that I kill the project for one of this two reasons:

Learning has ended, or my interest in the tech tools I used has waned.
I picked a technology, language or framework driven by the learning interest rather than the project needs taking me into a situation where I am trying to hammer a nail with a wrench.

What we need from a no-code tool

To avoid this problem, I have to find a pragmatic way to create my POC as fast as possible, focusing on the functional part and letting aside all the skills from my dev side as much as possible.

Yes, I know you are starting to smell the no-code and codeless fragrance in the air, and I also know that you will begin to say that using a low-code / no-code approach can force me to create a less tailored solution.

To understand if a low-code approach could be a real option and not just a wishful thought, I had to formalize which were the requirements.

Let's imagine that I want to create a backend for creating the feed for my podcast, and let's assume that I want to shift this need to make it reusable and multitenant to allow other podcasters to host their podcast on this platform, a sort of alternative to Anchor or Spreaker.

Those requirements are:

A rest or graphql crud api to make the interaction from other applications possible
A simple and extensible CRUD UI for entering data
A system that handles roles and rights also at the record level, like "this user can see just his records or records where his ID is in the receiver field."
A system for creating custom endpoints that run custom business logic
A system for creating workflows based on the time (cron), data entry or changes, etc
A system for creating custom UI
A file storage system that manages the data according to roles and rights.
A user management system

Ok, I can hear your thought, "ehi Mauro, you are looking for the moon", but this is the minimal structure of a minimal SaaS product.

Let's do a little exercise. Firstly, let's estimate the work needed to implement these features from scratch or with a framework.

Ok, no matter if you use a fast framework like rails o laravel, just one answer is correct: "too much code, too much time".

Directus

After searching for a while, trying different open-source, closed-source, on-premise and "as a service" products, I finally found an open-source solution that fits nearly 100% of my needs.

Directus has all the requirements mentioned above and more features with a special plus that it has a complete set of possible extensions.

We can use our developer superpowers.
If we reach the limit of the "out of the box API or UI."
We can extend our instance with custom UIs for fields, list pages, email templates, interfaces and complete custom modules, everything possible just by creating a simple package that uses Vue js.

We can also extend the backend part by writing simple js or typescript code creating knexjs database migrations, custom hooks and endpoints.

With Directus, I felt a new feeling. Finally, being a developer could help my entrepreneurial desires without introducing too much complexity or time needs.

The limits

Ok, All that glitters is not gold.
Until now, I spotted two limits:
A vuejs development experience is not suitable for me. Compared to react, it is more verbose and less ergonomic.
I wouldn't say I like express js. I cheer for fastify.

Do you feel them silly? 😂. I still am a developer, and cheering is our favourite sport.

Jokes aside, there is one big limit that Directus shows after a while you use it. So if you want to do something more complex than classic crud or if you want to create your extension, you need to rely on complete documentation.
Currently, the extensions docs miss some details in the extensions topic, to understand how things work, I dug into the source.
Since Directus is opensource, let's improve the docs together!

Ps. Here are the directus website and docs

I am tired of hearing banalities about remote working

BrainRepo — Sun, 04 Jun 2023 11:24:41 +0000

This article was previously hosted on brainrepo.dev and was written on January 30, 2023, at 03:57 PM.

All opinions I express here are my ones.*

As you know, I’ve been working for a full-remote tech company, and I love to do it!

I explain and share my opinion very calmly and moderately, but when I read statements that trivialize crucial topics, I lose self-control and rage.

That’s what happened a few days ago reading a post on LinkedIn.

But let’s see those bullshit one by one.

Come to think of it: the world is not built by people teleworking…
We want to invent and experiment with new ways of working.
Because, yes, the world is changing…
We want to train young people in our business on the ground.
We want to be there for our customers.
We want to share technical opinions in an open space that may be a little noisy but is full of life…
We want to shake hands.

We want to laugh at the coffee machine.
We don’t want this “trend” that isolates people.
It is a sad fate to don’t leave your home anymore!
Locked up at home… without an electronic bracelet… whatever 🤔…
So, yes, once again, I may push away the “charentaise” sellers and the “pilou pilou pyjama” suppliers.
We at **** are going back to real life.

And if you like living into cloistering, ordering online and living as an avatar in the metaverse, I think we’ll never meet, and that’s certainly not a bad thing!

Ok, firstly, a deep breath,

the world is changing, and how people interact also, I am not a 100% digital life fan, but I am for a good balance between the digital and physical parts. As you see, I don’t speak about real life because real life also contains a digital side!

“We want to be there for our customers”

what does it mean? Your customer needs your physical presence. Why? What can your body give more to your customer (😂)?

I think your customer prefers your attention, sensitivity, patience, expertise, pragmatism and effectiveness.

We can give all things with honest communication, commitment and empathy in front of a webcam (mandatorily on) or with a well-written email!

“We want to share technical opinions in an open space that may be a little noisy but is full of life.”

Hei man, if you would see our slack channels, where hundreds of people share opinions, experiences and their pets’ photos, you would understand. And you know, being a fully remote company allows us to amplify that because, in the channel, you see the true diversity, people from different places, with various religions or gender speaking together with respect, pleasure and open-mindedness.

I was born in a little village in the centre of Sardinia. Say to me when I could have had such an opportunity! Thanks to remote working, I have this opportunity and defend it with all my might!

Speaking about the opportunity, in a world without remote working, how a guy/girl can work for an international company and be part of a global impact without living in his/her hometown? This has a significant effect, you know? Little villages are dying, and people following the occasion are sacrificing their quality of life!

“We want to shake hands. We want to laugh at the coffee machine.”

How strong are the relations constructed in front of the coffee machine?

Quality rather than quantity determines the power of our relationships.

I am just coming back from a very intense company retirement with activities that helped me reach my limits, and I am pretty sure that the experiences I shared with my colleagues are stronger than years of coffee machine times! I can call them friends after seeing them just one time!

These shared experiences are necessary for professional remote working and not only that.

We at **** are going back to real life.

And if you like living into cloistering, ordering online and living as an avatar in the metaverse, I think we’ll never meet, and that’s certainly not a bad thing!

I am not isolated. I am remote and connected to the world.

If I am living with an electronic bracelet, I am happy to have it because my jail is the entire world!

Ps: There are another thousand good reasons why remote working well-made is a good thing, but I am nervous, tired, and at midnight. I’m going to bed. See you!

What would it be like if Italo Calvino and Sun Tzu explained refactoring?

BrainRepo — Sun, 04 Jun 2023 11:18:51 +0000

This article was previously hosted on brainrepo.dev and was written on February 19, 2023, at 03:57 PM.

Cities grow, cities evolve, cities have parts that simply die
while other parts flourish; each city has to be renewed in order
to meet the needs of its populace… Software-intensive systems
are like that. - Grady Booch (UML Inventor)

The day has flown by quickly, and I have a headache.

I spent the day refactoring the code. Refactoring is not a simple task - there are many things to keep in mind and many things to discover along the way.

Now is the time to relax and reread a beautiful book I read it many years ago.
"Invisible Cities" by Italo Calvino.

The city of Leonia refashions itself every day:
every morning the people wake between fresh sheets,
wash with just-unwrapped cakes of soap, wear
brand-new clothing, take from the latest model
refrigerator still unopened tins, listening to the
last-minute jingles from the most up-to-date radio.

As I read this, I'm reminded of Refactoring and all the conflicts and pains that come with it.

The Refactoring approach works similarly to Rudolph Giuliani's crime reduction strategy in New York City in the 90s.

Clean up the city, and the city will clean itself up.

Even though these simple words may sound like a magic formula, we have no magic wand. They hide commitment and dedication to the code/city we want to carry into the future.

What is refactoring?

Refactoring is an internal change or evolution (in the Darwinian sense) that does not alter the code's external behaviour and aims to improve the characteristics that make our code resilient, I am not refering to the bugfixing activity, but rather all the action for increase maintainability, performance and reduce the complexity.

It is said that to develop a good habit, we must repeat it, repeat it again, and repeat it some more, and the same goes for refactoring.

Refactoring shouldn't be a one off activity. We should create a routine following the infinite cycle of Test-Driven Development (TDD); write the test, develop the micro-functionality on the fly, just to make it work, and then refactorone off, just cleaning and refining what done.

Every time we refactor, we must do it in the best way possible, constantly guided by pragmatism (I come back to this issue later), in other words, organizing everything in order to complete the activity in the available timeframe.

And then, once again, a red test, a new feature, and refactoring...

Tactical and Strategic Programming

The first step to understanding refactoring is to realize there are two ways of programming: tactical and strategic.

Kent Beck uses the metaphor of two hats, one for writing code and the other for refactoring.

Whenever we write code, we put ourselves in the shoes of bricklayers -- the professionals who, in a tactical way, create something "that works" in the shortest time possible without being concerned about the complexity increase.

On the other hand, the strategic hat resembles the engineer's point of view and presupposes a holistic vision of the context. His priorities are to reduce or at least contain the system's complexity, even with a medium-long-term perspective.

Therefore, one of the engineer's tasks is to establish the goal of refactoring. To be meaningful and beneficial, the refactoring must always have a reason and a goal to achieve. This is the task from the engineer's perspective.

I have always said that a developer must be a monster with two personalities (that's why we are often thought of as crazy). We must learn to switch from one hat to the other quickly and with a good rhythm. In general, the more frequently we bounce between the two personalities, the better the quality of our code.

Developing a new feature or fixing a bug often increases coupling and complexity, sending style and coherence into hell. This is where we need to make a leap towards strategic thinking.

In "The Art of War", Sun Tzu says that the ideal future vision must influence present actions.

To achieve victory, the commander must know whether the battle he is fighting today will lead him there.

"Do not move unless you see an advantage; do not use your troops unless something is to be gained."

Unfortunately, the products we develop today often do not envision the future clearly. Our market moves quickly and is infamously mutable, so how can we guide our everyday work?

🧭 We can do it guided by Values.

Consistency

On the sidewalks, encased in spotless plastic bags,
the remains of yesterday's Leonia await the garbage
truck. Not only squeezed tubes of toothpaste,
blown-out light bulbs, newspapers, containers,
wrappings, but also boilers, encyclopedias, pianos,
procelain dinner services. It is not so much by the
things that each day are manufactured, sold, bought
that you can measure Leonia's opulence, but rather
by the things that each day are thrown out to make
room for the new.

To understand values, we must make our way through the garbage of Leonia. The first feeling we experience is disorientation. When we impact with this sensation, we notice the need for an essential value, a compass that we can call consistency.

The need for consistency usually arises when we sit behind our desks and face the monitor. Everything looks calm, but this is when we realize that our code looks alien to us.

Our efforts to understand become more and more, and we spend considerable time doing so.
Frustration accumulates when the code is not designed to accommodate the modification we want, letting us with just the dream of an ideal world where the quality of our codebase equals what we imagined during the design phase.

In this situation, a dangerous driver has fueled: fear. TDD helps us not be overwhelmed by this feeling, at least eliminating the uncertainty of what will happen once the modification is applied. But is it enough?
In a refactoring context where fear is present, refactoring turns into many little quick fixes. Small interventions that are driven by the intent to be less impactful as possible to avoid breaking things. Sounds scary, but here is a situation where a slight change in the code can have a ripple effect. Instead of bringing improvements, acting this way increases our code complexity, inconsistency, and instability. The code will need further refactoring again, triggering a negative spiral.

This is the result: the more Leonia expels goods,
the more it accumulates them; the scales of its past
are soldered into a cuirass that cannot be removed.
As the city is renewed each day, it preserves all of itself
in its only definitive form: yesterday's sweepings
piled up on the sweepings of the day before yesterday
and of all its days and years and decades.

This happens purely because we are not comfortable with our code. But why?

Because conventions help us orient ourselves and show us the way. What would a city be like if every neighbourhood had different street signs?

Consistency allows us to learn things in one place and reuse/recognize them everywhere.

Knowing the rules, we no longer need to rack our brains looking for patterns or making wrong assumptions.

Because of that, when we learn something that is not obvious from our code, we need to memorize it. But, unfortunately, our memorization ability is finite (mine is very limited), and, like the ancient scribes, when we achieve to solve the puzzle, we need to formalize our understood bringing it out from our minds and putting it into code (I mean not only on writing code but perhaps removing it is more important 😀).

The pitfalls in the search for consistency

The need for consistency leads us to a compulsive and often unconscious search, and here are two everyday actions that arise in this situation.

The comprehension refactoring and the aesthetic lifting (Yuch refactoring).

The violent attack of comprehension refactoring is activated when we don't understand what's happening. When we are entangled in a dense maze of classes and functions, and we don't know why they are there and what they do.

As Martin Fowler would say:

Any fool can write code that a computer can understand. Good programmers write code that humans can understand.

After all, I would say that our code is not a mystery novel.

The other trap lies behind the concept of aesthetic refactoring of our code.
When pronouncing “What is this rubbish, and who wrote it?” is the instant just before a lifting refactoring.

We, developers, dislike ugly or unfashionable code, and when we see something that does not adhere to our own standards of beauty, we refactor it.

Why do these two refactoring drivers hide a pitfall?

The desire to throw everything away and rewrite everything from scratch is just around the corner, and we often do it without worrying too much about the sustainability of our decision.

And so, to quote Calvino...

he greater its height grows, the more the danger
of a landslide looms: a tin can, an old tire, an unraveled
wine Bask, if it rolls toward Leonia, is enough
to bring with it an avalanche of unmated shoes, calendars
of bygone years, withered Bowers, submerging the city in
its own past, which it had tried in
vain to reject, mingling with the ~t of the neighboring cities,
finally clean. A cataclysm will Batten
the sordid mountain range, canceling every trace of
the metropolis always dressed in new clothes. In the
nearby cities they are all ready, waiting with bulldozers to
Batten the terrain, to push into the new
territory, expand, and drive the new street cleaners
still farther out.

By rewriting everything, we endanger our business. Our decision to rewrite code is often driven by an incomplete analysis of our codebase caused often by an incomplete/limited vision given by our current perspective. When we start the rewrite, we must reconsider all of the decisions we have taken over weeks, months, and years, facing a bill we cannot afford.

Rewriting everything means redoing the work that has brought us to this point, including the team discussions. We also need to do new training about the new conventions.

And then?

So, we have two paths to follow: accepting the compulsive refactoring cycle or recognizing what in our code is worth refactoring regardless of trends and/or compulsive attacks.

This is not an effortless decision, as it presupposes the development of a pragmatic awareness that leads to accepting imperfection and finding a balance between consistency and innovation.

To do this, it is essential to start with four questions:

Do I have enough information to justify my action (whether it is a consistency break or innovation introduction),
Wasn’t this information available in the design phase? And why was it not done that way if it was available in the design phase?
Am I breaking conventions? How much does this cost me regarding training for people involved in the codebase?
Does the change I am introducing really generate value in terms of maintenance from a product perspective?

Answering these questions will make it easy to see some considerations emerge.

Refactoring the code complex areas is just the refactoring that makes sense. On the simple and understandable parts is often an effort waste.
When estimating refactoring, we must always consider the full scope of the project, including edge cases. Often, the hidden costs and efforts are there.
Reconsidering conventions can often become a waste of time. New conventions can improve our code, but sometimes distributing knowledge is a hidden heavy cost.
When making decisions regarding refactoring, it is necessary to use a coordinated approach, involving all the stakeholders, sharing ownership and documenting decisions with ADR.

From what I have written so far, it might seem that I am not a fan of refactoring, but I like it.

I think that refactoring should be estimated implicitly in the project plan and diluted in the work of everyday life, avoiding the “formal refactoring moment” approach done to justify the "not done" in everyday work.

Refactoring should be automatic and probably should not have a name because it is implicit in the act of writing code (giving it a name mentally tends to purge it from the act of programming).

Refactoring should start by accepting imperfection, like in the Wabi-sabi Japanese philosophy. That vision or aesthetic is based on accepting the transience and imperfection of things. That vision of "imperfect, impermanent, and incomplete beauty" is typical of the Buddhist world.

So when can see refactoring as an evolution of conscious repair actions, which make our code tell a coherent story written over time, step by step.

Romantically, I like to imagine the value of code memory.

Restoring code not to a state of newness but also to a state that keeps the memory like done in Kintsugi, the art of repairing ceramics with gold.

Calvino describes this feeling and idea better than I in the final part of Invisible Cities when he says:

The inferno of the living is not something that will be; if there is one, it is what is already here, the inferno where we live every day, that we form by being together.
There are two ways to escape suffering it. The first is easy for many: accept the inferno and become such a part of it that you can no longer see it. The second is risky and demands constant vigilance and apprehension: seek and learn to recognize who and what, in the midst of inferno, are not inferno, then make them endure, give them space.

I let you some links if you wan't to read more about Italo Calvino or Sun Tzu

I also recorded one pocast episode / talk about it, but it is in Italian. If you come from the Dante's land you can enjoy it :D

https://www.gitbar.it/episodes/ep52-la-programmazione-e-il-refactoring-secondo-italo-calvino

How I converted a podcast into a knowledge base using Orama search and OpenAI whisper and Astro

BrainRepo — Tue, 23 May 2023 10:08:24 +0000

As many of you know, I have been hosting a podcast for the last three and a half years. It is one of the most exciting experiences of my life. I have produced approximately 200 hours of audio content in all these years.

The audio content has a drawback; if you want to search for something that was said, you often have to listen to hours and hours of old episodes without reaching the searched point.

How to solve this problem?

Transcript

The first step is to transcribe the episodes. Since the beginning, I have implemented a simple pipeline to transcribe each episode.

It turned out to be a total failure! The transcriptions were based on AWS Transcribe, and the service couldn't transcribe Italian audio correctly, perhaps due to the presence of technical English words or the remarkable Sardinian accent.

The outcome was terrible, it was impossible to read and understand the meaning, and they were not usable for my primary purpose. Aside from that, it also had a cost, each transcription was around 1 euro, and despite the low price, it was an absolute waste of money.

After one year, I stopped using the lambdas and decided not to transcribe the episodes anymore.

Also, the wise man retraces his steps, and who am I to not review my decisions?

Reviewing the decision is an outstanding practice, and doing it when the context changes could help us to stay tuned with the world around us and catch opportunities.

Since Open Ai started releasing its products, our industry was swept into a whirlwind of astonishment; chat GPT, copilot, and DALL•E were perceived as masterpieces, but another service caught my attention.

Whisper, as its name suggests, arrived without making much noise. While all the attention was on ChatGPT, Whisper was exceptionally interesting. Its quality, compared to other transcription services, is remarkable. It performs excellently in Italian, accurately recognising English words and technical jargon. I have never seen such precision before. Moreover, there is another non-trivial aspect—it is open source and released under the MIT license!

After conducting a test, I quickly embraced Whisper to transcribe the episodes. At first, I was tempted to set up a machine on AWS to run the entire process in the cloud. However, Whisper requires a massive amount of resources and time. Ultimately, I chose to run it on my gaming machine, which has been dormant for a while. My desktop computer, equipped with a GTX 3060, was the perfect candidate to put it to the test, especially after I stopped playing video games.

Whisper offers different pre-trained models, ranging from small to large. The largest one, which provides the best quality but is also the slowest, can utilise the mathematical capabilities of the GPU.

Since I am not a Python developer and PyTorch is unfamiliar to me, starting from scratch to implement the transcription script was nearly impossible.

Thankfully, a simple Docker image came to my rescue. This container simplifies all the steps and provides a REST API directly.

https://github.com/ahmetoner/whisper-asr-webservice

Now, it is enough to navigate to the web port exposed by the container to reach the swagger ui. From the Swagger UI, select the file, select the language (in my case, Italian), and wait around 30 minutes. Each episode is one and a half hours long, so Whisper needs quite a bit of time for transcribing. In the end, we will receive a well-structured JSON file containing the transcriptions with time references.

Cool, but now it's time to play with the code!

Make it searchable

With Whisper, I have completed the first half of the problem. Now it's time to discuss how to implement the search functionality.

Whisper can export in multiple formats, including TXT, VTT, SRT, TSV, and JSON.

In my case, I will be using the JSON data format. This format contains both the raw translated text and the time-coded text. The raw translated text is displayed on the episode page and is crucial for SEO purposes. The second part of the implementation revolves around the search functionality, which is one of the main pillars of this project.

The search process is fairly straightforward. There will be an input box where users can enter the words they wish to search for.

After submitting the search query, a list of audio samples that match the searched terms will be displayed.

The episode will be played by clicking on the "play" button next to each text slice, starting from when the word is pronounced.

The Gitbar website has no backend; it is entirely static and built using Astro, an excellent framework.

https://github.com/brainrepo/gitbar-2023

How to manage the search feature? Should I install Elasticsearch? How much will it cost? Or should I consider using Algolia?
These questions arose as I started implementing the feature.

From the beginning, using Elasticsearch was excluded as an option. Managing an Elasticsearch instance is not trivial, as it requires a server or computational capabilities. Similarly, using Algolia incurs additional costs, and since we rely on donations to support Gitbar's expenses, we need to minimise the expenditures.

Therefore, I needed to find an alternative solution.

I have been following Michele's project, Orama Search, since its inception, and I believe he, Paolo, Angela, and their "crime partners" are doing incredible work with it.

If JavaScript has democratised software development, I would say that Orama Search (also known as Lyra for nostalgic folks like me) has done the same for the search experience.

Initially, JavaScript may seem limiting, but thanks to it, we can run Orama Search everywhere, from the client to the server. It's truly amazing!

Another appealing aspect of Orama is its immutable nature, which makes it the perfect fit for my use case.

Since the Gitbar website is statically generated, it is not an issue for me to build the search index during the page generation process and share it as a simple JSON file.

To accomplish that, I created an Astro plugin inspired by the official one.

Now, let's dive into the details of what I have implemented.

Creating an Astro Plugin for Orama Search

Orama Search provides built-in support for Astro. It takes the generated files of the website and creates an index or database from the content within the HTML pages. However, my use case had specific requirements that differed from the common ones.

You can refer to the Astro plugin documentation for Orama for more information.

To meet my specific needs, I had to index a particular data structure that included the following fields:

text: The transcribed fragment, usually consisting of 10-15 words.
title: The episode title.
from: The timestamp indicating when the words are pronounced.
episodePath: The path of the episode page.

Given this requirement, I had to create a plugin from scratch to support it.

Astro provides a plugin API that allows us to extend its capabilities. It's important to note that the plugin API is relatively low-level. While it grants access to many internal details, it also requires caution when making changes to avoid unintended consequences.

Now, let's go through the steps involved in creating the plugin.

Initial Setup:

To start, create a new folder in the root directory of your project called /plugins. This folder will hold all the plugins for this project. Each Astro plugin is a JavaScript (or TypeScript) file that exports a single function.

export default () => ({
  name: "GITBAR-ASTRO-SEARCH",
  hooks: {
    "astro:server:start": async () => {
      await initDB("dev");
    },
    "astro:build:done": async () => {
      await initDB("prod");
    },
  },
});

Astro's core functionalities can be extended using hooks, which allow us to run custom JavaScript code at specific moments in Astro's lifecycle. In this case, we want to extend two pivotal moments: the server starting phase and the build done phase, when the website is fully built.

astro:server:start: Since Gitbar is a static website and will be served by Netlify servers, we don't require a Node server to run it. However, during the development environment, we want the plugin to build the Orama database for us to use in the development process.
astro:build:done: We use this hook to build the production database. When we release the website, along with the static pages, we also release a JSON file that contains a serialized **Orama database.

Data Preparation and Ingestion

To prepare the data for seeding the Orama database, I followed a multi-step process. Here's a breakdown of the steps you took:

Fetch the episodes from the podcast feed using the @podverse/podcast-feed-parser library. This allowed me to retrieve the necessary episode data.

const { episodes } = await getPodcastFeed(podcastFeed);

Iterate over the list of episodes and check if there is a corresponding translation file in the transcriptions folder. You searched for a file with the episode number as the filename.

const { episodes } = await getPodcastFeed(podcastURL);

const episodeSegments = episodes.map(async (episode) => {
  const episodeNumber = extractEpisodeNumber(episode.title);
  try {
      const json = await import(`../transcriptions/${episodeNumber}.json`);

      const segments = json.segments.map((s) => ({
        title: episode.title,
        path: getSlug(episode),
        from: s.start,
        text: s.text,
        mp3url: episode.enclosure.url,
      }));

      return segments;
  } catch (e) {
    console.log(`Transcription ${episodeNumber} not found`);
  }
  return [];
});

const results = await Promise.all(episodeSegments);
const segmentsToInsert = results.flat();

After obtaining the segments for each episode, I flattened them into a single array that contains all the segments of all the episodes.

After that, I created the Orama database by calling the create function providing the desired schema.

const db = await create({
  schema: {
    title: "string",
    text: "string",
    from: "number",
    path: "string",
    mp3url: "string",
  },
});

To efficiently insert the segments into the Orama database, I divided them into chunks of 500 units and used the insertMultiple function for batch insertion.

await insertMultiple(db, episodes, 500);

Finally, to complete the plugin, I serialised the database to a JSON file. This allowed me to share the database as a simple JSON file.

With these steps, I am able to prepare and ingest the necessary data into the Orama search index, using the appropriate schema and chunking techniques to optimise performance.

Search component

Now is the time to come out from the shadows and create the components for our search form. Since Astro supports React components, let's write our search component.

const Search = () => {

    return ...
}

I will skip some parts to keep the focus on the interesting ones.

The first thing I want to do is fetch the Orama database that we built earlier from the network. I want to do this during the mounting phase of the component.

I'll use a useEffect hook where:

I initialise the Orama instance with the same schema used before.
I load the database file and track the loading state to disable the search UI during the loading process.
I load the fetched data into the Orama instance.
I update the DB state to make the Orama instance available to the component.

import { search, create, load } from "@orama/orama";
...

const Search = () => {

    // State that holds the database
    const [DB, setDB] = useState(null);

    // State that holds the loading status
    const [isLoading, setIsLoading] = useState(true);

    useEffect(() => {
      const getData = async () => {
        const _db = await create({
          schema: {
            title: "string",
            text: "string",
            from: "number",
            path: "string",
            mp3url: "string",
          },
        });
        setIsLoading(true);
        const resp = await fetch(`/in_episode.db.json`);
        setIsLoading(false);
        const data = await resp.json();
        load(_db, data);
        setDB(_db);
      };

      getData();
    }, []);

    return ...
}

Lastly, we need to create our search function, which we'll call when there is a change in the search field input.

To avoid creating a new find function with every rendering, we'll use the useCallback hook to cache it and update it when DB or setResults changes.

The rest of the function calls Orama's search function with the search term and runs the search on all properties, retrieving the first 100 results.

const find = useCallback(
    async (term: string) => {
      const res = (
        await search(DB, {
          term,
          properties: "*",
          limit: 100,
        })
      )?.hits?.map((e) => e.document);

      setResults(res);
    },
    [DB, setResults]
  );

Now all that's left is to attach this function to an input field change event, and we're done!

<input
    className="bg-transparent w-full text-white p-6 text-2xl outline-yellow-300 outline-3 placeholder:text-yellow-300"
    placeholder="Search..."
    onChange={(e) => console.log(find(e.target.value))}
  />

The audio player features are beyond the scope of this article. Let me know if you would like me to write an article on that.

I intentionally left out some other implementation details. If you want a running code example, take a look at https://github.com/brainrepo/gitbar-2023/blob/main/src/plugins/lyra_in_episode.ts for the database creation and https://github.com/brainrepo/gitbar-2023/blob/main/src/components/searchAdvanced.tsx for the frontend JSX code.

The limits

Currently the search feature is in production on this url, but if you check the network inspector you will see that the Orama database size is more the 40MB, note that not all the episode are indexed, the transcription process takes lots of time and I transcribed and indexed from the episonde n*130 to the number 156.

The size is not negligible, and I expect that the size of the database can reach 200MB pretty soon.

Botstrapping a nodejs server for running it seems the most reasonable way to solve the problem, but I don’t want to do it than I need a plan B.

The B Plan

My Plan B has two levels:

The first solution is very straightforward. The JavaScript ecosystem offers some fantastic libraries that can assist with GZIP compression. It appears that the Netlify servers don't handle compression, so I can leverage libraries such as pako, tiny-inflate, uzip.js, or fflate to achieve this goal. I conducted some tests, and the compressed database size was reduced to just 10% of the original size. Implementing this solution requires only a few lines of code, specifically less than 4. By incorporating this solution, I can easily handle up to 300 episodes with sustainable download times. Considering that I have recorded around 160 episodes in the past three and a half years, I can sleep soundly because I have ample time ahead.
Whenever I encounter an upper limit, I usually engage in a mental exercise to find a workaround. What if I group the episodes into chunks of 10 or 20 elements and create an Orama database for each group? Furthermore, what if I begin my search with the most recent episodes and, once I have fetched and searched within the first group, I proceed to the older databases until the result limit is reached? This approach would compel the results to be sorted by date (which could even be considered a feature), and the sequential style of this search feature prevents the need to download all the databases in advance.

There is always a C Plan

Ok, these two B Plans have stimulated my rambling. What if I incorporate the Chat GPT API to provide answers in natural language, using Orama for the time-stamped results? That way, I can have a contextual conversation with the generated excerpt from Chat GPT and rely on the accurate source of information and timestamp reference from Orama search results.

Alright, alright, it's time to come back to reality now. @micheleriva, I'm blaming you for this mind-bending journey 😂!

Before wrapping up this article, I want to extend my heartfelt appreciation to Michele, Paolo, Angela, and everyone involved in the hard work on Orama. Hey folks, keep up the great work—I'm a big fan!