Forem: Amnish Singh Arora

Never lose valuable error context in JavaScript

Amnish Singh Arora — Thu, 22 May 2025 01:00:45 +0000

If you are a JavaScript developer, you might have ran into error messages like:

Failed to extract text from the file
Failed to save entity
And so on...

But what exactly led to failing of an entire high-level operation?

We don't know the answer because we didn't do a good enough job to preserve the error context, when throwing a specific error that is a symptom of a more low level cause.

For years, JavaScript developers have struggled to preserve appropriate error context when throwing exceptions, as there was no in-built mechanism provided by the language to attach error cause, unlike other languages like Java, Python and C#.

This problem was solved by Node, when they finally added a cause property to their Error constructor in v16.9.0, which means you could now throw exceptions like so:

try {
    await fetch("some-resource");
}
catch(error) {
    throw new Error("Specific error message", {
        cause: error
    });
}

This means you now get to preserve the entire context of the original error that led to throwing of a high level exception (including its message and stack trace).

1. The problem
2. The solution?
3. Installation
4. Configuration
4.1. VSCode Extension
5. Conclusion

The problem

This feature has been available in Node for a while now (added in 2022), and although it solves the big problem of not having a standard way to preserve error context, it introduces a new problem as its completely optional to attach the error cause to new exceptions.

The solution?

What we need is a static analysis tool to enforce this as a coding convention — ensuring developers always attach available causes when throwing new errors.

While ESLint is the go-to tool for code quality in JavaScript, it doesn’t provide any built-in rule for this.

So I decided to write one myself!

Introducing eslint-plugin-error-cause — an ESLint plugin that flags any throw new Error(...) calls that omit the cause when inside a catch block.

Let's talk about how we can set this up in our projects.

Installation

The very first thing you need to do is install the plugin and eslint as dev dependencies in your project. This example uses pnpm, but you could use a package manager of your choice.

pnpm add -D eslint eslint-plugin-error-cause

Configuration

1. Once installed, create an eslint.config.js file at the root of your project.

2. There are two ways to add this plugin to your eslint configuration.

Use the in-built recommended config, that enables no-swallowed-error-cause rule with a severity level of warn.

import errorCause from "eslint-plugin-error-cause";
import { defineConfig } from "eslint/config";

export default defineConfig([errorCause.configs.recommended]);

Enable the rule manually with a severity level of your choice.

import errorCause from "eslint-plugin-error-cause";
import { defineConfig } from "eslint/config";

export default defineConfig([
    {
        plugins: {
            "error-cause": errorCause,
        },
        rules: {
            "error-cause/no-swallowed-error-cause": "warn",
        },
    },
]);

3. Setup lint scripts in your package.json.

"scripts": {
    ...
    "lint": "eslint .",
    "lint:fix":  "pnpm lint --fix",
    ...
},

4. Use the lint script to report all the lint errors in your code. 🎊

> pnpm lint

3:10  error    'rootError' is defined but never used                                 no-unused-vars
4:5   warning  Include the original caught error as the `cause` of the custom error  error-cause/no-swallowed-error-cause

5. Use --fix to fix any instances on no-swallowed-error-cause.

> pnpm lint:fix

VSCode Extension

Almost everyone uses ESLint's VSCode extension to detect and fix lint errors in-place when coding.

If you don't have it setup, follow these steps:

1. Install the ESLint extension.

2. Add this setting to your workspace/user settings.json. You could add/omit the JavaScript flavours based on your project needs.

"eslint.validate": [
  "javascript",
  "javascriptreact",
  "typescript",
  "typescriptreact"
  ],

3. Now you can detect and fix swallowed error causes using VSCode's UI.

Conclusion

So that's all about how you could setup my new plugin in your projects, and never again worry about missing important error context when throwing custom exceptions!

This plugin is still very new and might not cover some edge-cases. I welcome any interested readers to visit the GitHub repo, open issues, send in pull requests, or even give it a star and share with your connections if this helped you in any way 😁

Github: https://github.com/Amnish04/eslint-plugin-error-cause

NPM: https://www.npmjs.com/package/eslint-plugin-error-cause

Concluding OSD700

Amnish Singh Arora — Sun, 21 Apr 2024 02:26:26 +0000

My open source journey started about 8 months ago when I enrolled myself in OSD600, a course delivered by the one and only David Humphrey.

In that course, we learned about the basics of open source, like how to make good PRs and contribute to random open-source projects, and how to use Git effectively in the process. We participated in events like Hacktoberfest that helped us embrace the spirit of open source.

This term, I enrolled myself in OSD700, that acts as a second part of 600. This time, instead of jumping across random projects every time, we stuck to one project and stuck to that.

That project was what I have been writing about for last 4 months - ChatCraft.

My journey with ChatCraft and OSD700 has been super rewarding.

Not only was it the most fun course I ever had, but:

I was able to learn about a lot of cool technologies that I had no idea of before this. Some of them being Cloudfare Pages, Cloudfare R2, PnPM, Secret Management with SOPS, Vite, and Web Assembly.
Working on ChatCraft was a great introduction to leveraging the power of AI models using OpenAI's API, as I was able to apply my learnings to a real project and implement cool features along the way.
I learned about common open source practices like running a triage meeting, the role of a Sheriff, and the recipe of how people are attracted to an open source project. People want to be involved with projects that are alive, where there's a community that is active.
And I realized the role blog posts play not only in an individual's career, but in the growth of your projects (if you write about them). Writing about your work and projects is one of the best way to marketing your skills and your work.

This has been a true privilege working on a cool project with a great team, and under an exceptional mentor.

Of course, this is not the end of my open source journey, but I see it as the best beginning possible.

Its very common to get comfortable with doing one thing after graduating (at a job), keep doing it until it becomes obsolete, and end your career by making that mistake.

- Taras

I feel like I have already had such an experience, even though I haven't officially started. I've been working as an Angular Developer for almost 2 years now, continuing at the place where I got my coops. Before this course, I had become too comfortable with doing that one thing (Angular UIs) that I felt uncomfortable in exploring different things, and I knew that was not good.

Working on ChatCraft pulled me out of that pit, and helped me realize that Open Source is a great way to explore different technologies and stay up to date with the latest trends even after graduating and starting a job.

I would like to end this post by once again thanking all my peers, Taras, and David for helping me have the have the best learnings possible on this journey 😄

Improve Download Speeds with Concurrency

Amnish Singh Arora — Sun, 21 Apr 2024 02:26:02 +0000

This week, I came across an interesting problem while working on an issue in ChatCraft.

A few weeks ago, I added text to speech support to the app, which when combined with other features, leads to some 😎 use cases.

I have already written a few posts about it. Feel free to take a look for more context.

In this post, I'll talk about a problem that I encountered with Text to Speech, how I solved it and leveraged Concurrency to optimize the performance of end result.

1. The Problem
2. Breaking larger messages into Chunks
       2.1. Algorithm for Natural Chunks
       2.2. Using Text Chunks to generate Audio
       2.3. Optimizing Download Speeds
3. Results
4. Release 2.0

The Problem

The Text to Speech feature had been working really well, until I encountered this error a few days ago.

What I was trying to do was download an interview as audio after importing its transcriptions to ChatCraft using Web Handlers (as simple as pasting Youtube video's url in prompt).

But while implementing the Download and Speak features for chat messages, I forgot to consider that OpenAI's TTS API might have a limit on how much it can process in a single request (4096 characters).

That is the reason I got this error as I was sending the entire chat content in one request for processing.

Breaking larger messages into Chunks

An obvious solution for this problem was to break larger messages into smaller chunks, send those chunks for audio generation, concatenate the generated audio blob pieces into a single blob, and finally download the weaved audio on user's file system.

On surface, this may seem like a very straightforward problem to solve, but there is another aspect to it.

For the TTS model to generate natural audio, the source it ingests also needs to be natural. In other words, I couldn't just break larger sentences to a certain max length without considering scenarios that could lead to inaccurate results.

For example,

Parts of a single word could end up in different chunks.
Parts of the same sentence could end up in different chunks.

Algorithm for Natural Chunks

To go around this problem, I leveraged the Natural Language Processing library we are using in the project (Compromise) to make sure that each chunk was less than a specified character length, and
the chunks ended with a full sentence (as long as the sentence itself wasn't longer than the limit).

Here's the algorithm:

First, I try to break the entire text into sentences using the NLP library mentioned above.
Then I initialize a text buffer variable and start iterating over each sentence.
If the adding the current sentence to the text buffer does not exceed the preferred chunk length, I add it to the buffer.
If it does exceed the limit, I push the buffer content as a new chunk to chunks array and assign the text buffer with the current sentence's value.
There is still one exception here. If the current sentence's length itself is greater than preferred chunk length, I have to force break it into smaller chunks as there is no way to have a natural meaningful chunk at that point.
Lastly, I return the list of generated chunks.

The code looks like this:

import nlp from "compromise/one";

export function tokenize(text: string) {
  const sentences: string[] = nlp(text)
    .json()
    .map((s: { text: string }) => s.text);
  const terms = nlp(text).terms().out("array");

  return { sentences, terms };
}

/**
 *
 * Tries to split the provided text into
 * an array of text chunks where
 * each chunk is composed of one or more sentences.
 *
 * The function attempts to limit each chunk to maximum
 * preferred characters.
 * If a single sentence exceeds preferred character length,
 * that sentence will be force broken into chunks of preferred length
 * with no guarantee that individual chunks make sense.
 *
 * @param text The text content that needs to be split into Chunks
 * @param maxCharsPerSentence Maximum number of characters preferred per chunk
 * @returns Array of text chunks
 */
export function getSentenceChunksFrom(text: string, maxCharsPerSentence: number = 4096): string[] {
  const { sentences } = tokenize(text);
  const chunks: string[] = [];

  let currentText = "";

  for (const sentence of sentences) {
    if (sentence.length >= maxCharsPerSentence) {
      // If the sentence itself is greater than maxCharsPerSentence

      // Flush existing text buffer as a chunk
      if (currentText.length) {
        chunks.push(currentText);
        currentText = "";
      }

      // Force break the long sentence without caring
      // about natural language
      const sentencePieces =
        sentence.match(new RegExp(`.{1,${maxCharsPerSentence}}\\b`, "g")) || [];

      chunks.push(...sentencePieces);
    } else {
      // Check if adding the new sentence to the buffer
      // exceeds the allowed limit.

      // If not, add another sentence to the buffer
      if (currentText.length + sentence.length < maxCharsPerSentence) {
        currentText += ` ${sentence.trim()}`;
      } else {
        // Flush the buffer as a chunk
        if (currentText.length) {
          chunks.push(currentText);
        }

        currentText = sentence;
      }
    }
  }

  if (currentText.length) {
    chunks.push(currentText);
    currentText = "";
  }

  return chunks;
}

Using Text Chunks to generate Audio

Now that I was able to break large messages into natural language chunks, it was time to generate audio clips using those and weave them together into full audio message for the user.

Here's what I did:

const handleDownloadAudio = useCallback(async () => {
  if (messageContent.current) {
    const text = messageContent.current.textContent;
    if (text) {
      const { loading, closeLoading } = await utilizeAlert();

      const alertId = loading({
        title: "Downloading...",
        message: "Please wait while we prepare your audio download.",
      });

      try {
        const textChunks = getSentenceChunksFrom(text, 4096);
        const audioClips: Blob[] = [];

        for (const textChunk of textChunks) {
          const audioClipUrl = await textToSpeech(
            textChunk,
            settings.textToSpeech.voice,
            "tts-1-hd"
          );

          audioClips.push(await fetch(audioClipUrl).then((r) => r.blob()));
        }

        const audioClip = new Blob(audioClips, { type: audioClips[0].type });

        download(
          audioClip,
          `${settings.currentProvider.name}_message.${audioClip.type.split("/")[1]}`,
          audioClip.type
        );

        closeLoading(alertId);
        info({
          title: "Downloaded",
          message: "Message was downloaded as Audio",
        });
      } catch (err: any) {
        console.error(err);

        closeLoading(alertId);
        error({ title: "Error while downloading audio", message: err.message });
      }
    }
  }
}, [error, info, settings.currentProvider.name, settings.textToSpeech.voice]);

If you are wondering about the textToSpeech function, I wrote about it in this post.

After this change, ChatCraft could download content of infinite length as audio.

But... there was still one more problem with this.

Optimizing Download Speeds

Even though breaking large messages into chunks solved our problem, the user experience was still not optimal as download speed for long messages was painfully low.

The reason was that the current algorithm only processed one chunk at a time.

for (const textChunk of textChunks) {
  const audioClipUrl = await textToSpeech(
    textChunk,
    settings.textToSpeech.voice,
    "tts-1-hd"
  );

  audioClips.push(await fetch(audioClipUrl).then((r) => r.blob()));
}

It never struck me before until my professor suggested that we could run these promises concurrently!

He also asked to use a library called p-limit to limit the number of concurrent promises at a time.

I am not that knowledgeable about concurrency, so I asked ChatCraft why would we need to limit concurrent execution.

This is what I got.

After understanding certain nuances of this approach, I modified my approach to concurrently generate 8 audio chunks at a time, and limited each message's length to a maximum of 500 characters.

Here's my implementation.

const textChunks = getSentenceChunksFrom(text, 500);
const audioClips: Blob[] = new Array<Blob>(textChunks.length);

// Limit the number of concurrent tasks
const pLimit = (await import("p-limit")).default;

const limit = pLimit(8); // Adjust the concurrency limit as needed

const tasks = textChunks.map((textChunk, index) => {
  return limit(async () => {
    const audioClipUrl = await textToSpeech(
      textChunk,
      settings.textToSpeech.voice,
      "tts-1-hd"
    );

    const audioClip = await fetch(audioClipUrl).then((r) => r.blob());
    audioClips[index] = audioClip;
  });
});

// Wait for all the tasks to complete
await Promise.all(tasks);

const audioClip = new Blob(audioClips, { type: audioClips[0].type });

download(
  audioClip,
  `${settings.currentProvider.name}_message.${audioClip.type.split("/")[1]}`,
  audioClip.type
);

You'll notice that I am using a dynamic import for p-limit, to avoid unnecessarily increasing initial bundle size of the application.

And I also had to make sure the generated chunks were in order by keeping track of indices, now that the execution was not sequential.

Results

With all these changes, I was able to download an audio clip that was more than 30 minutes long in just less than a minute!!!

I guess Taras is excited about downloading an Audio Book next 😝

Release 2.0

We have been building on ChatCraft 1.0 for about 4 months now. After 9 minor releases, it was finally time to do a major release this week and I am glad this work could go in that 🥳

ChatCraft 2.0 is almost Here!

Amnish Singh Arora — Sun, 14 Apr 2024 00:03:33 +0000

That's right. The BIG 2.0 release for ChatCraft is scheduled by the end of next week. We have been working to build on ChatCraft's existing functionality in its v1.0 state for almost 13 weeks now, and have successfully added lots of cool features (even this post's cover photo is generated by ChatCraft).

However, adding these features in such a short span of time has also led to some technical debt that needs to be cleaned up and subtle bugs that need to be fixed so any future contributors have a good experience working on this project.

This week, I did some of the refactoring work, fixed a bug and also helped my peers to do the same by reviewing their PRs and help wherever possible.

In this post, I will be briefly talking about what I've been up to, and what I plan to do this upcoming week so we can make our 2.0 release a success!

1. Fixing a Web Handlers related bug
2. Refactoring PasswordInput Component
3. Reviewing other PRs
4. Next Plans

Fixing a Web Handlers related bug

For the past few weeks, I had been working on adding a new feature called Web Handlers to ChatCraft. Even though there are many more ways to still extend that feature, I think its a good time to fix any bugs introduced with it before for now since a major release is coming up.

The issue was that when the user changed any of his previous messages and pressed Re-Ask button, it would not invoke any Web Handlers even if the updated prompt text matched with one of the registered handlers' match pattern.

The reason for this happening was that prompt text was not at all being passed to the resubmit function which led to the message being sent to LLM by default.

Hence, the fix was as simple as follows:

Refactoring PasswordInput Component

Recently, support for custom AI providers was added to ChatCraft and in implementing the UI for providers selection, changes were made to the existing RevealablePasswordInput component to make the appearance ideal for a particular use case.

However, I felt that the code had some design issues.

This is what it looked like:

import { useState, type ComponentPropsWithRef } from "react";
import { Input, InputGroup, InputRightElement, Button } from "@chakra-ui/react";

type PasswordInputProps = ComponentPropsWithRef<typeof Input> & {
  inputSize?: "sm" | "md";
  isInvalid?: boolean;
};

function PasswordInput({ inputSize = "md", isInvalid, ...props }: PasswordInputProps) {
  const [show, setShow] = useState(false);

  const paddingRight = inputSize === "sm" ? "2.5rem" : "4.5rem";
  const paddingLeft = inputSize === "sm" ? "0.4rem" : undefined;
  const inputFieldSize = inputSize === "sm" ? "sm" : "md";
  const buttonSize = inputSize === "sm" ? "xs" : "sm";

  return (
    <InputGroup size={inputFieldSize}>
      <Input
        {...props}
        pr={paddingRight}
        pl={paddingLeft}
        type={show ? "text" : "password"}
        focusBorderColor={isInvalid ? "red.500" : "blue.500"}
        borderColor={isInvalid ? "red.500" : "gray.200"}
        isInvalid={isInvalid}
      />
      <InputRightElement width={paddingRight}>
        <Button variant="ghost" h="1.75rem" size={buttonSize} onClick={() => setShow(!show)}>
          {show ? "Hide" : "Show"}
        </Button>
      </InputRightElement>
    </InputGroup>
  );
}

export default PasswordInput;

1. Programatically determining styles based on other styles and hard coding values inside component caused tight coupling between different styles making it hard to reuse in other cases in the future. (Would need to add more if statements every time a new appearance was required)

2. In a way, it overrode Chakra's in-built styling system which is not good.

Which is why I created a Pull Request to refactor that part.
I tried to make use of Chakra's own design system as much as possible turning it into a more generic wrapper around the regular Input component provided by Chakra.
In other words, PasswordInput now behaves more like a component provided by Chakra itself since most of the styles/props are being mapped directly.

Reviewing other PRs

I also reviewed some Pull Requests opened by other contributors.
Here's a list with hyperlinks.

1. Replaced Show/Hide button with Eye ReactIcons
2. Feed icon should open raw feed in a new tab
3. Fix toast message width for mobile devices

You can take a look if interested.

Next Plans

With all these contributions and many more, we were able to do a minor v1.9.0 release taking us one step closer 2.0.

We are very close to a successful major release 😎

Who else is excited? Any ChatCraft users here? Share your thoughts in comments!

Adding a Code Editor to your React App

Amnish Singh Arora — Mon, 08 Apr 2024 01:36:14 +0000

For the last couple of weeks, I have been working on implementing the idea of Web Handlers in ChatCraft. And last week, I finished setting up the initial functionality where users can save their configurations to extend ChatCraft's web handling capabilities.

Since the configuration is done in YAML, this week I decided to enhance that feature by replacing the regular TextArea element I was using for saving configurations with a dedicated Code Editor for a developer friendly experience, which I'll be talking about in this post.

1. How it looked before
2. Replacing with a Code Editor
2.1. Adding to ChatCraft
3. Conclusion

How it looked before

Before talking about what code editor component I used for this purpose, here's what the configuration modal looked like before this change.

Replacing with a Code Editor

Now imagine you were forced to write your code in notepad. Naturally, it wouldn't be a pleasant experience 😝

So our users don't have to go through that pain, I used a library called react-codemirror to replace it with a code editor written for the web.

From the official CodeMirror documentation:

CodeMirror is a code editor component for the web. It can be used in websites to implement a text input field with support for many editing features, and has a rich programming interface to allow further extension.

This editor is highly configurable

and supports a whole bunch of languages/frameworks

And it is super simple to add it to our projects.

Adding to ChatCraft

The first step whenever we want to use an external package is, of course 🥁🥁🥁

1. to install that package.

npm install @uiw/react-codemirror --save

2. Next, just use the component wherever you need in your code like so

import ReactCodeMirror from "@uiw/react-codemirror";


ReactCodeMirror
  value={webHandlerConfig}
  theme={colorMode}
  height={editorHeight}
  onChange={handleConfigValueChange}
/>

3. Lastly, you need to set the language extension that you want to be supported by the code editor.

What I noticed is that these language extension packages follow a pattern of the format

"@codemirror/lang-<language_name_here>"

Since I was dealing with YAML, I installed the following package

npm install @codemirror/lang-yaml

and in the JSX, just add to the extensions property of ReactCodeMirror.

import { yaml } from "@codemirror/lang-yaml";
...
...
ReactCodeMirror
  value={webHandlerConfig}
  theme={colorMode}
  extensions={[yaml()]}
  height={editorHeight}
  onChange={handleConfigValueChange}
/>

And that's all it took to make it look like this 🤩

Conclusion

This was a brief overview of how you can use CodeMirror editor in your React Applications.

For full documentation, you can visit
https://codemirror.net/docs/

This also went to ChatCraft's 1.8.0 release which happened this week.

Hope this post offered some value to you.
I am planning to work on some other cool stuff next week.

Will be writing on it soon!

Iterating on Web Handlers Implementation

Amnish Singh Arora — Sun, 31 Mar 2024 18:56:28 +0000

Last week, I wrote about introducing the idea of Web Handlers in ChatCraft, and how I implemented an initial proof of concept with a hard-coded Web Handler.

This week, after receiving some feedback, I extended upon that functionality by adding a form that allows users to configure as many Web Handlers as they wish in YAML, which are ultimately persisted in their browser's localStorage.

1. How it works 🏃
       1.1. The Model
       1.2. Ability to Configure
       1.3. Executing Handlers
       1.4. Making Sense of it
2. Looking Ahead 👀

How it works 🏃

At this point, you might have been a little confused about what exactly I've been talking about all this time. Trust me, I was also confused when I signed up for this task.

So, here's a quick demo 🥂

The Model

If you remember from my last post, I was mostly discussing about the model of Web Handlers.

It looks something like this in its current state.

export class WebHandler {
  handlerUrl: string;
  method: HttpMethod;
  matchPattern: RegExp;

  constructor({
    handlerUrl,
    method,
    matchPattern,
  }: {
    handlerUrl: string;
    method: HttpMethod;
    matchPattern: RegExp | string;
  }) {
    this.handlerUrl = handlerUrl;
    this.method = method;
    this.matchPattern = matchPattern instanceof RegExp ? matchPattern : new RegExp(matchPattern);
  }

  isMatchingHandler(message: string) {
    return this.matchPattern.test(message);
  }
  ...
  ...
  ...
}

Ability to Configure

Users will now have the ability to configure their own Web Handlers in YAML.

Here's an example:

Executing Handlers

And the first matching handler gets executed. In my case, I have configured YouTube URLs to be handled by a mock api just for demonstration purposes.

Prompt:

https://www.youtube.com/watch?v=bnFa4Mq5PAM

Result:

Making Sense of it

This might not make sense at first, but essentially, you can extend the functionality of ChatCraft without the project implementing new features. You can write your own Microservices to provide some functionality and register Web Handlers in ChatCraft to send HTTP requests to your Microservice with your entered message attached to it.

A more realistic example would be using Taras's Scrape2Md Microservice as handlerUrl.

Let's try another YouTube URL now with updated config:

Prompt:

https://www.youtube.com/watch?v=tfSS1e3kYeo

Result:

It gave me the lyrics of the song I pasted, and since we have Text to Speech in ChatCraft, I can ask one of our singers

to sing or download an instantly remixed song for me 🎶🪩🤩

Looking Ahead 👀

You must have noticed that it is already kinda powerful, and still holds a lot of potential as I said in my last post as well.

The coding part for this was fairly simple as I just had to implement a UI for the functionality I wrote last week.

If you really want to follow along, you can look this PR.

Implement initial version of web handlers #519

Amnish04 posted on Mar 21, 2024

This is my initial attempt in implementing the idea of what we can call web handlers as discussed in #507 .

I have:

Written a WebHandler class registering/searching web handlers and executing their logic i.e. sending request to handler url based on request config (so far request method)
Added logic to check if the prompt matches with match pattern of one of the registered Web Handlers. The search is sequential as mentioned in the issue, and the first matched web handler processes the prompt url. If there is no match, the prompt is released to the regular control flow (slash command or sent to AI).
Currently, I have hard coded 2 Web Handlers as mentioned in issue, but we can later create UI to configure as many as user wants in localStorage.

Demo:

Prompt: https://github.com/tarasglek/chatcraft.org/pull/519

What I don't like is how we are currently allowing prompt like import <url> to be registered as match patterns. This causes ambiguity and I had to write specialized logic to handle edge cases like

url = extractFirstUrl(url) ?? ""; // When the input is not a url itself

as an example.

I feel like we should restrict web handler match patterns to just urls, and let import command serve its own purpose.

Maybe I am wrong and don't understand fully where we are going with this.

@humphd @tarasglek Please let me know what you think.

View on GitHub

Next week, I am planning to replace the TextArea being used for configuration with a dedicated code editor called CodeMirror for a better developer experience.

I may also work on making the configuration more flexible by allowing more options to be configured for the handler.

If you have any questions or suggestions, feel free to start a conversation in the comments!

I will follow up with another post 🔜

Introducing the idea of Web Handlers in ChatCraft

Amnish Singh Arora — Sun, 24 Mar 2024 17:23:52 +0000

Have you ever considered the concept of a Chatbot that not only enables interaction with Language Models (LLMs), but also offers the capability to integrate your own microservices for the most personalized experience imaginable?

This week, I came across an issue in ChatCraft that was asking for something similar.

In this post, I'll discuss how I was able to design and implement an initial version of this feature.

1. Background 💭
1.1. Spin Off 🧐
2. Implementation 🧑🏻‍💻
3. Upcoming 🔮

Background 💭

I've been working on ChatCraft for a couple of months now and have already written many posts about it. It comes with many cool features apart from what a regular AI Chatbot offers, like ability to define custom ChatCraft Functions, in-built utility commands (use /commands to get a list), text-to-speech, and many more.

This is related to /import command (an existing feature) that allows you to make request to any URL and loads the text response as a system chat message.

Result:

Spin Off 🧐

The idea is to build something that extends this functionality.

Here's how it would work:

We can allow the users to register various Web Handlers with a match pattern.
If the message matches any of the registered web handlers, an HTTP request is sent to the Web Handlers's handler url based on the specified configuration.
The received response is streamed into a new Chat Message in Markdown format.

This might sound a little confusing, so let's try to understand with an example.

Let's say you want the YouTube URLs to be handled in a certain way. You can write your own service or a serverless cloud function for processing. Then configure a ChatCraft Web Handler matching YouTube URL chat messages (use match pattern) that makes a request to your service for custom functionality.
This would allow you to plug and play any custom behaviour in ChatCraft that is not supported by default. And if that feature becomes popular, maybe we can add if by default in ChatCraft 😉

Implementation 🧑🏻‍💻

To implement an initial version of this concept, I started by designing a Web Handler class defining the model and behaviour of this new entity.

The model consists of 3 main properties - handlerUrl, request method and matchPattern.

export class WebHandler {
  handlerUrl: string;
  method: HttpMethod;
  matchPattern: RegExp;

  constructor({
    handlerUrl,
    method,
    matchPattern,
  }: {
    handlerUrl: string;
    method: HttpMethod;
    matchPattern: RegExp;
  }) {
    this.handlerUrl = handlerUrl;
    this.method = method;
    this.matchPattern = matchPattern;
  }
  ...
  ...
  ...
}

export enum HttpMethod {
  CONNECT = "CONNECT",
  DELETE = "DELETE",
  GET = "GET",
  HEAD = "HEAD",
  OPTIONS = "OPTIONS",
  PATCH = "PATCH",
  POST = "POST",
  PUT = "PUT",
  TRACE = "TRACE",
}

and some methods defining its behavior

export class WebHandler {
  ...
  ...
  isMatchingHandler(message: string) {
    return this.matchPattern.test(message);
  }

  async executeHandler(message: string): Promise<string> {
    const requestUrl = new URL(this.handlerUrl);

    const params = new URLSearchParams();
    params.append("url", message);

    requestUrl.search = params.toString();

    const response = await fetch(requestUrl, {
      method: this.method,
    });

    const content = (await response.text()).trim();

    const resultHeader = `**Web Handler**: [${this.handlerUrl}](${this.handlerUrl})?url=[${message}](${message})`;
    const text = `${resultHeader}\n\n` + content;

    return text;
  }

  static getMatchingHandler(message: string): WebHandler | null {
    return (
      this.getRegisteredHandlers().find((handler) => handler.isMatchingHandler(message)) ?? null
    );
  }

  static getRegisteredHandlers(): WebHandler[] {
    // TODO: Fetch from localStorage

    const supportedHandlers = [
      new WebHandler({
        handlerUrl: "https://taras-scrape2md.web.val.run/",
        method: HttpMethod.GET,
        matchPattern: /^https:\/\/\S+$/,
      }),
    ];

    return supportedHandlers;
  }

This is a pretty simple model, and currently I just have a generic handler for all https urls that redirects to a service written by Taras (the product owner) i.e. https://taras-scrape2md.web.val.run/.

Here's a detailed blog post for this tool
https://www.val.town/v/taras/scrape2md

With Web Handlers, we can now use the power of this tool in ChatCraft itself. Isn't that cool?

Upcoming 🔮

There's still a lot of potential in this idea that needs to be harnessed. I am thinking of introducing a custom UI for registering and configuring these web handlers, just like we currently do with ChatCraft Functions.

I am pretty sure, there are things I can do better and maybe a better direction I can take this in. This concept is still too young for ChatCraft and if you guys have any suggestions, let me know in comments.

I'll follow up with another post soon!

Getting an Open Source Project ON Track for Release

Amnish Singh Arora — Sat, 09 Mar 2024 22:52:54 +0000

Over the past couple of months, we have released 4 minor versions of ChatCraft, but this last release was a little different.

Due to unforeseen circumstances, the release v1.4.0 had to be delayed and this week, we tried to cover up for the loss of one week.

In this post, I'll discuss my contributions towards the latest release.

1. Refactoring sentence parsing using an NLP library
2. Fixing an annoying UI bug 🐞
3. Reviewing a PR aimed at Refactoring Providers
4. Release v1.4.0

Refactoring sentence parsing using an NLP library

A few weeks ago, I implemented text-to-speech for ChatCraft and I have been writing about the related follow-ups in my past few posts.

One of the follow-ups required me to refactor the manual sentence parsing algorithm I implemented with the help of a Natural Language Processing library called compromise.

It was a good idea to use this library as it was already part of our project tree and provided a human-friendly API that is easy to understand thanks to its declarative nature.

// Set a maximum words in a sentence that we need to wait for.
// This reduces latency and number of TTS api calls
const TTS_BUFFER_THRESHOLD = 25;

// To calculate the current position in the AI generated text stream
let ttsCursor = 0;
let ttsWordsBuffer = "";

const chat = chatWithLLM(messages, {
  ...
  ...
  onPause() {
    setPaused(true);
  },
  onResume() {
    setPaused(false);
  },
  onData({ currentText }) {
    if (!pausedRef.current) {
      // Hook tts code here
      ttsWordsBuffer = currentText.slice(ttsCursor);

      const { sentences } = tokenize(ttsWordsBuffer);

      if (ttsSupported && getSettings().announceMessages) {
        if (
          sentences.length > 1 // Has one full sentence
        ) {
          // Pass the sentence to tts api for processing
          const textToBeProcessed = sentences[0];
          const audioClipUri = textToSpeech(textToBeProcessed);
          addToAudioQueue(audioClipUri);

          // Update the tts Cursor
          ttsCursor += sentences[0].length;
        } else if (nlp(ttsWordsBuffer).terms().out("array").length >= TTS_BUFFER_THRESHOLD) {
          // Try to break the large sentence into clauses
          const clauseToProcess = nlp(ttsWordsBuffer).clauses().out("array")[0];
          const audioClipUri = textToSpeech(clauseToProcess);
          addToAudioQueue(audioClipUri);

          ttsCursor += clauseToProcess.length;
        }
      }

      setStreamingMessage(
        new ChatCraftAiMessage({
          ...
          ...
        })
      );
      incrementScrollProgress();
    }
  },
});

There was no change in behaviour, but it helped make the code easy to maintain. For more details, check out this Pull Request

Fixing an annoying UI bug 🐞

A few weeks ago, we replaced the existing Chakra Menus with the one provided by szhsin/react-menu library.

While we were able to successfully create a wrapper that very closely mimicked the previous menus, it introduced a pretty annoying auto-scroll bug, as demonstrated in this issue with the help of GIFs

This is what I found with my research:

1. The scrollbar issue was happening as the menu was positioned in a manner that made if overflow horizontal in its parent container. To fix that I have used the align property to make sure it stays inside the container's available space.
2. The auto-scroll issue was occurring when the list of options in the submenu was pretty long and the first list element tried to gain focus automatically. I have capped the max-height of submenus and added vertical scrollbars if the list grows beyond that limit. I made sure that it aligns with recommended UX guidelines before making the decision.

For more details, visit this Pull Request that fixes the issue.

Reviewing a PR aimed at Refactoring Providers

Alright, this week's been full of refactoring. I guess we do have to stop and pay our technical debts at some point :D

The work on this PR has been going on for 2 weeks, and I have been reviewing it as more changes come in.

Katie has written about this Pull Request in this post. Feel free to check out for more details.

Release v1.4.0

Finally, even after a week of delay, we were able to release v.1.4.0 for ChatCraft.

Although I was mostly focused on refactoring and follow-ups, this release added pretty cool features to the app, like support for running python in the browser with the help of a library called python-wasi that leverages web assembly to make it happen.

I have been planning to work on a cool feature for next week. Will write about it when it's done.

In the meantime, STAY TUNED!!!

An Audio Player hook for your React App

Amnish Singh Arora — Sun, 25 Feb 2024 01:37:18 +0000

This was my 7th week of contributing to ChatCraft, and so far, I've successfully shipped various improvements to the existing system. Every week, I've tried to add new features to the app and since we are shipping code quickly, many time its not possible to handle each aspect of the issue perfectly and follow ups have to be filed.

A few weeks ago, I used OpenAI's text to speech API enabling ChatCraft to announce LLM responses as they were generated.

I tried to parse the responses into sentences, pass those chunks to tts api for audio generation, and ultimately play the generated audio clips in sequence.

I've already written about the entire process in another post. In this post, I'll discuss one of the shortcomings of that feature, and how I updated my AudioPlayer hook to manage and orchestrate the ordered playback of audio chunks throughout the application.

1. The Problem ⁉️
2. Implementing a global audio Queue
       2.1. Defining an Audio Player Context
       2.2. Wrapping the audio queue in a Context Provider
       2.3. Providing the context
       2.4. Defining useAudioPlayer Hook
       2.5. Using the Hook 🪝
3. The Pull Request 📝
4. Reviewing a PR that adds Custom Provider Classes
5. What's Ahead 🔜

The Problem ⁉️

If you went through my post, you might have noticed that in order to preserve the order of audio chunks, I was maintaining a classic queue data structure inside my hook.

Here's the code for quick recap:

import { useState, useEffect } from "react";

const useAudioPlayer = () => {
  // We are managing promises of audio urls instead of directly storing strings
  // because there is no guarantee when openai tts api finishes processing and resolves a specific url
  // For more info, check this comment:
  // https://github.com/tarasglek/chatcraft.org/pull/357#discussion_r1473470003
  const [queue, setQueue] = useState<Promise<string>[]>([]);
  const [isPlaying, setIsPlaying] = useState<boolean>(false);

  useEffect(() => {
    if (!isPlaying && queue.length > 0) {
      playAudio(queue[0]);
    }
  }, [queue, isPlaying]);

  const playAudio = async (audioClipUri: Promise<string>) => {
    setIsPlaying(true);
    const audioUrl: string = await audioClipUri;
    const audio = new Audio(audioUrl);
    audio.onended = () => {
      URL.revokeObjectURL(audioUrl); // To avoid memory leaks
      setQueue((oldQueue) => oldQueue.slice(1));
      setIsPlaying(false);
    };
    audio.play();
  };

  const addToAudioQueue = (audioClipUri: Promise<string>) => {
    setQueue((oldQueue) => [...oldQueue, audioClipUri]);
  };

  return { addToAudioQueue };
};

export default useAudioPlayer;

Now this works like a Charm, when you are only using this hook at just one place in the app. But imagine what would happen if I was to use this hook from some other part of the app as well.

const { addToAudioQueue } = useAudioPlayer();

const audioClipUri = textToSpeech(ttsWordsBuffer);
addToAudioQueue(audioClipUri);

Would it still maintain the order?
Leave the order aside, is there a guarantee there is only a single audio clip playing at a point of time.

The answer, as you might have guessed is a NO ❌

Reason being, every time we call the useAudioPlayer hook, it returns a fresh instance of an audio queue. Let's say you use it at 3 different places in your app, you now have 3 audio players active at the same time.

And if you push audio clips to one of them while another is already playing something, you get into a similar situation as if you there are 3 speakers in a room, and each one is playing a different song.

This didn't prevent me from merging the existing code as I was only initializing an audio queue at one place throughout the application.

But with this, there was no way to access and operate with the audio queue from other parts of the application, like stopping the audio playback when a new question is asked.

The details are documented in this follow up.

Stop playing audio when tts is turned off #391

Amnish04 posted on Jan 31, 2024

Follow up for #357

Currently, when text to speech is turned off

The announcement of response does not stop. We need to make sure any tts audio playing is stopped and audio player queue needs to be cleared.

View on GitHub

Implementing a global audio Queue

To fix this issue, I had to make sure that there was always only one instance of the audio queue throughout the application, while the core logic and interface would remain the same.

This had to be done in the following steps.

Defining an Audio Player Context

The first step was to define a React Context, that would expose the relevant functions to operate with the audio queue.

type AudioPlayerContextType = {
  addToAudioQueue: (audioClipUri: Promise<string>) => void;
  clearAudioQueue: () => void;
};

const AudioPlayerContext = createContext<AudioPlayerContextType>({
  addToAudioQueue: () => {},
  clearAudioQueue: () => {},
});

Notice that I am also exposing a new method called clearAudioQueue now. We'll look at the implementation soon.

Wrapping the audio queue in a Context Provider

For any of the components in a React Application to read from a context, it first needs to be provided to one of its ancestors.

This is done with the help of a React Context Provider. I wrapped the existing audio player logic into a function that returns an AudioPlayerContext Provider exposing the functions that will be used to operate on the single Audio Queue in the app.

import { useState, useEffect, createContext, useContext, ReactNode, FC } from "react";

type AudioClip = {
  audioUrl: string;
  audioElement: HTMLAudioElement;
};

export const AudioPlayerProvider: FC<{ children: ReactNode }> = ({ children }) => {
  // We are managing promises of audio urls instead of directly storing strings
  // because there is no guarantee when openai tts api finishes processing and resolves a specific url
  // For more info, check this comment:
  // https://github.com/tarasglek/chatcraft.org/pull/357#discussion_r1473470003
  const [queue, setQueue] = useState<Promise<string>[]>([]);
  const [isPlaying, setIsPlaying] = useState<boolean>(false);
  const [currentAudioClip, setCurrentAudioClip] = useState<AudioClip | null>();

  useEffect(() => {
    if (!isPlaying && queue.length > 0) {
      playAudio(queue[0]);
    }
  }, [queue, isPlaying]);

  const playAudio = async (audioClipUri: Promise<string>) => {
    setIsPlaying(true);
    const audioUrl: string = await audioClipUri;
    const audio = new Audio(audioUrl);
    audio.preload = "auto";
    audio.onended = () => {
      URL.revokeObjectURL(audioUrl);
      setQueue((oldQueue) => oldQueue.slice(1));
      setIsPlaying(false);

      setCurrentAudioClip(null);
    };
    audio.play();
    setCurrentAudioClip({
      audioElement: audio,
      audioUrl: audioUrl,
    });
  };

  const addToAudioQueue = (audioClipUri: Promise<string>) => {
    setQueue((oldQueue) => [...oldQueue, audioClipUri]);
  };

  const clearAudioQueue = () => {
    if (currentAudioClip) {
      // Stop currently playing audio
      currentAudioClip.audioElement.pause();
      URL.revokeObjectURL(currentAudioClip.audioUrl);

      setCurrentAudioClip(null);
      setIsPlaying(false);
    }

    // Flush all the remaining audio clips
    setQueue([]);
  };

  const value = { addToAudioQueue, clearAudioQueue };

  return <AudioPlayerContext.Provider value={value}>{children}</AudioPlayerContext.Provider>;
};

You'll notice the implementation of the clearAudioQueue function, which is meant to be called from any application component to pause the current audio clip, and flush the remaining clips in the queue.

I am also managing a new state now

type AudioClip = {
  audioUrl: string;
  audioElement: HTMLAudioElement;
};

const [currentAudioClip, setCurrentAudioClip] = useState<AudioClip | null>();

This allows me to pause the current playing audio as soon as the clearAudioQueue function is called.

Providing the context

The next step is to provide the AudioPlayerContext at the root of the application.

In main.tsx

import { AudioPlayerProvider } from "./hooks/use-audio-player";

ReactDOM.createRoot(document.querySelector("main") as HTMLElement).render(
  <React.StrictMode>
    <ChakraProvider theme={theme}>
      <AudioPlayerProvider>
        <SettingsProvider>
          <CostProvider>
            <ModelsProvider>
              <UserProvider>
                <ColorModeScript initialColorMode={theme.config.initialColorMode} />
                <RouterProvider router={router} />
              </UserProvider>
            </ModelsProvider>
          </CostProvider>
        </SettingsProvider>
      </AudioPlayerProvider>
    </ChakraProvider>
  </React.StrictMode>
);

Defining useAudioPlayer Hook

After defining the context and provider, we can finally write the hook to use the AudioPlayerContext provided by the closest context provider.

const useAudioPlayer = () => useContext(AudioPlayerContext);

export default useAudioPlayer;

Using the Hook 🪝

And the final step was to actually use the global Audio Player context at places I would like to.

Currently, I am using it for 2 scenarios:
1. The audio for previous message will instantly stop playing as soon as the user asks a new question.

import useAudioPlayer from "../../hooks/use-audio-player";
...
...
const { clearAudioQueue } = useAudioPlayer();
...
...
 // NOTE: we strip out the ChatCraft App messages before sending to OpenAI.
const messages = chat.messages({ includeAppMessages: false });
// Clear any previous audio clips
clearAudioQueue();
const response = await callChatApi(messages, {
  functions,
  functionToCall,
});

2. Any audio clips being played via useAudioPlayer will instantly stop as soon as the user disables the TTS setting.

import useAudioPlayer from "../../hooks/use-audio-player";
...
...
const { clearAudioQueue } = useAudioPlayer();
...
...
{isTtsSupported && (
  <Tooltip
    label={settings.announceMessages ? "Text-to-Speech Enabled" : "Text-to-Speech Disabled"}
  >
    <IconButton
      type="button"
      size="lg"
      variant="solid"
      aria-label={
        settings.announceMessages ? "Text-to-Speech Enabled" : "Text-to-Speech Disabled"
      }
      icon={
        settings.announceMessages ? <MdVolumeUp size={25} /> : <MdVolumeOff size={25} />
      }
      onClick={() => {
        if (settings.announceMessages) {
          // Flush any remaining audio clips being announced
          clearAudioQueue();
        }
        setSettings({ ...settings, announceMessages: !settings.announceMessages });
      }}
    />
  </Tooltip>
)}

And that is all it takes to create and use a custom audio player hook in a React Application. I am pretty sure there are audio player hooks already published on npm, but I doubt any of them will fit my needs as I need to maintain a queue.

Let me know in the comments if you know a better approach.

The Pull Request 📝

My PR for this improvement is already up awaiting reviews before it lands.
You can look at the entire source code and a detailed explanation here.

Make AudioPlayer queue global #484

Amnish04 posted on Feb 23, 2024

In #357, I was able to stream audio responses as the LLM response was generated and also optimize the playback.

But there were some problems with it, and follow ups were filed.

This is regarding one of those follow ups - #391.

The problem was that once an audio clip was added to the audio queue, there was no way to stop it from other parts of the application since every invocation to useAudioPlayer hook resulted in a fresh audio queue. This was pretty annoying and made the TTS feature unusable as users had to wait for the previous message announcement to finish before it started playing for the next one.

To fix this issue, I have made the Audio Player queue global, by creating and providing an AudioPlayerContext at the root of the application.

https://github.com/tarasglek/chatcraft.org/compare/amnish04/global-audio-player?expand=1#diff-1cd8b18798a1a103bfe13bef54354c1f3a3bea29a31c8eea1a0c67a3a839b811

This allowed me to expose another function from the hook called clearAudioQueue which can now be called from any application component to pause the current audio clip, and flush the remaining clips in the queue.

Currently, I am using it for 2 scenarios:

The audio for previous message will instantly stop playing as soon as the user asks a new question.
Any audio clips being played via useAudioPlayer will instantly stop as soon as the user disables the TTS setting.

This approach also makes the audio system more robust any we now have a single source of truth for audio clips, and the risk to play multiple audio clips at once is also reduced.

This fixes #391

View on GitHub

Reviewing a PR that adds Custom Provider Classes

I also reviewed a PR from Katie this week.

Right now, we just have to supported AI providers - OpenAI and OpenRouter. But since we are expecting to have more in the future, it would become impractical to manage all of them in a generic ChatCraftProvider class.
Katie is working on creating a separate concrete class for each AI provider inheriting from the current ChatCraftProvider class, such that if new providers do some things differently, existing methods can be overridden and new methods can be added to their own classes.

Katie LiuFollow

I am a software developer with an interest in open source and web development :)

I went through the code and left a few comments for changes. The PR is still up. If the PR is still up at the time of your reading, please take a look and let us know if you can come up with a better approach!

What's Ahead 🔜

The TTS functionality I've been working on still needs some more work to make best use of the API OpenAI provides.

I'll start out by allowing the users to choose from different voices, ability to re-announce a message, and even downloading the audio narration of a response.

Everything is documented in this issue

Make TTS more flexible #400

tarasglek posted on Feb 03, 2024

Add a Speak submenu to both human and bot messages
In the Speak submenu allow one to select voice, which causes message to be spoken,
once message is spoke, allow generated message to be downloaded..can add a download to the speak submenu after speech completes. It's ugly but not sure re other options
Changing voice in speech menu also changes it for the speak icon when automatic tts is on
should move speak icon to the menu on the left and change it to be "options" or 3 vertical dots on mobile

View on GitHub

In this post, I shared my approach of creating and managing an audio queue with a custom React Hook, and how I am leveraging it to stream OpenAI's LLM responses to their TTS API, and ultimately playing the generated clips in order.

I'll follow up about the other planned functionalities soon.

In the meantime, STAY TUNED!

Sheriffing in Open Source

Amnish Singh Arora — Sun, 18 Feb 2024 01:33:36 +0000

For past few weeks, I've been writing about my experiences working on ChatCraft, an open source AI chat bot, specifically catering to Software Developers.

So far, my weeks were pretty similar with little change in routine. I worked on some feature or fixing bugs, and reviewed a couple of Pull Requests opened by others. But this week was a little different as I got to be the Sheriff and was responsible for Release v1.3.0.

What is a Sheriff ⭐

Managing a serious open source project is not a walk in the park by any means. With various team and community members continuously contributing to the project, it gets pretty challenging to maintain the new and existing issues, incoming pull requests and the quality of code they bring that will ultimately land in production.

Which is why we need a Sheriff to help developers shepherd their code through the development and review process so that it can be landed cleanly in git, and to do so without breaking any tests, CI automation, or infrastructure. (See How to be a Sheriff)

Responsibilities

People often confuse a Sheriff as someone being responsible for fixing each and every problem, which is not the case.
Open Source is driven by community effort, and a Sheriff is responsible for coordinating those community efforts to quickly land fixes, and make decisions to keep the project going.

In ChatCraft, we have a rotation policy and every week there are two Sheriffs - one Primary and a Backup sheriff in case the person scheduled for the week is unable to perform duties. The primary responsibility is being available as the first line of defence in case anything anything goes wrong, keeping concerned people aware of ongoing problems, and getting necessary help to resolve them.

There are many responsibilities of a Sheriff and a good list of those duties can be found in this wiki:
https://github.com/humphd/osd700-winter-2024/wiki/How-to-be-a-Sheriff#responsibilities

Me as a Sheriff

I mentioned at the very beginning that I was scheduled to be ChatCraft's Sheriff this week. And now that we know the associated responsibilities, I'd like to share my experience being in the role.

Running the weekly Triage

Every Wednesday, we have a Weekly Triage meeting where the entire team gathers to discuss existing issues, pull requests and subsequently plan for the upcoming release.

This was when the Sheriff from previous week handed over the responsibilities to me, and I was to run the Triage.

Here's how it went:

I started by looking at the existing Pull Requests opened for issues in the current release cycle (v1.3.0). We discussed 3 main things - how was the progress, if there are any roadblocks, and if need to schedule it for a further milestone.
Next, I opened the list of issues scheduled for the week's milestone and again, we discussed if the issues could be closed within the milestone or if there were any showstoppers.
The last step was to plan for the upcoming release v1.4.0. I asked everyone to move any of their assigned issues that they would like to Milestone 1.4 and also did it for myself. Once we had some initial issues scheduled, I started going through the list of existing unassigned issues - anyone aware of it explained the context, anyone interested asked to be assigned and the issue was moved to the upcoming Milestone if the assignee was comfortable with it.

Going through issues and pull requests also sparked some healthy discussions keeping the team engaged and up to date with what's happening with the project.

Here are the meeting notes if you're interested:

https://github.com/humphd/osd700-winter-2024/wiki/Meeting-Minutes-%E2%80%90-Milestone-1.3

Making Contributions

Being a Sheriff does not mean you're exempt from contributing yourself. You need to put some effort in striking a balance between helping others and delivering your own work.

In total, I got 3 contributions this week and a breakdown is as follows:

1. Improve audio recording UX: This one was a follow up to another issue in which I changed the mic recording behaviour to click instead of drag. All I had to do was add better indicators to the mic button, so users could easily identify the recording state of the control. So I added a Tooltip, a basic size and color transition to fix the problem.

2. Fix viewport pushed by switching chats: This was another instance of a pretty old issue that I fixed back last October. Since I had almost forgotten what I did back then, I revisited by blog post where the solution was documented, and quickly opened a PR with the fix. As this was essentially a regression, Dave (my professor) suggested we need to find a better fix for this. So I filed a new issue for it.

3. Make Sidebar Responsive: This was a continuation to an old Pull Request that I had put up last November. I have blogged about the problem and solution in my last post. But this was the final result on mobile.

Refactoring: Since this introduced a lot of duplication for sidebar related styles and logic, I tried to minimize that duplication by creating separate components for mobile and desktop sidebars and exposing them through a common Sidebar component.

I opened the Pull Request as soon as the sidebar changes were merged, but since it was late in the week and review were pending, I decided to move it to the next release.

Getting the PRs Reviewed 👁️

This was not too different for me as even normally we're expected to do code review for PRs that request our review.

But since a Sheriff is the one responsible for the either reviewing or getting the code reviewed as soon as possible, I had to make myself available all time. And as soon as a new PR was up, I went through the it to make myself of the changes and assigned reviewers appropriately based on their expertise.

I don't know WHY, but I felt like I wasn't entitled to request people for reviews if I didn't do myself as well. So I tried to review as much as I could. Normally I do around 2 code reviews each week. But this week was a little crazy as I reviewed 9 PRs in total and didn't realize until I counted.

Here's a list if you're interested:

Isn't that Crazy!

Time for the Release 🚀

All that hard work finally paid off as we were able to close most of the planned issues

And got to release ChatCraft v1.3.0 at the end of the week.

In this post, I discussed about the roles and responsibilities of a Sheriff in Open Source, and my experience Sheriffing a quality open source project for this week 🚀

Hope you enjoyed the post!

And here's one for me creating my first Software Release 🍻

Refactor and Reboot: Time to Clear Backlog in Open Source

Amnish Singh Arora — Sat, 10 Feb 2024 22:10:42 +0000

Last week, I filed and fixed follow up issues for Text to Speech feature I recently integrated with ChatCraft.There are still many issues open related to Text to Speech that I need to address.

But this week, I decided to prioritize some smaller issues and pull requests I had under my name, as my backlog was growing out of control. In this post, I will discuss how I was able to get many of those issues closed, and review other Pull Requests, effectively contributing to ChatCraft's Release 1.2 this week.

1. Improving Mic UX
2. Helping with Image Styling
3. Working on a Custom Command 💻
4. Refactoring the approach to determine TTS Support 🔧
5. Sidebar Enhancement REBOOT
5.1. Challenges with Git
5.2. Sidebar Changes
6. Reviewing Other PRs
7. Release 1.2.0 🚀

Improving Mic UX

I had a couple of ongoing PRs, but I chose to kick off the week with an easy one. I opened a Pull Request last week to change the Audio Recording behaviour of ChatCraft to "Click to Start, Click to Stop" from "Drag and Hold" as it was pretty challenging to make drag behaviour work seamlessly on mobile.

Last week, I only changed it for smaller screens as I did not want to dump all the work done by David to implement the drag behaviour.

However, everyone believed that click/touch was a better approach

I made the change for all screen sizes. This was pretty easy as all I had to do was REMOVE drag code, and add SIMPLE click handler for recording toggle.

This was merged as it was good enough for the PR, but I got the following recommendation for further improvement:

So I opened a follow up issue for that,

and started looking for the next thing to work on.

Helping with Image Styling

During this week's Triage meeting, we were discussing about a Pull Request made by Mingming that would allow ChatCraft to accept images as input and use OpenAI's Vision model to generate responses based on that. This was a truly killer feature and the PR had had been in progress for about 4 months now. While almost ready to merge, there were some final touch ups left and one of those was that image previews were not centred in the chat and the width was fixed to 50%.

and in the Modal Preview, it was overlapping with the close button.

Everyone decided to fix that in a follow up issue. But I thought that could be a quick fix, so immediately started working on that.

For Call of Duty fans 😉

By the end of the meeting, I was able to come up with some style changes and opened a Pull Request to be merged in to Mingming's branch.

Now the images looked like this:

In the Chat:

In Modal Preview:

No more overlap with close button

Katie asked me to make a minor change which I did shortly.

And since everyone was happy with the fix,

it was merged shortly after, closing the corresponding follow up issue.

Working on a Custom Command 💻

ChatCraft has some in-built commands to quickly execute certain operation like opening a new chat, deleting existing etc. without the need to even touch your 🖱️
A few weeks ago, I opened an issue to show a list of commands when someone entered an invalid command, i.e., not in ChatCraft Command Registry.

I opened the Pull Request last week, and it had been in review since.

Roy did an initial review and was curious why we need error handling in the following code since we know that /commands should run without any problems.

// We are sure that this won't return null
// since prompt is definitely a command
const { command } = ChatCraftCommand.parseCommand(prompt)!;
const commandFunction = ChatCraftCommandRegistry.getCommand(`/commands ${command}`)!;
setShouldAutoScroll(true);
try {
  await commandFunction(chat, user);
  forceScroll();
} catch (err: any) {
  error({
    title: `Error Running Command`,
    message: `There was an error running the command: ${err.message}.`,
  });
}

which Professor clarified. He also helped me notice that I forgot to add the new command to the text for general help text.

Mingming later suggested we should have a Single Source of Truth for the commands help text so we don't have to change it twice.

which I fixed in this commit.

There were some more discussions after which we decided to also show the command which failed in a toast message, so the user knows if they typed something wrong.

But Taras thought that it the unrecognized command should show up in inline with the list of commands instead of toasting.

Now as simple as it seems, it WASN'T. I had to understand the complicated system of how commands were accepting and using arguments to get this done. But finally I succeeded and this was the result.

Passing arguments to this command would allow us to add a feature later on to query a specific command like so

/commands some-command

But for now, everyone seemed to like it and hence, this was also merged.

Refactoring the approach to determine TTS Support 🔧

As a follow up for Add TTS Support issue, I was supposed to modify my approach for determining if TTS was supported by the user's AI provider.

Originally, I was checking if the user had chosen OpenAI as their provider, but this wasn't a good approach as in the future even OpenRouter could allow using the TTS mode.

So the approach suggested by Taras was that I should rather check if the list of current models included the model I was using for audio streaming.

I spent a while in addressing this issue and opened a Pull Request.

This was also a long conversation and if you're interested in code changes, you can follow this link.

https://github.com/tarasglek/chatcraft.org/pull/420/files

I was able to get it merged yesterday.

Sidebar Enhancement REBOOT

I had a Pull Request regarding Sidebar enhancement opened since November, but the requirements kept changing and it became really hard to continue working on the same branch when it was decided to drop a lot of things.

Challenges with Git

I initially tried to continue in the same PR, by following this plan.

Branch off of main
Work now new changes
Rebase on the original sidebar branch and fast-forward merge into it.

But the problem at hand was that my original sidebar branch was in my downstream fork, while my new branch was in the original repository. And I wasn't sure if I could merge into a downstream repo branch from an upstream repo branch.

One approach I could use would be:

Make sure both sidebar branches (downstream and upstream) are rebased on upstream main.
Create a new branch in downstream based on the sidebar branch in upstream.
Fast-forward the new downstream branch into the downstream sidebar branch.
Push new commits to downstream Gitub repo

And this should have done it. But I thought NAHH, its easier to create a new PR from the branch in upstream as this would be cleaner and we could also benefit from Cloudfare Previews in reviews.

Sidebar Changes

But anyways, lets take a quick look at the changes I made.

The overlay effect is only applied to mobile devices now, and Chakra Drawer is being used for that purpose.
The search field has also been moved to the sidebar on mobile devices due to the lack of real estate in the header.
The behaviour on desktop remains the same, except a few animations have been added to make the push content effect smoother.
It has also been ensured that there is no redundant animation on page reloads, as it was one of the problems in the last PR.
The Pin Sidebar feature has also been removed.

Here's a Demo 🎬

On Desktop:

The gif was too large so couln't attach
Link

On Mobile:

I would suggest looking at the PR directly.
https://github.com/tarasglek/chatcraft.org/pull/437

Reviewing Other PRs

These were all my direct contributions to this week's release, but
I also reviewed Pull Requests made by others.

The first one was by Professor, to allow image downloads of messages.

The second was by Katie, that would allow users to store multiple providers at once.

Release 1.2.0 🚀

This sure was a pretty busy week. I got 4 pull requests merged, 1 Reboot, and 2 Reviews.

Everyone worked pretty hard, and with all contributions, we were able to successfully release ChatCraft v1.2.0 this week.

Cheers, and looking forward to 1.3 next week 💪

Taking an Open Source Project to Release 1.1 🚀

Amnish Singh Arora — Sat, 03 Feb 2024 22:59:25 +0000

In my last post, I discussed how I integrated Text to Speech support into ChatCraft using OpenAI's TTS API.

In this post, I'll share my progress on that Pull Request and other contributions I made as a part of releasing v1.1 of the project.

1. Text to Speech Support
2. Filing and fixing follow ups
       2.1. Using Better icons for TTS button
       2.2. Fixing TTS toggle behaviour
       2.3. TTS should abort when switched off
3. Helping with migrating the Menu Component
       3.1. Suggesting changes on GitHub
       3.2. Pushing Fixes
4. Reviewing a Pull Request
5. Release v1.1 🚀

Text to Speech Support

Before talking about other contributions, I was able to get an initial version of text to speech landed successfully after a few discussions and changes.

My professor was a little confused why I was managing Promises of audio urls instead of directly storing url strings in my audio queue. I tried to explain it with the help of an example.

Here's the link if you're interested:
https://github.com/tarasglek/chatcraft.org/pull/357#discussion_r1473470003

I embedded a video demo in my last post, and you can try it yourself by visiting ChatCraft.

It was not perfect by any chance, and so I had to open a couple of follow up issues aimed to fix the technical shortcomings.

As soon as it was merged, I started receiving more feedback that I couldn't even get in original reviews because now people were forced to use it 😉.

And so, it was time to file even more follow ups aimed to address the issues brought up by Taras.

Filing and fixing follow ups

Various issues were encountered after my PR was merged.

Using Better icons for TTS button

The first one was that the icons I used for TTS enable/disable button didn't do a good job at indicating the state of the feature.

I quickly opened an issue for that

and replaced the icons with Material Design variants as suggested.

After a quick review, the Pull Request was merged.

Time to look at the other issues!

Fixing TTS toggle behaviour

The next thing on the list was a little annoying for the users.

You read that RIGHT! For some reason, the text to speech functionality was always enabled no matter if the state of the toggle button.

Another follow up issue had to be opened.

The problem was that I wasn't checking for the TTS setting in the else if branch in the following code:

if (isTtsSupported() && getSettings().announceMessages) {
  if (
    sentenceEndRegex.test(ttsWordsBuffer) // Has full sentence
  ) {
    // Reset lastIndex before calling exec
    sentenceEndRegex.lastIndex = 0;
    const sentenceEndIndex = sentenceEndRegex.exec(ttsWordsBuffer)!.index;

    // Pass the sentence to tts api for processing
    const textToBeProcessed = ttsWordsBuffer.slice(0, sentenceEndIndex + 1);
    const audioClipUri = textToSpeech(textToBeProcessed);
    addToAudioQueue(audioClipUri);

    // Update the tts Cursor
    ttsCursor += sentenceEndIndex + 1;
  } else if (ttsWordsBuffer.split(" ").length >= TTS_BUFFER_THRESHOLD) {
    // Flush the entire buffer into tts api
    const audioClipUri = textToSpeech(ttsWordsBuffer);
    addToAudioQueue(audioClipUri);

    ttsCursor += ttsWordsBuffer.length;
  }
}

I quickly opened a Pull Request for the fix.

and felt the need for adding a hotfix label to the repo

as this sure was one.

TTS should abort when switched off

The third problem was that the TTS announcement always kept playing, even when the feature was turned off using the toggle button.

I filed an issue for that

but still need to work on it.

Here's the state of follow ups so far.

Helping with migrating the Menu Component

The next big thing I worked on this week was helping Rachit with the creation of a new Menu Component for the application using the react-menu package.

Here's some context for you guys.

Rachit was able to get the functionality working for our component wrapper, but needed some help with the styling part.

Since I was asked for help,

I had to get into action.

Suggesting changes on GitHub

The first step was to review behaviour and suggest any initial changes that came to my mind.

I quickly noticed a weird behaviour that did not exist before

I posted a suggestion on GitHub that could be directly committed to fix that issue.

Fixed Behaviour:

We continued with this GitHub Suggestions for a while

Pushing Fixes

Until professor suggested us another approach of collaboration.

This was exactly what I wanted to do, and now that I knew it was acceptable, I asked Rachit if could push directly to his branch. And pushed the fixes for anything I could think of.

For full details, please follow the Pull Request directly. This was the biggest conversation I ever I had in a PR.

Reviewing a Pull Request

Last but not least, I reviewed a Pull Request Katie was working on, that would allow users to store metaData for multiple AI providers at once.

I made a simple suggestion on this one to create a url to provider name mapping instead of manually determining provider name with a switch statement.

This suggestion was accepted

And after reviews from professor, this PR was also merged 🎊

Release v1.1 🚀

All this work, was just a part of my contributions for Release 1.1 that happened yesterday.

We have made significant progress if count changes made by all others

and here's one for the new contributors 🍾🥂

Forem: Amnish Singh Arora

Never lose valuable error context in JavaScript

Table of Contents

The problem

The solution?

Installation

Configuration

VSCode Extension

Conclusion

Concluding OSD700

Improve Download Speeds with Concurrency

Table of Contents

The Problem

Breaking larger messages into Chunks

Algorithm for Natural Chunks

Using Text Chunks to generate Audio

Optimizing Download Speeds

Results

Release 2.0

ChatCraft 2.0 is almost Here!

Table of Contents

Fixing a Web Handlers related bug

Refactoring PasswordInput Component

Reviewing other PRs

Next Plans

Adding a Code Editor to your React App

Table of Contents

How it looked before

Replacing with a Code Editor

Adding to ChatCraft

Conclusion

Iterating on Web Handlers Implementation

Table of Contents

How it works 🏃

The Model

Ability to Configure

Executing Handlers

Making Sense of it

Looking Ahead 👀

Implement initial version of web handlers #519

Introducing the idea of Web Handlers in ChatCraft

Table of Contents

Background 💭

Spin Off 🧐

Implementation 🧑🏻‍💻

Upcoming 🔮

Getting an Open Source Project ON Track for Release

Table of Contents

Refactoring sentence parsing using an NLP library

Fixing an annoying UI bug 🐞

Reviewing a PR aimed at Refactoring Providers

Release v1.4.0

An Audio Player hook for your React App

Table of Contents

The Problem ⁉️

Stop playing audio when tts is turned off #391

Implementing a global audio Queue

Defining an Audio Player Context

Wrapping the audio queue in a Context Provider

Providing the context

Defining useAudioPlayer Hook

Using the Hook 🪝

The Pull Request 📝

Make AudioPlayer queue global #484

Reviewing a PR that adds Custom Provider Classes

Katie LiuFollow

What's Ahead 🔜

Make TTS more flexible #400

Sheriffing in Open Source

What is a Sheriff ⭐

Responsibilities

Me as a Sheriff

Running the weekly Triage

Making Contributions

Getting the PRs Reviewed 👁️

Time for the Release 🚀

Refactor and Reboot: Time to Clear Backlog in Open Source

Table of Contents

Improving Mic UX

Helping with Image Styling