Indeterminacy of the `mime` Package in Go: Trust But Verify

if err != nil { — Sun, 28 Jul 2024 16:18:46 +0000

Posted originally in my blog: https://lazy.bearblog.dev/go-mime-wtf/

Background

One weekend, I decided to write a Telegram bot for myself to automate to-do or shopping lists.

The idea is simple. I send the bot a list of items to buy, and it creates a webpage with checkboxes that I can tick off while holding my phone in one hand and a shopping basket in the other.

The bot was written pretty quickly, but then I wanted to experiment with OpenAI's Whisper model. So, I decided to add speech recognition to my application.

Now, the program works as follows:

I send the bot a voice message like this: "buy a kilogram of potatoes, a bunch of dill, a head of cabbage, a couple of beers, oh, don't need the cabbage, we have it"
The bot recognizes the speech and creates a page with checkboxes:

[ ] potatoes - 1 kg
[ ] dill - 1 bunch
[ ] beer - 2 bottles

Now, about the article. In go-openai, you can send audio using the openai.AudioRequest structure, which allows you to specify an audio data stream in the Reader field or a file path in the FilePath field. Well, I thought these fields were mutually exclusive, but it turns out you need to specify FilePath even when using an audio stream. The API probably uses this to determine the stream type.

On the Telegram side, we receive a tgbotapi.Voice structure, which has a MimeType field. For voice messages, it is audio/ogg, but it could easily change in the future.

So, we have the MIME type of the audio stream, and we need to determine the file extension from it, creating a file name to ensure the OpenAI API doesn't complain about an unknown file type.

What Could Be Simpler?

The code seems trivial:

extensions, err := mime.ExtensionsByType(mimeType)
if err != nil {
    return err
}
if len(extensions) == 0 {
    return fmt.Errorf("unsupported mime type: %s", mimeType)
}
ext := extensions[0]
log.Printf("Assumed extension: %s", ext)
fakeFileName := fmt.Sprintf("voice_message.%s", ext)

Testing locally - it works:

2024/07/28 17:49:06 Assumed extension: oga

Deploying to the cloud - it doesn't work:

2024/07/28 17:55:32 unsupported mime type: audio/ogg

Checking the Go versions, locally and in the CI/CD used for building - they are the same.

Let's Cut to the Chase

It turns out the behavior of the mime package is not deterministic and depends at runtime on whether there is a /etc/mime.types file in the operating system and its contents.

That's where the mime package reads the runtime table of MIME type to file extension mappings.

I'm dead serious: you compile a binary, run all the tests, everything seems great, but this binary will determine the assumed file extensions for the same MIME type differently in different environments.

How to Fix It

Let's take known MIME types of audio files and their extensions, and manually add them in the init function using mime.AddExtensionType.


var mimetypes = [][]string{
    {"audio/amr", "amr"},
    {"audio/amr-wb", "awb"},
    {"audio/annodex", "axa"},
    {"audio/basic", "au", "snd"},
    {"audio/csound", "csd", "orc", "sco"},
    {"audio/flac", "flac"},
    {"audio/midi", "mid", "midi", "kar"},
    {"audio/mpeg", "mpga", "mpega", "mp2", "mp3", "m4a"},
    {"audio/mpegurl", "m3u"},
    {"audio/ogg", "oga", "ogg", "opus", "spx"},
    {"audio/prs.sid", "sid"},
    {"audio/x-aiff", "aif", "aiff", "aifc"},
    {"audio/x-gsm", "gsm"},
    {"audio/x-mpegurl", "m3u"},
    {"audio/x-ms-wma", "wma"},
    {"audio/x-ms-wax", "wax"},
    {"audio/x-pn-realaudio", "ra", "rm", "ram"},
    {"audio/x-realaudio", "ra"},
    {"audio/x-scpls", "pls"},
    {"audio/x-sd2", "sd2"},
    {"audio/x-wav", "wav"},
}

func init() {
    log.Println("init mimetypes")
    for _, v := range mimetypes {
        typ := v[0]
        extensions := v[1:]

        for _, ext := range extensions {
            err := mime.AddExtensionType("."+ext, typ)
            if err != nil {
                log.Fatalf("mime: %s", err.Error())
            }
        }
    }
}

This will make the behavior of our application deterministic regarding the domain it will work with (in my case, audio files).

Conclusion

This experience showed that things in Go are not always deterministic. If something seems odd, don't hesitate to dive into the library's source code and see how it's implemented. This can save you a lot of time and trouble in the long run.

Implementing Golang's chan in TypeScript with @harnyk/chan

if err != nil { — Thu, 18 Jul 2024 14:05:12 +0000

Introduction

Asynchronous programming in TypeScript can be challenging, especially when dealing with concurrency. Inspired by Golang's chan, I created a library, @harnyk/chan, to bring similar functionality to TypeScript, enabling efficient and manageable concurrency.

What is `@harnyk/chan`?

@harnyk/chan is a TypeScript library that mimics Golang's channel mechanism. It allows for safe, concurrent communication between asynchronous tasks, similar to Go’s chan.

Features

Basic Channel Operations:
- Send and Receive: Channels in @harnyk/chan can be used to send and receive values between asynchronous functions.
- Buffered Channels: Support for buffered channels to handle multiple values.
Asynchronous Iteration:
- Channels can be iterated asynchronously using for-await-of, making it easy to process values as they are received.
Select Statement:
- The select statement allows you to wait on multiple channel operations, choosing whichever operation is ready first, similar to Go's select.

Example Usage

Here’s a simple example demonstrating how to use @harnyk/chan:

import { chan, select } from '@harnyk/chan';

const ch = chan<number>();

// Producer
(async () => {
  for (let i = 0; i < 5; i++) {
    await ch.send(i);
  }
  ch.close();
})();

// Consumer
(async () => {
  for await (const value of ch) {
    console.log(value);
  }
})();

Why Use `@harnyk/chan`?

Improved Concurrency Management: Easily manage concurrent tasks and communications.
Enhanced Readability: Clear and concise syntax inspired by Go’s chan.
Robust Asynchronous Patterns: Leverage the power of channels to build more complex and reliable async workflows.

Conclusion

@harnyk/chan brings the power of Golang's chan to TypeScript, making asynchronous programming more manageable and efficient. Whether you are dealing with complex async workflows or just need a better way to handle concurrency, @harnyk/chan can be a valuable tool in your TypeScript toolkit.

Links

Check out the GitHub repository for more examples and documentation. For a deeper dive, read the original blog post. Happy coding!

Forem: if err != nil {

Indeterminacy of the `mime` Package in Go: Trust But Verify

Background

What Could Be Simpler?

Let's Cut to the Chase

How to Fix It

Conclusion

Implementing Golang's chan in TypeScript with @harnyk/chan

Introduction

What is @harnyk/chan?

Features

Example Usage

Why Use @harnyk/chan?

Conclusion

Links

What is `@harnyk/chan`?

Why Use `@harnyk/chan`?