Forem: Anton Dosov

Pipe a file stream from AWS S3 to OpenAI Whisper in Node.js

Anton Dosov — Thu, 02 Mar 2023 18:45:52 +0000

Current (2023-03-01) OpenAI Whisper API expects a file uploaded as part of multipart/form-data in POST request. I initially struggled to use this API inside Node.js with an audio file stored in an AWS S3 bucket, so thought I'd share a working snippet:

The API usage example from the OpenAI docs in the following:

curl --request POST \
  --url https://api.openai.com/v1/audio/transcriptions \
  --header 'Authorization: Bearer TOKEN' \
  --header 'Content-Type: multipart/form-data' \
  --form file=@/path/to/file/openai.mp3 \
  --form model=whisper-1

The Node.js + AWS S3 snippet that seemed to work for looks like this:

import fetch from "node-fetch";
import AWS from "aws-sdk";
import fs from "fs";
import FormData from "form-data";

// Set the region and access keys for your AWS account
AWS.config.update({
    region: 'eu-central-1',
    accessKeyId: '***',
    secretAccessKey: '***'
});

// Create an S3 client
const s3 = new AWS.S3();

// Set the bucket and file key
const bucketName = 'openai-sample';
const fileKey = 'path/to/file/openai.mp3';

// Set the parameters for the S3 getObject operation
const params = {
    Bucket: bucketName,
    Key: fileKey
};

// Get audio metadata to retrieve size and type
s3.headObject(params, function(err, data) {
    if (err) throw err;

    // Get read object stream
    const s3Stream = s3.getObject(params)
      .createReadStream();

    // create form data to be send to whisper api
    const formData = new FormData();
    // append stream with a file
    formData.append('file', s3Stream, {
        contentType: data.ContentType,
        knownLength: data.ContentLength,
        filename: fileKey
    });
    formData.append('model', 'whisper-1');


    fetch('https://api.openai.com/v1/audio/transcriptions', {
        method: 'POST',
        body: formData,
        headers: {
            'Authorization': 'Bearer TOKEN',
            ...formData.getHeaders()
        }
    }).then(res => res.json()).then(json => console.log(json.text));
});

ZOOM-like video gallery with CSS Houdini 🎩

Anton Dosov — Mon, 08 Jun 2020 17:07:09 +0000

TLDR

Complete solution is here.

⚠️ This demo uses experimental API. Check browser support before using it in production.

If you use Chrome, make sure you have experimental-web-platform-features flag enabled. Check support for other browsers here.

Video in case you use a browser without CSS Layout API support:

Intro

Hi DEV community 👋

Last week I've built a video gallery just like in ZOOM.

Building a video gallery just like in ZOOM

Anton Dosov ・ Jun 1 '20

#javascript #css #webdev #html

I tried to find a solution using pure CSS but failed. This week I still don't have a pure CSS solution. Instead, I rebuilt the video gallery using experimental CSS Layout API from CSS Houdini 🎩.

Problem

image from zoom.us

Having videoCount videos with fixed aspectRatio and fixed container size (containerWidth, containerHeight), fit all the videos inside the container to occupy as much area as possible. Videos should have the same size and can't overflow the container.

CSS Houdini 🎩

CSS Houdini is a set of experimental browser APIs which allow to hook into the browser rendering process. We are going to use CSS Layout API for positioning and sizing video elements.

⚠️ This API is available only with an experimental flag. So it can't be used in production just yet!

Solution

Starting from following HTML structure:

<div id="gallery">
  <div class="video-container">
    <video/>
  </div>
  <div class="video-container">
    <video/>
  </div>
</div>

And making sure #gallery takes up the whole screen:

body {
  margin: 0;
  height: 100vh;
}
#gallery {
  height: 100%;
}

display: layout(zoom-like-gallery)

This is the moment where Houdini 🎩 does his magic:

#gallery {
  height: 100%;
  display: layout(zoom-like-gallery); // 💥
}

Normally we would use display property with one of the predefined values. Like grid, flex or inline-block. But CSS Layout API allows developers to implement their custom layouts 😱. And we are going to implement our custom zoom-like-gallery layout.

// check for CSS Layout API support
if ("layoutWorklet" in CSS) {
  // import a module with our custom layout
  CSS.layoutWorklet.addModule("zoom-like-gallery-layout.js");
}

Then in zoom-like-gallery-layout.js we register a layout:

registerLayout(
  "zoom-like-gallery",
  class {
    // array of CSS custom properties that belong to the container (to the `#gallery` in our case)
    // look at this like at parameters for custom layout
    // we will use this later to make aspect ratio configurable from CSS 
    static get inputProperties() {
      return [];
    }

    // array of CSS custom properties that belong to children (to `.video-container` elements in our case).
    static get childrenInputProperties() {
      return [];
    }

    // properties for layout,
    // see: https://drafts.css-houdini.org/css-layout-api/#dictdef-layoutoptions 
    static get layoutOptions() {  }

    // allows to define min-content / max-content sizes for a container (for `#gallery` in our case).
    // see: https://drafts.csswg.org/css-sizing-3/#intrinsic-sizes
    async intrinsicSizes(children, edges, styleMap) {}

    // finally function to perform a layout
    // (`children` is an array of `.video-container` elements in our case)
    async layout(children, edges, constraints, styleMap) {

    }
  }
);

⬆️ The API is complex, but to reach the goal we can just focus on layout function. This is where we have to write the code for sizing and positioning video elements. The browser will call this function whenever it needs to perform the layout.


async layout(children, edges, constraints, styleMap) {
  const containerWidth = constraints.fixedInlineSize; // width of a `#gallery`. Equals to the width of the screen in our case.
  const containerHeight = constraints.fixedBlockSize; // height of a `#gallery`. Equals to the height of the screen in our case.
  const videosCount = children.length;
  const aspectRatio = 16 / 9; // hardcode this for now. will improve later

If you followed the original post, you may notice we have the same input parameters as we had in the original solution. So we can reuse the layout algorithm from the original post to calculate the gallery layout.

async layout(children, edges, constraints, styleMap) {
  const containerWidth = constraints.fixedInlineSize; // width of a `#gallery. Equals to the weight of the screen in our case.
  const containerHeight = constraints.fixedBlockSize; // height of a `#gallery`. Equals to the height of the screen in our case.
  const videosCount = children.length;
  const aspectRatio = 16 / 9; // just hardcode this for now

  // `calculateLayout` finds layout where equally sized videos with predefined aspect ratio occupy the largest area
  // see implementation in codesandbox https://codesandbox.io/s/zoom-like-gallery-with-css-houdini-0nb1m?file=/layout.js:1840-2787
  // see explanation in the original post https://dev.to/antondosov/building-a-video-gallery-just-like-in-zoom-4mam
  const { width, height, cols, rows } = calculateLayout(containerWidth, containerHeight, videosCount, aspectRatio);
  // width: fixed width for each video
  // height: fixed height for each video 
}

Now when we have fixed width and height for all video elements, we can layout them using:

// layout video containers using calculated fixed width / height
const childFragments = await Promise.all(
  children.map(child => {
    return child.layoutNextFragment({
      fixedInlineSize: width,
      fixedBlockSize: height
     });
    })
   );

layoutNextFragment() is part of CSS Layout API. It performs layout on child elements (.video-container in our case). It returns children as an array of LayoutFragments.

At this point all videos inside a container are laid out with sizes we calculated. The only thing left is to position them within a container (#gallery).

Positioning childFragments within the container is done by setting its inlineOffset and `block offset attributes. If not set by the author they default to zero.

image from here

`js
childFragments.forEach(childFragment => {
childFragment.inlineOffset = // assign x position for a video container
childFragment.blockOffset = // assign y position for a video container
})

return { childFragments }; // finish layout function by returning childFragments
`
Refer to codesandbox for implementation ⬆️.

On this point, everything should work, but we can make it a bit better. We hardcoded aspectRatio inside the layout code:

const aspectRatio = 16 / 9;
To make this configurable from CSS:
`js
static get inputProperties() {
return ["--aspectRatio"];
}

async layout(children, edges, constraints, styleMap) {
const containerWidth = constraints.fixedInlineSize;
const containerHeight = constraints.fixedBlockSize;
const videosCount = children.length;
// const aspectRatio = 16 / 9;
const aspectRatio = parseFloat(styleMap.get("--aspectRatio").toString());

// ...
return childFragments
}
` And now pass it from CSS: `css

gallery {

height: 100%;
display: layout(zoom-like-gallery);
--aspectRatio: 1.77; /* 16 / 9 */ 👈
}
`
That's a wrap 🥇. Working solution is here. If you use Chrome, make sure you have experimental-web-platform-features flag enabled. Check support for other browsers here.
{% codesandbox zoom-like-gallery-with-css-houdini-0nb1m runonclick=1 %}

Video in case you use a browser without CSS Layout API support:

{% vimeo 426310990 %}

How is it different from the original solution?

Both implementations use the same algorithm to calculate the layout for the #gallery.

Nevertheless, there are a couple of notable differences:

When #gallery layout is recalculated.
What triggers the recalculation.
How #gallery layout values propagate to the screen.

In the original implementation, we added a debounced event listener to the resize event on a window. We recalculated the gallery layout on a main thread whenever an event fired. Then we changed CSS using calculated layout values and this triggered the browser rendering engine to re-layout videos for new screen dimensions.

resize event -> recalculate -> change CSS -> browser performs re-layout
In the implementation with CSS Layout API, the browser rendering engine calls layout() on its own whenever it decides it needs to recalculate the layout for #gallery. We didn't have to listen for resizes and didn't have to manually manipulate DOM. Our code to calculate layout for the #gallery is being executed as part of a browser rendering engine process. Browser may even decide to execute it in a separate thread leaving less work to perform on the main thread, and our UI may become more stable and performant 🎉.

Conclusion

Unfortunately, we can't deploy this to production just yet (support). Have to leave original implementation for now. But the future is exciting! Developers soon will have an API to hook into browser rendering engine making their apps more stable and performant!

Learn more

Practical overview of CSS Houdini
Houdini: Demystifying CSS
ishoudinireadyyet.com
CSS Layout API examples
CSS Layout API spec
I skipped the concept of Worklets trying to keep this hands-on post simpler.

Building a video gallery just like in ZOOM

Anton Dosov — Mon, 01 Jun 2020 18:39:52 +0000

Update

I posted a fun part 2, where I am building the same thing using experimental CSS Layout API from CSS Houdini 🎩. Check this out!

ZOOM-like video gallery with CSS Houdini 🎩

Anton Dosov ・ Jun 8 '20

#houdini #css #javascript #webdev

TLDR

Complete solution is here. Change videos count and resize the screen to see it in action.

Intro

Hi folks 👋

I had to build a video gallery view similar to the one ZOOM has for a video-conferencing app.

I've spent quite a bit of time trying to figure out how to build it with pure CSS and I... FAILED 😕.

Sharing my current solution with

a bit of JavaScript
CSS custom properties
display: flex

If someone has an idea on how to achieve similar result without using JavaScript, please share 🙏

Problem

Solution

Calculating size for a video

First I needed to make sure videos are not overflowing the container and occupying as much area as possible.

function calculateLayout(
    containerWidth: number,
    containerHeight: number,
    videoCount: number,
    aspectRatio: number
  ): { width: number; height: number, cols: number } {
 // see implementation in codesandbox
}

Current implementation brute-force searches the layout which occupies the most of the available space. It compares the sum of video areas for every possible number of columns.

// pseudocode, see codesandbox for complete version
let bestArea;
for (let cols = 1; cols <= videoCount; cols++) {
   const currentArea = /* sum of video areas in this layout */
   if (bestArea < currentArea) {
      bestArea = currentArea;
   }
}

There is also an npm module that does just that!
fzembow / rect-scaler

Find largest rectangle and square sizes when fitting them into a container
rect-scaler

A set of javascript functions for calculating how large a set of equally sized squares or rectangles can be to fit within an arbitrary rectangular container, to cover it as fully as possible.

Useful for graphical layouts where you need to space items in a nice way. This algorithm does not allow for rotations, and is not generic bin packing.

Usage

Install from npm:
npm install rect-scaler
Fitting squares

Pass the size of the container and the number of squares that need to be placed to largestSquare(), resulting in an object containing the optimal solution.
import { largestSquare } from "rect-scaler";

const containerWidth = 100;
const containerHeight = 100;
const numSquares = 8;
const { rows, cols, width, height, area } = largestSquare(
  containerWidth,
  containerHeight,
  numSquares
);
Fitting rectangles

Pass the size of the container and…
View on GitHub
But is there a better way then brute-force? Is it worth it if we assume the maximum videoCount is 50? 🤔

Markup, styles & CSS custom properties

The HTML structure I went with:

<body>
    <div id="gallery">
      <div class="video-container">
        <video></video>
      </div>
      <div class="video-container">
        <video></video>
      </div>
    </div>
</body

I applied calculated width and height to .video-container.

.video-container {
  width: var(--width);
  height: var(--height);
}

I used CSS custom properties to pass values calculated in JavaScript.

const gallery = document.getElementById('gallery');
gallery.style.setProperty("--width", width + "px");
gallery.style.setProperty("--height", height + "px");
gallery.style.setProperty("--cols", cols + "");

⚠️ Don't forget to recalculate these values when the screen size or number of videos changes.

Then I used display: flex to layout .video-container elements

#gallery {
  display: flex;
  justify-content: center;
  flex-wrap: wrap;
  max-width: calc(var(--width) * var(--cols));
}

As .video-container sizes are calculated to fit into the container, I didn't have to worry about anything else here.
I also found justify-content: center; to work best for the use case, as it nicely centers not fully filled rows.
The purpose of max-width is to force line breaks.

Handling different aspect ratios

This gallery has a constraint of a fixed aspect ratio for all elements. But videos could have different ratios. This is where having .video-container wrapping <video/> is handy.

.video-container {
  width: var(--width);
  height: var(--height);
}

video {
  height: 100%;
  width: 100%;
}

This way video occupies as much space as possible within its container and preserves its original aspect ratio.

For example, changing the ratio of a container from 16:9 to 1:1 doesn't distort videos with the original 16:9 ratio.

Result

This is how it looks in the real world:

Please find complete solution here. Change videos count and resize the screen to see it in action.

Open questions❓

Is it possible to achieve a similar result without calculating video size in JavaScript? 🤔
Is there a better way of calculating video sizes than brute-force search? Is it worth it if a number of videos can't exceed 50? 🤔