Forem: Oscar Nord

Generate subtitles with OpenAI Whisper and Eyevinn OSC

Oscar Nord — Thu, 08 Feb 2024 15:21:01 +0000

With AI and LLMs being on (almost) everyone's mind these days, we at Eyevinn thought that it would be interesting to explore the possibilities of using OpenAI's speech-to-text model to generate subtitles for video/audio content.

The initial idea was that it would be cool if we could generate subtitles on the fly when a user requests them for a movie, they won't be perfect of course but maybe they can be just good enough to provide basic translations for content that otherwise wouldn't have subtitles in that specific language available.

With that idea in mind, we have created a small POC that is available on GitHub. It's a super simple API that makes it possible to provide a link to a video/audio segment where the response is the transcribed content. We've also added the possibility of uploading the transcribed file to an S3 bucket.

To transcribe content you will do a POST to the /transcribe endpoint:

{
  "url": "https://example.net/vod-audio_en=128000.aac"
  "language": "en" // ISO 639-1 language code (https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (optional)
  "format": "vtt" // Supported formats: json, text, srt, verbose_json, or vtt (optional)
}

The response will then be the transcribed content:

{
  "workerId": "BFabbcCi3IYuWOj6LfsgK",
  "result": "WEBVTT\n\n00:00:00.000 --> 00:00:04.180\nor into transcoding I mean, I could probably add just the keyframe in the start and just\n\n00:00:04.180 --> 00:00:06.920\nskip I-frames and the rest of that.\n\n"
}

Formatted output:

WEBVTT

00:00:00.000 --> 00:00:01.940
So into transcoding, I mean, I could

00:00:01.940 --> 00:00:03.700
probably add just a keyframe at the start

00:00:03.700 --> 00:00:06.700
and then just skip iFrames in the rest of the scenes.

The API uses the OpenAI API so an OpenAI account and API key are required.

The easiest and best way to try this out is to use our newly released (currently in beta) platform that we call Eyevinn Open Source
as a service. It's a cloud platform that makes it super easy to prototype, monetize, and take open-source services to production. The idea behind this is that when you sign up, which is completely free, you can easily try out and deploy a selection of open-source services with a click of a button. What makes this platform special is that you only pay for the cost of running the services and a small community fee that will be shared with the maintainers of the open-source service that you use.

Take the auto-subtitles API as an example, when you have created an account you will be presented with a dashboard containing multiple services.

When creating an instance of auto-subtitles you'll need to input your OpenAI key (this is only used when making calls with the OpenAI API)

That's it now you are up and running and can try it out!

Eyevinn OSaaS is currently in beta so let us know what you think about it.

As always if you need assistance in the development and implementation of this, our team of video developers are happy to help you out. If you have any questions or comments just drop a line in the comments section to this post.

Upload content to AWS S3 with Rust 🦀

Oscar Nord — Thu, 27 Jan 2022 16:48:55 +0000

Introduction

In this post I'm going to describe how you can upload content to an S3 bucket with the AWS SDK for Rust.

I'm going to assume that you have already installed Rust, if not you can follow this guide.

Setup

Start by creating a new package with Cargo:

cargo new s3_uploader --bin

This initialises a new package with the following structure:

s3_uploader
├── Cargo.toml
└── src
    └── main.rs

The Cargo.toml is the manifest file for the project and contains all metadata necessary to compile the project. If you are familiar with Node.js it's the Rust version of the package.json file.

Now that we have a basic project to work with we are ready to start coding.

AWS and Rust

AWS have somewhat recently released a SDK for Rust that is currently a developer preview. It's easy to get started with and can be found on GitHub.

To be able to use the AWS SDK we need to include it under [dependencies] in our Cargo.toml. To be able to easily authenticate against AWS services we will use the aws-config crate and to be able to execute asynchronous code we will also add Tokio to our dependancies.

Your Cargo.toml should now look something like this:

[package]
name = "s3_uploader"
version = "0.1.0"
edition = "2018"

[dependencies]
aws-config = "0.3.0"
aws-sdk-s3 = "0.3.0"

tokio = { version = "1", features = ["full"] }

Implementation

Start with creating a new file in the src folder and name it s3_uploader.rs. In the newly created file import the following:

use std::path::Path;
use std::process;

use aws_sdk_s3::{ByteStream, Client, Error};

We are now ready to start implementing the function to upload content.

create a new async function:

pub async fn upload(path: &str, bucket: &str, key: &str) -> ResultResult<aws_sdk_s3::output::PutObjectOutput, Error> {...}

This function will take a path, name of the input bucket and the name of the file when uploaded to S3. The return type is an enum with either output from the PutObject action or an error.

Start by loading the necessary environment variables that we need via aws_config. After that is done we can create a new client and load in the source file from the path.

let config = aws_config::load_from_env().await;
let client = Client::new(&config);
let file = ByteStream::from_path(Path::new(path)).await;

The next step is to create a variable in which we can store the response from AWS.

let resp;

Now we have everything we need to upload the file to S3:

match file {
  Ok(f) => {
    resp = client
      .put_object()
      .bucket(bucket)
      .key(key)
      .body(f)
      .send()
      .await?;
  },
  Err(e) => {
    panic!("Error uploading file: {:?}", e);
  }
};
Ok(resp)

The match expression implies that we expect the put_object() to either return a successful response or an error when we upload content. If everything works as expected we will return the response from AWS.

The upload function is now finished and we can use it as shown below.

mod s3_uploader;

#[tokio::main]
async fn main() {
  let upload = s3_uploader::upload(
    "path-to-file",
    "S3-bucket",
    "filename-in-bucket",
  ).await;
  println!("{:?}", upload);
}

The #[tokio::main] macro is used to make main async.

Conclusion

This small function can of course be extended further but I hope that this small introduction have sparked some curiosity to continue to use and explore the Rust language.

If you need assistance in the development and implementation of this, our team of video developers are happy to help you out. If you have any questions or comments just drop a line in the comments section to this post.

Locally transcode VOD content with Eyevinn Ingest Application Framework and SVT Encore

Oscar Nord — Fri, 15 Oct 2021 08:57:35 +0000

In a previous post I described how to utilise AWS MediaConvert and MediaPackage to transcode VOD content.

AWS is great for this kind of applications but if you want to be able to do the video transcoding locally an alternative to AWS MediaConvert is SVT Encore which is a open source video transcoder build on FFmpeg and Redis and is developed by Swedish Television.

The IAF Plugins

To make it a bit easier to automate the process of using SVT Encore together with AWS services like S3 and MediaPackage we here at Eyevinn have developed an IAF plugin for local video transcoding with SVT Encore that we have made available as open source.

The general idea is that it should be as easy to use this plugin together with the other IAF plugins, and to chain them together as we did with the IAF AWS MediaConvert plugin in the previous post to push local VOD content to AWS.

The plugin interacts with a local or remote instance of Encore via Encore's API/endpoints which makes it easy to expand upon in the future.

The quickest way to get a local instance of Encore up and running:

Clone Encore and follow the install instructions.
Navigate to encore/src copy the profile folder from encore/src/test/resource to encore/src/main/resource.
Open encore/src/main/resources/application-local.yml and update it with the content below. (We are currently only using the default profiles defined in profiles.yml to test things with)

service:
  name: encore-local

spring:
  redis:
    host: localhost
    port: 6379

profile:
  location: classpath:profile/profiles.yml

encore-settings:
  concurrency: 3
  local-temporary-encode: false
  poll-initial-delay: 1s
  poll-delay: 1s

  audio-mix-presets:
    default:
      pan-mapping:
        6:
          2: stereo|c0=1.0*c0+0.707*c2+0.707*c4|c1=1.0*c1+0.707*c2+0.707*c5

Now run $ SPRING_PROFILES_ACTIVE=local ./gradlew clean bootRun in the Encore root folder.
You should now have a local copy of Encore running to test with.

To use the IAF Encore plugin in an IAF setup it should look something like this:

// other imports
import { EncoreUploadModule } from "@eyevinn/iaf-plugin-encore-local";

const encoreUploader = new EncoreUploadModule(/** args **/);
const fileWatcher = /** initialize your file watcher of choice**/

fileWatcher.onAdd(encoreUploader.onFileAdd);

The IAF Encore plugin will generate a SMIL file when the transcoding job have finished.
To be able to upload this content to an S3 bucket as we did in the MediaConvert example we need another newly developed IAF plugin to upload files as they have been transcoded.

The IAF AWS S3 plugin's fileWatcher will in this case monitor the Encore output folder for files and upload these when added. The MediaPackage plugin can be injected via the fileUploadedDelegate callback function in the same way as when we chain the MediaConvert and MediaPackage plugins together.

We can now create the resource name for the SMIL-file in the S3 bucket and provide that to the IAF MediaPackage plugin

MPUploadModule.onFileAdd("arn:aws:s3:::AWS_BUCKET/file.smil", null);

The MediaPackage plugin will if succeeded return the endpoints that we then can use to stream the content.

Conclusion

As mentioned in the Eyevinn Ingest Application Framework articles previously the IAF is still a work in progress and we encourage readers to keep an eye on the website for changes and news. We hope that the IAF will be able to help developers build their own ingest applications faster, and remove some of the complexity from the process.

Building plugins for the Eyevinn Ingest Application Framework

Oscar Nord — Thu, 23 Sep 2021 13:36:24 +0000

Introduction

As introduced by my colleague Fredrik the Eyevinn Ingest Application Framework (IAF) allows you to build plugins in a modular way that can interact with different storage or transcoding solutions.
In this short walkthrough we will show you how you can use two of the IAF plugins that we have developed to easily transcode and make on demand video content (VOD) available on multiple devices using AWS Elemental MediaConvert and AWS Elemental MediaPackage.

Lets for example say that we have a couple of local video files in MP4 format that we want our users to be able to stream on a wide range of devices. How would we in an easy way make the publicly available? One way is to use AWS MediaConvert to first transcode the files into Apple HLS and then utilise AWS MediaPackage to re-package it to two endpoint types; DASH-ISO and Apple HLS. We can then use a CDN for example CloudFront to deliver the video to the output devices.

The plugins

To automate this process as much as possible the IAF AWS transcoding plugin and the newly added IAF MediaPackage plugin can be used to push transcoded content to MediaPackage.

Both plugins main modules are the AwsUploadModule that implements the IafUploadModule interface. This module interacts with the S3 storage, MediaConvert and MediaPackage. Our IAF applications only needs the onFileAdd method to start uploading the file and start the transcoding. When instantiating the AwsUploadModule it's possible to provide a custom encoder profile. An example when this would be useful could be when we have a video file with only on audio track in 5.1 but we want the output to have two, one in stereo and one in surround. When the file have been uploaded a watcher starts monitoring the output bucket for the transcoded master manifest(s).

When the transcoding plugin have finished uploading the content and a transcoding job have been successfully completed the watcher will pass the AWS source ARN:s as an object to the fileUploadedDelegate. This callback function can be injected when we instantiate AwsUploadModule this is what makes the framework really great and modular because now we can chain two IAF plugins together.

The IAF MediaPackage plugin's onFileAdd method expect the ARN of the transcoded HLS master manifest in this case (it's possible to transcode into multiple formats depending on the configuration JSON) and will start call the dispatcher to create a re-packaging job in MediaPackage. Currently you need to have created a packaging group via the AWS console and provide the packagingGroupId this is something what we would like to update so that we can create the packaging group "on the fly" and pass that to the plugin. The fileUploadedDelegate in this plugin will if the dispatcher have succeeded get the result from MediaPackage including the endpoints.

Example of the iaf-aws and iaf-aws-mediapackage plugins:

The possibility to chain plugins together and/or add other services on top of this gives us the advantage that we can for example instead of transcoding VOD content with MediaPackage add another IAF plugin for MediaLive or some other on-premise live encoder and push that into our MediaPackage plugin without modifying the underlying structure of the framework.

Conclusion

As mentioned in the Eyevinn Ingest Application Framework article by Fredrik the IAF is still a work in progress and we encourage readers to keep an eye on the website for changes and news. We hope that the IAF will be able to help developers build their own ingest applications faster, and remove some of the complexity from the process.

Introduction to text detection using OpenCV and pytesseract

Oscar Nord — Tue, 02 Mar 2021 10:38:56 +0000

Introduction

Detecting and reading text from images is a complex task with a multitude of factors that is needed to take into account, it can be something like detecting handwritten text on a piece of paper to detect subtitles in a movie.

In this short tutorial we are going to utilise OpenCVs image manipulation tools and the Python wrapper for Google’s Tesseract-OCR Engine Python-tesseract or pytesseract to detect a specific number of predefined words in a video sequence.

The fun part

You need to have OpenCV for Python and pytesseract on your system, so start with installing them via pip:

pip install opencv-python
pip install pytesseract

The idea here is to detect when (on what frame) the pre/post credits in a video shows up. For the simplicity of this tutorial just print the word and the frame number to the console.

We start by importing OpenCV and pytesseract and define our list with words.

from cv2 import cv2
import pytesseract

LIST = ['director', 'directed', 'written', 'writer',
        'producer', 'produced', 'production', 'lead']

The input will be a path to a local video file or a url, the url can also be a HLS manifest. Then we extract the first frame and define our frame counter.

path = input('Enter path to file or hls manifest: ')

vidcap = cv2.VideoCapture(path)
success,frame = vidcap.read()
framenbr = 0

What we want to do now is to read the next frame and process it as long as there is content left in our vidcap variable. If no content is present we are done and can break out of the reading loop.

while success:
    success,frame = vidcap.read()
    if not success:
        break

We are now going to process the frame using OpenCV to make it easier for pytesseract to extract the text by convert the image to grayscale.

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

We can also add layers of processing to the image at this stage but I will skip this for the simplicity of this tutorial.

Now on to text extraction using pytesseract:

data = pytesseract.image_to_string(gray, lang='eng', config='--psm 6').lower()

Here gray is the grayscale image, lang='eng' is of course the language that we expect the text which we are trying to detect to be in, and config='--psm 6' defines what setting pytesseract will use when extracting the text. Page segmentation mode 6 will make pytesseract assume a single uniform block of text on extraction.

The available page segmentation modes are:

0 Orientation and script detection (OSD) only.
1 Automatic page segmentation with OSD.
2 Automatic page segmentation, but no OSD, or OCR. (not implemented)
3 Fully automatic page segmentation, but no OSD. (Default)
4 Assume a single column of text of variable sizes.
5 Assume a single uniform block of vertically aligned text.
6 Assume a single uniform block of text.
7 Treat the image as a single text line.
8 Treat the image as a single word.
9 Treat the image as a single word in a circle.
10 Treat the image as a single character.
11 Sparse text. Find as much text as possible in no particular order.
12 Sparse text with OSD.
13 Raw line. Treat the image as a single text line, bypassing hacks that are Tesseract-specific.

The only thing left is to compare what we have in our data variable with our predefined list and continue to the next frame.

if any(word in data for word in LIST):
    print(data + '\nFrame number: ' + str(framenbr))
framenbr += 1

Here is the complete example code:

from cv2 import cv2
import pytesseract

LIST = ['director', 'directed', 'written', 'writer',
        'producer', 'produced', 'production', 'lead']

path = input('Enter path to file or hls manifest: ')

vidcap = cv2.VideoCapture(path)
success,frame = vidcap.read()
framenbr = 0

while success:
    success,frame = vidcap.read()
    if not success:
        break
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    gray = cv2.resize(gray, None, fx=2, fy=2, interpolation=cv2.INTER_LINEAR)
    data = pytesseract.image_to_string(gray, lang='eng', config='--psm 6').lower()
    if any(word in data for word in LIST):
        print(data + '\nFrame number: ' + str(framenbr))
    framenbr += 1

Conclusion

If we would take this to the next step it could be used for detecting when an intro occurs in a tv-serie to make it easier for the viewer to skip it or to continue to the next episode.