Forem: Davkharbayar

Generate Subtitles for Audio and Video Easily with AssemblyAI Speech-to-Text

Davkharbayar — Fri, 22 Nov 2024 18:27:48 +0000

This is a submission for the AssemblyAI Challenge : Sophisticated Speech-to-Text.

What I Built

Subtitles are critical for improving accessibility, engagement, and global reach for videos and audio. As a content creator and developer, I often faced challenges when generating subtitles manually. I wanted an automated, tech-driven solution that could handle this process efficiently.

Demo

Live Demo

Code

Home screen

mp3 file create subtitle

create SRT file

Journey: Incorporating Universal-2, AssemblyAI's Speech-to-Text Model into My Application

Starting Point: The Problem

Manually creating subtitles for audio and video files was a tedious and time-consuming task. It required listening to recordings, transcribing speech into text, and carefully syncing subtitles with audio. For long or complex recordings, this process was not only error-prone but also impractical.

I envisioned building an automated solution that could handle this entire workflow seamlessly. The goals were ambitious yet practical:

Key Objectives

1. Accurately Transcribe Speech into Text
Leverage AI to precisely convert spoken words into text, even in noisy or multi-speaker environments.

2. Generate Subtitles in Popular Formats like SRT
Ensure compatibility with platforms like YouTube, social media, and video editing software.

3. Create Subtitled Videos Using FFMPEG
Integrate the subtitles directly into video files, saving users the hassle of separate configurations.

4. Add Subtitles with a Background for Audio Files
For users with audio-only content, generate a video with subtitles displayed on a beautiful, customizable background.

5. Enhance Content with Thumbnail Images and Animated WebP Files
Utilize FFMPEG to create visually engaging thumbnail images and lightweight, animated WebP files for promotional use.

The Solution
To achieve these goals, I combined AssemblyAI's Universal-2 Speech-to-Text model with the powerful media-processing capabilities of FFMPEG. The workflow ensures speed, accuracy, and flexibility, making it ideal for content creators, educators, and businesses alike.

Tools Used
Here’s an overview of the tools and technologies that powered this project:

1. AssemblyAI
Role: Core transcription engine.
Features Used:

Transcription API: Converts audio and video into text with high accuracy, providing timestamps, speaker diarization, and punctuation.
Sentence API: Extracts transcription data at the sentence level, making it easier to format and sync subtitles.

Node.js (Express and EJS Engine) Role: Backend server and template engine. Features Used:

Express.js: Built the API endpoints for handling user requests, file uploads, and processing workflows.
EJS Template Engine: Rendered dynamic web pages for the user interface, allowing seamless file uploads and result display.

FFMPEG Role: Media processing and editing. Features Used:

Subtitled Videos: Burned SRT subtitles directly into video files.
Audio-to-Video Conversion: Added subtitles to audio files by generating a video with a custom background.
Thumbnail Generation: Captured still images from videos for thumbnails.
Animated WebP Files: Created lightweight animations for social media or marketing.

Final Thoughts
Building an automated subtitle generator using AssemblyAI and FFMPEG was both an exciting challenge and a rewarding journey. By integrating state-of-the-art speech-to-text technology with powerful media processing tools, I was able to create a solution that simplifies subtitle creation, enhances accessibility, and delivers professional results effortlessly.

Key Takeaways
The Power of AI: AssemblyAI’s Universal-2 model proved to be a game-changer, offering high accuracy and advanced features like speaker diarization and timestamping.
Automation Matters: Automating tedious tasks like transcription and subtitle generation saves time and eliminates errors, making life easier for content creators and professionals.
FFMPEG’s Versatility: Whether it’s burning subtitles into videos, adding visuals to audio, or creating animated media, FFMPEG brought flexibility and polish to the project.

AI-Powered CV Analysis Platform

Davkharbayar — Sun, 10 Nov 2024 08:21:42 +0000

What I Built

The AI-Powered CV Analysis Platform is a groundbreaking tool that connects job seekers and employers through the power of artificial intelligence. With this platform, users can upload their resumes and job advertisements to receive a detailed analysis. It provides real-time insights, practical recommendations, and personalized guidance to enhance career prospects. Moreover, it helps users discover suitable job opportunities tailored to their skills and experiences, making the job search process smarter and more efficient.

Demo

Live Demo

Repo

Video

Home Page Job List: Explore tailored job opportunities directly from the platform's homepage.

Job detail

Resume analysis in progress: Turning your CV into actionable insights for better job opportunities.

AI Suitability Score and Proficiency Levels: Matching your CV to the perfect job with precision and insights.

Discover similar job opportunities tailored to your skills and experience.

Tools Used

TimescaleDB

Used as the primary database for efficient management and retrieval of document and job-related data.

PGAI

Acts as a core database for document storage and processing. It provides the foundation for managing AI-related data integrations.

pgvector

Utilized to store vector embeddings of documents, enabling advanced AI-driven search and similarity matching.

pgvectorscale

Employed to create indexes on the vector embeddings, optimizing performance for vector similarity queries and enhancing the search experience.

OpenAI API

Powers the AI capabilities for document analysis, embedding generation, and providing intelligent insights and recommendations.

Tech Stack

Backend

Node.js: Server-side runtime environment.
Express.js: Web framework for building server logic and handling routing.

Frontend

EJS (Embedded JavaScript): Used to render dynamic HTML pages. Combines server-side data with frontend templates for seamless integration.

Database

TimescaleDB: Primary database for managing and retrieving document data.
PostgreSQL: Base relational database supporting structured queries.
pgvector: For storing document embeddings.
pgvectorscale: To create and optimize indexes for vector searches.

Final Thoughts

Building the AI-Powered CV Analysis Platform has been an incredible journey, blending traditional database systems with cutting-edge AI capabilities. At first, the idea of integrating advanced AI, like embedding models and similarity searches, directly within the database seemed unconventional. However, as I delved deeper into the project, I realized how transformative this approach could be, especially for simplifying workflows for non-technical users like recruiters and HR professionals.

What excites me most about this project is how it bridges the gap between technical complexity and user-friendly solutions. By utilizing tools like TimescaleDB, pgvector, and OpenAI's API, we've created a system that can streamline CV analysis and job matching without the need for external AI frameworks. This approach not only simplifies integration but also ensures faster, more efficient insights directly from the database.

One of the highlights of this experience was learning to optimize PostgreSQL with extensions like pgvector, and creating a seamless interaction between AI embeddings and SQL queries.

This project has shown me that innovation often lies in reimagining familiar tools in new contexts—and that AI-powered solutions can democratize access to advanced analytics in ways we’re just beginning to explore.

Working with TimescaleDB felt like discovering an exciting new frontier in data management.

TimescaleDB made it feel like the limitations of traditional databases could be overcome without compromising on performance or scalability. Its integration into this project enabled a seamless flow of AI-powered insights, reinforcing the idea that innovation is as much about leveraging the right tools as it is about having the right ideas.

This project has shown me that innovation often lies in reimagining familiar tools in new contexts—and TimescaleDB was instrumental in pushing those boundaries. It turned what seemed like a technical challenge into an exciting opportunity to create something truly transformative.