Forem: Awal Hossain

Behind the Stream: Building a Scalable Video Server Solution

Awal Hossain — Mon, 21 Apr 2025 04:51:15 +0000

Alright, in Part 1, we got the 30,000-foot view of our microservices architecture for video streaming. We talked about the why (handling uploads, conversion, smooth playback) and the what (API Gateway, API Server, Video Conversion Service, HLS).

If you missed Part 1, you can find the link here: Part-1

Now, it's time to roll up our sleeves and get our hands dirty with the how. In this part, we'll focus on:

Setting up your development environment.
Implementing the direct-to-cloud upload flow using pre-signed URLs.
Wiring up the video conversion pipeline using FFmpeg, BullMQ, and RabbitMQ.

1. Setting Up Shop: Your Development Environment

First things first, let's make sure you have the necessary tools installed.

Prerequisites:

Node.js: v18.0.0 or later (check with node -v).
Yarn: (Or npm) for package management (check with yarn -v).
Docker & Docker Compose: To easily run our services and dependencies like Redis and RabbitMQ. Install Docker.
FFmpeg: The video processing powerhouse. If you missed it in Part 1, installation guides are often found online (e.g., brew install ffmpeg on Mac, or check the official FFmpeg site).
Code Editor: VS Code, WebStorm, Sublime Text - your choice!
Git: For cloning the project repository.

Get the Code:

git clone https://github.com/AwalHossain/video-streaming-server
cd video-streaming-server

Environment Variables (.env): The Secret Sauce

Each of our services needs some configuration, like database connection strings, API keys, and secrets. We manage these using .env files. You'll find a .env.example file in each service directory (api-gateway, api-server, video-conversion).

Action: For each service directory:

Copy .env.example to a new file named .env.
Open the .env file and fill in the values specific to your setup.

Here are some key variables you'll need to configure:

PORT: The port each service will run on (e.g., 8000, 8001, 8002).
MONGO_URI: Your MongoDB connection string (for api-server).
RABBITMQ_ENDPOINT: Your RabbitMQ server address (e.g., amqp://user:password@localhost:5672).
REDIS_URL: Your Redis connection URL (for video-conversion, e.g., redis://localhost:6379).
JWT_SECRET: A long, random secret string for signing authentication tokens (used by api-gateway and api-server).
Cloud Storage Credentials (e.g., for Digital Ocean Spaces or AWS S3):
- DO_SPACES_ACCESS_KEY / AWS_ACCESS_KEY_ID
- DO_SPACES_SECRET_KEY / AWS_SECRET_ACCESS_KEY
- DO_SPACES_ENDPOINT / AWS_ENDPOINT (if using non-AWS S3 compatible)
- DO_SPACES_BUCKET_NAME / AWS_BUCKET_NAME
- DO_SPACES_REGION / AWS_REGION
CLIENT_URL1, CLIENT_URL2: The URLs of your frontend application(s) for CORS configuration.

2. The Grand Upload: Pre-signed URLs in Action

Remember how we wanted to avoid bogging down our servers with large file uploads? Pre-signed URLs are the answer! The client gets a temporary, secure link to upload directly to cloud storage.

API Gateway - The Gatekeeper:

Routing: We need an endpoint to request the pre-signed URL.

// api-gateway/src/app/routes/index.ts
import { Router } from 'express';
import { AuthRoutes } from '../modules/auth/auth.routes';
import { VideoRoutes } from '../modules/video/video.route';

const router = Router();
const ModuleRoutes = [
  { path: '/auth', route: AuthRoutes },
  { path: '/videos', route: VideoRoutes }, // Our video routes
];
ModuleRoutes.forEach((route) => router.use(route.path, route.route));
export default router;

// api-gateway/src/app/modules/video/video.route.ts
import express from 'express';
import auth from '../../middleware/auth'; // <-- Authentication middleware
import { VideoController } from './video.controller';

const router = express.Router();
// POST endpoint protected by auth middleware
router.post(
    '/presigned-url',
     auth(), // Ensures only logged-in users can get a URL
     VideoController.getPresignedUrl
    );
router.post(
    '/confirm-upload',
     auth(),
     VideoController.confirmUpload
     );
// ... other video routes
export const VideoRoutes = router;

Controller (getPresignedUrl): This function handles the request.

// api-gateway/src/app/modules/video/video.controller.ts
import { Request, Response } from 'express';
import { v4 as uuidv4 } from 'uuid';
import catchAsync from '../../../shared/catchAsyncError';
import sendResponse from '../../../shared/response';
import doSpacesUpload from '../../../utils/doSpacesUpload'; // The utility function

const getPresignedUrl = catchAsync(async (req: Request, res: Response) => {
  const { filename, contentType } = req.body; // Get info from client

  if (!filename || !contentType) {
    return res.status(400).json({ /* ... error */ });
  }

  const userId = req.user.id; // From auth middleware

  // Clean filename and make it unique
  const cleanFilename = filename
    .split('.')[0]
    .replace(/\s+/g, '-')
    // ... more cleaning ...
    + '-' + uuidv4() + '.' + filename.split('.').pop();

  // Call the utility to generate the URL
  const result = await doSpacesUpload.generatePresignedUrl({
    filename: cleanFilename,
    contentType: contentType,
    userId: userId,
    expirySeconds: 3600, // URL valid for 1 hour
  });

  sendResponse(res, {
    statusCode: 200,
    success: true,
    message: 'Presigned URL generated successfully',
    data: result, // Send URL and fileKey back to client
  });
});

Utility (doSpacesUpload.ts): This interacts with the cloud storage SDK.

// api-gateway/src/utils/doSpacesUpload.ts
import AWS from 'aws-sdk'; // Using AWS SDK for S3 compatible storage
import config from '../config';

const s3 = new AWS.S3({ // Configure S3 client
  endpoint: config.doSpaces.endpoint,
  accessKeyId: config.doSpaces.accessKey,
  secretAccessKey: config.doSpaces.secretKey,
  region: config.doSpaces.region,
  s3ForcePathStyle: true,
  signatureVersion: 'v4'
});

const generatePresignedUrl = async (options) => {
  const { filename, contentType, userId, expirySeconds = 3600 } = options;
  const timestamp = Date.now();
  // Define the path/key in the bucket
  const key = `uploads/${userId}/${timestamp}-${filename}`;

  const params = {
    Bucket: config.doSpaces.bucketName,
    Key: key,
    Expires: expirySeconds,
    // ACL: 'public-read', // Optional: If you want uploaded files to be public
    // ContentType: contentType // Often needed for direct browser uploads
  };

  try {
    // Generate the signed URL for PUT operation
    const url = await s3.getSignedUrlPromise('putObject', params);
    const fileUrl = `https://${config.doSpaces.bucketName}.${config.doSpaces.endpoint}/${key}`;

    return {
      uploadUrl: url, // The URL the client will use to PUT the file
      fileKey: key,   // The path/key of the file in the bucket
      bucketName: config.doSpaces.bucketName,
      fileUrl, // Direct link to the file (if public)
      fileName: key.split('/').pop(),
    };
  } catch (error) {
    console.error('Error generating presigned URL:', error);
    throw error;
  }
};

export default { generatePresignedUrl };

Client-Side Upload (Conceptual):

Once the client receives the uploadUrl, fileKey, etc., they perform the upload.

// Simple example using fetch
async function uploadFile(presignedData, file) {
  const { uploadUrl, contentType } = presignedData; // Assume contentType was passed back or known

  try {
    const response = await fetch(uploadUrl, {
      method: 'PUT',
      headers: {
        'Content-Type': file.type, // Use the actual file type
        // 'x-amz-acl': 'public-read' // If ACL was set during URL generation
      },
      body: file
    });

    if (!response.ok) {
      throw new Error(`Upload failed: ${response.statusText}`);
    }
    console.log('Upload successful!');
    // Now call the confirmUpload endpoint
  } catch (error) {
    console.error('Upload error:', error);
  }
}

Confirming the Upload:

After the direct upload finishes, the client must notify our backend.

// api-gateway/src/app/modules/video/video.controller.ts (confirmUpload)
const confirmUpload = catchAsync(async (req: Request, res: Response) => {
  // Client sends back fileKey, originalName, etc. received earlier
  const { fileKey, originalName, fileName, bucketName } = req.body;
  const userId = req.user.id;

  // 1. Notify user upload is complete (via RabbitMQ -> WebSocket)
  RabbitMQ.sendToQueue(NOTIFY_EVENTS.NOTIFT_VIDEO_UPLOADING_BUCKET, {
     userId, status: 'completed', /* ... */
  });

  // 2. Prepare metadata payload for API Server
  const metadataPayload = {
    originalName,
    recordingDate: Date.now(),
    author: userId,
    fileName,
    title: originalName.split('.')[0].replace(/[_]/g, ''), // Basic title generation
    // ... other initial metadata
  };

  // 3. Send event to API Server to save metadata
  broadcastVideoEvent(API_SERVER_EVENTS.INSERT_VIDEO_METADATA_EVENT, metadataPayload);

  // 4. Prepare data for Video Conversion Service
  const conversionData = { userId, fileName, bucketName, fileKey };

  // 5. Send event to Video Conversion Service to start processing
  broadcastVideoEvent(
    VIDEO_CONVERSION_SERVER.SEND_VIDEO_METADATA_EVENT,
    conversionData
  );

  sendResponse(res, { /* ... success response */ });
});

Phew! That's the upload flow. The key is the client uploads directly to storage, and then notifies our backend to kick off the next steps via RabbitMQ events.

3. Conversion Central: FFmpeg, BullMQ, and RabbitMQ

Now, the Video Conversion service takes over.

Receiving the Signal:

The service listens for the SEND_VIDEO_METADATA_EVENT on RabbitMQ. When it receives a message, it triggers the download process.

// video-conversion/src/server.ts (Conceptual listener setup)
import RabbitMQ from './shared/rabbitMQ';
import downloadBlob from './processor/downloadFile';
import { VIDEO_CONVERSION_SERVER } from './constant/events';

async function startServer() {
 // ... other setup ...

 await RabbitMQ.connect();

 // Listen for the event from API Gateway
 await RabbitMQ.consume(
   VIDEO_CONVERSION_SERVER.SEND_VIDEO_METADATA_EVENT,
   async (msg, ack) => {
     try {
       const data = JSON.parse(msg.content.toString());
       logger.info('Received video metadata event:', data);
       // Trigger the download process
       await downloadBlob(data.bucketName, data.fileKey, data.userId);
       ack(); // Acknowledge the message
     } catch (error) {
       errorLogger.error('Failed processing metadata event:', error);
       // Decide if you should ack or nack based on the error
       ack(); // Ack for now to prevent reprocessing loop on simple errors
     }
   }
 );

 // ... start express app ...
}

Downloading the File (downloadFile.ts):

This function grabs the file from cloud storage.

// video-conversion/src/processor/downloadFile.ts
import fs from 'fs';
import path from 'path';
import { v4 as uuidv4 } from 'uuid';
import s3 from '../shared/s3Client'; // Configured S3 client
import initiateVideoProcessing from './initiateVideoProcessing'; // Next step
import RabbitMQ from '../shared/rabbitMQ';
import { API_GATEWAY_EVENTS } from '../constant/events';

async function downloadBlob(bucketName, fileKey, userId) {
  try {
    const fileName = path.basename(fileKey);
    const originalName = fileName.substring(fileName.indexOf('-') + 1); // Extract original name

    // Notify download start
    RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_VIDEO_DOWNLOADING, { /* ... */ });

    // Create a unique local directory for processing
    const uploadFolder = `container-${uuidv4()}`;
    const destination = path.normalize(path.join('uploads', uploadFolder, 'videos')).replace(/\\\\/g, '/');
    fs.mkdirSync(destination, { recursive: true });
    const videoPath = path.normalize(path.join(destination, fileName)).replace(/\\\\/g, '/');

    logger.info(`Downloading file from DO Spaces to ${videoPath}`);

    // Perform the download using the S3 SDK stream
    await new Promise<void>((resolve, reject) => {
      const writeStream = fs.createWriteStream(videoPath);
      s3.getObject({ Bucket: bucketName, Key: fileKey })
        .createReadStream()
        .pipe(writeStream)
        .on('error', reject)
        .on('finish', resolve);
    });

    // Notify download complete
    RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_VIDEO_DOWNLOADED, { /* ... */ }); // Typo in original? Should be NOTIFY_VIDEO_DOWNLOADED

    // --- Trigger the next step: Add to processing queue ---
    // (Note: Fetching metadata via API_SERVER_EVENTS.GET_VIDEO_METADATA_EVENT
    // might happen here or be passed in the initial event)
    const videoMetadata = { _id: 'some-id-from-db', /* ... other fields */ }; // Placeholder

    await initiateVideoProcessing({ videoPath, destination, userId, videoMetadata });

  } catch (error) {
    errorLogger.error('Error in downloadBlob:', error);
    RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_EVENTS_FAILED, { /* ... */ });
    throw error;
  }
}

Kicking Off the Queue (initiateVideoProcessing.ts):

After downloading, we add the first job to our BullMQ processing queue.

// video-conversion/src/processor/initiateVideoProcessing.ts
import { QUEUE_EVENTS, API_SERVER_EVENTS } from '../constant/events';
import { addQueueItem } from '../queues/addJobToQueue'; // BullMQ helper
import { getVideoDurationAndResolution } from './videoProcessingHandler';
import RabbitMQ from '../shared/rabbitMQ';

const initiateVideoProcessing = async ({ videoPath, destination, userId, videoMetadata }) => {
  // Get basic video info like duration
  const { videoDuration } = await getVideoDurationAndResolution(videoPath);

  const payload = {
    ...videoMetadata,
    duration: videoDuration,
    id: videoMetadata._id, // Crucial: DB id for tracking
    path: videoPath,       // Local path after download
    destination,           // Local processing directory
    userId,
  };

  // Optional: Update DB immediately with duration/local path
  RabbitMQ.sendToQueue(API_SERVER_EVENTS.UPDATE_METADATA_EVENT, {
    id: videoMetadata._id,
    videoPath, // May not be needed in DB long-term
    duration: videoDuration,
  });

  // Add the first job to the queue!
  await addQueueItem(QUEUE_EVENTS.VIDEO_UPLOADED, payload); // Start the chain
};

export default initiateVideoProcessing;

The BullMQ Workforce:

BullMQ manages our jobs.

Queue Setup (addJobToQueue.ts): Defines the queues based on event names.

// video-conversion/src/queues/addJobToQueue.ts
import { Queue } from 'bullmq';
import { ALL_EVENTS as QUEUE_EVENTS } from '../constant/events';
import { RedisClient } from '../shared/redis'; // Redis connection

export const queues = Object.values(QUEUE_EVENTS).map((queueName) => {
  return {
    name: queueName,
    queueObj: new Queue(queueName, { connection: RedisClient.redisConnection }),
  };
});

export const addQueueItem = async (queueName, item) => {
  const queue = queues.find((q) => q.name === queueName);
  if (!queue) throw new Error(`Queue ${queueName} not found`);
  // Add job to the queue
  await queue.queueObj.add(queueName, item, {
    removeOnComplete: true, // Clean up successful jobs
    removeOnFail: false,    // Keep failed jobs for inspection
  });
};

Worker Setup (jobWorker.ts): Listens for jobs and calls the right handler.

// video-conversion/src/worker/jobWorker.ts
import { Worker } from 'bullmq';
import { QUEUE_EVENTS } from '../constant/events';
import { QUEUE_EVENT_HANDLERS } from '../processor/videoEventHandler'; // Maps event name to function
import { RedisClient } from '../shared/redis';
import { logger } from '../shared/logger';

export const listenQueueEvent = (queueName) => {
  const worker = new Worker(
    queueName,
    async (job) => { // Job processor function
      const handler = QUEUE_EVENT_HANDLERS[queueName]; // Find the right handler
      if (handler) {
        logger.info(`Processing job ${job.id} from queue ${queueName}`);
        return await handler(job); // Execute the handler
      }
      throw new Error('No handler found for queue: ' + queueName);
    },
    { connection: RedisClient.redisConnection, concurrency: 3 } // Process up to 3 jobs concurrently
  );

  worker.on('failed', (job, err) => {
    logger.error(`Job ${job?.id} failed in queue ${queueName}: ${err.message}`, err);
  });

  logger.info(`${queueName} worker started.`);
};

export const setupAllQueueEvent = () => { // Called at startup
  Object.values(QUEUE_EVENTS).forEach((queueName) => {
    listenQueueEvent(queueName);
  });
  // videoLifecycleHandler(); // Setup lifecycle listeners (e.g., for uploads)
  return true;
};

The Processing Pipeline (videoEventHandler.ts):

This file contains the functions (handlers) that execute for each job type. They often add the next job in the sequence upon completion.

// video-conversion/src/processor/videoEventHandler.ts
import { Job } from 'bullmq';
import { QUEUE_EVENTS } from '../constant/events';
import { addQueueItem } from '../queues/addJobToQueue';
import processMp4ToHls from './hlsConvertProcessor';
import { generateThumbnail, processRawFileToMp4WithWatermark } from './mp4ConvertProcessor';
// ... other imports like fs, path, logger

const uploadedHandler = async (job: Job) => {
  logger.info('Handler: VIDEO_UPLOADED', job.data.id);
  // Simply trigger the next step
  await addQueueItem(QUEUE_EVENTS.VIDEO_PROCESSING, job.data);
};

const processingHandler = async (job: Job) => { // Convert to MP4 if needed
  logger.info('Handler: VIDEO_PROCESSING', job.data.id);
  const filePath = job.data.path; // Local path of downloaded file
  const uploadPath = `${job.data.destination}/processed`; // Output dir for MP4
  const thumbnailPath = `${job.data.destination}/thumbnails`; // Output dir for thumbs
  fs.mkdirSync(uploadPath, { recursive: true });
  fs.mkdirSync(thumbnailPath, { recursive: true });

  // Generate thumbnail (can happen in parallel or sequence)
  await generateThumbnail(filePath, thumbnailPath, job.data); // Sends RabbitMQ event on completion

  if (needsConversion(filePath)) { // Check if it's already mp4/webm etc.
     logger.info(`Converting ${filePath} to MP4...`);
     // Calls ffmpeg internally, adds VIDEO_PROCESSED job on completion
     await processRawFileToMp4WithWatermark(filePath, uploadPath, {
       ...job.data, next: QUEUE_EVENTS.VIDEO_PROCESSED
     });
  } else {
     logger.info(`Skipping MP4 conversion for ${filePath}.`);
     // File is okay, move to next step directly
     const destPath = `${uploadPath}/${path.basename(filePath)}`; // Define destination
     fs.copyFileSync(filePath, destPath); // Just copy it
     // Add the next job manually since we skipped the conversion function
     await addQueueItem(QUEUE_EVENTS.VIDEO_PROCESSED, { ...job.data, path: destPath });
  }
};

const processedHandler = async (job: Job) => { // MP4 ready, start HLS
   logger.info('Handler: VIDEO_PROCESSED', job.data.id);
   await addQueueItem(QUEUE_EVENTS.VIDEO_HLS_CONVERTING, job.data);
};

const hlsConvertingHandler = async (job: Job) => { // The core HLS conversion
  logger.info('Handler: VIDEO_HLS_CONVERTING', job.data.id);
  const hlsUploadPath = `${job.data.destination}/hls`; // Output dir for HLS files
  fs.mkdirSync(hlsUploadPath, { recursive: true });

  // Calls the main ffmpeg HLS logic, adds VIDEO_HLS_CONVERTED job on completion
  await processMp4ToHls(job.data.path, hlsUploadPath, { // job.data.path is now the MP4 path
      ...job.data, next: QUEUE_EVENTS.VIDEO_HLS_CONVERTED
  });
};

const hlsConvertedHandler = async (job: Job) => { // HLS done, trigger upload
    logger.info('Handler: VIDEO_HLS_CONVERTED', job.data.id);
    // This handler might trigger the upload of the HLS files to cloud storage
    // via processHLSFile called from videoLifecycleHandler.ts
    // It should also send a final notification event.
};

export const QUEUE_EVENT_HANDLERS = { // Mapping used by the worker
  [QUEUE_EVENTS.VIDEO_UPLOADED]: uploadedHandler,
  [QUEUE_EVENTS.VIDEO_PROCESSING]: processingHandler,
  [QUEUE_EVENTS.VIDEO_PROCESSED]: processedHandler,
  [QUEUE_EVENTS.VIDEO_HLS_CONVERTING]: hlsConvertingHandler,
  [QUEUE_EVENTS.VIDEO_HLS_CONVERTED]: hlsConvertedHandler,
};

FFmpeg Magic (hlsConvertProcessor.ts):

This is where the actual HLS conversion command is built and executed.

// video-conversion/src/processor/hlsConvertProcessor.ts
import ffmpegPath from 'ffmpeg-static';
import ffmpeg from 'fluent-ffmpeg';
import path from 'path';
import RabbitMQ from '../shared/rabbitMQ';
import { API_GATEWAY_EVENTS } from '../constant/events';
// ... other imports

ffmpeg.setFfmpegPath(ffmpegPath); // Tell fluent-ffmpeg where ffmpeg is

const processMp4ToHls = async (filePath, outputFolder, jobData) => {
  const fileNameWithoutExt = path.basename(filePath, path.extname(filePath));

  // --- Define HLS Renditions (Quality Levels) ---
  // (Includes logic for vertical vs. landscape videos from your code)
  const renditions = [
      { resolution: '854x480', bitrate: '800k', name: '480p' },
      { resolution: '1920x1080', bitrate: '5000k', name: '1080p' },
      // Add more if needed
  ];

  const renditionProgress = {}; // Track progress per rendition
  renditions.forEach(r => renditionProgress[r.name] = 0);
  let lastReportedProgress = 0;

  const promises = renditions.map((rendition) => {
    return new Promise<void>((resolve, reject) => {
      const outputFileName = `${fileNameWithoutExt}_${rendition.name}.m3u8`;
      const segmentFileName = `${outputFolder}/${fileNameWithoutExt}_${rendition.name}_%03d.ts`;

      ffmpeg(filePath) // Input MP4 file
        .output(path.join(outputFolder, outputFileName)) // Output M3U8 playlist for this rendition
        .outputOptions([
          `-s ${rendition.resolution}`,    // Set resolution
          `-c:v libx264`,                 // Video codec
          `-crf 23`,                      // Quality (lower is better)
          `-preset fast`,                 // Encoding speed vs compression
          `-b:v ${rendition.bitrate}`,    // Target video bitrate
          `-g 48`,                        // Keyframe interval
          `-hls_time 10`,                 // Segment duration (seconds)
          `-hls_playlist_type vod`,       // Video on Demand playlist
          `-hls_flags independent_segments`,
          `-hls_segment_filename ${segmentFileName}`, // Naming pattern for TS files
        ])
        .on('start', (cmd) => logger.info(`FFmpeg started for ${rendition.name}: ${cmd}`))
        .on('progress', (progress) => { // --- Progress Reporting ---
          renditionProgress[rendition.name] = progress.percent || 0;
          const totalProgress = Object.values(renditionProgress).reduce((a, b) => a + b, 0);
          const overallProgress = Math.round(totalProgress / renditions.length);

          // Report progress in ~10% increments
          if (overallProgress > 0 && overallProgress - lastReportedProgress >= 10) {
            lastReportedProgress = overallProgress;
            RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_EVENTS_VIDEO_BIT_RATE_PROCESSING, {
                userId: jobData.userId,
                status: 'processing',
                progress: overallProgress,
                fileName: fileNameWithoutExt,
                /* ... */
            });
          }
        })
        .on('end', () => {
          logger.info(`FFmpeg finished for ${rendition.name}`);
          resolve();
        })
        .on('error', (err) => {
          errorLogger.error(`FFmpeg error for ${rendition.name}:`, err);
          RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_EVENTS_FAILED, { /* ... failure data */ });
          reject(err);
        })
        .run();
    });
  });

  await Promise.all(promises); // Wait for all renditions

  // --- Create Master Playlist ---
  // (Code to generate the master manifest referencing all rendition playlists)
  // fs.writeFileSync(path.join(outputFolder, `${fileNameWithoutExt}_master.m3u8`), masterPlaylistContent);

  logger.info('HLS conversion completed for:', fileNameWithoutExt);

  // Send final success notification
  RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_EVENTS_VIDEO_BIT_RATE_PROCESSED, {
       userId: jobData.userId, status: 'completed', /* ... */
  });

  // Add the next job (e.g., upload HLS files) or mark as complete
  // This is handled by videoLifecycleHandler in your code
  // which listens for QUEUE_EVENTS.VIDEO_HLS_CONVERTED and calls processHLSFile
};

Uploading Processed Files:

The videoLifecycleHandler.ts listens for the QUEUE_EVENTS.VIDEO_HLS_CONVERTED event emitted by BullMQ (when the hlsConvertedHandler completes). It then calls processHLSFile.ts, which in turn likely uses uploadToCloud.ts to upload the generated .m3u8 playlists and .ts segment files back to your cloud storage.

4. Tying it Together: RabbitMQ & Real-time Updates

We saw RabbitMQ used everywhere!

API Gateway -> API Server (Save Metadata)
API Gateway -> Video Conversion (Start Processing)
Video Conversion -> API Gateway (Progress/Status Updates)
Video Conversion -> API Server (Update Metadata like duration, status)

The key for real-time updates is:

Video Conversion sends progress events (e.g., NOTIFY_EVENTS_VIDEO_BIT_RATE_PROCESSING) to RabbitMQ, including the userId.
API Gateway consumes these events from RabbitMQ.
Using Socket.io, the API Gateway finds the WebSocket connection associated with that userId and emits the progress data directly to the connected client's browser.

// api-gateway/src/socket/index.ts (Conceptual Example)
import { Server as SocketIOServer, Socket } from 'socket.io';
import http from 'http';
import RabbitMQ from '../shared/rabbitMQ';
import { API_GATEWAY_EVENTS } from '../constants/event'; // Or NOTIFY_EVENTS

let io: SocketIOServer;
const users: Map<string, string> = new Map(); // Map userId to socketId

export const initSocketIO = (server: http.Server) => {
  io = new SocketIOServer(server, { cors: { /* ... */ } });

  io.on('connection', (socket: Socket) => {
    console.log('A user connected:', socket.id);

    // Store user association (needs authentication, e.g., JWT verification on connect)
    const userId = getUserIdFromSocket(socket); // Implement this securely
    if (userId) {
        users.set(userId, socket.id);
        console.log(`User ${userId} mapped to socket ${socket.id}`);
    }


    socket.on('disconnect', () => {
      console.log('User disconnected:', socket.id);
      // Clean up user mapping
      users.forEach((socketId, uId) => {
          if (socketId === socket.id) {
              users.delete(uId);
          }
      });
    });
  });

  // Listen for RabbitMQ progress events
  listenForProgressUpdates();

  return io;
};

const listenForProgressUpdates = async () => {
    await RabbitMQ.consume(
        API_GATEWAY_EVENTS.NOTIFY_EVENTS_VIDEO_BIT_RATE_PROCESSING, // Example event
        (msg, ack) => {
            try {
                const data = JSON.parse(msg.content.toString());
                const userId = data.userId;
                const socketId = users.get(userId); // Find the user's socket

                if (socketId && io) {
                     console.log(`Sending progress to user ${userId} (socket ${socketId})`);
                     // Emit event directly to the specific user's socket
                     io.to(socketId).emit('video_progress', data);
                } else {
                    console.log(`Socket not found for user ${userId}, unable to send progress.`);
                }
                ack();
            } catch (error) {
                console.error("Error processing progress update:", error);
                ack(); // Ack to avoid requeue loops
            }
        }
    );
    // ... listen for other notification events ...
};

// Helper to securely get userId from socket handshake/query
function getUserIdFromSocket(socket: Socket): string | null {
    // IMPORTANT: Implement proper authentication here!
    // E.g., verify JWT passed in handshake query or auth object
    return socket.handshake.query.userId as string || null;
}

export { io }; // Export for potential use elsewhere

Docker Installation

The easiest way to run all services and their dependencies (MongoDB, RabbitMQ, Redis) is using Docker Compose. We have docker-compose.yml files defined. Let's look at the one in the root directory:

# docker-compose.yml (Root Directory - Simplified Example)
version: "3.7"
services:
  api-gateway:
    build: ./api-gateway
    ports:
      - "8000:8000" # Example port
    env_file:
      - ./api-gateway/.env
    depends_on:
      - rabbitmq
    volumes:
      - ./api-gateway:/app
      - /app/node_modules

  api-server:
    build: ./api-server
    ports:
      - "8001:8001" # Example port
    env_file:
      - ./api-server/.env
    depends_on:
      - mongodb
      - rabbitmq
    volumes:
      - ./api-server:/app
      - /app/node_modules

  video-conversion:
    build: ./video-conversion
    ports:
      - "8002:8002" # Example port
    env_file:
      - ./video-conversion/.env
    depends_on:
      - redis
      - rabbitmq
    volumes:
      - ./video-conversion:/app
      - ./video-conversion/uploads:/app/uploads # Mount uploads volume
      - /app/node_modules

  mongodb:
    image: mongo:latest
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

  rabbitmq:
    image: rabbitmq:3-management
    ports:
      - "5672:5672"  # For AMQP
      - "15672:15672" # For Management UI

  redis:
    image: redis:latest
    ports:
      - "6379:6379"

volumes:
  mongo-data:

To start everything:

docker-compose up -d --build

This command builds the images for our services (if they don't exist) and starts all containers in the background.

5. Conclusion & What's Next

We covered a lot of ground! We set up the environment, built the direct-to-cloud upload flow with pre-signed URLs, and wired up the complex video conversion pipeline using FFmpeg, BullMQ for queuing, and RabbitMQ for communication and real-time updates.
You should now have a much clearer understanding of the core mechanics.

Coming up in Part 3:

Diving deeper into the API Server and MongoDB schema.
Exploring deployment strategies using Docker Swarm or Kubernetes.
Discussing error handling and monitoring.
Potential optimizations and scaling considerations.

Part-3: Check out

Explore the full implementation code from the GitHub repository:

Server-Code
Server-Client

Do follow me on my Socials to stay updated on tech stuff:

💼 LinkedIn
🐙 Github

Building Your Own Scalable Video Streaming Server: Part 3 - Data, Deployment & Durability

Awal Hossain — Sun, 13 Apr 2025 10:44:17 +0000

Welcome back! In Part 1, we looked at the overall structure, and in Part 2, we jumped into the code — setting up our environment and building the main upload and HLS conversion features.

If you missed previous parts, you can find them here:

Part-1: https://dev.to/awalhossain/building-your-own-scalable-video-streaming-server-part-1-the-big-picture-5fa7
Part-2:https://dev.to/awalhossain/building-your-own-scalable-video-streaming-server-part-2-setup-core-implementation-32gf

Now, in this final part, we’ll work on the key parts needed to make our project stronger and ready to deploy:

The API Server's Role: How we manage and store video metadata using MongoDB.
Taking Flight: Strategies for deploying our microservices.
Handling Hiccups: Making our system resilient with error handling and monitoring.
Supercharging: Potential optimizations and scaling considerations.

Let's get this system ready for the real world!

1. The Librarian's Bookshelf: API Server & MongoDB Deep Dive

While the API Gateway handles requests and the Video Conversion service does the heavy lifting, the API Server acts as the central source of truth for our data. Its main jobs are:

Managing video metadata (titles, descriptions, status, etc.).
Handling user data (if you have user accounts).
Providing APIs for fetching video information.
Keeping the video processing status up-to-date based on events.

MongoDB Schema - The Blueprint:

We use Mongoose to define the structure of our video data in MongoDB. Let's look at the core VideoSchema:

// api-server/src/app/modules/video/video.model.ts
import mongoose, { model } from "mongoose";
import { VIDEO_STATUS, VIDEO_VISIBILITIES } from "./video.constant"; // Import status constants

const VideoSchema = new mongoose.Schema(
  {
    title: { type: String /* required: true */ },
    description: { type: String, default: "" },
    author: { type: mongoose.Schema.Types.ObjectId, required: true, ref: "User" }, // Link to User model
    duration: { type: String }, // e.g., "10:35"
    viewsCount: { type: Number, min: 0, default: 0 },
    // ... other fields like playlistId, language, category ...
    videoLink: { type: String }, // URL to the final HLS master playlist (e.g., ...master.m3u8)
    rawVideoLink: { type: String }, // Link to original uploaded file (optional)
    videoPath: { type: String }, // Temporary local path during processing (might not be needed long-term)
    fileName: { type: String }, // The unique filename used in storage/processing
    originalName: { type: String, required: true }, // User's original filename
    thumbnailUrl: { type: String }, // URL to the generated thumbnail
    history: { type: Array, default: [] }, // Array to track processing steps
    status: { // <-- Crucial for tracking
      type: String,
      enum: Object.values(VIDEO_STATUS),
      default: VIDEO_STATUS.PENDING, // Starts as pending
    },
    visibility: { type: String, enum: Object.values(VIDEO_VISIBILITIES), default: VIDEO_VISIBILITIES.PUBLIC },
    tags: { type: [String], default: [] },
    size: { type: Number }, // Original file size
    videoConversionTime: { type: String }, // How long conversion took
  },
  { timestamps: true } // Adds createdAt and updatedAt fields automatically
);

export const Video = model("Video", VideoSchema);

Key Fields Explained:

author: Links the video to the user who uploaded it.

status: Tracks the video's journey. We use constants for this:

// api-server/src/app/modules/video/video.constant.ts
export const VIDEO_STATUS = {
    PENDING: "Pending",
    PROCESSING: "Processing", // Overall processing started
    UPLOADING_TO_CLOUD: "Uploading to cloud", // (If tracking HLS upload separately)
    SUCCESS: "Success", // Ready for playback
    FAILED: "Failed", // Something went wrong
} as const; // Using "as const" for stricter typing

history: An array where we can log timestamps and stages (e.g., { status: 'HLS Conversion Complete', timestamp: Date.now() }). This is useful for debugging.
videoLink: The most important field for playback – the URL to the master HLS playlist (.m3u8) file in cloud storage.

API Server Logic - Responding to Events:

The API Server doesn't just sit there; it actively listens for messages from RabbitMQ, primarily sent by the API Gateway and Video Conversion service.

// api-server/src/app/events/video.events.ts (Conceptual Structure)
import RabbitMQ from '../../shared/rabbitMQ';
import { API_SERVER_EVENTS } from '../../constants/events'; // Consuming events meant for it
import { VideoService } from '../modules/video/video.service'; // The service handling DB operations

const setupVideoEventListeners = async () => {
  // Listen for the initial metadata insertion request from API Gateway
  await RabbitMQ.consume(API_SERVER_EVENTS.INSERT_VIDEO_METADATA_EVENT, async (msg, ack) => {
    try {
      const videoData = JSON.parse(msg.content.toString());
      await VideoService.createVideo(videoData); // Call service to save to DB
      ack();
    } catch (error) { /* Handle error, maybe Nack */ ack(); }
  });

  // Listen for updates from Video Conversion Service (e.g., duration, status)
  await RabbitMQ.consume(API_SERVER_EVENTS.UPDATE_METADATA_EVENT, async (msg, ack) => {
     try {
        const { id, ...updateData } = JSON.parse(msg.content.toString());
        // Could also update status to PROCESSING here
        await VideoService.updateVideo(id, updateData);
        ack();
     } catch (error) { /* ... */ ack(); }
  });

  // Listen for thumbnail generation completion
   await RabbitMQ.consume(API_SERVER_EVENTS.VIDEO_THUMBNAIL_GENERATED_EVENT, async (msg, ack) => {
     try {
        const { id, thumbnailUrl } = JSON.parse(msg.content.toString());
        await VideoService.updateVideo(id, { thumbnailUrl, /* Update history maybe */ });
        ack();
     } catch (error) { /* ... */ ack(); }
  });

   // Listen for final HLS conversion completion
   await RabbitMQ.consume(API_SERVER_EVENTS.VIDEO_HLS_CONVERTED_EVENT, async (msg, ack) => {
     try {
        const { id, videoLink } = JSON.parse(msg.content.toString());
        // Update status to SUCCESS and add the final HLS link!
        await VideoService.updateVideo(id, { status: 'Success', videoLink });
        ack();
     } catch (error) { /* ... */ ack(); }
  });


  // ... potentially listen for failure events ...
};

The VideoService (video.service.ts) contains the actual Mongoose queries (Video.create(), Video.findByIdAndUpdate(), etc.) to interact with the database.

2. Taking Flight: Deployment Strategies

Running docker-compose up on your machine is great for development, but how do you get this running reliably for others to use? Deployment!

Docker is Key:

We've already created Dockerfiles for each service. These are the blueprints for building container images – standardized packages containing our code and dependencies.

# Example: video-conversion/Dockerfile (Multi-stage build)
# Stage 1: Build the application
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN yarn install --frozen-lockfile
COPY . .
RUN yarn build # Compile TypeScript to JavaScript in /dist

# Stage 2: Create the final, smaller image
FROM node:18-alpine
WORKDIR /app
# Install FFmpeg directly in the final image!
RUN apk update && apk add --no-cache ffmpeg
COPY package*.json ./
# Only install production dependencies
RUN yarn install --production --frozen-lockfile
# Copy built code from the builder stage
COPY --from=builder /app/dist ./dist
# Copy entrypoint script if needed
COPY entrypoint.sh .
RUN chmod +x entrypoint.sh

ENV NODE_ENV=production
EXPOSE 8002 # Expose the port the service runs on
# Command to run when the container starts
CMD ["node", "dist/server.js"]
# Or use entrypoint script: ENTRYPOINT ["./entrypoint.sh"]

Now, let's look at how to run these containers in a production-like environment.

Option 1: Docker Compose (Single Server)

Concept: Run all containers (services, DB, Redis, RabbitMQ) on a single virtual machine using the docker-compose.yml file we used for development.
Pros: Simplest approach, good for small projects or internal tools.
Cons: Doesn't scale easily beyond one machine, single point of failure (if the machine goes down, everything goes down).

Option 2: Docker Swarm (Basic Orchestration)

Concept: Docker's built-in orchestrator. You can join multiple machines (nodes) into a "swarm" and deploy services across them.
How: Initialize a swarm (docker swarm init), join worker nodes (docker swarm join), and deploy your docker-compose.yml as a "stack" (docker stack deploy -c docker-compose.yml myapp). Swarm handles basic load balancing and service discovery.
Pros: Built into Docker, relatively easy transition from Compose, provides redundancy if a node fails (if services have replicas).
Cons: Less feature-rich and less widely adopted than Kubernetes.

Option 3: Kubernetes (K8s - Advanced Orchestration)

Concept: The industry standard for managing containerized applications at scale. It's powerful but more complex.
How: You define your application using Kubernetes objects (written in YAML):
- Deployment: Manages replicas (pods) of your application, handles rolling updates.
- Service: Provides a stable IP address and DNS name for accessing your deployments.
- Ingress: Manages external access (HTTP/S routing) to your services, often handling SSL termination.
- ConfigMap/Secret: Manage configuration and sensitive data separately from your container images.
- PersistentVolume/PersistentVolumeClaim: For stateful data like MongoDB.
- You likely have basic manifests in the k8s/ directories within each service. You'd apply these using kubectl apply -f <directory>.
Pros: Highly scalable, resilient (auto-healing, rescheduling), rolling updates, large community and ecosystem, standard across cloud providers (like AWS EKS, Google GKE, Azure AKS).
Cons: Steeper learning curve, more complex setup and management.

Choosing Your Path:

Starting out / Small project? Docker Compose on a single VM is fine.
Need basic scaling and redundancy? Docker Swarm might be sufficient.
Building for large scale, high availability, and adopting cloud-native practices? Kubernetes is the way to go, despite the complexity.

3. Handling Hiccups: Error Handling & Monitoring

Things will go wrong in production. Being prepared is key.

Error Handling:

Catching Async Errors: We use utilities like catchAsync in controllers to avoid repetitive try...catch blocks around async route handlers.

// api-gateway/src/shared/catchAsyncError.ts
import { Request, Response, NextFunction, RequestHandler } from 'express';
const catchAsync = (fn: RequestHandler) => {
  return (req: Request, res: Response, next: NextFunction) => {
    Promise.resolve(fn(req, res, next)).catch((err) => next(err));
  };
};
export default catchAsync;

Global Error Handler: Each service's app.ts should have a final middleware that catches any unhandled errors passed via next(err). This handler standardizes error responses.
Specific Errors: try...catch blocks are still essential for critical operations like file system access, database calls, or FFmpeg execution (hlsConvertProcessor.ts has .on('error', ...)).
Queue Failures: BullMQ workers have .on('failed', ...) listeners (jobWorker.ts). Log these errors! You might need strategies to automatically retry certain failures or flag them for manual review.

Monitoring:

You can't fix what you can't see!

Logging: We're using Winston (shared/logger.ts) for structured logging (JSON format is best for machines to parse). winston-daily-rotate-file helps manage log file size. Ensure you log meaningful information, including request IDs to trace requests across services.
Application Performance Monitoring (APM): Tools like Sentry (which you have integrated in video-conversion/src/app.ts) are invaluable. They automatically capture unhandled exceptions, performance bottlenecks, and provide rich context (stack traces, request data) to help you debug faster.
Infrastructure Metrics: Monitor the health of your containers and nodes (CPU, memory, disk I/O, network). Cloud providers offer built-in monitoring, or you can use tools like Prometheus and Grafana.
Queue/Broker Monitoring: Keep an eye on RabbitMQ (the management UI on port 15672 is helpful) and Redis/BullMQ to spot backed-up queues or high error rates.

4. Supercharging Your Server: Optimizations & Scaling

As your platform grows, you'll need to optimize.

Video Conversion Service:
- Scale Horizontally: This service is CPU-bound due to FFmpeg. Run multiple instances (replicas/pods) behind a load balancer (handled by Swarm/K8s Services). BullMQ naturally distributes jobs across available workers.
- Hardware: Consider using VMs/nodes optimized for CPU performance. GPU acceleration for FFmpeg is possible but adds complexity.
- Concurrency: Fine-tune the BullMQ worker concurrency (jobWorker.ts) based on your server resources. Too high can overload the server; too low underutilizes it.
Database (MongoDB):
- Indexing: Ensure you have indexes on fields used frequently in queries (e.g., author, status, createdAt). Use MongoDB's explain() to analyze query performance.
- Read Replicas: If you have many users browsing videos (read-heavy), set up MongoDB read replicas to distribute the load.
API Services (Gateway/Server):
- Statelessness: These are generally stateless (don't store session data in memory), making them easy to scale horizontally by adding more replicas.
- Caching: Use Redis (which you already have for BullMQ!) to cache frequently accessed, non-critical data – user sessions, results of expensive queries, maybe even pre-signed URL results briefly.
Content Delivery Network (CDN):
- Crucial for Playback: Put a CDN (like AWS CloudFront, Cloudflare, Akamai) in front of your cloud storage bucket (DO Spaces/S3).
- Benefits: The CDN caches the HLS playlist (.m3u8) and video segment (.ts) files in edge locations closer to your users worldwide. This dramatically improves video start times, reduces buffering, and lowers your cloud storage egress costs.

5. Conclusion: Where to Go From Here?

And that's a wrap on our three-part journey! We've gone from a high-level architectural idea (Part 1), through the nitty-gritty code implementation (Part 2), to managing data, deploying, and ensuring robustness (Part 3).

You now have a solid foundation for building a scalable video streaming server. The specific implementation details might evolve, but the core concepts – microservices, direct uploads, asynchronous processing queues, HLS conversion, and monitoring – are fundamental.

Next Steps:

Explore the Code: Clone the repository (if available) and experiment.
Deploy It: Try deploying using Docker Compose or tackle Kubernetes.
Contribute/Improve: Identify areas for enhancement.

Future Features to Consider:

Live Streaming (Requires different FFmpeg setup and protocols like RTMP/WebRTC)
Video Playlists
User Comments & Likes
Detailed Analytics
Digital Rights Management (DRM) for content protection
Admin Interface

Building complex systems is challenging but incredibly rewarding. Hopefully, this series has demystified the process and equipped you to build your own amazing video platform. Happy coding!

Explore the full implementation and grab the code from the GitHub repository:

Github-Server: https://github.com/AwalHossain/video-streaming-server
Github-Client: https://github.com/awalHossain/video-streaming-client

Do follow me on my Socials to stay updated on tech stuff:

💼 LinkedIn: [https://www.linkedin.com/in/awal-hossain-0bb650108/]
🐙 Github: [https://github.com/awalHossain/]

Building Your Own Scalable Video Streaming Server: Part 2 - Setup & Core Implementation

Awal Hossain — Sun, 13 Apr 2025 10:43:29 +0000

If you missed Part 1, you can find the link here: Part-1

Now, it's time to roll up our sleeves and get our hands dirty with the how. In this part, we'll focus on:

Setting up your development environment.
Implementing the direct-to-cloud upload flow using pre-signed URLs.
Wiring up the video conversion pipeline using FFmpeg, BullMQ, and RabbitMQ.

1. Setting Up Shop: Your Development Environment

First things first, let's make sure you have the necessary tools installed.

Prerequisites:

Node.js: v18.0.0 or later (check with node -v).
Yarn: (Or npm) for package management (check with yarn -v).
Docker & Docker Compose: To easily run our services and dependencies like Redis and RabbitMQ. Install Docker.
FFmpeg: The video processing powerhouse. If you missed it in Part 1, installation guides are often found online (e.g., brew install ffmpeg on Mac, or check the official FFmpeg site).
Code Editor: VS Code, WebStorm, Sublime Text - your choice!
Git: For cloning the project repository.

Get the Code:

git clone https://github.com/AwalHossain/video-streaming-server
cd video-streaming-server

Environment Variables (.env): The Secret Sauce

Action: For each service directory:

Copy .env.example to a new file named .env.
Open the .env file and fill in the values specific to your setup.

Here are some key variables you'll need to configure:

PORT: The port each service will run on (e.g., 8000, 8001, 8002).
MONGO_URI: Your MongoDB connection string (for api-server).
RABBITMQ_ENDPOINT: Your RabbitMQ server address (e.g., amqp://user:password@localhost:5672).
REDIS_URL: Your Redis connection URL (for video-conversion, e.g., redis://localhost:6379).
JWT_SECRET: A long, random secret string for signing authentication tokens (used by api-gateway and api-server).
Cloud Storage Credentials (e.g., for Digital Ocean Spaces or AWS S3):
- DO_SPACES_ACCESS_KEY / AWS_ACCESS_KEY_ID
- DO_SPACES_SECRET_KEY / AWS_SECRET_ACCESS_KEY
- DO_SPACES_ENDPOINT / AWS_ENDPOINT (if using non-AWS S3 compatible)
- DO_SPACES_BUCKET_NAME / AWS_BUCKET_NAME
- DO_SPACES_REGION / AWS_REGION
CLIENT_URL1, CLIENT_URL2: The URLs of your frontend application(s) for CORS configuration.

2. The Grand Upload: Pre-signed URLs in Action

Remember how we wanted to avoid bogging down our servers with large file uploads? Pre-signed URLs are the answer! The client gets a temporary, secure link to upload directly to cloud storage.

API Gateway - The Gatekeeper:

Routing: We need an endpoint to request the pre-signed URL.

// api-gateway/src/app/routes/index.ts
import { Router } from 'express';
import { AuthRoutes } from '../modules/auth/auth.routes';
import { VideoRoutes } from '../modules/video/video.route';

const router = Router();
const ModuleRoutes = [
  { path: '/auth', route: AuthRoutes },
  { path: '/videos', route: VideoRoutes }, // Our video routes
];
ModuleRoutes.forEach((route) => router.use(route.path, route.route));
export default router;

// api-gateway/src/app/modules/video/video.route.ts
import express from 'express';
import auth from '../../middleware/auth'; // <-- Authentication middleware
import { VideoController } from './video.controller';

const router = express.Router();
// POST endpoint protected by auth middleware
router.post(
    '/presigned-url',
     auth(), // Ensures only logged-in users can get a URL
     VideoController.getPresignedUrl
    );
router.post(
    '/confirm-upload',
     auth(),
     VideoController.confirmUpload
     );
// ... other video routes
export const VideoRoutes = router;

Controller (getPresignedUrl): This function handles the request.

// api-gateway/src/app/modules/video/video.controller.ts
import { Request, Response } from 'express';
import { v4 as uuidv4 } from 'uuid';
import catchAsync from '../../../shared/catchAsyncError';
import sendResponse from '../../../shared/response';
import doSpacesUpload from '../../../utils/doSpacesUpload'; // The utility function

const getPresignedUrl = catchAsync(async (req: Request, res: Response) => {
  const { filename, contentType } = req.body; // Get info from client

  if (!filename || !contentType) {
    return res.status(400).json({ /* ... error */ });
  }

  const userId = req.user.id; // From auth middleware

  // Clean filename and make it unique
  const cleanFilename = filename
    .split('.')[0]
    .replace(/\s+/g, '-')
    // ... more cleaning ...
    + '-' + uuidv4() + '.' + filename.split('.').pop();

  // Call the utility to generate the URL
  const result = await doSpacesUpload.generatePresignedUrl({
    filename: cleanFilename,
    contentType: contentType,
    userId: userId,
    expirySeconds: 3600, // URL valid for 1 hour
  });

  sendResponse(res, {
    statusCode: 200,
    success: true,
    message: 'Presigned URL generated successfully',
    data: result, // Send URL and fileKey back to client
  });
});

Utility (doSpacesUpload.ts): This interacts with the cloud storage SDK.

// api-gateway/src/utils/doSpacesUpload.ts
import AWS from 'aws-sdk'; // Using AWS SDK for S3 compatible storage
import config from '../config';

const s3 = new AWS.S3({ // Configure S3 client
  endpoint: config.doSpaces.endpoint,
  accessKeyId: config.doSpaces.accessKey,
  secretAccessKey: config.doSpaces.secretKey,
  region: config.doSpaces.region,
  s3ForcePathStyle: true,
  signatureVersion: 'v4'
});

const generatePresignedUrl = async (options) => {
  const { filename, contentType, userId, expirySeconds = 3600 } = options;
  const timestamp = Date.now();
  // Define the path/key in the bucket
  const key = `uploads/${userId}/${timestamp}-${filename}`;

  const params = {
    Bucket: config.doSpaces.bucketName,
    Key: key,
    Expires: expirySeconds,
    // ACL: 'public-read', // Optional: If you want uploaded files to be public
    // ContentType: contentType // Often needed for direct browser uploads
  };

  try {
    // Generate the signed URL for PUT operation
    const url = await s3.getSignedUrlPromise('putObject', params);
    const fileUrl = `https://${config.doSpaces.bucketName}.${config.doSpaces.endpoint}/${key}`;

    return {
      uploadUrl: url, // The URL the client will use to PUT the file
      fileKey: key,   // The path/key of the file in the bucket
      bucketName: config.doSpaces.bucketName,
      fileUrl, // Direct link to the file (if public)
      fileName: key.split('/').pop(),
    };
  } catch (error) {
    console.error('Error generating presigned URL:', error);
    throw error;
  }
};

export default { generatePresignedUrl };

Client-Side Upload (Conceptual):

Once the client receives the uploadUrl, fileKey, etc., they perform the upload.

// Simple example using fetch
async function uploadFile(presignedData, file) {
  const { uploadUrl, contentType } = presignedData; // Assume contentType was passed back or known

  try {
    const response = await fetch(uploadUrl, {
      method: 'PUT',
      headers: {
        'Content-Type': file.type, // Use the actual file type
        // 'x-amz-acl': 'public-read' // If ACL was set during URL generation
      },
      body: file
    });

    if (!response.ok) {
      throw new Error(`Upload failed: ${response.statusText}`);
    }
    console.log('Upload successful!');
    // Now call the confirmUpload endpoint
  } catch (error) {
    console.error('Upload error:', error);
  }
}

Confirming the Upload:

After the direct upload finishes, the client must notify our backend.

// api-gateway/src/app/modules/video/video.controller.ts (confirmUpload)
const confirmUpload = catchAsync(async (req: Request, res: Response) => {
  // Client sends back fileKey, originalName, etc. received earlier
  const { fileKey, originalName, fileName, bucketName } = req.body;
  const userId = req.user.id;

  // 1. Notify user upload is complete (via RabbitMQ -> WebSocket)
  RabbitMQ.sendToQueue(NOTIFY_EVENTS.NOTIFT_VIDEO_UPLOADING_BUCKET, {
     userId, status: 'completed', /* ... */
  });

  // 2. Prepare metadata payload for API Server
  const metadataPayload = {
    originalName,
    recordingDate: Date.now(),
    author: userId,
    fileName,
    title: originalName.split('.')[0].replace(/[_]/g, ''), // Basic title generation
    // ... other initial metadata
  };

  // 3. Send event to API Server to save metadata
  broadcastVideoEvent(API_SERVER_EVENTS.INSERT_VIDEO_METADATA_EVENT, metadataPayload);

  // 4. Prepare data for Video Conversion Service
  const conversionData = { userId, fileName, bucketName, fileKey };

  // 5. Send event to Video Conversion Service to start processing
  broadcastVideoEvent(
    VIDEO_CONVERSION_SERVER.SEND_VIDEO_METADATA_EVENT,
    conversionData
  );

  sendResponse(res, { /* ... success response */ });
});

Phew! That's the upload flow. The key is the client uploads directly to storage, and then notifies our backend to kick off the next steps via RabbitMQ events.

3. Conversion Central: FFmpeg, BullMQ, and RabbitMQ

Now, the Video Conversion service takes over.

Receiving the Signal:

The service listens for the SEND_VIDEO_METADATA_EVENT on RabbitMQ. When it receives a message, it triggers the download process.

// video-conversion/src/server.ts (Conceptual listener setup)
import RabbitMQ from './shared/rabbitMQ';
import downloadBlob from './processor/downloadFile';
import { VIDEO_CONVERSION_SERVER } from './constant/events';

async function startServer() {
 // ... other setup ...

 await RabbitMQ.connect();

 // Listen for the event from API Gateway
 await RabbitMQ.consume(
   VIDEO_CONVERSION_SERVER.SEND_VIDEO_METADATA_EVENT,
   async (msg, ack) => {
     try {
       const data = JSON.parse(msg.content.toString());
       logger.info('Received video metadata event:', data);
       // Trigger the download process
       await downloadBlob(data.bucketName, data.fileKey, data.userId);
       ack(); // Acknowledge the message
     } catch (error) {
       errorLogger.error('Failed processing metadata event:', error);
       // Decide if you should ack or nack based on the error
       ack(); // Ack for now to prevent reprocessing loop on simple errors
     }
   }
 );

 // ... start express app ...
}

Downloading the File (downloadFile.ts):

This function grabs the file from cloud storage.

// video-conversion/src/processor/downloadFile.ts
import fs from 'fs';
import path from 'path';
import { v4 as uuidv4 } from 'uuid';
import s3 from '../shared/s3Client'; // Configured S3 client
import initiateVideoProcessing from './initiateVideoProcessing'; // Next step
import RabbitMQ from '../shared/rabbitMQ';
import { API_GATEWAY_EVENTS } from '../constant/events';

async function downloadBlob(bucketName, fileKey, userId) {
  try {
    const fileName = path.basename(fileKey);
    const originalName = fileName.substring(fileName.indexOf('-') + 1); // Extract original name

    // Notify download start
    RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_VIDEO_DOWNLOADING, { /* ... */ });

    // Create a unique local directory for processing
    const uploadFolder = `container-${uuidv4()}`;
    const destination = path.normalize(path.join('uploads', uploadFolder, 'videos')).replace(/\\\\/g, '/');
    fs.mkdirSync(destination, { recursive: true });
    const videoPath = path.normalize(path.join(destination, fileName)).replace(/\\\\/g, '/');

    logger.info(`Downloading file from DO Spaces to ${videoPath}`);

    // Perform the download using the S3 SDK stream
    await new Promise<void>((resolve, reject) => {
      const writeStream = fs.createWriteStream(videoPath);
      s3.getObject({ Bucket: bucketName, Key: fileKey })
        .createReadStream()
        .pipe(writeStream)
        .on('error', reject)
        .on('finish', resolve);
    });

    // Notify download complete
    RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_VIDEO_DOWNLOADED, { /* ... */ }); // Typo in original? Should be NOTIFY_VIDEO_DOWNLOADED

    // --- Trigger the next step: Add to processing queue ---
    // (Note: Fetching metadata via API_SERVER_EVENTS.GET_VIDEO_METADATA_EVENT
    // might happen here or be passed in the initial event)
    const videoMetadata = { _id: 'some-id-from-db', /* ... other fields */ }; // Placeholder

    await initiateVideoProcessing({ videoPath, destination, userId, videoMetadata });

  } catch (error) {
    errorLogger.error('Error in downloadBlob:', error);
    RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_EVENTS_FAILED, { /* ... */ });
    throw error;
  }
}

Kicking Off the Queue (initiateVideoProcessing.ts):

After downloading, we add the first job to our BullMQ processing queue.

// video-conversion/src/processor/initiateVideoProcessing.ts
import { QUEUE_EVENTS, API_SERVER_EVENTS } from '../constant/events';
import { addQueueItem } from '../queues/addJobToQueue'; // BullMQ helper
import { getVideoDurationAndResolution } from './videoProcessingHandler';
import RabbitMQ from '../shared/rabbitMQ';

const initiateVideoProcessing = async ({ videoPath, destination, userId, videoMetadata }) => {
  // Get basic video info like duration
  const { videoDuration } = await getVideoDurationAndResolution(videoPath);

  const payload = {
    ...videoMetadata,
    duration: videoDuration,
    id: videoMetadata._id, // Crucial: DB id for tracking
    path: videoPath,       // Local path after download
    destination,           // Local processing directory
    userId,
  };

  // Optional: Update DB immediately with duration/local path
  RabbitMQ.sendToQueue(API_SERVER_EVENTS.UPDATE_METADATA_EVENT, {
    id: videoMetadata._id,
    videoPath, // May not be needed in DB long-term
    duration: videoDuration,
  });

  // Add the first job to the queue!
  await addQueueItem(QUEUE_EVENTS.VIDEO_UPLOADED, payload); // Start the chain
};

export default initiateVideoProcessing;

The BullMQ Workforce:

BullMQ manages our jobs.

Queue Setup (addJobToQueue.ts): Defines the queues based on event names.

// video-conversion/src/queues/addJobToQueue.ts
import { Queue } from 'bullmq';
import { ALL_EVENTS as QUEUE_EVENTS } from '../constant/events';
import { RedisClient } from '../shared/redis'; // Redis connection

export const queues = Object.values(QUEUE_EVENTS).map((queueName) => {
  return {
    name: queueName,
    queueObj: new Queue(queueName, { connection: RedisClient.redisConnection }),
  };
});

export const addQueueItem = async (queueName, item) => {
  const queue = queues.find((q) => q.name === queueName);
  if (!queue) throw new Error(`Queue ${queueName} not found`);
  // Add job to the queue
  await queue.queueObj.add(queueName, item, {
    removeOnComplete: true, // Clean up successful jobs
    removeOnFail: false,    // Keep failed jobs for inspection
  });
};

Worker Setup (jobWorker.ts): Listens for jobs and calls the right handler.

// video-conversion/src/worker/jobWorker.ts
import { Worker } from 'bullmq';
import { QUEUE_EVENTS } from '../constant/events';
import { QUEUE_EVENT_HANDLERS } from '../processor/videoEventHandler'; // Maps event name to function
import { RedisClient } from '../shared/redis';
import { logger } from '../shared/logger';

export const listenQueueEvent = (queueName) => {
  const worker = new Worker(
    queueName,
    async (job) => { // Job processor function
      const handler = QUEUE_EVENT_HANDLERS[queueName]; // Find the right handler
      if (handler) {
        logger.info(`Processing job ${job.id} from queue ${queueName}`);
        return await handler(job); // Execute the handler
      }
      throw new Error('No handler found for queue: ' + queueName);
    },
    { connection: RedisClient.redisConnection, concurrency: 3 } // Process up to 3 jobs concurrently
  );

  worker.on('failed', (job, err) => {
    logger.error(`Job ${job?.id} failed in queue ${queueName}: ${err.message}`, err);
  });

  logger.info(`${queueName} worker started.`);
};

export const setupAllQueueEvent = () => { // Called at startup
  Object.values(QUEUE_EVENTS).forEach((queueName) => {
    listenQueueEvent(queueName);
  });
  // videoLifecycleHandler(); // Setup lifecycle listeners (e.g., for uploads)
  return true;
};

The Processing Pipeline (videoEventHandler.ts):

This file contains the functions (handlers) that execute for each job type. They often add the next job in the sequence upon completion.

// video-conversion/src/processor/videoEventHandler.ts
import { Job } from 'bullmq';
import { QUEUE_EVENTS } from '../constant/events';
import { addQueueItem } from '../queues/addJobToQueue';
import processMp4ToHls from './hlsConvertProcessor';
import { generateThumbnail, processRawFileToMp4WithWatermark } from './mp4ConvertProcessor';
// ... other imports like fs, path, logger

const uploadedHandler = async (job: Job) => {
  logger.info('Handler: VIDEO_UPLOADED', job.data.id);
  // Simply trigger the next step
  await addQueueItem(QUEUE_EVENTS.VIDEO_PROCESSING, job.data);
};

const processingHandler = async (job: Job) => { // Convert to MP4 if needed
  logger.info('Handler: VIDEO_PROCESSING', job.data.id);
  const filePath = job.data.path; // Local path of downloaded file
  const uploadPath = `${job.data.destination}/processed`; // Output dir for MP4
  const thumbnailPath = `${job.data.destination}/thumbnails`; // Output dir for thumbs
  fs.mkdirSync(uploadPath, { recursive: true });
  fs.mkdirSync(thumbnailPath, { recursive: true });

  // Generate thumbnail (can happen in parallel or sequence)
  await generateThumbnail(filePath, thumbnailPath, job.data); // Sends RabbitMQ event on completion

  if (needsConversion(filePath)) { // Check if it's already mp4/webm etc.
     logger.info(`Converting ${filePath} to MP4...`);
     // Calls ffmpeg internally, adds VIDEO_PROCESSED job on completion
     await processRawFileToMp4WithWatermark(filePath, uploadPath, {
       ...job.data, next: QUEUE_EVENTS.VIDEO_PROCESSED
     });
  } else {
     logger.info(`Skipping MP4 conversion for ${filePath}.`);
     // File is okay, move to next step directly
     const destPath = `${uploadPath}/${path.basename(filePath)}`; // Define destination
     fs.copyFileSync(filePath, destPath); // Just copy it
     // Add the next job manually since we skipped the conversion function
     await addQueueItem(QUEUE_EVENTS.VIDEO_PROCESSED, { ...job.data, path: destPath });
  }
};

const processedHandler = async (job: Job) => { // MP4 ready, start HLS
   logger.info('Handler: VIDEO_PROCESSED', job.data.id);
   await addQueueItem(QUEUE_EVENTS.VIDEO_HLS_CONVERTING, job.data);
};

const hlsConvertingHandler = async (job: Job) => { // The core HLS conversion
  logger.info('Handler: VIDEO_HLS_CONVERTING', job.data.id);
  const hlsUploadPath = `${job.data.destination}/hls`; // Output dir for HLS files
  fs.mkdirSync(hlsUploadPath, { recursive: true });

  // Calls the main ffmpeg HLS logic, adds VIDEO_HLS_CONVERTED job on completion
  await processMp4ToHls(job.data.path, hlsUploadPath, { // job.data.path is now the MP4 path
      ...job.data, next: QUEUE_EVENTS.VIDEO_HLS_CONVERTED
  });
};

const hlsConvertedHandler = async (job: Job) => { // HLS done, trigger upload
    logger.info('Handler: VIDEO_HLS_CONVERTED', job.data.id);
    // This handler might trigger the upload of the HLS files to cloud storage
    // via processHLSFile called from videoLifecycleHandler.ts
    // It should also send a final notification event.
};

export const QUEUE_EVENT_HANDLERS = { // Mapping used by the worker
  [QUEUE_EVENTS.VIDEO_UPLOADED]: uploadedHandler,
  [QUEUE_EVENTS.VIDEO_PROCESSING]: processingHandler,
  [QUEUE_EVENTS.VIDEO_PROCESSED]: processedHandler,
  [QUEUE_EVENTS.VIDEO_HLS_CONVERTING]: hlsConvertingHandler,
  [QUEUE_EVENTS.VIDEO_HLS_CONVERTED]: hlsConvertedHandler,
};

FFmpeg Magic (hlsConvertProcessor.ts):

This is where the actual HLS conversion command is built and executed.

// video-conversion/src/processor/hlsConvertProcessor.ts
import ffmpegPath from 'ffmpeg-static';
import ffmpeg from 'fluent-ffmpeg';
import path from 'path';
import RabbitMQ from '../shared/rabbitMQ';
import { API_GATEWAY_EVENTS } from '../constant/events';
// ... other imports

ffmpeg.setFfmpegPath(ffmpegPath); // Tell fluent-ffmpeg where ffmpeg is

const processMp4ToHls = async (filePath, outputFolder, jobData) => {
  const fileNameWithoutExt = path.basename(filePath, path.extname(filePath));

  // --- Define HLS Renditions (Quality Levels) ---
  // (Includes logic for vertical vs. landscape videos from your code)
  const renditions = [
      { resolution: '854x480', bitrate: '800k', name: '480p' },
      { resolution: '1920x1080', bitrate: '5000k', name: '1080p' },
      // Add more if needed
  ];

  const renditionProgress = {}; // Track progress per rendition
  renditions.forEach(r => renditionProgress[r.name] = 0);
  let lastReportedProgress = 0;

  const promises = renditions.map((rendition) => {
    return new Promise<void>((resolve, reject) => {
      const outputFileName = `${fileNameWithoutExt}_${rendition.name}.m3u8`;
      const segmentFileName = `${outputFolder}/${fileNameWithoutExt}_${rendition.name}_%03d.ts`;

      ffmpeg(filePath) // Input MP4 file
        .output(path.join(outputFolder, outputFileName)) // Output M3U8 playlist for this rendition
        .outputOptions([
          `-s ${rendition.resolution}`,    // Set resolution
          `-c:v libx264`,                 // Video codec
          `-crf 23`,                      // Quality (lower is better)
          `-preset fast`,                 // Encoding speed vs compression
          `-b:v ${rendition.bitrate}`,    // Target video bitrate
          `-g 48`,                        // Keyframe interval
          `-hls_time 10`,                 // Segment duration (seconds)
          `-hls_playlist_type vod`,       // Video on Demand playlist
          `-hls_flags independent_segments`,
          `-hls_segment_filename ${segmentFileName}`, // Naming pattern for TS files
        ])
        .on('start', (cmd) => logger.info(`FFmpeg started for ${rendition.name}: ${cmd}`))
        .on('progress', (progress) => { // --- Progress Reporting ---
          renditionProgress[rendition.name] = progress.percent || 0;
          const totalProgress = Object.values(renditionProgress).reduce((a, b) => a + b, 0);
          const overallProgress = Math.round(totalProgress / renditions.length);

          // Report progress in ~10% increments
          if (overallProgress > 0 && overallProgress - lastReportedProgress >= 10) {
            lastReportedProgress = overallProgress;
            RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_EVENTS_VIDEO_BIT_RATE_PROCESSING, {
                userId: jobData.userId,
                status: 'processing',
                progress: overallProgress,
                fileName: fileNameWithoutExt,
                /* ... */
            });
          }
        })
        .on('end', () => {
          logger.info(`FFmpeg finished for ${rendition.name}`);
          resolve();
        })
        .on('error', (err) => {
          errorLogger.error(`FFmpeg error for ${rendition.name}:`, err);
          RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_EVENTS_FAILED, { /* ... failure data */ });
          reject(err);
        })
        .run();
    });
  });

  await Promise.all(promises); // Wait for all renditions

  // --- Create Master Playlist ---
  // (Code to generate the master manifest referencing all rendition playlists)
  // fs.writeFileSync(path.join(outputFolder, `${fileNameWithoutExt}_master.m3u8`), masterPlaylistContent);

  logger.info('HLS conversion completed for:', fileNameWithoutExt);

  // Send final success notification
  RabbitMQ.sendToQueue(API_GATEWAY_EVENTS.NOTIFY_EVENTS_VIDEO_BIT_RATE_PROCESSED, {
       userId: jobData.userId, status: 'completed', /* ... */
  });

  // Add the next job (e.g., upload HLS files) or mark as complete
  // This is handled by videoLifecycleHandler in your code
  // which listens for QUEUE_EVENTS.VIDEO_HLS_CONVERTED and calls processHLSFile
};

Uploading Processed Files:

4. Tying it Together: RabbitMQ & Real-time Updates

We saw RabbitMQ used everywhere!

API Gateway -> API Server (Save Metadata)
API Gateway -> Video Conversion (Start Processing)
Video Conversion -> API Gateway (Progress/Status Updates)
Video Conversion -> API Server (Update Metadata like duration, status)

The key for real-time updates is:

Video Conversion sends progress events (e.g., NOTIFY_EVENTS_VIDEO_BIT_RATE_PROCESSING) to RabbitMQ, including the userId.
API Gateway consumes these events from RabbitMQ.
Using Socket.io, the API Gateway finds the WebSocket connection associated with that userId and emits the progress data directly to the connected client's browser.

// api-gateway/src/socket/index.ts (Conceptual Example)
import { Server as SocketIOServer, Socket } from 'socket.io';
import http from 'http';
import RabbitMQ from '../shared/rabbitMQ';
import { API_GATEWAY_EVENTS } from '../constants/event'; // Or NOTIFY_EVENTS

let io: SocketIOServer;
const users: Map<string, string> = new Map(); // Map userId to socketId

export const initSocketIO = (server: http.Server) => {
  io = new SocketIOServer(server, { cors: { /* ... */ } });

  io.on('connection', (socket: Socket) => {
    console.log('A user connected:', socket.id);

    // Store user association (needs authentication, e.g., JWT verification on connect)
    const userId = getUserIdFromSocket(socket); // Implement this securely
    if (userId) {
        users.set(userId, socket.id);
        console.log(`User ${userId} mapped to socket ${socket.id}`);
    }


    socket.on('disconnect', () => {
      console.log('User disconnected:', socket.id);
      // Clean up user mapping
      users.forEach((socketId, uId) => {
          if (socketId === socket.id) {
              users.delete(uId);
          }
      });
    });
  });

  // Listen for RabbitMQ progress events
  listenForProgressUpdates();

  return io;
};

const listenForProgressUpdates = async () => {
    await RabbitMQ.consume(
        API_GATEWAY_EVENTS.NOTIFY_EVENTS_VIDEO_BIT_RATE_PROCESSING, // Example event
        (msg, ack) => {
            try {
                const data = JSON.parse(msg.content.toString());
                const userId = data.userId;
                const socketId = users.get(userId); // Find the user's socket

                if (socketId && io) {
                     console.log(`Sending progress to user ${userId} (socket ${socketId})`);
                     // Emit event directly to the specific user's socket
                     io.to(socketId).emit('video_progress', data);
                } else {
                    console.log(`Socket not found for user ${userId}, unable to send progress.`);
                }
                ack();
            } catch (error) {
                console.error("Error processing progress update:", error);
                ack(); // Ack to avoid requeue loops
            }
        }
    );
    // ... listen for other notification events ...
};

// Helper to securely get userId from socket handshake/query
function getUserIdFromSocket(socket: Socket): string | null {
    // IMPORTANT: Implement proper authentication here!
    // E.g., verify JWT passed in handshake query or auth object
    return socket.handshake.query.userId as string || null;
}

export { io }; // Export for potential use elsewhere

Docker Installation

The easiest way to run all services and their dependencies (MongoDB, RabbitMQ, Redis) is using Docker Compose. We have docker-compose.yml files defined. Let's look at the one in the root directory:

# docker-compose.yml (Root Directory - Simplified Example)
version: "3.7"
services:
  api-gateway:
    build: ./api-gateway
    ports:
      - "8000:8000" # Example port
    env_file:
      - ./api-gateway/.env
    depends_on:
      - rabbitmq
    volumes:
      - ./api-gateway:/app
      - /app/node_modules

  api-server:
    build: ./api-server
    ports:
      - "8001:8001" # Example port
    env_file:
      - ./api-server/.env
    depends_on:
      - mongodb
      - rabbitmq
    volumes:
      - ./api-server:/app
      - /app/node_modules

  video-conversion:
    build: ./video-conversion
    ports:
      - "8002:8002" # Example port
    env_file:
      - ./video-conversion/.env
    depends_on:
      - redis
      - rabbitmq
    volumes:
      - ./video-conversion:/app
      - ./video-conversion/uploads:/app/uploads # Mount uploads volume
      - /app/node_modules

  mongodb:
    image: mongo:latest
    ports:
      - "27017:27017"
    volumes:
      - mongo-data:/data/db

  rabbitmq:
    image: rabbitmq:3-management
    ports:
      - "5672:5672"  # For AMQP
      - "15672:15672" # For Management UI

  redis:
    image: redis:latest
    ports:
      - "6379:6379"

volumes:
  mongo-data:

To start everything:

docker-compose up -d --build

This command builds the images for our services (if they don't exist) and starts all containers in the background.

5. Conclusion & What's Next

Coming up in Part 3:

Diving deeper into the API Server and MongoDB schema.
Exploring deployment strategies using Docker Swarm or Kubernetes.
Discussing error handling and monitoring.
Potential optimizations and scaling considerations.

Part-3: Check out

Explore the full implementation code from the GitHub repository:

Server-Code
Server-Client

Do follow me on my Socials to stay updated on tech stuff:

💼 LinkedIn
🐙 Github

Building Your Own Scalable Video Streaming Server: Part 1 - The Big Picture

Awal Hossain — Sun, 13 Apr 2025 10:36:31 +0000

Ever wondered how platforms like YouTube or Netflix deliver video so smoothly, adjusting quality on the fly based on your internet speed? Or maybe you're building an app and your users are complaining about slow video loading times? 🤔

If you've faced these challenges or just want to build the next big video platform, you're in the right place! In this series, we'll break down how to build a robust video streaming server from scratch using a modern microservices approach.

We'll tackle common problems like:

Handling various video formats (MKV, FLV, etc.)
Delivering smooth playback even on slower connections
Processing multiple videos without bringing the system to its knees
Keeping users informed about their video's progress

By the end of this series, you'll have a blueprint for a system that converts videos into a format called HLS (HTTP Live Streaming).

What's HLS and Why Should You Care?

HLS is a technology developed by Apple that breaks down videos into small chunks. Instead of downloading one huge file, your video player requests these small chunks one after another.

The magic? HLS creates multiple versions of these chunks at different quality levels (like 480p, 1080p). The player can then automatically switch between these quality levels based on your internet speed, preventing buffering and giving you the best possible viewing experience. ✨

Our system will automate this conversion process!

Our Architecture: The 3 Musketeers

Instead of building one massive application (a monolith), we're using a microservices architecture. Think of it like having specialized teams:

The Bouncer (API Gateway): This service is the front door. It handles incoming requests, checks user permissions (authentication), figures out where the request needs to go, and even helps users upload files directly to cloud storage safely using something called pre-signed URLs. It also keeps users updated in real-time using WebSockets.
The Librarian (API Server): This service manages all the data. It keeps track of users, video details (like title, description, status), and stores this information neatly in a MongoDB database.
The Video Wizard (Video Conversion Service): This is where the heavy lifting happens. It grabs the uploaded video, converts it to the standard MP4 format if needed, and then works its magic with FFmpeg to create those HLS chunks in different qualities. It uses a job queue (BullMQ/Redis) to handle multiple videos efficiently.

Here's a simplified view of how they talk to each other:

How Does a Video Get Processed?

Let's follow a video's journey:

Upload Time: You (the user) tell the API Gateway, "I want to upload a video!"
Get Your Ticket: The API Gateway talks to the cloud storage (like Digital Ocean Spaces or AWS S3) and gets a special, temporary "ticket" (a pre-signed URL).
Direct Flight: You upload your video directly to the cloud storage using this ticket. This is faster and keeps the load off our servers.
"I'm Done!": You tell the API Gateway the upload is complete.
Tell Everyone: The API Gateway tells the API Server to save the video details and tells the Video Conversion service, "Hey, new video ready for processing!"
Conversion Magic: The Video Conversion service downloads the video, converts it to MP4 (if needed), and then chops it up into HLS format (multiple qualities).
Back to Storage: The HLS files are uploaded back to cloud storage.
Status Updates: Throughout the process, the Video Conversion service sends updates (like "Processing started," "HLS converting," "Done!") via RabbitMQ (our messenger). The API Gateway picks these up and sends them to you in real-time via WebSockets!

The Tech Toolkit 🛠️

Here are the main tools we're using:

Node.js & Express: For building our backend services.
TypeScript: For safer, more maintainable code.
MongoDB & Mongoose: Our database for storing video info.
FFmpeg: The powerhouse for video conversion.
RabbitMQ: Our trusty messenger for communication between services.
BullMQ & Redis: To manage the video processing jobs queue.
Digital Ocean Spaces / AWS S3: Where we store the video files.
JWT (JSON Web Tokens): For secure user authentication.
Socket.io: For those real-time progress updates.
Docker & Kubernetes: To package and run our services easily.

Sneak Peek: What Your App Might Look Like

Here are some simple wireframes to visualize the user experience:

(Wireframes remain the same as the previous version)

1. Video Upload Interface

┌─────────────────────────────────────────────────┐
│ ┌─────────────────────────────────────────────┐ │
│ │                                             │ │
│ │        Drag & Drop Video Files Here         │ │
│ │                     or                      │ │
│ │           [  Browse Files Button  ]         │ │
│ └─────────────────────────────────────────────┘ │
│  Title: [                               ]       │
│  Description: [                         ]       │
│                [ Upload Video ]                 │
└─────────────────────────────────────────────────┘

2. Video Processing Status

┌─────────────────────────────────────────────────┐
│  My Videos > Processing                         │
│  ┌─────────────────────────────────────────┐    │
│  │ Video: My_Cool_Vid.mp4                  │    │
│  │ [██████████░░░░░░░░░░] 45%              │    │
│  │ Step: HLS Transcoding                   │    │
│  │ ✓ Upload Complete                       │    │
│  │ ✓ MP4 Conversion                        │    │
│  │ ⌛ HLS Transcoding                       │    │
│  │ ⬚ Ready for Playback                    │    │
│  └─────────────────────────────────────────┘    │
└─────────────────────────────────────────────────┘

3. Video Player

┌─────────────────────────────────────────────────┐
│ ┌─────────────────────────────────────────────┐ │
│ │                                             │ │
│ │                 Video Player                │ │
│ │                                             │ │
│ └─────────────────────────────────────────────┘ │
│  Title: My Awesome Video                        │
│  ┌─────┐ ┌─────────┐ ┌─────┐  ┌───────────────┐ │
│  │ 480p│ │ 720p    │ │1080p│  │    Auto ▼     │ │
│  └─────┘ └─────────┘ └─────┘  └───────────────┘ │
└─────────────────────────────────────────────────┘

What's Next? (Part 2 Preview)

Okay, that's the big picture! We've covered the why and the what. In Part 2, we'll get our hands dirty with the how:

Setting up your development environment.
Diving into the code for the pre-signed URL upload.
Implementing the FFmpeg conversion logic.
Configuring RabbitMQ and BullMQ.
Running everything with Docker.

Ready to build? Let's go!

Part-2 (https://dev.to/awalhossain/building-your-own-scalable-video-streaming-server-part-2-setup-core-implementation-32gf)

Do follow me on my Socials to stay updated on tech stuff:

💼 LinkedIn: [https://www.linkedin.com/in/awal-hossain-0bb650108/]
🐙 Github: [https://github.com/awalHossain/]

ReadMe File

Awal Hossain — Wed, 28 Aug 2024 08:56:23 +0000

📚 Data Collector Frontend - Guidelines

This project is a Next.js frontend application aimed at collecting social media data from various platforms (Facebook, YouTube, Twitter, Telegram, Instagram) through a web interface. It works in collaboration with the backend of this projec 🚀

Project Introduction Video
GitHub Branch Maintenance Guidelines
Softice Solutions User Guidelines
Code Style and Formatting
React and Next.js Best Practices
State Management
Form Handling
Styling
Performance
Testing
Version Control
Documentation
Accessibility
Security

🎥 Project Introduction Video

🚀 Quick Start

Prerequisites

Node.js (version X.X.X or higher)
NPM (Node package manager)

Installation

Install and run the project with yarn:

npm install
npm run dev

📚 Documentation

🌿 GitHub Branch Maintenance Guidelines

📹 Github Guidelines Video

To ensure consistency and maintain the quality of our codebase, please adhere to the following branch maintenance rules:

Branch Structure

Our repository is organized with three main branches: main, production, and development.
Always create new branches from the development branch.
Once your work is tested and stable in the development branch, it will be merged into the production branch for deployment.

Branch Naming Conventions

Use meaningful branch names that include at least two parts:
- The first part should describe the type of work (e.g., features, bugFix).
- The second part should specify the issue or task (e.g., login-page-design, update-font-size).
Examples:
- features/login-page-design
- bugFix/update-font-size

Pull Requests

After completing your task:
1. Build your application.
2. Fix all build errors.
3. Create a Pull Request (PR) to the development branch.
Your code will be reviewed before it's merged into the development branch.

🛠 Softice Solutions User Guidelines

We manage our projects in Softice Solutions, powered by Redmine. Follow these steps to get started:

Getting Started

Create an Account:
- Open an account on Softice Solutions.
- Request contributor access from your team lead.
Accessing Issues:
- Navigate to Projects > Bijoy 2024 - National Public Archive > Issues.
- Check the issues created by your team lead.

Working on Issues

Assigning Issues:
- If an issue is not assigned to anyone, you can assign it to yourself and start working on it.
Logging Progress:
- Regularly add comments in the Log Time section so that other team members and your team lead are aware of your progress and the current status of the issue.

Completing Issues

Pull Request:
- Once you complete an issue, follow the instructions in the GitHub guidelines to create a pull request.
Updating Issue Status:
- After submitting a pull request, change the issue status in Softice Solutions to Feedback.
Review and Merge:
- Your team lead will review the changes. If everything is satisfactory, the issue will be resolved, and your sub-branch will be merged into the development branch.

👨‍💻 Developer Guidelines

This guide will help you maintain high-quality code and follow best practices. Let's build something amazing together! 🚀

Code Style and Formatting

✅ Do:

Follow the Airbnb JavaScript Style Guide
Use ESLint to catch and fix style issues
Leverage TypeScript for type safety
Write self-documenting code with clear names

❌ Don't:

Ignore ESLint warnings or errors
Use any type in TypeScript unnecessarily
Leave commented-out code in the codebase

React and Next.js Best Practices

✅ Do:

Use functional components and hooks
Implement proper error boundaries
Utilize Next.js built-in optimizations
Keep components small and focused

❌ Don't:

Use class components without good reason
Modify the DOM directly
Ignore React key warnings

State Management

✅ Do:

Use React's useState and useContext
Consider Redux for complex state

❌ Don't:

Overuse global state
Mutate state directly

Form Handling

✅ Do:

Leverage Formik for form management
Implement Yup schemas for validation
Create reusable form components

❌ Don't:

Create forms without proper validation
Ignore accessibility in form design

Styling

✅ Do:

Use Tailwind CSS for styling
Utilize Ant Design (antd) components
Maintain a consistent design system

❌ Don't:

Mix different styling approaches
Unnecessarily override Ant Design styles

Performance

✅ Do:

Optimize images and assets
Use React.memo, useMemo, and useCallback wisely
Implement loading states and skeleton screens

❌ Don't:

Prematurely optimize
Load unnecessary assets

Testing

✅ Do:

Write unit tests for utils and hooks
Implement integration tests for key flows
Use React Testing Library

❌ Don't:

Skip tests for new features or bug fixes
Test implementation details

Documentation

✅ Do:

Document complex logic or algorithms
Keep README and docs up-to-date
Use JSDoc comments

❌ Don't:

Neglect updating docs with changes
Write unnecessary comments

Accessibility

✅ Do:

Ensure proper keyboard navigation
Use semantic HTML elements
Implement ARIA attributes when needed

❌ Don't:

Ignore color contrast issues
Use non-standard UI patterns carelessly

Security

✅ Do:

Sanitize user inputs
Use environment variables for sensitive info
Keep dependencies up-to-date

❌ Don't:

Store sensitive information in frontend code
Ignore security warnings

Thank you for following these guidelines to ensure smooth collaboration and project management. Happy coding! 🎉

Deply Node app with github action Automatically

Awal Hossain — Sat, 03 Feb 2024 11:33:15 +0000

After successfully connecting with your VPS,. Let’s first do some basic setup on our Ubuntu VPS. The first task is to update and upgrade necessary package

sudo apt update 
sudo apt upgrade

after that install nginx

apt install nginx

to deploy your project with github actions, you have to create a user and give permission to that user to create folder and write file


sudo adduser [your_user_name]

it will ask you to give the password and some other information; after that,, you have to give sudo permission to that user

 sudo usermod -aG sudo [your_user_name]

after that you have to give permission to that user to create folder and write file in your vps directory

sudo chmod -R 777 [your_directory_path]

for me it's [your_directory_path] is /var/www,

this sudo chmod -R 777 is the command to provide permission to write or create folder as user, inside square bracket you need to put directory path,after that you have to switch to that user

su - [your_user_name]

now you will have to create a folder where you will do your necessary work, for me i'll work as name streaming, you need to navigate to your directory where you want to create folder, for me it's /var/www, and this directory only available after nginx install.

cd /var/www 

mkdir streaming

cd streaming

As we are going to use github actions to deoploy our project automatically, so we need to create a repository in github and push our project there. i've already created a repository name streaming. go to your repository setting and find action, you will see a option call runner, click that and you will see a option call add runner, click that and after that click self-hosted runner, here you will see differen os option, choose linux and follow the instruction.

At very last step you need to install some packgae to make your runner active all the time. if you run ./run.sh, it will stay active for momentary but if exit it will be in offline. Sohere we have install a packgae to make it active all the time

sudo ./svc.sh install

after that run it

sudo ./svc.sh start

Environment(.env) Setup

It's important step to add environment variable in your project before deploying it automatically,
you can add environment variable in your repository setting, go to your repository setting and find secret,

now if you check back to your runner, you'll see it's activated but idle. now we need to install some package to run our project in vps, install nodejs lts version

curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash - &&\
sudo apt-get install -y nodejs

it will install both node and npm to check

node -v
npm -v

now we need to install pm2 to run our project in background

npm install -g pm2@latest

you might get into an error while download this package as user, so you could you switch to root user by run exit command, then install it. after that let's configure nginx. you can do these as root user or as user, i'll do it as root user. so let's go to root cd /root as root user

cd /etc/nginx/sites-available/

ls

you'll see default file now let's edit this default file

location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
proxy_pass http://localhost:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}

proxy_pass http://localhost:5000;: This command tells Nginx to forward requests it receives to http://localhost:5000, which is the address of the application you're running. Replace 5000 with the port number your application is running on.

proxy_http_version 1.1;: This command sets the version of the HTTP protocol that will be used for the proxy request. In this case, it's HTTP/1.1.

proxy_set_header Upgrade $http_upgrade;: This command sets the Upgrade HTTP header to the value of the $http_upgrade variable. This is used for handling WebSocket connections.

proxy_set_header Connection 'upgrade';: This command sets the Connection HTTP header to the value 'upgrade'. This is also used for handling WebSocket connections.

proxy_set_header Host $host;: This command sets the Host HTTP header to the value of the $host variable. This is used to pass the original host header to the proxied server.

proxy_cache_bypass $http_upgrade;: This command tells Nginx to bypass the cache when the $http_upgrade variable is set. This is useful for WebSocket connections, which should not be cached.

After you've replaced localhost:5000 with your port number, you need to save the configuration and restart Nginx for the changes to take effect. You can check your port number by running the pm2 list command, which will show you your project name and port number.

systemctl restart nignx

and then check nginx status to check nginx running successfully

nginx -t

Domain Configuration

If you want add any domain all you have to do is put your registered domain name as server_name in default file. so let's go again to default file and add our domain name

cd /etc/nginx/sites-available/
nano default

and add your domain name as server_name and save it.

        server_name api.reely.tech;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                proxy_pass http://localhost:5000;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
                proxy_cache_bypass $http_upgrade;
        }

Firewall Configuration

it's very important to configure firewall to protect your vps from any kind of attack. Here we are going to use ufw to configure basic firewall setting.

sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable

SSL Certification

To have https connection we need to install ssl certification. we are going to use Let's Encrypt to install ssl certification.

apt install certbot python3-certbot-nginx

Make sure that Nginx Full rule is available

ufw status

certbot --nginx -d example.com -d www.example.com

Let’s Encrypt’s certificates are only valid for ninety days. To set a timer to validate automatically:

systemctl status certbot.timer

there is one single thing to set, basically nginx allow only few mb file to upload so if you want allow more than you have go to

sudo nano /etc/nginx/nginx.conf

and paste down there and save

client_max_body_size 100M;

reload nginx

sudo systemctl reload nginx

Github Actions

Now we are going to use github actions to deploy our project automatically. so let's go to our github repository and create a folder name .github and inside that folder create a folder name workflows and inside that folder create a file name main.yml. so the path will be .github/workflows/main.yml. now let's copy and paste this code in main.yml file

name: Node.js CI

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs: 
        build:
        runs-on: self-hosted
        strategy:
        matrix:
                node-version: [14.x]
        steps:
        - uses: actions/checkout@v2
        - name: Use Node.js ${{ matrix.node-version }}
                uses: actions/setup-node@v1
                with:
                node-version: ${{ matrix.node-version }}
        - name: npm install, build, and test
                run: |
                npm install
                npm run build --if-present
                npm run test --if-present
        - name: Deploy to VPS
                pm2 start npm --name "streaming" -- start
        ```



configure githbu actions according to your need. This one is for very

How does the Javascript engine work?

Awal Hossain — Sun, 05 Jun 2022 05:23:24 +0000

Ryan Dhal created NodeJs in 2009 using the Chrome V8 engine and a C++ library called libuv. But before a deep dive into NodeJs, you must first understand how Javascript works. Let's start by talking about the chrome V8 engine.

V8 is the name of the Javascript engine that powers the Google Chrome browser. This engine takes our Javascript code and executes it while browsing with Chrome. V8 provides the runtime environment in which JavaScript executes. There are several Javascript engines, which you will find for each major internet browser.

V8 – Open-source JavaScript Engine developed by Google
SpiderMonkey – The JavaScript Engine powering Mozilla Firefox
JavaScriptCore–Open-source JavaScript Engine developed by Apple
Rhino – Open-source JavaScript Engine managed by Mozilla
Chakra– A JavaScript Engine for Microsoft Edge
JerryScript – A JavaScript engine for the Internet of Things (Iot).

Now,Let's see what's actually happening inside these engine.

When we deliver a Javascript file inside this engine, first it does something called lexical analysis, which breaks the code into something called tokens to identify their meaning, so that the engine can know what the code is trying to do.

These tokens are formed into what we call an AST (abstract syntax tree) and once it is formed, it goes through something called interpreter, profiler, and compiler.

The interpreter converts each high-level program statement one by one into machine code, but when it interprets the same lines of code a few times, it gets slower. But when the compiler sees the same code running again and again, it just replaces the function with the exact result that we want to return. And these sorts of edits that compiler do are called optimizations.

Interpreter Or Compiler. Which one is Faster??

Both the interpreter and the compiler have advantages and disadvantages. It takes a little longer for the compiler to start up, but the code will eventually execute quicker. An interpreter, on the other hand, can get up and running quite quickly. However, it does not do any optimizations.

JIT Compiler
In order to resolve this issue, Google came up with a solution. They invented a JIT compiler, or just in time compiler, by combining compiler and interpreter, and this is exactly what browsers started doing. Browsers started mixing compilers, specifically this JIT compilers, for just in time compilation to make the engines faster.

Now, if the same lines of code are executed a few times, the interpreter will tell the compiler, "Hey, here's some code for you to optimize," and the compiler will take it on. The compiler either compiles or modifies the code. So, interpreter allows us to run the code right away, and the compiler and profiler allows us to optimize the code.

After going through the interpreter, profiler, and compiler, the entire engine will spit out code that our CPU will understand, which will then be shown on our monitor and that's how Javascript engine actually works.

In the following part, I'll explain how libuv interacts with NodeJs.

My journey over the last two months

Awal Hossain — Thu, 26 May 2022 05:34:14 +0000

Our End-Game Journey
Hi there! For the past two months, I've been working with the Progamming Hero Job Placement Team. We were actually offered a shadow internship. It's approximately equivalent to 70% of an actual internship. There were 11 teams in this internship, each with six team members. Except on Saturday and Sunday, we were scheduled to join two scrum meetings each day. Few of the students were becoming irritated. Simply because they had to attend scrum meetings twice a day. Well, I became bored with this scrum as well, but I eventually got used to it.
End-Game Project
Each team was given an unique project, such as e-commerce, medical, hostel finder, car share, and issue tracker. The Issue Tracker Project was given to our group. Our project was unlike any other, and we were using technology that we had never touched before in our lives. The following is a list of the technologies we used to construct this app.
• NextJs
• TypeScript
• Socket.io
• WebRtc
I was completely unfamiliar with these technologies. As a result, I had to learn all of these technologies from scratch and then integrate them into our program. I was getting a lot of errors while working with socket.io and WebRTC, and I had no idea how to fix them. In particular, while I was working with socket.io, it was giving me a large number of connections at once. However, after trying for two whole weeks, I was able to implement real-time chatting, audio and video calling, push notifications, and screen sharing.

How does Issue Tracker work??
Allow me to tell you a story. XYZ is a new start-up that has just released its application. It is currently experiencing performance, user experience, and bug issues with the application. As a result, we've arrived to this point. Using our program, they can quickly fix their problem.

Now I'll show you how our app can help them solve their problem and provide a great user experience for their customers.

Section: My projects
When you log in, you land on this page which will show all the projects. You can create your own project or be invited to a project. You can click on one and it will take you to the issue board.
Section - Issue board
Issue board is a submenu under the project. You can change the status of an issue, check the names of the reporters, see the deadlines, countdown timers, and download files that have been uploaded to a certain issue. You may also use email to invite individuals to this issue board.
Section - User assignment
You can assign different roles to different users here. If you are an admin, you can assign different developers to different issues with countdown timers, so that they can finish the bug or issue within the bounded time.
Section - Project details
Here, you can see all the users and the issues reported against a project and also the details of all the issues and their details
Now, I am going to show you all the feature that we have added to make XYZ start-up problem more easy.

** Feature**
• User role management
• Graphs and Charts
• Real-time chatting
• Audio and video calling, live screenshare in video call
• File upload and download
• Dynamic data update with drag and drop
• Email invitation
• OTP validation
• Push notification
• SMS to mobile when a critical issue arises
• Responsive

Conclusion
I'm aware that our application has many faults. However, we are working nonstop to make this program more user-friendly so that even a non-technical person may use it without concern. Ohh, By the way, after seeing how our application can handle the problem of XYZ start - up company, they decided to collaborate with us.

What the Hell is a JWT Token??

Awal Hossain — Fri, 15 Apr 2022 17:18:47 +0000

When I first heard the jwt token name, it gave me a headache. I can still remember how terrified I was that day. I was telling myself what the hell is this new thing that I need to implement in order to verify the user and api.

However, after working on a few projects, I feel forced into making this statement

Jwt token is an excellent approach to validate any user against a database.

What is jwt token??

Jwt is an abbreviation for Json Web Token. A JWT is a mechanism to verify the owner/user. It’s encoded. When a server receives a JWT, it can guarantee the data it contains can be trusted because it’s signed by the source. No middleman can modify a JWT once it’s sent.

Creating JWT token

If you take a look at the above picture closely, you will get an idea how developers generate the jwt token.

When a developer creates a JWT token with a method called jwt.sign(), it contains elements such as the user id, JWT_SECRET, and expiration date. JWT_SECRET helps to make a sign token that won't work anywhere unless the secret is given. The expiration date will determine how long this token will be valid.
After successfully generating a jwt token, All you have to do now is transmit the jwt token to the frontend and save it in localstorage or Cookie.
.

How does JWT Token work ?

Let's have a look at how it works now. Let's say you need to access your data from the dashboard, but how will that website validate your identity and prevent a hacker from gaining access to your information?

Simply said, the provided jwt token will be checked against the database.
If that website discovers a user id associated with the jwt token, it will grant you access to the dashboard.

So, I used the jwt.verify() function, which accepts two parameters: token and JWT SECRET, to check the jwt token. It will allow you to enter if both input values are true.

Take a look at the above picture to better understand.

Forem: Awal Hossain

Behind the Stream: Building a Scalable Video Server Solution

1. Setting Up Shop: Your Development Environment

2. The Grand Upload: Pre-signed URLs in Action

3. Conversion Central: FFmpeg, BullMQ, and RabbitMQ

4. Tying it Together: RabbitMQ & Real-time Updates

Docker Installation

5. Conclusion & What's Next

Building Your Own Scalable Video Streaming Server: Part 3 - Data, Deployment & Durability

1. The Librarian's Bookshelf: API Server & MongoDB Deep Dive

2. Taking Flight: Deployment Strategies

3. Handling Hiccups: Error Handling & Monitoring

4. Supercharging Your Server: Optimizations & Scaling

5. Conclusion: Where to Go From Here?

Building Your Own Scalable Video Streaming Server: Part 2 - Setup & Core Implementation

1. Setting Up Shop: Your Development Environment

2. The Grand Upload: Pre-signed URLs in Action

3. Conversion Central: FFmpeg, BullMQ, and RabbitMQ

4. Tying it Together: RabbitMQ & Real-time Updates

Docker Installation

5. Conclusion & What's Next

Building Your Own Scalable Video Streaming Server: Part 1 - The Big Picture

What's HLS and Why Should You Care?

Our Architecture: The 3 Musketeers

How Does a Video Get Processed?

The Tech Toolkit 🛠️

Sneak Peek: What Your App Might Look Like

1. Video Upload Interface

2. Video Processing Status

3. Video Player

What's Next? (Part 2 Preview)

ReadMe File

📚 Data Collector Frontend - Guidelines

Table of Contents

🎥 Project Introduction Video

🚀 Quick Start

Prerequisites

Installation

📚 Documentation

🌿 GitHub Branch Maintenance Guidelines

📹 Github Guidelines Video

Branch Structure

Branch Naming Conventions

Pull Requests

🛠 Softice Solutions User Guidelines

Getting Started

Working on Issues

Completing Issues

👨‍💻 Developer Guidelines

Code Style and Formatting

✅ Do:

❌ Don't:

React and Next.js Best Practices

✅ Do:

❌ Don't:

State Management

✅ Do:

❌ Don't:

Form Handling

✅ Do:

❌ Don't:

Styling

✅ Do:

❌ Don't:

Performance

✅ Do:

❌ Don't:

Testing

✅ Do:

❌ Don't:

Documentation

✅ Do:

❌ Don't:

Accessibility

✅ Do:

❌ Don't:

Security

✅ Do:

❌ Don't:

Deply Node app with github action Automatically