Forem: longtk26

Overview: Change Data Capture (CDC)

longtk26 — Fri, 17 Apr 2026 14:03:41 +0000

Overview: Change Data Capture (CDC)

Welcome! In this post, we'll explore Change Data Capture (CDC) — one of those foundational concepts that can genuinely transform the way you think about data synchronization in distributed systems. Let's dive in!

What is Change Data Capture?
Use Cases
- Cache Invalidation
- Data Integration
- Real-Time Analysis
Pros and Cons
- Pros
- Cons
When Should You Use CDC?
When Should You NOT Use CDC?
Wrapping Up

What is Change Data Capture?

Change Data Capture (CDC) is a method used in databases to track and record changes made to data.

CDC captures modifications like inserts, updates, and deletes, and makes them available for downstream systems to consume — either for analysis, replication, or synchronization purposes. CDC helps maintain data consistency across different systems by recording alterations in real time.

Think of it like a digital detective sitting quietly inside your database, logging every change that happens and when it occurred — so that other systems never fall behind.

Use Cases

Cache Invalidation

Automatically invalidate entries in a cache as soon as the corresponding record changes or is removed.

If your cache (such as Redis, Memcached, or Infinispan) runs as a separate process, the invalidation logic can be placed into a dedicated service — completely decoupled from your main application. In more advanced setups, you can even use the data from the change event itself to update the affected cache entries directly, rather than just evicting them.

Data Integration

Data is often stored in multiple places, especially when used for different purposes or in slightly different shapes. Keeping those multiple systems in sync can quickly become a maintenance nightmare.

Example: When a system uses a separate search service like Elasticsearch, it's critical to keep the search index synchronized with the primary database whenever records are created or updated.

CDC makes this elegant: changes are captured from the database and automatically propagated to the search service. Instead of sprinkling Elasticsearch update calls throughout your business logic, a lightweight event-processing layer handles it asynchronously. This simplifies the application design while improving scalability and maintainability.

Real-Time Analysis

CDC powers real-time analytics platforms by continuously feeding data changes into analytical systems. Organizations can then derive insights from fresh data and respond swiftly to changing conditions — all without expensive batch jobs or manual ETL pipelines.

Pros and Cons

Pros

1. Decouples Application Logic

The main application no longer needs to explicitly publish events or sync data to downstream systems.
Reduces duplicated logic across services — one source of truth emits events, and consumers react independently.

2. Near Real-Time Data Synchronization

Changes from the database are streamed almost instantly.
Ideal for search indexing, analytics pipelines, and cache management.

3. Non-Intrusive to Existing Systems

Works at the database level (via WAL / binlog), so there's usually no need to heavily modify existing application code.
Particularly useful for integrating with legacy systems where source code changes are risky or costly.

4. Event-Driven Architecture Enablement

Naturally integrates with streaming platforms like Apache Kafka.
Makes it easier to build reactive microservices and real-time data pipelines.

5. Data Consistency (Eventually Consistent)

Ensures downstream systems gradually converge toward the source of truth over time.
Change events are durable and ordered, making them reliable for replication.

6. Supports Multiple Consumers

One stream of change events can feed multiple services simultaneously — search indexing, analytics, auditing, cache invalidation, and more.

Cons

1. Increased System Complexity

Requires additional infrastructure: a message broker (e.g., Kafka), connectors, and monitoring tooling.
Debugging issues that span multiple components can be significantly harder than debugging a simple API call.

2. Schema Evolution Challenges

Changes in the database schema (adding/removing columns, renaming tables) can break downstream consumers if not handled carefully.
A solid schema management strategy — such as a schema registry — becomes essential at scale.

3. Operational Overhead

Someone needs to manage connectors, track offsets, handle failures, and configure retries.
This requires a level of DevOps maturity that smaller teams may not yet have.

4. Learning Curve

Teams need to invest time in understanding event-driven design principles, CDC concepts, and the tooling involved.
The mental model shift from "call an API" to "react to events" can take time to internalize.

5. Data Filtering and Transformation Are Limited at the Source

Raw database change events may not perfectly match what downstream consumers need.
An additional transformation or processing layer is often required to shape, filter, or enrich events before they reach their destination.

When Should You Use CDC?

Use CDC when:

You need to sync data to systems like Elasticsearch, a data warehouse, or a cache layer
You are building an event-driven or microservices architecture
You want to reduce coupling between services
You need audit logs or historical change tracking
You have multiple downstream consumers of the same data
You want near real-time data propagation without modifying core application logic

** Examples:**

Sync user records from PostgreSQL → Elasticsearch for full-text search
Stream order events into an analytics pipeline for real-time dashboards

When Should You NOT Use CDC?

Avoid CDC when:

Your system requires strong consistency — CDC is eventually consistent and introduces some propagation delay
The system is simple or monolithic and doesn't benefit from event streaming
You only have one consumer and a direct API call or a database trigger would suffice
Your team lacks experience in operating distributed systems
Infrastructure cost and operational complexity are a concern
Your database is already under heavy load (CDC adds overhead via WAL reading)

Examples:

A simple CRUD app with basic search — just update Elasticsearch directly from the application
A low-scale system with no real-time data requirements

Wrapping Up

CDC is a powerful pattern, but like any tool, it shines brightest when applied to the right problem. If you are building systems where data needs to flow reliably across multiple components in near real time, CDC is absolutely worth the investment.

Thank you so much for reading! I hope this overview gave you a clear and practical understanding of Change Data Capture. If you found this helpful, feel free to explore the Debezium setup guide to see how to put these concepts into practice with real infrastructure. Happy building! 🚀

Debezium: CDC in Practice

longtk26 — Fri, 17 Apr 2026 14:03:35 +0000

If you've read the Overview CDC post, you already know why Change Data Capture is such a compelling pattern. Now it's time to get our hands dirty. In this guide, we'll walk through a complete, working Debezium setup — from standing up Kafka and Debezium Connect with Docker, all the way to watching live database changes flow into Elasticsearch in real time. Let's go! 🛠️

What is Debezium?
Setup: PostgreSQL Connector for Tracking Database Changes
- Step 1 – Create the Docker Network
- Step 2 – Prepare the Custom Debezium Connect Image
- Step 3 – Prepare the Docker Compose File
- Step 4 – Start the Services
- Step 5 – Configure the PostgreSQL Connector via Debezium UI
Setup: Synchronization from PostgreSQL to Elasticsearch
- Step 1 – Start Elasticsearch and Kibana
- Step 2 – Register the Connectors
- Step 3 – Verify the Pipeline
Wrapping Up

What is Debezium?

Debezium is an open-source project that provides a low-latency data streaming platform for Change Data Capture (CDC). It connects to your database, monitors its transaction log (e.g., PostgreSQL's Write-Ahead Log), and emits a stream of change events for every row-level modification.

Your applications then consume these events and react accordingly — syncing to a search index, invalidating a cache, updating a data warehouse, or triggering downstream workflows.

Key characteristics:

Open source under the Apache 2.0 license
Built on top of Kafka Connect — battle-tested, scalable, and extensible
Supports many popular databases: PostgreSQL, MySQL, MongoDB, SQL Server, Oracle, and more
Guarantees at-least-once delivery of change events with configurable offset management

Setup: PostgreSQL Connector for Tracking Database Changes

Step 1 – Create the Docker Network

All services in this setup share a common Docker bridge network. Create it once before starting any containers:

docker network create base-network

Step 2 – Prepare the Custom Debezium Connect Image

We need a custom Debezium Connect image that includes the Confluent Elasticsearch Sink Connector plugin. Create a Dockerfile inside a folder named connect/:

FROM debezium/connect:3.0.0.Final

# Install the Confluent Elasticsearch Sink Connector for Kafka Connect.

ARG ELASTICSEARCH_SINK_VERSION=15.1.1

RUN set -eux; \

    curl -fsSL "https://hub-downloads.confluent.io/api/plugins/confluentinc/kafka-connect-elasticsearch/versions/${ELASTICSEARCH_SINK_VERSION}/confluentinc-kafka-connect-elasticsearch-${ELASTICSEARCH_SINK_VERSION}.zip" -o /tmp/es-sink.zip; \

    mkdir -p /kafka/connect/confluentinc-kafka-connect-elasticsearch; \

    unzip -q /tmp/es-sink.zip -d /kafka/connect/confluentinc-kafka-connect-elasticsearch; \

    rm -f /tmp/es-sink.zip

Note: If you don't need Elasticsearch, skip the custom image and use the official debezium/connect:3.0.0.Final image directly in your Compose file.

Step 3 – Prepare the Docker Compose File

Create a docker-compose.yml at the project root with the following content:

version: '3.9'
services:
  kafka-ui:
    container_name: kafka-ui
    image: ghcr.io/kafbat/kafka-ui:latest
    ports:
      - "8070:8080"
    depends_on:
      - kafka
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092
      DYNAMIC_CONFIG_ENABLED: 'true'
    networks:
      - base-network

  kafka:
    image: apache/kafka:4.0.2
    container_name: kafka
    ports:
      - "9092:9092"
      - "9094:9094"
    environment:

      ############################################
      # KRaft Metadata & Node Identity
      ############################################

      # Unique ID for this Kafka node
      KAFKA_NODE_ID: 1

      # Unique Kafka cluster identifier (required for KRaft mode)
      KAFKA_CLUSTER_ID: 'energy-tracker-cluster-1'

      # Node roles — this instance acts as BOTH a broker and a controller
      KAFKA_PROCESS_ROLES: 'broker,controller'

      ############################################
      # Controller Quorum Configuration (Raft)
      ############################################

      # List of controller nodes that form the Raft quorum
      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:9093'

      # Listener name used by controllers for internal Raft communication
      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'

      ############################################
      # Network Listeners (Internal, External & Controller)
      ############################################

      # Define all listener endpoints for this Kafka node
      KAFKA_LISTENERS: >
        PLAINTEXT://0.0.0.0:9092,
        EXTERNAL://0.0.0.0:9094,
        CONTROLLER://0.0.0.0:9093

      # What each listener advertises to connecting clients
      # Internal (Docker containers): kafka:9092
      # External (host machine): localhost:9094
      KAFKA_ADVERTISED_LISTENERS: >
        PLAINTEXT://kafka:9092,
        EXTERNAL://localhost:9094

      # Maps listener names to their security protocols
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: >
        CONTROLLER:PLAINTEXT,
        PLAINTEXT:PLAINTEXT,
        EXTERNAL:PLAINTEXT

      # Which listener brokers use for internal broker-to-broker communication
      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'

      ############################################
      # Single-Node Cluster Safety Settings
      # (replication factor > 1 would fail on a single node)
      ############################################
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

      ############################################
      # Broker Defaults & Quality-of-Life Settings
      ############################################

      # Speed up consumer group startup
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

      # Auto-create topics if they don't already exist
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

      # Default number of partitions per topic
      KAFKA_NUM_PARTITIONS: 1

    networks:
      - base-network

  debezium-connect:
    build:
      context: ./connect
    container_name: debezium-connect
    ports:
      - "8083:8083"
    environment:
      - BOOTSTRAP_SERVERS=kafka:9092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_connect_statuses
      - CONFIG_STORAGE_REPLICATION_FACTOR=1
      - OFFSET_STORAGE_REPLICATION_FACTOR=1
      - STATUS_STORAGE_REPLICATION_FACTOR=1
      - REST_ADVERTISED_HOST_NAME=debezium-connect
      - KEY_CONVERTER=org.apache.kafka.connect.storage.StringConverter
      - VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
      - VALUE_CONVERTER_SCHEMAS_ENABLE=false
      - PLUGIN_PATH=/kafka/connect,/usr/share/java,/usr/share/confluent-hub-components
    depends_on:
      - kafka
    networks:
      - base-network

  debezium-ui:
    container_name: debezium-ui
    image: debezium/debezium-ui
    ports:
      - "8080:8080"
    environment:
      - KAFKA_CONNECT_URIS=http://debezium-connect:8083
    depends_on:
      - debezium-connect
    networks:
      - base-network

networks:
  base-network:
    external: true

Service summary:

Service	Role
`debezium-connect`	Hosts the Kafka Connect worker. Connectors registered here read the PostgreSQL WAL and publish change events to Kafka topics.
`debezium-ui`	A web UI that simplifies connector management by wrapping the Debezium Connect REST API.
`kafka`	The message broker that durably stores change event streams. Runs in KRaft mode (no Zookeeper required).
`kafka-ui`	A web UI for browsing Kafka topics, messages, and consumer groups — invaluable for debugging.

Step 4 – Start the Services

docker compose up -d --build

Once the containers are healthy:

Debezium UI → http://localhost:8080
Kafka UI → http://localhost:8070

Step 5 – Configure the PostgreSQL Connector via Debezium UI

Before creating the connector, make sure you have a users table in your PostgreSQL database.

Open Debezium UI at http://localhost:8080 and follow the steps shown in the screenshots below to create a new PostgreSQL source connector:

Click Create Connector

Choose the PostgreSQL connector type

Fill in the basic connection settings (hostname, port, database name, credentials)

Configure advanced settings (table filter, plugin name: pgoutput)

Validate the configuration and finish

Confirm the connector is running

Now try inserting a record into the users table and open Kafka UI at http://localhost:8070 to see the change event appear on the Kafka topic:

Setup: Synchronization from PostgreSQL to Elasticsearch

With Debezium already streaming changes to Kafka, adding Elasticsearch as a sync target is straightforward — we just need to:

Start an Elasticsearch (+ Kibana) stack
Register a sink connector that reads from the Kafka topic and writes to an Elasticsearch index

Step 1 – Start Elasticsearch and Kibana

Create a separate docker-compose-elk.yml file:

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.15
    container_name: elasticsearch
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
    networks:
      - base-network

  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.15
    container_name: kibana
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch
    networks:
      - base-network

networks:
  base-network:
    external: true

Start the stack:

docker compose -f docker-compose-elk.yml up -d --build

Elasticsearch → http://localhost:9200
Kibana → http://localhost:5601

Step 2 – Register the Connectors

Note: Make sure the users table exists in your PostgreSQL database before registering the connectors.

Use the following curl commands to register both connectors via the Debezium Connect REST API.

Elasticsearch Sink Connector — reads the users Kafka topic and writes to the users Elasticsearch index:

curl --location 'http://localhost:8083/connectors' \
--header 'Content-Type: application/json' \
--data '{
  "name": "elasticsearch-sink-connector",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "topics": "users",
    "connection.url": "http://elasticsearch:9200",

    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",

    "schema.ignore": "true",
    "key.ignore": "false",

    "write.method": "upsert",
    "behavior.on.null.values": "delete",

    "transforms": "extractKey,removeDeleted",
    "transforms.extractKey.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
    "transforms.extractKey.field": "id",

    "transforms.removeDeleted.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
    "transforms.removeDeleted.blacklist": "__deleted",

    "errors.tolerance": "all",
    "errors.deadletterqueue.topic.name": "dlq-users",
    "errors.deadletterqueue.topic.replication.factor": "1"
  }
}'

PostgreSQL Source Connector — reads WAL changes from PostgreSQL and publishes them to the users Kafka topic:

curl --location 'http://localhost:8083/connectors' \
--header 'Content-Type: application/json' \
--data '{
  "name": "postgres-source-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgresdb",
    "database.port": "5432",
    "database.user": "postgres",
    "database.password": "postgres",
    "database.dbname": "template",
    "topic.prefix": "users_table",
    "table.include.list": "public.users",
    "plugin.name": "pgoutput",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "transforms": "route,unwrap",
    "transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.route.regex": "users_table.public.users",
    "transforms.route.replacement": "users",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.delete.handling.mode": "rewrite"
  }
}'

Step 3 – Verify the Pipeline

After registering both connectors, check their status in Debezium UI and verify that a consumer group has been created in Kafka UI:

Now run some INSERT, UPDATE, or DELETE statements on the users table in PostgreSQL and follow the data through the pipeline:

1. Run a search in Kibana DevTools before inserting any data:

2. Insert a new record into the users table:

3. Check the users topic in Kafka UI — the change event should appear:

4. Query the users index in Elasticsearch via Kibana DevTools — the new document should be there:

That's the full pipeline working end-to-end: PostgreSQL → Debezium → Kafka → Elasticsearch. 🎉

Reference

Repository: Setup CDC

Wrapping Up

You've just set up a fully functional CDC pipeline with Debezium! From a single row update in PostgreSQL, a change event flows through Kafka and lands in Elasticsearch — all automatically, with zero application code involved.

This is just the beginning. From here you can explore:

Adding more sink connectors (e.g., a data warehouse, another database)
Schema evolution strategies using a schema registry
Monitoring connector health and lag in production

Thank you so much for following along! I hope this guide saved you hours of trial and error. If you haven't read it yet, the Overview CDC post provides great background context on the concepts behind what we built here. Happy streaming! 🚀

Subtitle to Audio Tutorial

longtk26 — Sun, 18 Jan 2026 16:17:50 +0000

Subtitle to Audio Tutorial

Hi everyone! In this tutorial, we will explore how to convert subtitles into audio using text-to-speech (TTS) technology. This process can be particularly useful for creating audiobooks, podcasts, or enhancing accessibility for video content.

Subtitle to Audio Tutorial
- Table of Contents
- Introduction
- Workflow Overview
- Problems with workflow
  - 1. Limited execution time of lambda functions
  - 2. Limited resources within lambda functions
  - 3. Synchronous time and speed between subtile and audio
  - 4. Popping noises between audio segments
- Solutions
  - Algorithm diagram
  - Key Optimizations
- Conclusion

Introduction

When working with applications that involve video content and translation, it is often necessary to translate an original subtitle file (e.g., SRT format) into another language.
To enhance user experience, we can convert these translated subtitles into audio using TTS technology.
This tutorial will guide you through the process of converting subtitles to audio, addressing common challenges and providing solutions.

Workflow Overview

Convert from subtitle to audio is a job that need several steps to complete that why we will handle it asynchronously.
We choose lamba functions for saving cost and easy in implementation.
The workflow consist of the following steps:
1. Choose a subtitle file from current system: Assume our system already show list of subtitle files that user can choose from.
2. Choose voice settings: User can choose voice settings like male or female voice.
3. Request to convert subtitle to audio: After user choose subtitle file and voice settings we will send request to backend to start the conversion process.
4. Server request to s3 to get subtitle file: Because in real project we usually store files in s3, so our server need to get subtitle file url from s3 first.
5. Server request to lambda function to convert subtitle to audio: After get subtitle file url server will send request to lambda function to start convert subtitle to audio.
6. Lambda function process subtitle file: Lambda function will download subtitle file from s3, parse subtitle file to get text and timing information, then use google text to speech to convert text to audio, finally combine audio segments according to timing information and save the final audio file to s3.

Problems with workflow

1. Limited execution time of lambda functions

AWS Lambda functions have a maximum execution time limit (currently 15 minutes). If the subtitle file is large or the conversion process takes too long, the lambda function may time out before completing the task.

2. Limited resources within lambda functions

Lambda functions have limited CPU and memory resources. If the TTS engine requires more resources or inprogress the size of audio become to large, the conversion process may fail or produce error out of memory.

3. Synchronous time and speed between subtile and audio

Ensuring that the audio segments align perfectly with the timing information in the subtitle file can be challenging. Any discrepancies can lead to audio that is out of sync with the original video content.

4. Popping noises between audio segments

When combining multiple audio segments generated from subtitles, popping noises may occur at the boundaries between segments.

Solutions

This section will show you solutions for the problems mentioned above and successully in converting subtitle file more than 30 minutes to audio only in some minutes. Also implement it with at least resources, we only use default lambda function settings (128MB memory) and timeout is 15 minutes.

Algorithm diagram

Link algorithm diagram: Image

The flowchart above illustrates the complete subtitle-to-audio conversion algorithm with optimized memory management and streaming architecture. Here's how it works:

1. Initialization Phase

The process begins by downloading and parsing the SRT subtitle file from the provided URL
It pre-calculates the total PCM (Pulse Code Modulation) audio size based on subtitle duration and timing information
An S3 Multipart Upload is initialized to enable streaming upload without holding the entire file in memory
A WAV header is created and added to the part buffer, initializing tracking variables (part_number=1, content_id index=0, current_part=b"")

2. Batch Processing Loop

Instead of processing all subtitles at once, the algorithm processes them in batches of approximately 15 subtitles
For each batch, it retrieves the next set of subtitle entries to process
This batching approach prevents memory overflow by keeping only a small subset of subtitles in memory at any time

3. Audio Generation for Each Subtitle

For each subtitle in the current batch:
- Calculate speech speed: Determines the appropriate speaking rate to fit the subtitle text within its designated time window
- Generate audio using Google Cloud TTS: Converts the subtitle text to audio using the selected voice settings
- Apply fade effects: Adds 250ms fade-in and fade-out to prevent popping noises between audio segments
- Calculate and add silence padding: Adds appropriate silence gaps to ensure precise alignment with subtitle timing

4. Streaming Upload Mechanism

As audio segments are generated, PCM data is appended to the part buffer
When the buffer reaches 5MB or when processing the last batch, the current part is uploaded to S3 immediately
After each upload, the buffer is cleared and the part_number is incremented
This streaming approach ensures that Lambda memory usage remains constant (around 5-10MB) regardless of the total audio file size

5. Finalization Phase

Once all subtitle batches have been processed and all parts uploaded, the algorithm completes the S3 Multipart Upload
The final audio file is assembled from all uploaded parts and becomes available on S3
Error handling paths are included to manage upload failures or conversion issues (shown by the red error node)

Key Flow Control

The algorithm uses two main decision loops:
- "More batches to process?": Controls the outer loop for batch processing
- "More subtitles in batch?": Controls the inner loop for processing individual subtitles within a batch
"Buffer size >= 5MB OR last batch?": Triggers S3 part uploads at optimal intervals

Key Optimizations

Batch Processing: Subtitles are processed in batches (default: 15 subtitles per batch) instead of loading all at once. This prevents memory overflow in Lambda.
S3 Multipart Upload:
- Initialize multipart upload at the start
- Upload parts incrementally when buffer reaches 5MB
- Each part is uploaded immediately and buffer is cleared
- Complete multipart upload at the end
- This approach ensures Lambda never holds the entire audio file in memory
Streaming Architecture: Audio data flows directly from TTS → batch buffer → S3 parts, with minimal memory footprint at any given time.
Fade Effects: 250ms fade-in/out applied to each audio segment to eliminate popping noises between segments.
Memory Management: After each part upload, the buffer is reset, keeping memory usage constant regardless of total audio length.
Avoid FFmpeg and audio processing libraries:

We should avoid using FFmpeg (both the binary and Python wrappers like pydub, ffmpeg-python) in Lambda functions because they significantly increase deployment package size.
FFmpeg binaries are typically 50-100MB, and audio processing libraries add another 20-50MB, easily exceeding Lambda's package size limits.
Instead, we should work directly with WAV file format using Python's built-in struct module and pure Python code to manipulate raw PCM audio bytes.
This approach keeps the deployment package minimal (<15MB) while maintaining full control over audio processing operations like fade-in/fade-out, silence padding, and concatenation.

Conclusion

Thank you for following this tutorial on converting subtitles to audio using TTS technology.
I hope this information can help you to implement this feature in your own projects efficiently. You could use these ideas and use any AI tools for generating code to implement it.
This blog is written by my experience in real project that we need to save cost with lambda function, if you have any good ideas or suggestions please feel free to share with me!
You could find more on my blogs: Website

Understanding the Factory Method Pattern

longtk26 — Sat, 04 Jan 2025 15:18:34 +0000

Introduction

Hi everyone, I am writing this post to share my knowledge as I continue learning about design patterns. Today, I will present the Factory Method Pattern, which is a design pattern commonly used in real-world applications. If there are any mistakes in my post, please feel free to comment below, and I will gladly fix and update it.

Factory method pattern provides an interface for creating objects in a superclass, but allow subclasses to alter the type of objects that will be created.

Problem

Assume you have a bank application, and you’re building a feature for transferring money through various methods like bank transfer, paypal transfer,…

Before using the Factory Method pattern, let’s examine the scenario without it.

I will give an example implemented in Java.

Situation: Person1 sends money to Person2 using a transfer method (Bank Transfer or PayPal Transfer).

Folder structure:

problem/
├─ BankApp.java
├─ service/
│  ├─ PaypalTransferPayment.java
│  ├─ BankTransferPayment.java
├─ data/
│  ├─ Person.java

In the main application, create two persons with default amounts of money.

package problem;

import problem.data.Person;

public class BankApp {
    public static void main(String[] args) {
        Person person1 = new Person("John", 1000);
        Person person2 = new Person("Jane", 500);
    }
}

Create BankTransferPayment and PaypalTransferPayment classes.

package problem.service;

import problem.data.Person;

public class BankTransferPayment {
    public void processPayment(Person fromAccount, Person toAccount, float amount) {
        fromAccount.withdraw(amount);
        toAccount.deposit(amount);
        System.out.println("Bank transfer payment success.");
    }
}

package problem.service;

import problem.data.Person;

public class PaypalPayment {
    public void processPayment(Person fromAccount, Person toAccount, float amount) {
        fromAccount.withdraw(amount);
        toAccount.deposit(amount);
        System.out.println("Paypal transfer payment success.");
    }
}

Implement the logic in the main function.

package problem;

import problem.data.Person;
import problem.service.BankTransferPayment;
import problem.service.PaypalPayment;

public class BankApp {
    public static void main(String[] args) {
        Person person1 = new Person("John", 1000);
        Person person2 = new Person("Jane", 500);

        String paymentMethod = "BANK_TRANSFER";

        if (paymentMethod.equals("BANK_TRANSFER")) {
            BankTransferPayment bankTransferPayment = new BankTransferPayment();
            bankTransferPayment.processPayment(person1, person2, 100);

            System.out.println("===Method bank_transfer===");
            System.out.println(person1.getName() + " has " + person1.getAmount());
            System.out.println(person2.getName() + " has " + person2.getAmount());
        } else if (paymentMethod.equals("PAYPAL")) {
            PaypalPayment paypalPayment = new PaypalPayment();
            paypalPayment.processPayment(person1, person2, 100);

            System.out.println("===Method paypal===");
            System.out.println(person1.getName() + " has " + person1.getAmount());
            System.out.println(person2.getName() + " has " + person2.getAmount());
        }
    }
}

Problems with the current implementation:

Repetitive code: The processPayment method logic is repeated for every payment method.
Tightly coupled code: The application needs to create the payment method objects itself, making it hard to extend the application.
Scalability issues: If new payment methods are added, the source code becomes more complex and harder to maintain.

Solution

The solution to the above situation is to use factory method pattern. So, how do we apply it ?

In the example above:

Each if-else block calls the processPayment method, which leads to repetitive code.
Objects are created based on the payment type condition, making the code messy with excessive if-else statements.

To solve these issues, the Factory Method pattern will be implemented step by step.

Folder structure (solution):

solution/
├─ BankApp.java
├─ service/
│  ├─ payments/
│  │  ├─ Payment.java
│  │  ├─ PaymentFactory.java
│  │  ├─ BankTransferPayment.java
│  │  ├─ PaypalTransferPayment.java
├─ data/
│  ├─ Person.java

Step 1: Create Payment interface, declares common method processPayment

package solution.service.payments;

import solution.data.Person;

// Step 1: Create an interface for the payment
public interface Payment {
    void processPayment(Person fromAccount, Person toAccount,float amount);
}

Step 2: Create BankTransferPayment and PaypalTransferPayment classes implement Payment interface.

package solution.service.payments;

import solution.data.Person;

// Step 2: Create a class that implements the Payment interface
public class BankTransferPayment implements Payment {
    @Override
    public void processPayment(Person fromAccount, Person toAccount, float amount) {
        fromAccount.withdraw(amount);
        toAccount.deposit(amount);
        System.out.println("Bank transfer payment success.");
    }
}

package solution.service.payments;

import solution.data.Person;

public class PaypalPayment implements Payment{
    @Override
    public void processPayment(Person fromAccount, Person toAccount, float amount) {
        fromAccount.withdraw(amount);
        toAccount.deposit(amount);
        System.out.println("Paypal transfer payment success.");
    }
}

Step 3: Create PaymentFactory class. This class is responsible for creating objects based on payment type condition.

package solution.service.payments;

public class PaymentFactory {
    public Payment createPayment(String paymentType) {
        if (paymentType == null) {
            return null;
        }
        if (paymentType.equalsIgnoreCase("BANK_TRANSFER")) {
            return new BankTransferPayment();
        } else if (paymentType.equalsIgnoreCase("PAYPAL")) {
            return new PaypalPayment();
        }
        return null;
    }
}

Step 4: Use the Factory in the Main Application.

Modify the main function to use the Factory Method pattern.

package solution;

import solution.data.Person;
import solution.service.payments.Payment;
import solution.service.payments.PaymentFactory;

public class BankApp {
    public static void main(String[] args) {
        Person person1 = new Person("John", 1000);
        Person person2 = new Person("Jane", 500);

        String paymentMethod = "PAYPAL";

        Payment payment = new PaymentFactory().createPayment(paymentMethod);
        payment.processPayment(person1, person2, 100);
    }
}

Benefits of Using the Factory Method Pattern

The code is cleaner and more structured.
Repetitive calls to processPayment in multiple if-else blocks are eliminated.
Object creation is delegated to the factory, improving maintainability.

Bonus

To make the PaymentFactory class comply with the Open/Closed Principle (from SOLID principles), you can implement a dynamic registration mechanism using the Strategy Pattern.

Updated PaymentFactory.java:

package solution.service.payments;

import java.util.HashMap;
import java.util.Map;

public class PaymentFactory {
    private Map<String, Payment> paymentMaps;

    public PaymentFactory() {
        this.paymentMaps = new HashMap<>();
    }

    public Payment createPayment(String paymentType) {
        return paymentMaps.get(paymentType);
    }

    public void registerPayment(String paymentType, Payment payment) {
        paymentMaps.put(paymentType, payment);
    }

    public PaymentFactory initializePaymentMethods() {
        Payment bankTransferPayment = new BankTransferPayment();
        Payment paypalPayment = new PaypalPayment();

        this.registerPayment("BANK_TRANSFER", bankTransferPayment);
        this.registerPayment("PAYPAL", paypalPayment);

        return this;
    }
}

Using the Updated Factory in the Main Application.

package solution;

import solution.data.Person;
import solution.service.payments.Payment;
import solution.service.payments.PaymentFactory;

public class BankApp {
    public static void main(String[] args) {
        Person person1 = new Person("John", 1000);
        Person person2 = new Person("Jane", 500);

        String paymentMethod = "BANK_TRANSFER";

        Payment payment = new PaymentFactory()
                .initializePaymentMethods()
                .createPayment(paymentMethod);
        payment.processPayment(person1, person2, 100);
    }
}

By applying this approach, the code adheres to the Open/Closed Principle, enabling the addition of new payment methods without modifying the PaymentFactory logic.

I hope this post will be helpful to you.

References:

guru-design-patterns

Forem: longtk26

Overview: Change Data Capture (CDC)

Overview: Change Data Capture (CDC)

Table of Contents

What is Change Data Capture?

Use Cases

Cache Invalidation

Data Integration

Real-Time Analysis

Pros and Cons

Pros

1. Decouples Application Logic

2. Near Real-Time Data Synchronization

3. Non-Intrusive to Existing Systems

4. Event-Driven Architecture Enablement

5. Data Consistency (Eventually Consistent)

6. Supports Multiple Consumers

Cons

1. Increased System Complexity

2. Schema Evolution Challenges

3. Operational Overhead

4. Learning Curve

5. Data Filtering and Transformation Are Limited at the Source

When Should You Use CDC?

Use CDC when:

When Should You NOT Use CDC?

Avoid CDC when:

Wrapping Up

Debezium: CDC in Practice

Table of Contents

What is Debezium?

Setup: PostgreSQL Connector for Tracking Database Changes

Step 1 – Create the Docker Network

Step 2 – Prepare the Custom Debezium Connect Image

Step 3 – Prepare the Docker Compose File

Step 4 – Start the Services

Step 5 – Configure the PostgreSQL Connector via Debezium UI

Setup: Synchronization from PostgreSQL to Elasticsearch

Step 1 – Start Elasticsearch and Kibana

Step 2 – Register the Connectors

Step 3 – Verify the Pipeline

Reference

Wrapping Up

Subtitle to Audio Tutorial

Subtitle to Audio Tutorial

Table of Contents

Introduction

Workflow Overview

Problems with workflow

1. Limited execution time of lambda functions

2. Limited resources within lambda functions

3. Synchronous time and speed between subtile and audio

4. Popping noises between audio segments

Solutions

Algorithm diagram

Key Optimizations

Conclusion

Understanding the Factory Method Pattern

Introduction

Problem

Solution

Benefits of Using the Factory Method Pattern

Bonus

References: