Forem: Homer

ElasticRelay: Reliably Streaming Multi‑Source Database Changes into Elasticsearch

Homer — Thu, 23 Apr 2026 15:37:05 +0000

ElasticRelay is an open-source CDC gateway that streamlines real-time data synchronization from MySQL, PostgreSQL, and MongoDB into Elasticsearch. It provides a lightweight, reliable alternative to heavy streaming platforms by integrating data governance, batch writing, and failure recovery into a single, Go-based pipeline

In many teams, Elasticsearch is no longer “just a search engine.”
It has become a core piece of infrastructure for search, operational analytics, log correlation, real‑time dashboards, and accelerated business queries.

With that evolution comes a familiar challenge: upstream data is usually scattered across OLTP databases such as MySQL, PostgreSQL, and MongoDB. How do you reliably, continuously, and with low operational overhead synchronize those changes into Elasticsearch?

Traditional approaches can solve the problem, but often at a high cost. You may need to maintain complex data pipelines, handle the transition between initial full imports and incremental updates, manage checkpoints and retries, tune indexing performance, and deal with field cleanup, masking, filtering, and index routing. For teams that do not want to introduce a heavy streaming platform, these tasks can quickly consume project time and energy.

This is exactly the problem ElasticRelay is designed to solve.

What Is ElasticRelay?

From its codebase and repository structure, ElasticRelay’s positioning is very clear:
It is a multi‑source CDC (Change Data Capture) gateway purpose‑built for Elasticsearch, continuously synchronizing data changes from MySQL, PostgreSQL, and MongoDB into Elasticsearch.

ElasticRelay is also fully open-sourced and independently developed by Yogoo Software Co., Ltd.
For teams that care about open‑source ecosystems and long‑term sustainability, this matters: you can use it out of the box, or extend, integrate, and customize it based on real, production‑grade code.

Currently, ElasticRelay supports three common database types:

MySQL:binlog‑based change capture, with support for initial sync and parallel snapshots
PostgreSQL:logical replication / WAL‑based incremental capture with LSN management
MongoDB:Change Streams‑based real‑time subscriptions, compatible with replica sets and sharded clusters

In one sentence, ElasticRelay can be described as:
A CDC middleware layer designed specifically for Elasticsearch, making database‑to‑index synchronization simpler, more direct, and easier for backend teams to operate independently.

Why the Elastic Community Should Care
For Elasticsearch users, the hardest part is rarely “can I write data into ES?”
The real pain point is “can I write it continuously, reliably, and at low cost?”
ElasticRelay addresses this with several very pragmatic design choices.

1. Unified Multi‑Source Configuration Model

ElasticRelay consolidates multiple database sources into a single Go service.
Its MultiConfig model separates data_sources, sinks, jobs, and global settings, allowing teams to define—using a single configuration approach—which source writes to which Elasticsearch target and under which job.

2. A Write Path Designed Around Elasticsearch

The ES sink uses the official go-elasticsearch/v8 client and BulkIndexer for batched writes.
Index names are dynamically generated based on _table or _collection metadata in events, for example:

elasticrelay-users
elasticrelay-orders

This is especially friendly for teams that want to split indices by business entities.

3. Built‑In Data Governance

ElasticRelay embeds “governance” directly into the synchronization pipeline.
Its Transform Engine supports rule‑based matching by source and table/collection, enabling:

filtering
field mapping
type conversion
expression processing
data masking

As a result, much of the “pre‑index cleanup” work no longer needs to live in ad‑hoc scripts or external services.

4. Failure‑ and Recovery‑Oriented Design

When sink writes fail, ElasticRelay persists events into a durable **DLQ (Dead Letter Queue) **and ties them to checkpoints for recovery and retry. Failures are not silently dropped.

How the Internal Data Pipeline Works

ElasticRelay is not an abstract concept—it is a very concrete data pipeline.

Core Components

Connector: reads changes from databases
Orchestrator: manages job lifecycle and synchronization flow
Transform Engine: applies rule‑based data governance
ES Sink: handles batched writes into Elasticsearch
DLQ: persists failed events and manages retries and cleanup

Runtime Flow

A typical runtime flow looks like this:

Create a synchronization job defining source, sink, and job configuration
If required, run an initial snapshot to import existing data into Elasticsearch
Start CDC to continuously receive incremental changes
Events enter an asynchronous batching queue, decoupling database reads from sink write speed
Batched events pass through the Transform Engine for filtering, mapping, and masking
Processed events are streamed to the ES sink using bulk writes
On success, checkpoints are committed; on failure, events are written to the DLQ for retry

A key design choice here is the explicit decoupling of database change reading from Elasticsearch writes.
Binlog / WAL consumption is not directly blocked by temporary ES slowdowns, which is critical for system stability.

Features Especially Friendly to Elasticsearch Use Cases

1. Dynamic Index Naming
ElasticRelay extracts table or collection names from events and automatically generates target index names using a prefix.
This allows a single pipeline to naturally route different business entities into separate indices.
This is valuable for scenarios such as:

Writing orders, users, and products into separate indices
Applying different mappings and query strategies per entity
Aligning index lifecycle management with business domains

2. Batch Writes Instead of Per‑Document Writes

The ES sink relies on BulkIndexer, which aligns with Elasticsearch’s throughput model.
For continuous CDC streams, batch writes are a baseline requirement for production readiness.

3. Built‑In Data Governance

In real‑world projects, OLTP data is rarely indexed “as‑is.” Common needs include:

removing internal fields
unifying field names
handling nulls and type normalization
masking sensitive data
filtering out records that should not be indexed

ElasticRelay’s Transform Engine abstracts these into a rule system matched by source and table patterns, making it a synchronization layer tailored specifically for search index construction—not just database replication.

4. Failure‑Aware by Design

Real pipelines encounter failures: ES outages, index creation errors, network hiccups, rule evaluation exceptions.
ElasticRelay’s philosophy is not “ignore failures,” but to persist them in DLQ with error context, retry counts, and checkpoints.
For Elasticsearch users, this turns silent data loss into an observable, traceable, and recoverable engineering problem.

Who Is ElasticRelay For?

ElasticRelay is particularly well‑suited for teams that:

Have production databases and want to quickly sync data into Elasticsearch for search or analytics
Need to integrate MySQL, PostgreSQL, and MongoDB without maintaining separate solutions
Do not want to introduce heavy infrastructure like Kafka or Flink solely for CDC
Want field governance, masking, and filtering built into the pipeline
Prefer a dedicated service responsible for “database → Elasticsearch” synchronization

Its engineering style is also very clear: a single Go binary, gRPC for internal APIs, JSON‑driven configuration, and multi‑stage Docker builds.
This form factor is especially friendly for small to mid‑sized teams in terms of deployment and operational cost.

A Practical Way to Get Started

If this is your first time working with ElasticRelay, you do not need to understand all of its internal implementation up front. A more practical approach is to first run a minimal end-to-end pipeline: prepare the configuration, fill in the database and Elasticsearch connection details, start the service, and then check whether data is beginning to flow into your index.

Using standalone deployment as an example, the entire process can be reduced to just a few steps:

Create the config, logs, and dlq directories, and copy a sample configuration file as your starting point.
Fill in data_sources, sinks, and jobs in the configuration, linking MySQL, PostgreSQL, or MongoDB to the target Elasticsearch cluster.
Start the service with docker-compose -f docker-compose.elasticrelay.yml up -d.
Verify that both the initial sync and subsequent incremental sync are working by checking the logs, checkpoints, and Elasticsearch index status.

What makes the experience user-friendly is not just the small number of steps, but also the clarity of the configuration model itself. You do not need to set up a full messaging or stream-processing stack first. Instead, you can organize everything around three simple questions: where the data comes from, where it goes, and under what rules it is synchronized.

For many teams, this means they can build a working proof of concept in a very short time. Start with a single table or collection, sync its fields into a target index such as myapp-users, and once the pipeline is confirmed to be stable, gradually expand to more tables, more rules, and more complete data-governance logic. That “start simple, then refine” path is exactly what makes ElasticRelay easy to adopt.

Why Elastic Users Would Care About ElasticRelay

For many Elasticsearch projects, the real challenge is often not how to write queries or design indexes, but how to reliably and continuously synchronize data from upstream business databases.

This is exactly where ElasticRelay shows its value. It is not a general-purpose data platform; instead, it is a lighter, more focused solution built around typical Elasticsearch integration needs. It captures changes from multiple source databases, processes them through bulk writes, rule-based transformations, checkpoint management, and failure recovery, and ultimately forms a synchronization pipeline that can be put into practice.

That is also why it deserves attention from the Elastic community. For teams building search, analytics, or real-time query capabilities, ElasticRelay offers more than just the ability to "write data into ES" - it provides an engineering solution that is easier to deploy, easier to maintain, and better aligned with real-world business scenarios.

Conclusion

From an implementation perspective, ElasticRelay already shows a solid production‑oriented shape: multi‑source ingestion, initial sync, incremental CDC, batch writes, rule‑based transformation, DLQ, and checkpointing are all part of a single main pipeline.

For teams looking to reduce the complexity of Elasticsearch data ingestion, tools like this offer immediate, tangible value.

For Elastic Meetup discussions, ElasticRelay represents more than “yet another sync tool.”

It reflects a growing engineering preference:
Instead of building ever‑larger pipelines, build focused systems that make the database‑to‑Elasticsearch path deep, stable, and understandable.

Looking ahead, two promising directions would be richer observability and a more mature control plane with visual configuration. Even in its current state, however, ElasticRelay clearly demonstrates one idea:

Making Elasticsearch data ingestion simpler is itself a valuable form of innovation.

As a fully open‑source project developed by Shanghai Yogoo Software Co., Ltd., ElasticRelay welcomes more developers and users from the Elastic community to explore, discuss, and contribute.

Project: https://github.com/YogooSoft/elasticrelay
Discussions: https://github.com/YogooSoft/elasticrelay/discussions
Twitter: https://twitter.com/elasticrelay

Why We Chose Go to Rewrite Our DB-to-Elasticsearch Sync Tool

Homer — Sun, 23 Nov 2025 10:38:00 +0000

Why We Chose Go to Rewrite Our DB-to-Elasticsearch Sync Tool

The Challenge: Building a Better CDC Tool

In the modern data landscape, real-time synchronization from databases to search engines has become a critical requirement. Whether you're building e-commerce search, analytics dashboards, or log aggregation systems, you need reliable, fast, and maintainable CDC (Change Data Capture) solutions.

When we started ElasticRelay, we looked at existing solutions like Logstash, Debezium + Kafka Connect, and Apache Flink. While powerful, they often came with significant overhead:

Complex deployment: Multi-service architectures requiring Kafka clusters, Zookeeper coordination, and JVM tuning
Resource intensive: High memory footprint and CPU usage, especially for smaller workloads
Configuration complexity: YAML/JSON configurations that quickly become unwieldy
Operational burden: Multiple moving parts, each with their own failure modes

We decided to build something different: a lightweight, reliable, and developer-friendly CDC tool that just works™.

Why Go? The Technical Decision

After evaluating several languages including Java, Python, and Rust, we chose Go for ElasticRelay's core data plane. Here's why:

1. Goroutines: Built-in Concurrency Without the Complexity

CDC workloads are inherently concurrent. You're reading from multiple database tables, transforming data in parallel, and writing to multiple Elasticsearch indices simultaneously. Go's goroutine model made this natural:

// ElasticRelay's parallel snapshot processing
func (m *ParallelSnapshotManager) Start(ctx context.Context, tables []string) error {
    // Create worker pool
    m.workers = make([]*SnapshotWorker, m.config.WorkerPoolSize)
    for i := 0; i < m.config.WorkerPoolSize; i++ {
        worker := NewSnapshotWorker(i, m)
        m.workers[i] = worker
        go worker.Run(m.ctx)  // Each worker runs in its own goroutine
    }

    // Process table chunks concurrently
    for _, tableName := range tables {
        go m.processTable(tableName) // Parallel table processing
    }

    return nil
}

What would require thread pools, executors, and complex synchronization in Java becomes elegant and readable in Go. Our parallel snapshot processing can handle millions of records across dozens of tables with just a few hundred lines of code.

2. Channels: Elegant Data Pipeline Architecture

CDC systems are essentially data pipelines. Go's channels provided the perfect abstraction for building our processing stages:

type ParallelSnapshotManager struct {
    tableQueue chan *TableTask    // Tables waiting to be processed
    chunkQueue chan *ChunkTask    // Data chunks ready for processing
    resultChan chan *ProcessResult // Completed chunks
}

// Data flows naturally through the pipeline
func (w *SnapshotWorker) Run(ctx context.Context) {
    for {
        select {
        case chunk := <-w.manager.chunkQueue:
            result := w.processChunk(chunk)
            w.manager.resultChan <- result
        case <-ctx.Done():
            return
        }
    }
}

This channel-based architecture makes our system naturally backpressure-aware and resource-bounded. If Elasticsearch is slow, the channels fill up and upstream processors automatically slow down.

3. Single Binary Deployment: DevOps Simplicity

One of Go's killer features for infrastructure tools is single binary deployment:

# Build once, run anywhere
go build -o elasticrelay ./cmd/elasticrelay

# Docker deployment is trivial
FROM scratch
COPY elasticrelay /elasticrelay  
ENTRYPOINT ["/elasticrelay"]

Compare this to a typical Kafka Connect + Debezium setup:

JVM with specific version requirements
Kafka cluster (3+ nodes for production)
Zookeeper ensemble (3+ nodes)
Connect worker nodes
Plugin management and classpath configuration

ElasticRelay runs as a single process with minimal resource requirements. Our users report production deployments running stably on 2-core, 4GB RAM instances handling millions of daily events.

4. Memory Efficiency: Streaming Without the Bloat

JVM-based tools often struggle with memory efficiency due to garbage collection overhead and object allocation patterns. Go's efficient memory model and garbage collector allowed us to build truly streaming processors:

// Stream processing with controlled memory usage
func (w *SnapshotWorker) processChunkStream(chunk *ChunkTask) error {
    // Process in configurable batches to control memory
    batchSize := w.config.BatchSize // Typically 1000-10000 records

    for {
        batch, err := w.fetchBatch(chunk, batchSize)
        if err != nil || len(batch) == 0 {
            break
        }

        // Transform and send immediately - no accumulation
        if err := w.processBatch(batch); err != nil {
            return err
        }

        batch = nil // Help GC
    }

    return nil
}

This approach keeps memory usage constant regardless of table size. We've successfully synchronized tables with 100+ million records while maintaining memory usage under 4GB.

5. Rich Ecosystem: Standing on Giants' Shoulders

Go's ecosystem provided excellent libraries for our specific use case:

go-mysql: Battle-tested MySQL binlog parsing
elastic/go-elasticsearch: Official Elasticsearch client with bulk operations
gRPC-Go: High-performance service communication
Testify: Comprehensive testing framework

// MySQL binlog parsing with go-mysql
syncer := replication.NewBinlogSyncer(replication.BinlogSyncerConfig{
    ServerID: cfg.ServerID,
    Flavor:   "mysql",
    Host:     cfg.DBHost,
    Port:     uint16(cfg.DBPort),
    User:     cfg.DBUser,
    Password: cfg.DBPassword,
})

// Elasticsearch bulk operations
res, err := es.Bulk(
    es.Bulk.WithIndex(indexName),
    es.Bulk.WithBody(bulkBody),
    es.Bulk.WithRefresh("wait_for"),
)

The integration was seamless, and the libraries' Go-idiomatic APIs made our code clean and maintainable.

Real-World Performance: The Numbers Don't Lie

The Go rewrite delivered significant performance improvements:

Metric	Legacy Solution	ElasticRelay (Go)	Improvement
Initial Sync Time	27 hours (100M records)	2-4 hours	85%+ faster
Memory Usage	8-16GB (unbounded)	2-4GB (controlled)	75% reduction
Binary Size	200MB+ (with dependencies)	15MB (static binary)	90% smaller
Cold Start Time	2-3 minutes	5-10 seconds	95%+ faster
Resource Requirements	8 cores, 16GB RAM	2 cores, 4GB RAM	75% reduction

Architecture Highlights: Go-Powered Design Patterns

Graceful Degradation with Interface-Based Design

Go's interfaces enabled us to build a system that gracefully handles failures:

type SinkServiceServer interface {
    BulkWrite(stream pb.SinkService_BulkWriteServer) error
    DescribeIndex(context.Context, *pb.DescribeIndexRequest) (*pb.DescribeIndexResponse, error)
}

// Real implementation
type ElasticsearchSink struct { /* ... */ }

// Fallback implementation for DLQ
type DummySinkServer struct {}

func (d *DummySinkServer) BulkWrite(stream pb.SinkService_BulkWriteServer) error {
    // Immediately fail to trigger DLQ processing
    return fmt.Errorf("sink unavailable - triggering DLQ")
}

When Elasticsearch is unavailable, ElasticRelay automatically routes events to a Dead Letter Queue (DLQ) and continues processing. This resilience-by-default approach prevents data loss during outages.

Context-Driven Cancellation

Go's context package provided elegant cancellation and timeout handling:

func (m *ParallelSnapshotManager) processWithTimeout(
    ctx context.Context, 
    table string,
) error {
    // Create timeout context for this specific table
    tableCtx, cancel := context.WithTimeout(ctx, 30*time.Minute)
    defer cancel()

    select {
    case result := <-m.processTable(tableCtx, table):
        return result
    case <-tableCtx.Done():
        return fmt.Errorf("table %s processing timeout", table)
    case <-ctx.Done():
        return ctx.Err() // Global cancellation
    }
}

This pattern ensures that no operation can hang indefinitely, and cancellations propagate cleanly through the entire system.

The Developer Experience Factor

Beyond performance, Go significantly improved our development experience:

1. Fast Build Times

# Complete rebuild in seconds, not minutes
time make build
real    0m3.245s
user    0m5.234s
sys     0m1.456s

2. Excellent Tooling

go fmt        # Consistent formatting
go vet        # Static analysis
go test -race # Race condition detection
go mod tidy   # Dependency management

3. Cross-Platform Builds

# Build for multiple platforms from one machine
make build-all
# Produces: linux/amd64, darwin/amd64, darwin/arm64, windows/amd64

Challenges and Trade-offs

Go wasn't perfect for every aspect of our system:

1. Error Handling Verbosity

Go's explicit error handling can be verbose:

// Typical Go error handling pattern
config, err := config.LoadMultiConfig(configFile)
if err != nil {
    return fmt.Errorf("failed to load config: %w", err)
}

orchServer, err := orchestrator.NewMultiOrchestrator(grpcAddr)
if err != nil {
    return fmt.Errorf("failed to create orchestrator: %w", err)
}

While verbose, this explicitness helped us build more robust error handling and better observability.

2. Generics Adoption

Before Go 1.18, the lack of generics led to some code duplication. Post-1.18, we've been gradually adopting generics for type-safe collections and algorithms.

3. Dynamic Configuration

Go's strong typing sometimes clashes with the need for dynamic configuration. We solved this with interface-based plugin systems:

type TransformRule interface {
    Apply(record map[string]interface{}) (map[string]interface{}, error)
    Validate() error
}

// Different rule implementations
type FieldRenameRule struct { /* ... */ }
type DataTypeConversionRule struct { /* ... */ }
type CustomScriptRule struct { /* ... */ }

Lessons Learned: Go Best Practices for Infrastructure Tools

1. Start with Interfaces

Define your interfaces first, implementations second. This enables testing, mocking, and graceful degradation patterns.

2. Embrace Channels for Pipeline Architecture

Channels naturally model data flow and provide backpressure handling for free.

3. Use Context Everywhere

Context enables clean cancellation, timeouts, and tracing throughout your system.

4. Design for Single Binary Deployment

Minimize external dependencies and embrace Go's static linking capabilities.

5. Profile Early and Often

Go's built-in profiling tools (go tool pprof) make performance optimization straightforward.

The Road Ahead: Go's Role in ElasticRelay's Future

As ElasticRelay evolves toward supporting PostgreSQL, MongoDB, and advanced data governance features, Go continues to be the right choice:

Performance: Our parallel processing architecture scales linearly with core count
Reliability: Explicit error handling and testing culture reduce production issues
Maintainability: Go's simplicity keeps our codebase approachable for new team members
Ecosystem: Rich libraries for databases, message queues, and cloud services

Conclusion: Go for the Win

Choosing Go for ElasticRelay's rewrite was one of our best technical decisions. The combination of:

Built-in concurrency (goroutines + channels)
Memory efficiency (streaming processing + efficient GC)
Deployment simplicity (single binary)
Developer productivity (fast builds + excellent tooling)
Rich ecosystem (mature libraries for our use case)

...enabled us to build a CDC tool that's 5x faster, 4x smaller, and 10x easier to deploy than traditional solutions.

If you're building infrastructure tools and considering Go, we highly recommend it. The language's design philosophy of simplicity, clarity, and pragmatism aligns perfectly with the needs of reliable, high-performance systems.

Want to try ElasticRelay? Check out our GitHub repository or read our Getting Started guide.

Questions? Join our community discussions or reach out on Twitter.

The ElasticRelay team is passionate about building better data infrastructure tools. Follow our journey as we make real-time data synchronization simple, reliable, and accessible to every developer.

Related Articles:

Tags: #golang #cdc #elasticsearch #dataengineering #opensource #mysql #performance