Forem: Ayodeji Ogundare

Building a GlobalStream: Replication Challenges and Optimizations

Ayodeji Ogundare — Fri, 10 Oct 2025 10:50:13 +0000

By Gaurav Singh & Luciano Sabença, Streaming Platform Team

Global data streaming requires careful tradeoffs between performance, compliance, and reliability. At Adyen, replication is essential: data must remain close to producers to reduce latency and meet regulations, while also being centralized for analytics.

In this blog, we share our experience optimizing MirrorMaker 2 for large-scale, cross-continent Kafka replication, the parameter tuning we applied, and the hidden behaviors in Kafka Connect that ultimately unlocked the required throughput.

Why Replicate?

There are many things you need to take into account when designing a data streaming platform for a company such as Adyen. One of the most important aspects of it is data location and replication. Due to performance, reliability, and compliance, keeping data close to producers is usually a good idea. After all, you don't want to wait in line at the cashier to confirm the payment for that fat burger you just bought on that nice beach in Australia while your payment was being sent to Europe. However, it's also a common practice to have all data in a centralized cluster(s) in centralized locations for analytical purposes. These two need a bridge, which is where mirroring comes in.

Mirror Maker

When it comes to copying data between Kafka clusters, Mirror Maker is the default open source tool. MirrorMaker 2 is the current version, and it's built on top of Kafka Connect, a platform designed to make it easier to integrate Kafka with other tools such as databases and distributed file systems, and - why not? - another Kafka cluster. Setting up Mirror Maker is fairly straightforward, but tuning it to the performance Adyen requires isn't. Let's go over the journey into the depths of Kafka Connect and parameter tuning to find a solution!

Playing the Volume Game

Kafka is a highly flexible tool and serves as the backbone for many different architectural patterns. It can be used for batch processing, real time processing, or any combination of these two extremes. With default parameters, Kafka producers and consumers are optimized for low-latency use cases: batch sizes are not large, and records are sent immediately to the server (i.e., linger.ms = 0). Given that our volume is on the order of several hundred thousand messages per second, any optimization has a huge impact on replication latency. In our case, the default settings were not good enough to keep the latency bounded within a few seconds; in fact, our replication latency was on the order of minutes. Since we are transferring this data across continents, we don't expect very low latency for replication, but such high latency indicates that we were at the maximum possible throughput for this configuration, so we needed to revisit those parameters.

We made the following adjustments on the producer side of MirrorMaker:

batch.size: Increased from the default 16384 to 409600. This leads to larger batches per request, and less requests to replicate the same number of messages
linger.ms: Increased from 0 to 200. This parameter sets how long the Kafka driver can wait before dispatching the messages. 0 means send immediately. Giving more time to the driver allows it to batch more messages in the same batch.
compression.type: Changed from 'none' to 'zstd'. Activating compression means more messages per batch.
These changes didn't yield the desired results. There was another bottleneck. We then turned our attention to the consumer parameters. After all, MirrorMaker is both a consumer from the source cluster and a producer to the destination.

We adjusted:

fetch.min.bytes: Increased from 1 to 5000. So we wait a bit more to ensure that more data is available for consumption, thus decreasing the number of consumer requests and optimizing throughput.
max.fetch.records: Increased from 500 to 2000 with the objective of fetching more records per requests and sending more requests to the producer.

The goal here was clear: have the consumers fetch as much data as possible at once and send it in very large batches to the destination cluster. The idea is to optimize network traffic and amortize large distance latencies when measured per unit of replicated byte.

Despite these efforts, the throughput remained stubbornly the same. Each time we restarted with new parameters, the record rate would spike briefly before settling back to its previous baseline, indicating that our changes were not having a lasting impact. This led us to believe there was a deeper, underlying issue we were missing.

The Missing Piece

There was, however, a missing piece of the puzzle to make it work as we expect: how Kafka Connect changes are persisted and saved to the nodes. As we mentioned before, MM2 is built on top of Kafka Connect. Kafka Connect uses some internal Kafka topics to store and persist its state and configuration. This gives us three possible layers of configuration:

Kafka driver's parameters
Mirrormaker configuration
Internal Kafka Connect state

We needed to be sure that whenever any configuration was changed in any of these three places it would be reflected in the others. It soon became clear that this wasn't really the case: whilst our MM2 consumer was able to consume a quite decent amount of data, the producer wasn't really able to keep the throughput high and wasn't creating big enough batches for our use-case.

After digging a bit into the internal configuration topics and on MM docs, we understood the root cause: configuration changes on the connectors are not updated while any instance of the connector is up and running. Instead, they are read from the internal mm configuration topics and applied to the connectors. This was our case: we are running MM2 in distributed mode, so we didn't stop all nodes and start them again. Although the behaviour makes sense, it prioritizes consistency when the application is running; it's not a very intuitive behavior and it almost led us to explore weird paths in our search for better throughput.

The Results: A Tale of Two Graphs

The results were dramatic once we understood the necessity of a complete restart. As seen in the graph below, the number of records per request skyrocketed. We were also able to validate via logs and via the data in the topic that the parameters being used by the producers and consumers were the same as we originally intended.

As a consequence of those parameters, the total number of requests sent by the producer dropped significantly and each request now was able to replicate more messages.

This experience was a valuable lesson in the intricacies of managing a distributed system like Kafka Connect. Understanding the operational details of how configurations are managed was the key to unlocking the needed performance. This journey highlights that sometimes the most significant gains come not from simply tweaking parameters, but from a deeper understanding of the tools themselves.

Guide: Upgrade and customization for Salesforce Commerce Cloud cartridge

Ayodeji Ogundare — Fri, 01 Sep 2023 13:52:28 +0000

In this guide, discover the importance and process of upgrading your Salesforce Commerce Cloud (SFCC) cartridge to the latest version to create a hassle-free integration and seamless checkout experience benefitting both you and your customers. We'll highlight best practices for customizations and walk you through the process of adding custom code to a new cartridge, ensuring a smooth transition to the latest releases.

As an online business owner, ensuring a smooth and seamless checkout experience for your shoppers is vital to success. To achieve this, customizations are often necessary to tailor your checkout process to meet specific needs. To take advantage of new features, bug fixes, and security patches, you’ll need to stay up-to-date with the latest cartridge version.

This guide is specifically tailored to users of the Salesforce Commerce Cloud Storefront Reference Architecture (SFRA). If your e-commerce platform does not utilize SFRA, some of the outlined steps and procedures may not be directly applicable.

The benefits of upgrading your cartridge

Upgrading your cartridge to the latest version offers several benefits that can significantly enhance your checkout experience. Here's why you should consider upgrading:

Access to new features: Each cartridge release comes with exciting new features that can improve the overall functionality and usability of your checkout process. This ensures that you offer your customers the best shopping experience.
Bug fixes and security patches: As with any software, cartridges might have bugs or security vulnerabilities. Regular updates and upgrades address these issues, not only ensuring that your checkout process remains secure, but also improving the authorization rate.
Compatibility with integrations: Third-party integrations and platforms continually evolve, and cartridge updates often ensure compatibility with the latest versions of these integrations. Upgrading your cartridge reduces the risk of conflicts and ensures a seamless connection with your preferred tools.
Continued Cartridge support: We provide support and assistance for the latest versions, making it easier to troubleshoot issues and seek guidance whenever needed.

Customization guide for smoother upgrade (SFRA)

Merchants use customizations to communicate their brand value to shoppers. Customizations are a great way to offer a unique shopper experience. Unfortunately, if customizations are implemented inside of the cartridge, they obscure future cartridge upgrades and introduce technology bottlenecks. We've created a 4-minute video to help you decouple your customizations and upgrade a customized cartridge to the latest version. You can skip the video if you prefer a step-by-step text guide.

Step 1: Add custom code to a new cartridge

To make future upgrades and troubleshooting easier, add your custom code to the int_custom_cartridge folder in your src/cartridges directory. If this folder is not available in your version, create a new folder and name it; int_custom_cartridge and add your custom code. This setup will allow you to temporarily remove custom code during upgrades or troubleshooting.

# Add your custom code to the int_custom_cartridge folder
# Modify the code as needed to customize the checkout experience. For example, in JavaScript:

# File: int_custom_cartridge/scripts/customization.js

function customizeCheckout() {
  // Your custom code here
  console.log("Custom checkout code executed.");
}

Step 2: Include custom cartridge in package.json file

Update the name property in the package.json file to int_custom_cartridge or the name you chose for your custom cartridge.

# Update package.json to support the custom cartridge
# File: int_custom_cartridge/package.json
{
  "name": "int_custom_cartridge",
  "version": "1.0.0",
  "description": "Custom cartridge for checkout customization",
  // Other package.json properties
}

# Update root package.json to contain the new cartridge in the build step
# Make sure you are in the root directory of your project and run

npm install
npm run build

Step 3: Add the new cartridge to your cartridge path

In the Business Manager, navigate to Administration > Sites > Manage Sites > [yourSite] > Settings. In the Cartridges field, add the new cartridge before the Adyen cartridges. Click “Apply” to save your changes.

Step 4: Modify or add end-to-end tests (optional)

To ensure your custom code works correctly, modify or add end-to-end tests as needed. Running these tests will validate the functionality of your customizations.

Upgrade guide for different integrations

Depending on your integration type (Default or Customized), the upgrade process varies:

Default integration

Check which SFRA version is required for the upgrade and migrate if necessary, aiming to use the highest available SFRA version
Download the desired Adyen cartridge version from GitHub
Transpile, compile, and upload the compatible, auto-generated code using npm run build
Re-upload the metadata

# Download the desired Adyen cartridge version from GitHub
# Replace '23.1.0' with the version you want to upgrade to
git clone https://github.com/adyen/adyen-salesforce-commerce-cloud.git --branch 23.1.0

# Re-upload the metadata
# Make sure you are in the root directory of your project
cd adyen-salesforce-commerce-cloud

# Transpile, compile, and upload the compatible, auto-generated code
npm run build

# Zip the site_import folder
zip -r adyen_integration.zip package/metadata/site_import/sites

# In Business Manager, go to Administration > Site Development > Site Import & Export
# Import the zipped file (site_import.zip)

Customized integration

For a quick demo of how to upgrade the SFCC cartridge with customization, watch this 2 mins video.

Steps:

Extract any custom code from the cartridge and place it in the custom cartridge (int_custom_cartridge) folder. Create a new folder if not available in your version.
Check SFRA version requirements and migrate if needed
Download the desired Adyen cartridge version from GitHub
Update the package.json as explained above and run npm run build
Re-upload the metadata
Update the cartridge path in the Business Manager to include your customizations

Conclusion

In today's competitive e-commerce landscape, the checkout experience can make or break a customer's decision to complete a purchase. As an online business owner, customizing your checkout process is essential to cater to your unique requirements and create a branded experience that resonates with your audience. However, these customizations must not come at the cost of missing out on crucial updates and improvements.

By following this guide to upgrading your cartridge to the latest version, you can strike the perfect balance between customization and innovation. Embrace the latest features, bug fixes, and security patches to ensure a seamless and secure checkout journey for your shoppers.

Upgrading your cartridge to the latest version is more than just a technical necessity - it's an investment in your business's success. Stay ahead of the competition by accessing new capabilities that improve the functionality of your checkout process and enhance customer satisfaction. Take your e-commerce venture one step closer to excellence.

So, are you ready to unlock the full potential of your checkout process? Your dedication to providing the best checkout experience will not only increase conversion rates but also foster customer loyalty.

Upgrade today and let your checkout experience set the gold standard for online shopping!