Forem: Isaac Ojerumu

Designing a Reliable Notification System for 1M+ Users (Push, SMS, Email)

Isaac Ojerumu — Sun, 24 May 2026 00:54:07 +0000

In fintech, notifications are not a “nice-to-have” feature.

They’re part of the product’s trust layer.

If a user transfers money and doesn’t get a confirmation, they panic.
If an OTP arrives 3 minutes late, login fails.
If price alerts come twice, users lose confidence fast.

At small scale, sending notifications feels simple:

Application → Twilio/SendGrid → Done

But once you’re dealing with millions of users, multiple channels, retries, provider outages, and traffic spikes… notification systems become distributed systems problems.

And distributed systems are mostly about handling failure gracefully.

Imagine a fintech platform sending:

transaction alerts
OTPs
portfolio updates
price alerts
security notifications

…to over 1 million users across:

Push notifications
SMS
Email

The challenge isn’t just “sending messages.”

The real challenge is making sure the system:

doesn’t send duplicates
doesn’t silently lose messages
survives provider outages
scales during spikes
remains observable when things go wrong

Here’s the architecture I’d use.

High-Level Architecture

[App Service]
      │
      ▼
[Notification Queue (Redis Streams / SQS / Kafka)]
      │
      ▼
[Worker Pool]
      │
      ▼
[Provider Router]
      │
 ┌────┼───────────────────────────────┐
 ▼    ▼                               ▼
SMS  Email                           Push
 │      │                              │
 ▼      ▼                              ▼
Twilio SendGrid                      FCM/APNs
 │
 ▼
Fallback Providers
(Termii / Mailgun / Direct APNs)
      │
      ▼
[Delivery Log + Idempotency Store]
(PostgreSQL + Redis)

1. Queue-Based Architecture

One of the biggest mistakes teams make early on is sending notifications directly from the API request cycle.

That works… until traffic spikes.

Imagine Black Friday, a crypto market crash, or salary payment day.
Suddenly, millions of notifications need to go out almost at once.

If your application waits for Twilio or SendGrid to respond before returning a response to the user, your entire app becomes hostage to external providers.

That’s dangerous.

Instead, the API should do one thing:

Accept the request quickly and push a notification event into a queue.

From there, worker processes handle delivery asynchronously.

This changes the system completely.

Queues give you:

horizontal scalability
retry capability
backpressure handling
failure isolation

If providers slow down, the queue absorbs the spike instead of crashing your application.

At this scale, queues stop being optional infrastructure.
They become the safety buffer protecting the rest of your system.

Recommended technologies:

Redis Streams
AWS SQS
Kafka (especially for very high throughput systems)

2. Reliability & Idempotency

The hardest problem in notification systems usually isn’t failed sends.

It’s duplicate sends.

Users are surprisingly tolerant of delayed notifications.
They are not tolerant of receiving the same debit alert three times.

Retries are where duplicates usually happen.

Example:

Worker sends SMS
Provider actually succeeds
Network timeout occurs before acknowledgment returns
Worker assumes failure
Worker retries
User receives duplicate SMS

To prevent this, every notification should carry an idempotency_key.

Before sending, workers check:

“Have we already processed this exact notification?”

Example constraint:

UNIQUE(user_id, notification_type, idempotency_key)

This is one of those small architectural decisions that saves massive operational pain later.

Even if retries happen multiple times, the database becomes the final protection layer against duplicates.

Every delivery attempt should also be logged.

Not just successes — everything.

notification_attempts

Including:

provider used
response status
retry count
timestamps
provider error payloads

Because when something goes wrong in production, you want evidence, not guesses.

3. Provider Routing & Fallbacks

A reality every senior engineer eventually learns:

Third-party providers fail more often than you expect.

Twilio can degrade.
SendGrid can throttle requests.
FCM can delay pushes.

The mistake is designing systems that assume providers are always available.

Reliable systems assume failure is normal.

So instead of hardcoding a single provider, introduce a provider routing layer.

Example Routing Strategy

Channel	Primary	Fallback
SMS	Twilio / Termii	Flutterwave SMS
Email	SendGrid	Mailgun
Push	Firebase FCM	Direct APNs

The worker flow becomes:

Attempt primary provider
Detect timeout or failure
Retry intelligently
Automatically fail over if necessary

Users shouldn’t notice your provider had a bad day.

That’s the goal.

4. Retry Strategy

Retries sound simple until they start causing damage.

Bad retry systems can:

spam users
overload providers
amplify outages
generate huge costs

A common mistake is retrying too aggressively.

If Twilio is already struggling, hammering it with thousands of immediate retries only makes things worse.

Instead, use exponential backoff.

Example:

Retry #1 → 30 seconds
Retry #2 → 2 minutes
Retry #3 → 10 minutes

This gives providers time to recover while keeping pressure manageable.

And after maximum retries?

Move the message into a Dead Letter Queue (DLQ).

That queue is basically your “something unusual happened here” bucket.

At that point, engineers should be alerted.

5. Reconciliation Jobs

One subtle issue in distributed systems:

Sometimes providers say “accepted” even though delivery eventually fails.

That creates dangerous blind spots.

A notification may look successful internally while the user never actually receives it.

This is why reconciliation jobs matter.

Every few minutes, background jobs should scan for suspicious states:

Notifications stuck in "pending" for too long

Then:

→ Re-query provider APIs
→ Update delivery status
→ Retry if needed

These jobs quietly save systems from edge cases caused by:

webhook failures
transient outages
network interruptions
delayed provider processing

A lot of reliability engineering is really just building systems that continuously self-correct.

6. User Preferences & Rate Limiting

Good notification systems are not just reliable.

They’re respectful.

Users should control how they’re contacted.

Examples:

disable marketing SMS
mute non-critical notifications after 10PM
push-only preferences
email digests instead of instant alerts

Simple table:

user_notification_settings

…can dramatically improve user experience.

Rate limiting matters too.

Without it, bugs or loops can become expensive very quickly.

Imagine accidentally sending OTPs in a retry loop to thousands of users.

Redis-based limits help protect against this.

Example:

Max 3 SMS/hour/user

That protects:

users from spam
providers from overload
the business from runaway costs

7. Monitoring & Observability

At scale, invisible systems are dangerous systems.

You need to know:

Are queues growing?
Are workers failing?
Which provider is slowing down?
Which region is affected?

The most important metrics are usually boring operational ones.

Golden Signals

queue latency
queue depth
worker error rate
provider response times

Then business-level metrics:

delivery success rate
percentage delivered within 30 seconds
provider costs
user opt-out rates

And finally: alerts.

Example:

Alert if SMS failure rate exceeds 5% for 2 minutes

The earlier you detect degradation, the smaller the incident becomes.

8. Testing Failure Scenarios

The biggest difference between systems that look reliable and systems that are reliable is failure testing.

Because everything works in happy-path demos.

The real question is:

What happens when dependencies misbehave?

One useful strategy is shadow testing.

Route a tiny percentage of production traffic through a new provider and compare results safely.

Example:

compare latency
compare delivery rates
validate formatting consistency

Chaos testing is also incredibly valuable.

Example:

Intentionally fail 10% of Twilio requests in staging

That sounds scary initially.

But it validates whether:

retries actually work
failovers trigger correctly
reconciliation jobs recover messages

Reliable systems are engineered through controlled failure exposure.

Why This Architecture Works

What makes this architecture resilient is that it assumes bad things will happen.

Because eventually:

providers fail
queues spike
retries happen
networks become unreliable

The system survives because reliability is built into the architecture itself.

By combining:

queues
idempotency
retries
fallback providers
reconciliation jobs
observability

…the platform continues operating even during partial outages and heavy traffic spikes.

And in fintech, reliability isn’t just infrastructure quality.

It directly affects user trust.

Tech Stack Example

Component	Suggested Tech
Queue	Redis Streams / Kafka / SQS
Workers	Laravel Queues / Go Workers / Node Consumers
Cache	Redis
Database	PostgreSQL
Monitoring	Prometheus + Grafana
Alerts	PagerDuty / Slack
SMS	Twilio / Termii
Email	SendGrid / Mailgun
Push	Firebase FCM

Final Thoughts

Most notification systems work during normal traffic.

That’s not the hard part.

The hard part is surviving:

provider outages
duplicate retry scenarios
partial failures
sudden spikes in traffic

That’s where architecture starts to matter.

Because users rarely remember the notifications that worked.

They remember the moments when communication failed during something important.

Integrating Flutterwave’s secure payment gateway into your website using PHP

Isaac Ojerumu — Sun, 23 Oct 2022 01:43:31 +0000

Flutterwave is one of the leading payment gateways in Nigeria and Africa. If you have a business that requires secure online payment from customers, then I suggest you try them out. You can collect payments in USD, GBP, EUR, NGN and a host of other African currencies. They have so many payment solutions with which you can complete your payment easily and secured. In this guide, I’ll be showing you how to integrate rave into your website. So let’s dive in.

A. Create a Flutterwave account
If you have not created an account yet, go to https://app.flutterwave.com/register to create an account. If you already have, login with your account details. Navigate to the settings page and click on the API keys tabs to get your public and secret key, we are going to need it.

B. Collect payment information and charge the customer

We have to collect the user’s email, the amount that will be paid and the currency if the payment will not be in naira(NGN). You can provide an input tag for the user to fill in the amount and their email or you can use a hidden input tag if the amount to be paid is static or you already have the user’s info. There are four (4) methods of doing this as seen in Flutterwave documentation;

(i) Flutterwave inline (ii) Flutterwave standard (iii) HTML chekout (iv) Direct charge

Let's look at integrating by Flutterwave standard method using the PHP language. You can refer to the documentation on the other methods: https://developer.flutterwave.com/docs/collecting-payments/overview

Flutterwave standard

Below is a sample form page I created with HTML and CSS (bootstrap) for demonstration. Here the user is to fund his wallet on a telecom (VTU) website.

<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Fund Wallet</title>
    <link rel="stylesheet" href="bootstrap.min.css" />
</head>
<body>
    <div class="container-xxl flex-grow-1 container-p-y">
     <div class="row">
       <div class="col-xl">
         <div class="card mb-4">
           <div class="card-header d-flex justify-content-between align-items-center">
             <h5 class="mb-0">Fund account with Flutterwave</h5>
            </div>
            <div class="card-body">
              <form method="post" action="process_payment.php" id="form">                  
                <div class="mb-3">
                  <label class="form-label" for="phone-number">Amount*</label>
                   <input type="tel" name="amount" class="form-control" required>
                </div>

                <div class="mb-3">
                  <input type="email" name="email" class="form-control"  required>
                </div>
                <div class="mb-3">
                  <input type="tel" name="phone" class="form-control" required>
                </div>

                <button type="submit" class="btn btn-primary action" name="pay"> Pay</button>
              </form>
            </div>
          </div>
        </div>

      </div>          
   </div>
  <script src="jquery.min.js"></script>
  <script src="bootstrap.min.js"></script>
</body>
</html>

Here the user information is submitted in a form to a backend script. I named my own script process_payment.php. This script will call the https://api.ravepay.co/flwv3-pug/getpaidx/api/v2/hosted/pay endpoint.

<?php

if (isset($_POST['transfer'])) {
  extract($_POST);

   $curl = curl_init();

    $customer_email = $email; 
    $amount_pay = $amount;
    $currency = "NGN";
    $txref = "rave" . uniqid(); // ensure you generate unique references per transaction.
    // get your public key from the dashboard.
    $PBFPubKey = "FLWPUBK-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-X"; 
    $redirect_url = "https://www.fltwvapp.herokuapp.com/redirect.php?email=$email&amount=$amount_pay"; // Set your own redirect URL

     curl_setopt_array($curl, array(
      CURLOPT_URL => "https://api.ravepay.co/flwv3-pug/getpaidx/api/v2/hosted/pay",
      CURLOPT_RETURNTRANSFER => true,
      CURLOPT_CUSTOMREQUEST => "POST",
      CURLOPT_POSTFIELDS => json_encode([
        'amount'=>$amount_pay,
        'customer_email'=>$customer_email,
        'currency'=>$currency,
        'txref'=>$txref,
        'PBFPubKey'=>$PBFPubKey,
        'redirect_url'=>$redirect_url,
      ]),
      CURLOPT_HTTPHEADER => [
        "content-type: application/json",
        "cache-control: no-cache"
      ],
    ));

    $response = curl_exec($curl);
    $err = curl_error($curl);

    if($err){
      // there was an error contacting the rave API
      die('Curl returned error: ' . $err);
    }

    $transaction = json_decode($response);

    if(!$transaction->data && !$transaction->data->link){
      // there was an error from the API
      print_r('API returned error: ' . $transaction->message);
    }

    // redirect to page so User can pay

    header('Location: ' . $transaction->data->link); 
}
?>

Assign the email and amount submitted by the POST request to variables. Make sure to validate those variables to make sure that the user has not done anything dubious. Generate a unique transaction reference. It is advisable to store this reference, amount, and any other information you may wish to collect about the product or service to be paid for in a database and then set the transaction status to pending. Paste in your public key and specify your redirect URL.

Note: Your redirect URL must be hosted on a web host

The curl is used to call the https://api.ravepay.co/flwv3-pug/getpaidx/api/v2/hosted/pay. The amount, email, phone, currency, unique reference, your public key and redirect URL is passed into it as well and then executed. Use json_decode to decode the response returned by the API and store in a transaction variable. Check the transaction object to see if it has the data property and if the link that will process the payment also exists(this link is from Flutterwave and not the one you provided earlier. don’t worry about it). We are now redirected to the link which is the checkout page for our card details.

When the transaction is completed, Flutterwave will now redirect to the URL provided earlier.

C. Verify the payment status

After the transaction has been completed, use the reference returned in the redirect URL to check for the status of the transaction. Below is the redirect.php file which does the verification for us.

<?php

if (isset($_GET['txref'])) {
  $ref = $_GET['txref'];
  $amount = $_GET['amount']; //Get the correct amount of your product
  $email = $_GET['email'];
  $currency = "NGN"; //Correct Currency from Server

  $query = array(
    "SECKEY" => "FLWSECK-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-X",
    "txref" => $ref
  );

  $data_string = json_encode($query);

  $ch = curl_init('https://api.ravepay.co/flwv3-pug/getpaidx/api/v2/verify');                                                                      
  curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
  curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);                                              
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
  curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));

  $response = curl_exec($ch);

  $header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
  $header = substr($response, 0, $header_size);
  $body = substr($response, $header_size);

  curl_close($ch);

  $resp = json_decode($response, true);

  $paymentStatus = $resp['data']['status'];
  $chargeResponsecode = $resp['data']['chargecode'];
  $chargeAmount = $resp['data']['amount'];
  $chargeCurrency = $resp['data']['currency'];

  if (($chargeResponsecode == "00" || $chargeResponsecode == "0") && ($chargeAmount == $amount)  && ($chargeCurrency == $currency)) {
    // transaction was successful...
  // please check other things like whether you already gave value for this ref
    // if the email matches the customer who owns the product etc
    //Give Value and return to Success page
    //   var_dump($resp);
    header('location: success.html');
  } else {
    //Dont Give Value and return to Failure page
    // var_dump($resp);
    header('location: error.html');
  }
}
else {
  die('No reference supplied');
}

?>

Use $_GET to collect the reference and other values passed to the redirect URL. Encode your secret key gotten from your dashboard and your txRef as JSON and call the https://api.ravepay.co/flwv3-pug/getpaidx/api/v2/verify endpoint with it to verify the payment. Decode the response returned. Check the chargecode, amount and currency to make sure they synchronize. Use the txRef to update the status of the transaction, redirect the user to the success or error page or do anything you’d like to do after a successful or failed transaction.

D. Use webhook to ensure stability

Webhooks are used for notifying our server that a payment event has just occurred. A web application implementing Webhooks will POST a message to a URL when certain things happen. Webhooks are optional. Without it, everything will work but it makes our application stable.

Let’s say that a user completes his payment successfully but maybe due to poor network signal was unable to get to your designated URL, or maybe closes the browser or the payment tab mistakenly, it will be difficult to ascertain the status of his payment unless we visit the dashboard manually and crosscheck. But with webhooks, you can still know if the transaction succeeded or failed and then you can update your transaction table and give or deny the user access to value being paid for whenever he comes back.

Another case is when the user is subscribing to a service and will be billed periodically, whenever the user is billed, the status of the transaction is sent to your webhook URL. You can now keep offering the service to the user or stop if he wasn’t billed. Other examples are for events — like getting paid via mobile money or USSD where the transaction is completed outside your application.

Enable webhook from your dashboard

Below is a sample webhook from the documentation page.

<?php

// Retrieve the request's body
$body = @file_get_contents("php://input");

// retrieve the signature sent in the reques header's.
$signature = (isset($_SERVER['verif-hash']) ? $_SERVER['verif-hash'] : '');

/* It is a good idea to log all events received. Add code *
 * here to log the signature and body to db or file       */

if (!$signature) {
    // only a post with rave signature header gets our attention
    exit();
}

// Store the same signature on your server as an env variable and check against what was sent in the headers
$local_signature = getenv('SECRET_HASH');

// confirm the event's signature
if( $signature !== $local_signature ){
  // silently forget this ever happened
  exit();
}

http_response_code(200); // PHP 5.4 or greater
// parse event (which is json string) as object
// Give value to your customer but don't give any output
// Remember that this is a call from rave's servers and 
// Your customer is not seeing the response here at all

$response = json_decode($body);
if ($response->body->status == 'successful') {
    # code...
    // Update your database and set the transaction status to successful
}
exit();

Note: Your redirect page and webhook page might be doing the same thing. In either your redirect page or webhook, you might first check to see if the payment status has been updated to avoid giving value twice.