Forem: Kiran Rongali

What is Temporal.AI and How it helps in Building Reliable Workflows in .NET

Kiran Rongali — Thu, 12 Feb 2026 16:25:06 +0000

Building Reliable Workflows in .NET with Temporal

Modern applications rarely perform just one simple action. A single business process might involve multiple microservices, APIs, databases, and external systems—and it might take minutes, hours, or even days to complete. Handling failures, retries, timeouts, and restarts in these long-running processes is one of the hardest parts of backend development.

This is where Temporal comes in.

Temporal is a workflow orchestration platform that helps you build reliable, fault-tolerant, long-running workflows using normal application code. With the Temporal .NET SDK, you can write workflows in C# while Temporal takes care of state management, retries, recovery, and scalability

What Problem Does Temporal Solve?

In traditional .NET systems, long-running or multi-step processes often rely on:

Background jobs
Message queues
Cron jobs or schedulers
Custom retry and state management logic
Database tables to track progress

This approach quickly becomes complex and fragile:

What happens if the service crashes mid-process?
How do you resume exactly where you left off?
How do you avoid duplicate processing?
How do you handle retries, timeouts, and compensation logic cleanly?

Temporal solves these problems by making your workflows durable, restartable, and fault-tolerant by design.

What Is Temporal, in Simple Terms?

Temporal is a workflow engine that runs your business processes reliably, even across failures, restarts, and long delays.

You write workflows in code (C# in .NET), and Temporal:

Persists the workflow state
Automatically retries failed steps
Resumes execution after crashes
Handles timeouts and delays
Ensures exactly-once execution semantics for workflow logic

Your code looks like normal code—but it runs with enterprise-grade reliability.

Key Concepts in Temporal
1-> Workflows

Workflows define the business process. For example:

Order processing
User onboarding
Data integration pipeline
Payment and refund flow
Multi-step approval process

Workflows are:

Deterministic
Durable
Can run for seconds, hours, or months
Automatically resumed after failures

2-> Activities

Activities are the actual work:

Call an API
Write to a database
Send an email
Process a file
Run a validation

Activities can fail and be retried automatically based on policies you define.

3-> Workers

Workers are .NET services that:
Poll Temporal for work
Execute workflows and activities
Report results back to Temporal

4-> Temporal Server

The Temporal server:
Stores workflow state
Tracks history
Handles retries, timers, and scheduling
Guarantees reliability and consistency

Why Use Temporal in .NET Applications?

Reliability by Default
If your .NET service crashes, restarts, or scales down:

Your workflow does not break
It resumes exactly where it left off

Built-in Retries and Timeouts
No more custom retry loops everywhere:

Configure retry policies per activity
Temporal handles backoff, limits, and failures

Long-Running Processes Are Easy
You can:

Wait for hours or days
Wait for human input
Wait for external systems
Continue without losing state

Clean Business Logic

Your workflow code stays:

Readable
Linear
Maintainable Instead of being split across queues, tables, and cron jobs.

A .NET-Oriented Example Use Case

Imagine a data integration pipeline in .NET:

Fetch data from a partner API
Validate and transform the data
Save it to your database
Call another system
If any step fails, retry with backoff
If the service crashes, resume from the last successful step

With Temporal:

Each step is an Activity
The whole flow is a Workflow
Temporal handles:

-> Retries
-> State persistence
-> Crash recovery
-> Timeouts
-> Exactly-once workflow execution

You don’t need to write custom “resume logic” or “status tables”.

How Temporal Fits into the .NET Ecosystem

Temporal works great with:

ASP.NET Core APIs
Background services / workers
Microservices architectures
Azure / AWS / GCP deployments
Message queues and event-driven systems
Data integration and ETL pipelines

Common patterns:

ASP.NET API starts a Temporal workflow
.NET worker executes activities
Workflow coordinates calls to multiple services
Temporal ensures reliability and consistency

Temporal vs Traditional Approaches

Traditional Approach	Problems	Temporal
Background jobs	Hard to resume state	Durable workflows
Message queues	You manage retries & state	Built-in reliability
Cron jobs	No workflow state	Full workflow history
Custom orchestration	Complex & error-prone	Simple, code-based orchestration

When Should You Use Temporal?

Temporal is a great fit when you have:

Long-running business processes
Multi-step workflows across services
Complex retry and failure handling needs
Integration pipelines
Distributed transactions (Saga pattern)
Human-in-the-loop workflows
Critical processes that must not break or lose state

If your system is “just a simple CRUD app,” you might not need it. But for enterprise workflows and integrations, Temporal can be a game-changer.

Conclusion

Temporal brings a powerful idea to .NET development: write normal code, get enterprise-grade reliability for free.

With the Temporal .NET SDK, you can build:

Durable workflows
Fault-tolerant integrations
Scalable background processes
Clean, maintainable orchestration logic

Instead of fighting failures, retries, and restarts, you let Temporal handle them—so you can focus on business logic, not plumbing.

Building Reliable and Scalable Data Integration Pipelines

Kiran Rongali — Fri, 06 Feb 2026 18:59:36 +0000

Building Reliable and Scalable Data Integration Pipelines

In today’s digital world, organizations rarely rely on a single system. Data flows continuously between applications, databases, cloud services, partners, and analytics platforms. Making sure this data moves accurately, securely, and efficiently is the job of a data integration pipeline.

A well-designed data integration pipeline is not just about moving data from Point A to Point B—it’s about ensuring quality, performance, scalability, and reliability across the entire data journey.

What Is a Data Integration Pipeline?

A data integration pipeline is an automated process that:

Extracts data from one or more source systems

Transforms the data into the required format or structure

Loads the data into a target system such as a database, data warehouse, API, or analytics platform

This pattern is often called ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform), depending on where transformations happen.

Common use cases include:

Syncing data between business applications

Feeding data into reporting and analytics systems

Integrating partner or third-party data

Migrating data between systems

Powering real-time or near-real-time workflows

Core Components of a Data Integration Pipeline

Data Sources

These can be:

Databases (SQL Server, Oracle, PostgreSQL, etc.)

APIs and web services

Files (CSV, JSON, XML)

Message queues and event streams

SaaS applications (CRM, ERP, billing systems)

Extraction Layer

This layer is responsible for:

Connecting to source systems

Pulling data in batches or streams

Handling authentication, pagination, and rate limits

Detecting changes (full load vs incremental load)

Transformation Layer

This is where data is:

Cleaned (remove duplicates, fix formats, handle nulls)

Validated (check data types, ranges, mandatory fields)

Mapped (convert source fields to target schema)

Enriched (join with other data, add derived fields)

Aggregated or filtered

Good transformation logic ensures data quality and consistency across systems.

Loading Layer

The final step writes data to:

Data warehouses or data lakes

Operational databases

Search indexes

Downstream applications or APIs

This layer must handle:

Bulk inserts vs upserts

Idempotency (safe re-runs)

Transaction management

Error handling and retries

Batch vs Real-Time Pipelines
Batch Pipelines

Run on schedules (hourly, daily, weekly)

Process large volumes of data at once

Simpler to design and maintain

Ideal for reporting, analytics, and historical processing

Real-Time (Streaming) Pipelines

Process data as it arrives

Lower latency

More complex architecture

Ideal for monitoring, alerts, personalization, and event-driven systems

Many modern systems use a hybrid approach: real-time for critical data, batch for heavy processing and analytics.

Performance and Scalability Considerations

As data volume grows, performance becomes critical. This is where understanding time and space complexity really matters.

Good practices include:

Streaming or batching instead of loading everything into memory

Avoiding nested loops over large datasets

Using indexes, hash sets, or dictionaries for fast lookups

Parallelizing work where possible

Minimizing network calls by batching requests

Processing only changed data (delta loads)

A pipeline that works for 10,000 records may fail or become painfully slow at 10 million if not designed properly.

Reliability and Error Handling

Production-grade pipelines must expect failures:

Network timeouts

API rate limits

Bad or unexpected data

Partial system outages

Key reliability patterns:

Retries with backoff

Dead-letter queues or error tables

Checkpointing and resume capability

Idempotent processing (safe to re-run)

Detailed logging and monitoring

Alerts for failures and data quality issues

A good pipeline is not one that never fails—it’s one that fails safely and recovers gracefully.

Security and Compliance

Since pipelines often move sensitive data, security is critical:

Encrypt data in transit and at rest

Secure credentials using vaults or managed identities

Apply least-privilege access

Mask or tokenize sensitive fields where needed

Maintain audit logs and data lineage

Tools and Technologies

Data integration pipelines can be built using:

Custom code (.NET, Java, Python, Node.js)

ETL/ELT tools (SSIS, Azure Data Factory, Informatica, Talend)

Streaming platforms (Kafka, Azure Event Hubs)

Cloud-native services (AWS Glue, Azure Synapse, GCP Dataflow)

Or a combination of these

The “best” tool depends on scale, complexity, budget, and team skills.

Conclusion

Data integration pipelines are the backbone of modern digital systems. A well-designed pipeline ensures that data is:

Accurate

Timely

Secure

Scalable

Reliable

By focusing not just on moving data, but on performance, quality, and resilience, organizations can build integration platforms that support growth, analytics, and real-time business decisions with confidence.

🚀 Accelerating Cloud Applications with Redis and Cloud-Based Caching Techniques

Kiran Rongali — Thu, 19 Jun 2025 20:22:10 +0000

🔍 Introduction
Modern cloud-native applications demand low-latency, high-throughput access to data. To meet these performance expectations, caching is a foundational technique used to temporarily store frequently accessed data in memory, reducing expensive round trips to databases or APIs.

One of the most widely adopted caching solutions in cloud environments is Redis, particularly in Azure through Azure Cache for Redis. But various cloud providers offer a rich landscape of caching options beyond Redis.

In this article, we’ll explore:

How Redis Cache works in Azure
When and why to use caching
Additional caching techniques in Azure
Caching in other cloud platforms (AWS, GCP)
Best practices for cloud caching

🌐 Redis Cache in Azure
⚙️ What is Azure Cache for Redis?
Azure Cache for Redis is a fully managed, open-source in-memory data store based on Redis. It’s built to handle:

Session state caching
Database query results
Static content (e.g., configuration, feature flags)
Rate-limiting and pub/sub messaging

💡 Key Features:

Geo-replication
Zone redundancy
Private endpoints and VNET integration
Enterprise tiers with Redis on Flash (for large datasets)

🧪 Sample Use Case: .NET App Integration
var cache = ConnectionMultiplexer.Connect("yourredis.redis.cache.windows.net:6380,password=yourPassword,ssl=True"); var db = cache.GetDatabase(); await db.StringSetAsync("key", "value"); string value = await db.StringGetAsync("key");

You can integrate Redis into a .NET Core or ASP.NET app for caching controller responses, user sessions, or computed results.

🧰 Other Caching Techniques in Azure

Technique	Description	Best Use Case
In-Memory Caching (MemoryCache)	Caching data in application memory (e.g., .NET `MemoryCache`)	Lightweight apps, single-instance services
Blob Cache/CDN	Use Azure Blob Storage + Azure CDN to cache static files	Media, documents, web assets
Output Caching	ASP.NET Output/Response caching	Web pages or API results
SQL Server Query Caching	Caching execution plans or result sets	Repetitive, read-heavy database queries

🌩️ Caching Across Other Cloud Platforms
🔷 AWS

Amazon ElastiCache (supports Redis and Memcached)
CloudFront (CDN caching)
DynamoDB Accelerator (DAX) – in-memory cache for DynamoDB

🔶 Google Cloud (GCP)

Cloud Memorystore (for Redis and Memcached)
Cloud CDN – integrated with Google Cloud Storage & Load Balancer
App Engine Memcache API

🛠️ Other Tools & Patterns

Local distributed cache (e.g., NCache for .NET, AppFabric legacy)
Hybrid cache (local memory backed by cloud store)
Write-through / write-behind cache for consistency with databases

✅ Caching Best Practices in the Cloud

Use TTL (time-to-live) to prevent stale data
Use cache aside pattern to populate cache on demand
Avoid caching sensitive data in shared caches
Partition large cache sets to reduce eviction pressure
Enable diagnostics and monitoring (Azure Monitor, Prometheus, etc.)

🧭 Conclusion
Caching remains one of the most effective techniques to boost performance, reduce costs, and scale applications in the cloud. Azure Cache for Redis is a robust option for .NET and other workloads, but the broader cloud ecosystem offers various other tools tailored to different caching needs.

By choosing the right caching strategy—whether distributed, local, or CDN—you can ensure your cloud applications are both performant and resilient.

Azure vs AWS vs Google Cloud: Which Cloud Platform is Right for You?

Kiran Rongali — Tue, 10 Jun 2025 20:00:03 +0000

As cloud computing becomes the backbone of modern IT infrastructure, businesses are increasingly faced with a critical question: Which cloud provider should we choose? The top three contenders—Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP)—each offer powerful capabilities, but choosing between them depends on your organization's goals, tech stack, and long-term vision.

Here’s a comprehensive comparison to help you make the best decision.

⚖️ Compute Services

AWS: EC2, ECS, EKS, Lambda
Azure: Virtual Machines, AKS, Azure Functions
GCP: Compute Engine, GKE, Cloud Functions

Takeaway: AWS offers the broadest options, Azure integrates best with Microsoft tools, and GCP shines with Kubernetes (GKE).

📀 Storage Services

AWS: S3, EFS, Glacier
Azure: Blob Storage, Files, Archive
GCP: Cloud Storage, Filestore, Archive

Takeaway: All three offer robust storage. AWS S3 is industry-standard, Azure is strong for enterprises, and GCP has excellent performance-tier options.

🧐 AI & Machine Learning

AWS: SageMaker, Comprehend, Polly
Azure: Cognitive Services, Azure Machine Learning
GCP: Vertex AI, AutoML, Vision API

Takeaway: GCP leads with intuitive and powerful ML tools, AWS offers breadth, and Azure is enterprise-friendly.

🚗 Databases

AWS: RDS, DynamoDB, Redshift
Azure: SQL Database, Cosmos DB, Synapse
GCP: Cloud SQL, Firestore, BigQuery

Takeaway: AWS has broad database support. Azure is optimal for SQL workloads. GCP's BigQuery is top-tier for analytics.

⚙️ DevOps & Tooling

AWS: CodePipeline, CloudFormation
Azure: Azure DevOps, GitHub Actions
GCP: Cloud Build, Deployment Manager

Takeaway: Azure DevOps stands out for integration. AWS is modular and flexible. GCP is strong in modern CI/CD.

🔐 Security & Identity

AWS: IAM, KMS, GuardDuty
Azure: Azure Active Directory, Key Vault
GCP: IAM, Cloud KMS, Security Command Center

Takeaway: Azure shines for identity (AAD), AWS offers granular security, GCP emphasizes security by design.

💸 Pricing Models

AWS: Pay-as-you-go, Reserved Instances, Spot Pricing
Azure: Hybrid Use Benefit, Reserved VM Instances
GCP: Sustained and Committed Use Discounts

Takeaway: GCP is most transparent with pricing. AWS offers the most options. Azure has discounts for Microsoft users.

🌐 Global Presence

AWS: 33+ regions, 500+ edge locations
Azure: 60+ regions, 200+ edge locations
GCP: 40+ regions, expanding rapidly

Takeaway: Azure leads in global coverage. AWS has the deepest infrastructure. GCP is growing steadily.

📈 Summary Table

🎯 Final Thoughts

Each cloud provider has unique strengths:

AWS is best for startups and enterprises needing a wide variety of services and scalability.

Azure is ideal for enterprises already invested in Microsoft tools.

GCP is perfect for organizations focused on AI, machine learning, and analytics.

For many businesses, multi-cloud or hybrid cloud strategies provide the best of all worlds

🔧 Why Support Teams Rely on Log Analytics, Bastion, and Firewalls in Azure

Kiran Rongali — Tue, 10 Jun 2025 14:11:40 +0000

In the fast-paced world of cloud infrastructure, support and operations teams play a crucial role in maintaining stability, security, and performance. Whether it's investigating a failed deployment, tracing suspicious activity, or accessing a virtual machine for emergency diagnostics — the right tools can make all the difference.

Here’s why Log Analytics, Azure Bastion, and Azure Firewall have become essential tools in every support engineer’s toolbox.

Log Analytics – The Eyes of Your Cloud
Azure Log Analytics is the go-to platform for collecting and analyzing telemetry data from across your environment. It centralizes logs from virtual machines, apps, networks, and services into one place — searchable with powerful queries.

Real-World Example:
A support engineer investigates why an app went down at midnight. Using Log Analytics, they correlate app service logs, backend SQL performance data, and CPU metrics — all in one query. Issue found. Fixed. Documented.

Why Support Teams Love It:

Real-time diagnostics using KQL
Custom alerts on resource health and anomalies
Unified visibility across resources

Azure Bastion – Secure VM Access Without the Risk
When remote access to virtual machines is required, security must come first. Azure Bastion allows secure RDP and SSH access directly from the Azure portal, without exposing public IPs.

Real-World Example:
A support engineer needs to inspect logs inside a production VM. Rather than opening ports to the public internet, they connect via Azure Bastion — directly through the portal, securely and instantly.

Why It’s Critical:

No public IPs required on VMs
No open RDP/SSH ports = reduced attack surface
Seamless browser-based access

Azure Firewall – Network Control with Visibility
Azure Firewall helps support teams enforce network security boundaries and analyze traffic flows. It ensures that only authorized communication happens between services and the outside world.

Real-World Example:
A database suddenly becomes unreachable. The support team checks Firewall logs and finds that a recently added rule accidentally blocked outbound access from the app subnet. Quick fix. Issue resolved.

Key Benefits:

Centralized rule management (inbound/outbound)
Threat Intelligence filtering (block known malicious IPs)
Deep traffic inspection and logging

Bringing It All Together
Here’s how these tools work together in a support scenario:

[User Issue] ─> [Log Analytics: trace logs] ─> [Bastion: inspect VM] ─> [Firewall: verify access rules]

These tools empower support teams to:

Troubleshoot faster
Access securely
Respond confidently

Final Thoughts:

Support teams are the first line of defense when systems behave unexpectedly. Equipping them with tools like Log Analytics, Bastion, and Azure Firewall ensures faster resolution, better observability, and iron-clad security.

🔧 These are not just tools — they’re your cloud safety net.

🌐 Why You Should Use Azure Front Door and WAF to Protect Your APIs

Kiran Rongali — Wed, 14 May 2025 19:07:16 +0000

In today's cloud-native world, securing and optimizing API access is critical for performance, scalability, and protection against web-based threats. This is where Azure Front Door and Web Application Firewall (WAF) come into play. Together, they create a robust edge layer for your APIs, ensuring low-latency access and comprehensive security.

What is Azure Front Door?

Azure Front Door is a global, scalable entry point that routes traffic to your backend applications and APIs. It combines:

Smart Traffic Routing
Load Balancing
Global Content Delivery
SSL Termination
Application Acceleration

It's designed to deliver high availability and low latency using Microsoft's global edge network.

What is Azure Web Application Firewall (WAF)?

Azure WAF is a firewall designed to protect HTTP(S) applications from common threats like:

SQL injection
Cross-site scripting (XSS)
Request smuggling
OWASP Top 10 attacks
Bot traffic

When enabled on Azure Front Door, it inspects and filters incoming requests before they reach your APIs.

Front Door + WAF: API Protection Architecture

Here's a high-level architecture diagram showing how Azure Front Door and WAF sit in front of your backend APIs:

[Client Devices]
⬇
[Azure Front Door]
└️ WAF (Security policies applied)
⬇
[Backend APIs - App Service / APIM / AKS]

This architecture ensures:

Security: Malicious traffic is filtered before it touches your app
Performance: Users are routed to the nearest healthy endpoint
Reliability: Front Door handles failover and retries

Key Benefits for APIs:

Feature - > Benefit

Global Load Balancing -> Distribute traffic across geo-redundant backends
WAF Protection - > Block OWASP top 10 vulnerabilities
Fast Failover - > Reroute requests during backend outages
Edge SSL Termination - > Secure and speed up client connections
Rate Limiting - > Throttle abusive traffic
Request Inspection - > Log and analyze malicious requests

🛠️ How to Implement

Create a Front Door resource in Azure
Add backend APIs (App Services, APIM, AKS)
Configure routing rules (e.g., path-based routing)
Attach WAF Policy with custom or default rules
Test your endpoints through Front Door endpoint URL
Monitor logs and metrics in Azure Monitor and Defender

Real-World Example:

A fintech company exposes APIs globally for mobile users. With Azure Front Door:

Traffic is routed to the closest backend based on latency

WAF blocks injection attacks before they reach the app

SSL termination is handled at the edge

API Management adds versioning and throttling

Tips and Best Practices

Enable logging to Application Insights
Monitor WAF logs for repeated attack patterns
Customize WAF rules (block, log, allow)
Use versioned paths (/v1, /v2) and route based on path
Integrate with APIM for rate limiting and developer portal

Conclusion

Using Azure Front Door + WAF is one of the most powerful patterns for protecting modern APIs. You gain:

📈 Better performance globally
🛡️ Strong security at the edge
🚀 Scalable and reliable API delivery

When combined with Azure API Management, you have a complete enterprise-grade solution for API gateway, monitoring, security, and analytics.

Start implementing Azure Front Door today, and build smarter, faster, and safer APIs for the cloud-native era.

🔁 How Azure Service Bus Handles Retries + Dead-Letter Queue (DLQ) Monitor Part - 2

Kiran Rongali — Wed, 07 May 2025 19:53:10 +0000

As posted in How Azure Service Bus Handles Retries Part- 1 , the below post focuses on the retry flow and monitoring the Dead-Letter Queue (DLQ)

Step-by-Step Upgrade
1) Simulate a Failure in Message Processing
Replace your original message processor with logic that randomly fails:

csharp

var processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions());

processor.ProcessMessageAsync += async args =>
{
    string body = args.Message.Body.ToString();
    Console.WriteLine($"Received: {body}");

    // Simulate a failure randomly
    if (new Random().Next(1, 4) == 1)
    {
        throw new Exception("Simulated processing failure.");
    }

    await args.CompleteMessageAsync(args.Message);
    Console.WriteLine("Message completed successfully.");
};

processor.ProcessErrorAsync += async args =>
{
    Console.WriteLine($" Error Handler: {args.Exception.Message}");
    await Task.CompletedTask;
};

This simulates real-world transient issues.

2) Read from the Dead-Letter Queue (DLQ)
This processor will read messages from the DLQ after they fail too many times:

csharp

Console.WriteLine("Starting Dead-Letter Queue Listener...");

var dlqReceiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions
{
    SubQueue = SubQueue.DeadLetter
});

var deadLetters = await dlqReceiver.ReceiveMessagesAsync(maxMessages: 10, TimeSpan.FromSeconds(5));

if (deadLetters.Count == 0)
{
    Console.WriteLine(" No messages in DLQ.");
}
else
{
    Console.WriteLine($" Found {deadLetters.Count} message(s) in DLQ:");
    foreach (var msg in deadLetters)
    {
        Console.WriteLine($"  ❗ Message: {msg.Body}");
        await dlqReceiver.CompleteMessageAsync(msg); // Mark as handled
    }
}

You can loop this or run it on a timer in a real app.

Test It Out:

Send messages with the original sender

Let the random failure simulate retries

Once the MaxDeliveryCount is hit (default: 10), the message will land in the DLQ

DLQ monitor will read and log them

Optional Improvements:

Push DLQ metrics to Application Insights
Automatically re-send DLQ messages to retry queue (with delay)
Add logging to a file or database

Question:
can we use even driven architecture when the message retry fails?

Answer:
Yes — using event-driven architecture even when message retries fail is not only possible, but it's also one of the best patterns to handle failure gracefully and reactively.

Here’s how it works and why it’s useful:

Using Event-Driven Architecture After Retry Fails

What it means:
Instead of letting failed messages silently die or sit in the DLQ, you can emit an event when a failure happens (after all retries), and other systems can react to that event.

Typical Flow with Retry Failure

Producer --> Queue --> Processor --> Fails N times
↓
Moved to DLQ
↓
Dead-letter handler raises an event → Notifier / Logger / Dashboard / Alert

Examples of Event-Driven Responses After Failure

How to Implement

Emit Custom Event on DLQ Handler After reading from DLQ:

csharp

foreach (var msg in deadLetters)
{
    var body = msg.Body.ToString();
    Console.WriteLine($" Dead-lettered: {body}");

    // Raise an event (example: publish to another queue or topic)
    var failureEvent = new ServiceBusMessage(JsonSerializer.Serialize(new
    {
        Type = "OrderProcessingFailed",
        OriginalPayload = body,
        Timestamp = DateTime.UtcNow
    }));

    await failureEventSender.SendMessageAsync(failureEvent);

    await dlqReceiver.CompleteMessageAsync(msg);
}

You could publish this to:

Another Service Bus queue or topic

Event Grid

A webhook or REST API

A logging pipeline (e.g., Application Insights, Datadog)

Why This Matters?
Benefits of triggering events after retries fail:

Real-time observability of critical failures
Prevent silent data loss in DLQ
Let multiple teams (notifications, support, analytics) respond independently
Enables automatic recovery, fallbacks, or audits

Best Practices:

🔁 How Azure Service Bus Handles Retries Part- 1

Kiran Rongali — Mon, 05 May 2025 20:23:56 +0000

In the post below, we discussed how Azure Service Bus integrates with a .NET application and the benefits it provides.
https://dev.to/kiranrongali/why-use-azure-service-bus-how-to-integrate-in-net-13f9

But what happens if a failure occurs during message processing? How does retry behavior work, and how many times will it attempt to process the message again? This post will help you understand that process.

When your message processor (receiver) throws an exception or fails to call CompleteMessageAsync, Azure Service Bus automatically retries processing that message, based on its configured max delivery count.

Retry Behavior (Default)
If your handler fails or throws an exception → message is re-delivered

This retry continues until:
The message is successfully completed (CompleteMessageAsync)
The max delivery count is exceeded

What Happens After Max Retries?
Once the message exceeds the MaxDeliveryCount (default: 10), it is moved to the Dead-Letter Queue (DLQ).

Dead-Letter Queue
This is a special sub-queue where failed messages are stored for inspection, manual or automated reprocessing.

You can access the DLQ like this:

#csharp
var receiver = client.CreateReceiver("your-queue-name", new ServiceBusReceiverOptions
{
    SubQueue = SubQueue.DeadLetter
});

What You Need to Do

Proper Error Handling Wrap your message processing in try/catch:

#csharp
processor.ProcessMessageAsync += async args =>
{
    try
    {
        var body = args.Message.Body.ToString();
        Console.WriteLine($"Processing: {body}");

        // Your logic here

        await args.CompleteMessageAsync(args.Message);
    }
    catch (Exception ex)
    {
        Console.WriteLine($"Error: {ex.Message}");
        // Do NOT complete the message → it will retry
        // Optionally log or notify
    }
};

Configure MaxDeliveryCount In Azure Portal → Service Bus Queue → Properties, you can set MaxDeliveryCount.

Or via ARM/CLI:

#json
"maxDeliveryCount": 5

Monitor the Dead-Letter Queue

Use: Azure Portal (Explore Dead-letter queue)

Azure Monitor / Alerts

Code-based reprocessing via SubQueue.DeadLetter

Optional: Auto Retry with Delay
Azure Service Bus doesn’t support native delayed retries (e.g., exponential backoff), but you can:

Use Scheduled Enqueue Time to requeue a message after a delay

Move failed messages to a custom retry queue with delay

Example of scheduling a retry:

#csharp
var retryMessage = new ServiceBusMessage("retry this")
{
    ScheduledEnqueueTime = DateTimeOffset.UtcNow.AddMinutes(5)
};
await sender.SendMessageAsync(retryMessage);

Best Practices:

Use try/catch to Prevent app crashes & enable retry logic
Use Monitor DLQ to Avoid silent message losses
Use custom retry queues for control over backoff and limits
Handle poison messages Log, alert, or notify for manual review

That's it for this Part1 and Part 2 will be covered in a separate post, focusing on the retry flow and monitoring the Dead-Letter Queue (DLQ).

💡Why Use Azure Service Bus? How to Integrate in .NET

Kiran Rongali — Wed, 30 Apr 2025 12:24:00 +0000

Azure Service Bus enables reliable, asynchronous communication between microservices or distributed components in your app. It helps in:

Decoupling services (e.g., API triggers a background worker)
Buffering traffic (e.g., spikes in request volume)
Enabling retries and fault tolerance
Scalable message handling with minimal coupling

Step-by-Step: Azure Service Bus Integration in .NET
Prerequisites:

An Azure subscription
An Azure Service Bus namespace and a queue or topic
A .NET 6 or later project (console, web API, etc.)

Step 1: Install the NuGet Package
In your terminal or Package Manager Console, run:

#bash
dotnet add package Azure.Messaging.ServiceBus

Or via Visual Studio's NuGet UI:
Manage NuGet Packages → Search for Azure.Messaging.ServiceBus → Install

Step 2: Configure the Connection String
Go to Azure Portal → Your Service Bus namespace → Shared Access Policies → RootManageSharedAccessKey → Copy the Connection String.

Add it to appsettings.json:

#json
{
  "ServiceBus": {
    "ConnectionString": "<your-connection-string>",
    "QueueName": "your-queue-name"
  }
}

Step 3: Send a Message
Sending a message using a ServiceBusSender

#csharp
using Azure.Messaging.ServiceBus;

var client = new ServiceBusClient("<your-connection-string>");
var sender = client.CreateSender("your-queue-name");

var message = new ServiceBusMessage("Hello from .NET");
await sender.SendMessageAsync(message);

Console.WriteLine("Message sent!");

You can also use dependency injection and IConfiguration to get config from appsettings.json.

Step 4: Receive Messages
Using ServiceBusProcessor (recommended for background processing)

#csharp
var processor = client.CreateProcessor("your-queue-name", new ServiceBusProcessorOptions());

processor.ProcessMessageAsync += async args =>
{
    string body = args.Message.Body.ToString();
    Console.WriteLine($"Received: {body}");

    await args.CompleteMessageAsync(args.Message);
};

processor.ProcessErrorAsync += args =>
{
    Console.WriteLine($"Error: {args.Exception.Message}");
    return Task.CompletedTask;
};

await processor.StartProcessingAsync();

// Optional: Stop when done
// await processor.StopProcessingAsync();

Step 5: Use Dependency Injection (in ASP.NET Core)
Add this to your Startup.cs or Program.cs:

#csharp
builder.Services.AddSingleton(serviceProvider =>
{
    var config = serviceProvider.GetRequiredService<IConfiguration>();
    return new ServiceBusClient(config["ServiceBus:ConnectionString"]);
});

builder.Services.AddSingleton(serviceProvider =>
{
    var client = serviceProvider.GetRequiredService<ServiceBusClient>();
    return client.CreateSender(config["ServiceBus:QueueName"]);
});

Step 6: Bonus – Send JSON or Custom Object

#csharp
var payload = new { OrderId = 123, Status = "Processed" };
var json = JsonSerializer.Serialize(payload);

var message = new ServiceBusMessage(json)
{
    ContentType = "application/json"
};
await sender.SendMessageAsync(message);

Sample Use Case:

Let’s say you're building a food delivery app:

The Order API sends an order placed event to the queue.
A Delivery Service listens to the queue and schedules a delivery.
A Notification Service sends an SMS or email, also triggered from the queue.

Summary:

💬📦 Why Every Modern App Needs a Message Queue

Kiran Rongali — Fri, 25 Apr 2025 02:00:40 +0000

In today’s software landscape, applications are expected to be highly available, scalable, and resilient — all while providing a fast, seamless user experience. But as complexity increases with the adoption of microservices, cloud-native architecture, and real-time data processing, direct communication between services becomes fragile and hard to scale.

That’s where message queues come in.

What is a Message Queue?
A message queue is a form of asynchronous service-to-service communication. Instead of a service calling another directly and waiting for a response, it places a message into a queue. The message is then picked up and processed by a separate service — either immediately or whenever it’s ready.

Think of it like:
A to-do list: someone adds tasks to it (producer), and someone else comes and does them (consumer) — without both needing to be there at the same time.

Why Use a Message Queue?

Decoupling Services Without message queues, services are tightly connected. If one fails, the whole process can collapse. With queues, the sender and receiver are independent — one can function even if the other is temporarily down.

Example:
An Order API accepts customer purchases.

It places a message in the queue for:

Inventory Service to update stock

Payment Service to charge the customer

Email Service to send confirmation

Each service works independently, and failures in one don't block the others.

Asynchronous Processing Some tasks take time (sending emails, generating reports). Message queues let you process those in the background, so the user isn’t kept waiting.

Example:
A user uploads a large image to be analyzed.

Your app queues the image for processing and returns a quick “We got it!” message.

A background worker picks up the task, processes it, and stores the result.

Scalability Queues naturally enable horizontal scaling — just add more consumers to process messages faster during peak load.

Example:
During Black Friday, your e-commerce site receives 10,000 orders per minute.

A queue holds all those orders.

You spin up 50 instances of the Order Processor service to handle them in parallel.

Reliability and Fault Tolerance Message queues offer durability and retry mechanisms. If a service fails or crashes mid-task, the queue holds the message and retries later or routes it to a dead-letter queue for review.

Example:
Your Payment Processor crashes while processing an order.

The message is re-queued and retried once the service is back up — no data loss.

Traffic Buffering Queues smooth out sudden traffic spikes. Instead of overwhelming your system, requests are stored and processed gradually.

Example:
Your app launches a viral marketing campaign.

Instead of crashing from overload, your queue handles the surge and processes it smoothly over time.

Real-World Scenario: With vs Without Message Queue
Without Queue:

User submits an order → API calls Billing → calls Inventory → sends Email
If any call fails, the whole process breaks
User may have to retry the entire flow manually

With Queue:

User submits an order → API places messages in queues
→ Billing, Inventory, and Email services process tasks independently
Failures handled via retries or logged in dead-letter queues

In a modern, cloud-native architecture, message queues aren’t optional — they’re essential. Whether you're building an e-commerce platform, a real-time analytics pipeline, or a scalable microservices backend, message queues offer the reliability, performance, and flexibility you need.

Decoupled
Scalable
Resilient
Efficient

In short, message queues give your architecture breathing room. Without them, your services become too tightly coupled and fragile — one broken link, and the entire chain suffers.

If you're building or scaling any serious application and beyond, a message queue isn’t just a “nice-to-have” — it's a must.

⚙️Integrate Datadog with a .NET Application

Kiran Rongali — Mon, 21 Apr 2025 17:28:28 +0000

As mentioned in a previous post https://dev.to/kiranrongali/how-datadog-helps-developers-teams-kkg, we discussed how Datadog can benefit developers and teams. In this post, we'll focus on integrating it with a .NET application.

Step 1: Create a Datadog Account

1.  Go to https://www.datadoghq.com.
2.  Sign up for a free trial.
3.  Once logged in, choose your region (important for API keys and endpoints).

Step 2: Install the Datadog Agent (on your dev machine, server, or container)

Download the agent from Datadog Agent Downloads.
Follow the OS-specific instructions:
- Windows: Run the .msi installer and enter your API Key during setup.
- Linux/macOS: Use the shell script provided.

🔑 Your API key can be found in Datadog → Integrations → APIs.

Step 3: Install Datadog Tracer for .NET In your .NET Core or ASP.NET Core project:

Install the NuGet package:

 #bash
 dotnet add package Datadog.Trace

Enable automatic instrumentation:

Add this to your environment (or launch profile):

  #bash

  DOTNET_Diagnostics=1
  DD_TRACE_ENABLED=true
  DD_ENV=dev
  DD_SERVICE=my-dotnet-app
  DD_VERSION=1.0.0
  DD_AGENT_HOST=localhost   # or your agent IP
  DD_TRACE_DEBUG=true       # optional for debugging

Or set in appsettings.json (if using Datadog.Config):

  #json
{
"DD_SERVICE": "my-dotnet-app",
"DD_ENV": "dev",
"DD_AGENT_HOST": "localhost",
"DD_TRACE_ENABLED": "true"
}

Step 4: Enable APM (Tracing) If using IIS or Kestrel, no code changes are needed but: To trace web requests, background jobs, or SQL calls:

 //csharp
  using Datadog.Trace;

  public IActionResult Index()
   {
    var span = Tracer.Instance.StartActive("custom.operation");
    try
     {
      // your logic here
     }
    finally
     {
      span.Dispose(); // ends trace
     }

     return View();
    }

Step 5: Enable Logging (Optional but Recommended)
Use Serilog, NLog, or Microsoft Logging to forward logs:

For Serilog:

#bash
dotnet add package Serilog.Sinks.Datadog.Logs

Example Config:

   //csharp
   Log.Logger = new LoggerConfiguration()
    .WriteTo.DatadogLogs("<your-api-key>", source: "csharp", service: "my-dotnet-app")
    .CreateLogger();

Step 6: Verify in Datadog Dashboard
- Go to APM → Services in Datadog.
- Select your service (my-dotnet-app) to view: Response times Error rates Traces and spans Host metrics (CPU, memory, etc.)
Step 7: Set Up Dashboards & Alerts
1. Create a custom dashboard:
  - Add widgets: charts, traces, logs, etc.
2. Add monitors/alerts:
  - Example: “Notify if error rate > 5% for 5 min”
3. Integrate with Slack, Teams, PagerDuty, etc. for alert delivery.
Optional: Monitor SQL, Redis, HTTP, etc.
- Install additional NuGet packages if needed (e.g., Datadog.Trace.ClrProfiler.Managed).
- You can trace:
  - SQL Server (SqlClient)
  - HTTP calls (HttpClient)
  - Entity Framework
  - Background Services (e.g., Hangfire)

That's it!
With this setup, you'll now have:

Real-time visibility into your .NET app’s performance
Detailed traces and logs
Dashboards and alerts

💡 How Datadog Helps Developers & Teams

Kiran Rongali — Thu, 17 Apr 2025 03:10:57 +0000

Datadog is more than just a monitoring tool — it’s a full-stack observability platform that empowers developers, DevOps engineers, SREs, and even business teams to monitor, debug, optimize, and secure applications and infrastructure.

Here’s a breakdown of how Datadog supports different stages and areas of the development and operations lifecycle:

Application Performance Monitoring (APM)

Use Case: Identify Slow Code & Optimize Bottlenecks
Tracks end-to-end performance of .NET Core, Java, Node.js, Python, and more.
Visualizes method-level execution time, external dependencies (DBs, APIs), and latency.
Provides service maps, flame graphs, and spans to isolate slow or failing areas.
Example: A POST /api/order/checkout request takes 5 seconds. APM shows 3.5 seconds are spent on a SQL join, helping the dev fix the DB query.

Logs Management & Analysis

Use Case: Centralize, Search & Correlate Logs Across Services
Ingest logs from apps, servers, containers, firewalls, and more.
Supports parsing, filtering, tagging, and alerting on logs.
You can correlate logs with traces, helping pinpoint context fast.

Example: A user reports a payment failure. Devs search logs for their email, see the stack trace, and click into the corresponding trace to understand what broke.

Real-Time Metrics Monitoring

Use Case: Monitor System Health & Custom Business Metrics
Tracks infrastructure (CPU, memory, I/O) and app-specific metrics.
Supports custom metrics via SDKs for .NET, Python, Java, etc.
Dashboards visualize real-time & historical data for fast decision-making.

Example: Developers track ItemsInCart.Count across time and regions to understand shopping trends or detect sudden drops.

Distributed Tracing

Use Case: Debug Across Microservices & Understand Dependencies
See how a request flows through multiple services (microservices or serverless).
Understand timing and failures at each step.
Detect high-latency internal calls or timeout-prone external APIs.

Example: A request flows through Auth → User → Billing → Notification services. You spot a 2s delay in Billing caused by an overloaded Redis instance.

Infrastructure Monitoring

Use Case: Visualize & Alert on Infrastructure in Cloud, Hybrid, or On-Prem
Automatically discovers EC2, Azure VMs, GCP instances, Kubernetes nodes, Docker containers, and more.
Provides live heatmaps, topology maps, and health overviews.
Tracks disk usage, open connections, network traffic, etc.

Example: A .NET app running in Azure AKS crashes intermittently. Datadog shows the node has disk pressure and evictions.

Error Tracking

Use Case: Identify, Aggregate & Prioritize Production Errors
Tracks exceptions in your app: .NET, JavaScript, etc.
Groups them by stack trace and frequency.
Allows you to set alerts on new or high-volume errors.

Example: After a deployment, 300+ NullReferenceException logs spike. Datadog notifies the dev team instantly, and they roll back before users notice.

Synthetic Monitoring

Use Case: Ensure Uptime & Measure End-User Experience
Automates browser or HTTP tests against APIs and websites.
Can simulate logins, form fills, and flows (e.g., checkout process).
Checks from multiple global locations for regional issues.

Example: Datadog detects a failed login flow at 2 AM via synthetics and alerts DevOps before users wake up.

Real User Monitoring (RUM)

Use Case: Understand How Users Experience Your Frontend
Tracks page load time, JS errors, AJAX calls, and user sessions.
Detects slow assets or unhandled exceptions affecting UX.
Filters by browser, location, OS, user ID.

Example: Users on Safari 17.3 report lag. Datadog shows high Largest Contentful Paint (LCP) times for that browser only.

Security Monitoring

Use Case: Detect Security Threats & Misconfigurations
Analyzes logs for suspicious activity (e.g., SQL injection attempts, brute-force login).
Monitors compliance (PCI, HIPAA, SOC 2).
Integrates with SIEM tools or your DevSecOps pipeline.

Example: Detects an unusual volume of login attempts from a single IP. Automatically blocks it via an integrated firewall rule.

CI/CD Pipeline Monitoring

Use Case: Improve Build/Deploy Reliability
Monitors builds, deploy times, failures, test coverage, and duration.
Tracks deployments across environments and correlates them with performance regressions.

Example: Deployment to prod causes latency spikes. Datadog tags traces/logs with that version, making rollback decision faster.

Alerts & Notifications

Use Case: Proactive Monitoring & Automated Response
Create monitors for any metric, log pattern, or error.
Supports anomaly detection and forecasting.
Alerting via Slack, Teams, PagerDuty, Webhooks, Email, etc.

Example: “Alert if HTTP 5xx rate > 1% for 5 mins” — Datadog auto-notifies the incident team.

Custom Dashboards for Teams

Use Case: Give Everyone the Insights They Need
Devs: Debug traces, view live logs.
Ops: View CPU, memory, and system alerts.
Business: Track sales, active users, or feature usage.
Security: Monitor unusual login attempts or audit trails.

Example: Product Managers view a dashboard tracking premium feature usage and correlate it with recent deploys.

Final Thoughts

Thank you for taking the time to read this. I hope you found these use cases helpful. Stay tuned for the next blog on integrating Datadog with a .NET application.**