Using AI Agents to Debug Distributed Systems in Under a Minute

tomas maiorino — Wed, 01 Apr 2026 22:30:57 +0000

Using AI Agents to Debug Distributed Systems Faster

At my company, we have a feature that allows customers to export large volumes of data to cloud providers.

Under the hood, this export process is split into multiple tasks, where each task is responsible for exporting a subset of objects. These tasks are executed by pods in a multi-tenant Kubernetes environment.

From time to time, we receive alerts indicating that some tasks are taking too long to start and remain in the queue for an extended period.

When that happens, an investigation begins.

The challenge is that this analysis is usually slow, manual, and repetitive.

A typical investigation involves:

Checking the status of each task and validating key attributes
Reviewing tenant configurations to identify values that may cause issues
Inspecting overall cluster health
Analyzing how many tasks each tenant has created
Cross-checking configuration in Bitbucket
Making multiple API calls across services

This process can easily take several minutes, and sometimes much longer, especially during active incidents.

The Idea: Automate the Investigation with an AI Agent

We decided to speed things up by building an AI-powered agent.

The goal was simple:

Automatically gather the relevant data, analyze it, and provide a probable root cause in seconds.

Architecture

To achieve this, we built two main components.

1. MCP Server

We created an MCP server that exposes a set of tools wrapping our internal APIs.

These tools allow the agent to:

Query task status
Fetch tenant configurations
Inspect system limits such as max replicas
Retrieve cluster-level information

2. AI Agent

On top of that, we built an AI agent that:

Uses the MCP tools
Analyzes the collected data
Produces a structured diagnostic report

Input: From Alert Logs to Insight

At the time, we did not have a direct integration with our alerting system.

Because of that, we designed the agent to interpret the log lines included in the alert and generate a report from them.

Example Input

Generate report for test environment
Long waiting in queue tasks amount:13 ; Affected tenants: tenant3, tenant2, tenant5, tenant4, tenant1,tenant6

Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant1 - groupId: 528f72d7-4f5d-4559-a825-b05014114dc7 - taskId: 528f72d7-4f5d-4559-a825-b05014114dc7 - 1h 33m 43s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant1 - groupId: e7b2449f-0ded-4627-a6b6-eb305e074503 - taskId: e7b2449f-0ded-4627-a6b6-eb305e074503 - 1h 33m 43s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant2 - groupId: b481240d-d166-45cd-92dd-9f45d99c17f0 - taskId: b481240d-d166-45cd-92dd-9f45d99c17f0 - 1h 33m 6s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant3 - groupId: e1514831-b1b3-444f-a209-118c82718fbe - taskId: e1514831-b1b3-444f-a209-118c82718fbe - 1h 32m 10s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant1 - groupId: bad9fb64-ec1a-46e8-b3cd-91ded32cd551 - taskId: bad9fb64-ec1a-46e8-b3cd-91ded32cd551 - 1h 32m 0s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant4 - groupId: c875921a-03d7-498b-aba4-848703e398d7 - taskId: c875921a-03d7-498b-aba4-848703e398d7 - 1h 24m 40s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant4 - groupId: b2eb4f12-439f-4027-9b07-be8d38c18d91 - taskId: b2eb4f12-439f-4027-9b07-be8d38c18d91 - 1h 24m 26s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant5 - groupId: e4361de7-280e-4906-9cc5-7a56c056a959 - taskId: 29b656a1-e77b-41b9-bcdc-4d0ee9edc282 - 1h 21m 43s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant5 - groupId: e4361de7-280e-4906-9cc5-7a56c056a959 - taskId: 2fe2d3a1-0ac1-42d2-996e-0edc5a075fe7 - 1h 21m 43s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant6 - groupId: e4361de7-280e-4906-9cc5-7a56c056a959 - taskId: 45e198f9-e629-47c0-ac5f-cf8eabbdcaf6 - 1h 21m 43s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant5 - groupId: e4361de7-280e-4906-9cc5-7a56c056a959 - taskId: 9ba638b7-ed61-4977-991b-2aa691a38d73 - 1h 21m 43s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant5 - groupId: e4361de7-280e-4906-9cc5-7a56c056a959 - taskId: cfc26a65-30eb-4264-88e0-307b878e4d3c - 1h 21m 43s - com.export.tasks.local.ExportTask
Mar 17 21:34:06 export-pod-1 export WARN test | Long waiting in queue task: tenant5 - groupId: e4361de7-280e-4906-9cc5-7a56c056a959 - taskId: d3748c08-baf5-43e4-98e7-7dd75d1ce43d - 1h 21m 43s - com.export.tasks.local.ExportTask

Example Response

Below is an example of the report generated by the agent.

Tenant Summary

Tenant ID: tenant11
Tenant Name: Unknown
Total Tasks: 58

Tasks Status

Group ID: e4361de7-280e-4906-9cc5-7a56c056a959
Status: SCHEDULED

Tasks in Queue

Task ID: 7b67dd23-fdca-4acb-ab28-88bf8597c7a0
- Group ID: 951b184b-e639-43cc-b837-0768b1f447ab
- parallelExecution: false

Tenant Config

maxPeriodicTasksPerTenant: 35
maxTaskPartsCount: 13

Credit Balance

priorityCredits: 23408
standardSyncCredits: 2796
standardAsyncCredits: -18368
internalCredits: -89636

Problem Areas

Tasks with `parallelExecution=false`

Tenant: tenant1
- Group ID: ec39f0e0-2093-43c0-93ed-73cc2675337f
- Task ID: 5b9280cd-9a10-4786-9ae0-000a26e6b0ce
Tenant: tenant3
- Group ID: 46588442-a969-4568-9a81-a7356727180f
- Task ID: 56193050-578e-4803-bca1-5c059e64fe3e
Tenant: tenant2
- Group ID: 951b184b-e639-43cc-b837-0768b1f447ab
- Task ID: 7b67dd23-fdca-4acb-ab28-88bf8597c7a0

Tenants with `maxTaskPartsCount` or `maxPeriodicTasksPerTenant` equal to or higher than 30% of the max replicas

Max Replicas: 84
Tenant: tenant6
- maxTaskPartsCount: 30
- Usage: 35.7% of max replicas
Tenant: tenant5
- maxPeriodicTasksPerTenant: 52
- Usage: 61.9% of max replicas

Another Example of Agent Output

In another test, the agent produced the following findings:

Problem Areas

Tasks with `parallelExecution=false`

Tenant ID: tenant6, Group ID: ab92df1b-9c82-4940-8e25-dfa77f275ebb
Tenant ID: tenant3, Group ID: e8143f64-11b1-48a7-960f-b005e5805871
Tenant ID: tenant5, Group ID: d5812c89-3961-4f4d-bf1b-b08c36833a06
Tenant ID: tenant4, Group ID: c875921a-03d7-498b-aba4-848703e398d7
Tenant ID: tenant1, Group ID: 8ed4824d-3333-4f9b-b442-123446b2006d
Tenant ID: tenant2, Group ID: 63801bb0-b5f7-44a9-bd94-84ccb9cc845e

Cluster Saturation

Total active tasks: 86
Max replicas: 38

This means the total number of active tasks was already higher than the available capacity for that environment.

Tasks with Lower Throughput

Tenant ID: tenant1
Group ID: 528f72d7-4f5d-4559-a825-b05014114dc7

Task Statuses Seen in the Analysis

CANCELED
COMPLETED
SCHEDULED
PROCESSING
PAUSED
SCHEDULED_POLL

Tenants with High Limits Relative to Cluster Capacity

Tenant: tenant1
- maxPeriodicTasksPerTenant: 86
- This is equal to or higher than 30% of the max replicas for that environment.

What the Agent Detects

From reports like the ones above, the agent is able to highlight:

Tasks that cannot run in parallel because parallelExecution=false
Cluster saturation issues, where active tasks exceed available replicas
Misconfigured tenants exceeding safe limits
Credit imbalances that may impact execution
Low-throughput or blocked task groups

Impact

This approach reduced investigation time from:

Manual analysis that could take several minutes or longer

to

automated analysis that usually takes less than one minute

It can take a bit longer when the agent needs to inspect more than 20 different tasks, but it is still significantly faster than the manual process.

Tech Stack

I created a Java project using Spring AI and the following modules:

Mockoon for generating mock API data
java-project for the MCP tools
agent-ui for the user interface
agent for the orchestration and reasoning layer

This agent was originally built in Python, but I thought it would be interesting to create a small project using Spring AI and get more knowledge about the tool.

Key Takeaways

A lot of operational analysis follows a repeatable pattern.

That makes it a good fit for an AI agent that can:

Gather data from multiple sources
Correlate information across APIs and configuration
Suggest a likely root cause
Produce a report that engineers can use immediately

The real value comes from combining tooling and reasoning.

What's Next

The next step is to rebuild the project using an agent + agent.md design pattern.

The goal is to make the solution:

More modular
Easier to maintain
Easier to evolve as new investigation paths are added

Final Thoughts

This project started from a very practical problem: incident investigations were taking too much time.

By turning the investigation flow into a set of tools plus an AI reasoning layer, we were able to automate a large part of the process and dramatically reduce the time to get useful answers.

Instead of manually checking dashboards, configs, and APIs, we now have an agent that can:

Read alert context
Investigate the system
Summarize the findings
Suggest likely root causes

All in under a minute.

⚠️ For Disclosure

At my company, the solution was originally designed in Python, as it was the primary language used for working with MCP (Model Context Protocol) and AI agents at the time.

However, I decided to build this demo in Java to gain hands-on experience with Spring AI while applying it to a real-world problem. You can find the demo code here.

All values and data shown in this article were generated using Mockoon and do not represent real production data. They are simplified examples meant to reflect the kind of information handled by the actual application.

Feedback

I would love to hear your thoughts.

Would you trust an AI agent to help debug production issues?
How would you design this differently?

Forem: tomas maiorino

Using AI Agents to Debug Distributed Systems in Under a Minute

Using AI Agents to Debug Distributed Systems Faster

The Idea: Automate the Investigation with an AI Agent

Architecture

1. MCP Server

2. AI Agent

Input: From Alert Logs to Insight

Example Input

Example Response

Tenant Summary

Tasks Status

Tasks in Queue

Tenant Config

Credit Balance

Problem Areas

Tasks with parallelExecution=false

Tenants with maxTaskPartsCount or maxPeriodicTasksPerTenant equal to or higher than 30% of the max replicas

Another Example of Agent Output

Problem Areas

Tasks with parallelExecution=false

Cluster Saturation

Tasks with Lower Throughput

Task Statuses Seen in the Analysis

Tenants with High Limits Relative to Cluster Capacity

What the Agent Detects

Impact

Tech Stack

Key Takeaways

What's Next

Final Thoughts

⚠️ For Disclosure

Feedback

Tasks with `parallelExecution=false`

Tenants with `maxTaskPartsCount` or `maxPeriodicTasksPerTenant` equal to or higher than 30% of the max replicas

Tasks with `parallelExecution=false`