Forem: Shivam Maurya

🚀 AI-Driven Failure Intelligence for 1000+ API Test Cases

Shivam Maurya — Mon, 02 Mar 2026 23:39:56 +0000

In large-scale API automation environments (1000+ scenarios), even 50–100 failures in a CI run can take hours to manually analyze.

I recently implemented an AI-assisted failure analysis layer within our CI pipeline to automatically interpret failed test cases and generate structured root-cause reasoning.

🔹 Standard Test Execution (Before AI Layer)

🟢 Test Execution Summary
-------------------------------------------------
Feature                      | Passed | Failed | Total
Customer Identity Suite      | 842 ✅  | 18 ❌  | 860
Order Processing Suite       | 97 ✅   | 12 ❌  | 109
Payment Validation Suite     | 31 ✅   | 5 ❌   | 36
-------------------------------------------------
TOTAL                        | 970    | 35     | 1005

When failures scale to 50–100+ cases:

Engineers typically:

Open HTML reports
Scan raw logs
Compare expected vs actual response
Identify assertion mismatches
Interpret schema failures
Trace backend logic impact

This increases:

Debug cycle time
Developer back-and-forth
CI triage effort

🔹 What Changed (AI Layer Enabled)

When optional AI analysis is enabled:

🚀 Running AI Failure Analysis...

📊 FINAL EXECUTION SUMMARY
----------------------------------
🚨 Total Failed Cases Analyzed: 35
🧠 AI Structured Reports Generated: 35
----------------------------------

Example AI-Generated Output

📂 Feature Name  : Customer Identity Suite
❌ Scenario Name : Validate getProfile API with expired token
💥 Failure Reason:
Assertion failed: expected HTTP 401 but received 200.
Possible cause: Token validation middleware not enforced.
Impact: Security validation gap in authentication flow.

📂 Feature Name  : Order Processing Suite
❌ Scenario Name : Validate order creation with invalid SKU
💥 Failure Reason:
Schema mismatch in response.data.errorCode.
Backend validation layer likely bypassed.

🔹 Behind the Scenes – Execution Flow

User Triggers CI Pipeline
        │
        ▼
Run 1000+ API Test Scenarios
        │
        ▼
Generate Standard Reports (HTML / JSON)
        │
        ▼
Extract failedScenarios.json (Only Failed Cases)
        │
        ▼
Build Structured Failure Payload
        │
        ▼
Send to AI Agent (Optimized Token Usage)
        │
        ▼
Async Polling Until Completion
        │
        ▼
Validate Structured Schema Output
        │
        ▼
Print Feature-Level Failure Intelligence Summary

🔹 Engineering Design Considerations

✅ Non-blocking integration (AI failure does not fail pipeline)
✅ Optional execution toggle
✅ Token optimization (only failed scenarios analyzed)
✅ Structured schema enforcement
✅ Feature-level grouping
✅ Polling-based async agent handling
✅ No impact to primary execution time

🔥 Real Impact (Measured)

In runs with 80–100 failures:

Before AI:

2–3 hours manual debugging
Multiple log scans
Repetitive analysis effort

After AI:

40–60% reduction in manual triage time
Immediate structured reasoning
Faster developer alignment
Reduced QA–Backend iteration cycle
Improved CI observability

🔹 What This Enabled

Instead of:

“Test Failed — Check Logs”

We now have:

“Test Failed — Here is structured reasoning and probable cause.”

This shifted automation from:

Execution-focused
to
Intelligence-enabled.

🔹 Tech Stack Blend

Test Automation × CI/CD × Structured AI Reasoning × Observability

Build a Scalable Karate API Automation Framework using Node.js & GitLab CI/CD

Shivam Maurya — Sun, 24 Aug 2025 02:25:19 +0000

📘 Karate API Automation Framework using Node.js + GitLab CI/CD
Author: [Shivam Maurya]
Repo Purpose: 🚀 A clean, modular, and scalable Karate test automation framework using Node.js, GitLab pipelines, and full parallel execution with tagging and logging support.
Tech Stack: Karate 🥋 | Node.js | GitLab CI/CD | Java (runtime only) | Markdown Docs

✅ STEP 1: Local Setup in VS Code (Node.js Environment)
We'll set up a modern Karate test automation framework using Node.js and VS Code — no Java coding needed, everything handled via npm.

🔧 1.1 Prerequisites
Make sure you have the following installed:
✅ Node.js (v18+ recommended)
✅ VS Code
✅ Java (JDK 11+ is used by Karate internally — not for coding, but for execution)

🔍 Check if installed
node -v
npm -v
java -version

✅ Sample Output:
v22.18.0
10.9.3
java version "11.0.18"
📦 If not installed, download Node.js (LTS version) from nodejs.org

🏗️ 1.2 Project Folder Structure
Run the following to set up a basic Node project:
mkdir Karate-Framework
cd Karate-Framework
npm init -y

Now, create the structure below:

Karate-Framework/
├── features/
│ └── sample.feature
├── utility/
│ └── karate.js
├── karate-config.js
├── package.json

Screenshot:

📦 1.3 Install Karate via NPM
npm install @karatelabs/karate --save-dev

📁 This installs the CLI & dependencies locally (no global install needed). You’ll see:
node_modules/
package-lock.json

🧠 1.4 Create Sample Test

📄 features/sample.feature

Feature: Sample API Test
       @sample
        Scenario: Call a public API
        Given url "https://jsonplaceholder.typicode.com/posts/1"
        When method get
        Then status 200
        And match response.id == 1

🧪 1.5 Create Config File (karate-config.js)
📄 karate-config.js

 function fn() {
    karate.configure('connectTimeout', 200000);
    karate.configure('readTimeout', 200000);
    karate.configure('printEnabled', true);
    karate.configure('logPrettyResponse', true);
    karate.configure('logPrettyRequest', true);

    var env = karate.env || 'dev';
    var config = {
        env: env
    };

    return config;
    }

🎯 This sets timeouts, logging preferences, and environment variables.
🧑‍💻 1.6 Create utility/karate.js (Programmatic Runner)
📄 utility/karate.js

// Utility script to execute Karate tests with dynamic args
    import karate from '@karatelabs/karate';
    const cliArgs = process.argv.slice(2);
    let threads = '5'; // Default thread count
    // Find numeric thread count passed as CLI arg
    const idx = cliArgs.findIndex(arg => /^\d+$/.test(arg));
    if (idx !== -1) {
    threads = cliArgs[idx];
    cliArgs.splice(idx, 1);
    }
    const options = `-T=${threads}` + (cliArgs.length ? ' ' + cliArgs.join(' ') : '');
    console.log('🎯 Running Karate with args:', options);
    karate.exec(options);

💡 Example:
node utility/karate.js ./features -t=@sample

📝 1.7 Add Scripts in package.json
📄 package.json

{
    "name": "karate-framework",
    "version": "1.0.0",
    "description": "Test Automation Framework for API",
    "type": "module",
    "scripts": {
        "karate": "node ./utility/karate.js",
        "test": "npm run karate -- ./features -f html,cucumber:json,junit:xml"
    },
    "devDependencies": {
        "@karatelabs/karate": "^0.3.2"
    }
    }

🚀 1.8 Run Karate Tests (With Tags & Parallel)
Run all tests:
npm test
Run tagged test with parallelism:
npm test -- -t=@sample 10

✅ STEP 2: Push Karate Project to GitLab

🧱 2.1 Create GitLab Repository
Go to 👉 https://gitlab.com
Create new project: karate-api-automation
(Do not initialize with README or .gitignore if present locally)

💻 2.2 Initialize Git Locally
git init
git remote add origin https://gitlab.com//karate-api-automation.git

📦 2.3 Commit All Files
git add .
git commit -m "🚀 Initial commit - Karate Framework setup"

✅ This includes:
package.json
karate-config.js
utility/karate.js
features/sample.feature
.gitlab-ci.yml (in next step)
README.md

🚀 2.4 Push to GitLab
git push -u origin main

📦 Now your entire framework is versioned and pipeline-ready.
Screenshot:

✅ STEP 3: GitLab CI/CD Pipeline Setup

📁 3.1 Create pipeline/ Folder
mkdir pipeline

📄 pipeline/.gitlab-ci.yml

image: node:20

    stages:
    - test

    variables:
    NODE_ENV: "dev"
    KARATE_TAG: "@sample"
    KARATE_PARALLEL: "5"

    cache:
    paths:
        - node_modules/

    before_script:
    - echo "Using Node.js version:"
    - node -v
    - echo "Installing dependencies..."
    - npm install

    karate_test:
    stage: test
    script:
        - echo "Running Karate tests with tag: $KARATE_TAG and parallelism: $KARATE_PARALLEL"
        - node ./utility/karate.js ./features -f html,cucumber:json,junit:xml $KARATE_PARALLEL -t=$KARATE_TAG
    artifacts:
        paths:
        - target/karate-reports/
        when: always
    retry: 1

🧩 3.2 Root .gitlab-ci.yml to Include Pipeline
📄 .gitlab-ci.yml

include:
    - local: 'pipeline/.gitlab-ci.yml'

✅ This structure allows multiple modular .yml files later.

🆓 3.3 Push CI Setup
git add .gitlab-ci.yml pipeline/.gitlab-ci.yml
git commit -m "🚀 Setup GitLab CI pipeline in pipeline folder with dynamic variables"
git push

✅ 🎉 Done!
Your Karate tests now run in GitLab CI/CD with:
Dynamic tag-based execution (@sample)
Configurable parallelism (via GitLab UI or .env)
Modular pipeline structure

Execute Pipeline Screenshot:

Pipeline Result Screenshot:

Execution Report Screenshot:

🔮 What’s next?
I’ll be sharing deeper insights on:

🔥 Advanced API testing scenarios with Karate Common utilities
🔐 Authentication flows and security testing
⚙️ Custom reusable functions and framework enhancements
📂 Real-world project examples and best practices

Stay tuned 🔔 and follow for updates! 🚀✨