Forem: shittuay

No Application Is Unhackable But Most Developers Are Making It Way Too Easy for Attackers

shittuay — Tue, 21 Apr 2026 00:21:35 +0000

A real-world lesson from the Vercel breach of April 2026

After 10+ years in cloud and DevOps engineering, I've noticed a pattern. When a breach happens, the first thing developers do is look at their own code. They audit their endpoints, check their authentication logic, review their database queries. Sometimes the vulnerability is right there.

But more and more, it's not. It's sitting quietly in a tool someone on your team trusted. A connection approved months ago without a second thought. A .env file on a developer's laptop. An OAuth permission screen someone clicked through in 30 seconds.

The Vercel breach in April 2026 is a perfect example of this — and every developer building and shipping software should pay close attention to it.

What Actually Happened to Vercel
Vercel — the company behind Next.js, trusted by millions of developers to host and deploy their applications — disclosed a serious security incident. Customer credentials were stolen. A threat actor posted the data for sale at $2 million.
But Vercel's own code was never the entry point. Here's how it actually unfolded:
A third-party AI tool called Context.ai had one of its employees infected with Lumma Stealer malware in February 2026. That malware harvested credentials — Google Workspace logins, API keys, Supabase tokens, Datadog tokens. The attacker then used a compromised OAuth token to access Vercel's Google Workspace.

Here's the detail that really matters: Vercel wasn't even a Context.ai customer. One Vercel employee had personally signed up for the tool using their enterprise account and granted "Allow All" permissions. That one action was enough to open a path into Vercel's internal environment.

From there, environment variables that weren't marked as sensitive were exposed. And those environment variables contained API keys, database credentials, and third-party service tokens. Crypto teams hosted on Vercel scrambled to rotate credentials. The Solana-based exchange Orca confirmed their frontend was on Vercel and rotated everything as a precaution. The attacker had nearly a month inside before anyone noticed.

Your Attack Surface Is Bigger Than Your Code
Most developers think about security in terms of their own code. But your real attack surface in 2026 includes every SaaS tool your team uses, every OAuth app connected to your Google, GitHub, or AWS accounts, every npm package in your node_modules, every Python package in your requirements.txt, every CI/CD integration in your pipeline, and every browser extension installed on your developers' machines.

You are only as secure as the least secure third-party tool in your ecosystem.
This is what a supply chain attack looks like in practice — and it's becoming the most common vector for serious breaches. SolarWinds. Log4Shell. The XZ Utils backdoor. Now Vercel. The attackers aren't breaking down the front door anymore. They find a side window left open by someone you trusted.

The OAuth Trap
OAuth is convenient. It's how you connect tools to your GitHub or log into apps with your Google account. The problem is that OAuth tokens can carry enormous permissions, and most people click through the authorization screen without reading what they're actually granting.

In the Vercel breach, one employee clicked "Allow All" on a third-party tool using their enterprise account. That was it. The attacker was in.

Go audit your OAuth connections right now. For Google, head to myaccount.google.com/connections. For GitHub, it's github.com/settings/applications. Check your AWS IAM Console under Identity Providers too. Revoke anything you don't recognize or haven't used recently.

When a tool asks for permissions, ask yourself whether it actually needs access to your entire workspace or just a small piece of it. Grant only what's necessary. And never use your company enterprise account to try out personal tools — that's how one curious employee becomes an incident report.

In your own application, be deliberate about the OAuth scopes you request:

 Too broad — never do this
SCOPES = ["https://www.googleapis.com/auth/cloud-platform"]

 Minimum required — do this
SCOPES = ["openid", "email", "profile"]

The .env File Risk Nobody Talks About Enough
Almost every developer has a .env file on their machine right now. It's convenient, it works, and it's in .gitignore — so most people feel fine about it.

The risk isn't that you're careless. The risk is that things go wrong in ways you don't expect.
You're moving fast on a new feature, you create a branch, you make a commit — and your .env file ends up in your Git history. Even if you remove it in the next commit, the history is there unless you force-rewrite it. That's one scenario.

Another is the malware scenario. Lumma Stealer — the same malware that started the Vercel chain — specifically targets browser-stored credentials and local files. If a machine gets infected, the .env file is one of the first things that gets sent out. And if you're using a shared dev environment, a cloud IDE, or you've deployed with your .env file on a remote server, you've expanded the risk further than you probably realize.

The simplest protection is a pre-commit hook that scans for secrets before any commit leaves your machine:

pip install detect-secrets pre-commit
detect-secrets scan > .secrets.baseline

 .pre-commit-config.yaml
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

pre-commit install

Every git commit will now scan for secrets automatically before anything gets staged.

Also keep a .env.example committed to your repo with placeholder values so teammates know what's needed without ever seeing the real credentials:

 .env.example — committed to Git
ANTHROPIC_API_KEY=your-key-here
DATABASE_URL=postgresql://user:password@localhost/yourdb
AWS_REGION=us-east-1

And scan your Git history to make sure nothing slipped through in the past:

pip install trufflehog
trufflehog git file://. --since-commit HEAD~100

For production, the cleanest approach is to pull secrets directly from AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager at runtime so secrets never touch the filesystem at all:

import boto3
import json
from functools import lru_cache

@lru_cache(maxsize=None)
def get_secret(secret_name: str, region: str = "us-east-1") -> dict:
    client = boto3.client("secretsmanager", region_name=region)
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])

 config.py
secrets = get_secret("myapp/production")
API_KEY = secrets["ANTHROPIC_API_KEY"]
DB_URL = secrets["DATABASE_URL"]

Your Dependencies Are Also Code You're Running
Every package you install is code executing inside your application. You've probably never read the source of requests, boto3, or express. Neither has most of your team. You're trusting that maintainers are doing the right thing and that the package hasn't been tampered with. Sometimes that trust gets broken.

Run regular dependency audits:

 Python
pip install pip-audit safety --break-system-packages
pip-audit
safety check -r requirements.txt

 Node.js
npm audit
npm audit fix

Add scanning to your CI/CD pipeline so it runs automatically on every push:

 .github/workflows/security.yml
name: Security Scan
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Scan for secrets
        uses: trufflesecurity/trufflehog@main
        with:
          path: ./
          base: main

      - name: Python audit
        run: |
          pip install pip-audit safety
          pip-audit
          safety check -r requirements.txt

      - name: Node audit
        run: npm audit --audit-level=high

Also pin your dependency versions. A floating version like requests>=2.0 means your next deployment could pull in a compromised update without you knowing:

 requirements.txt
requests==2.31.0
fastapi==0.110.0
anthropic==0.21.3

Detecting a Breach Before It Gets Worse
The Vercel attacker had nearly a month of access before anyone noticed. Thirty days of reading environment variables, moving through internal systems, and collecting data while everything looked normal on the outside.
Prevention matters, but detection matters just as much. You want to know the moment something unusual happens in your environment.
Enable AWS GuardDuty on your account. It uses machine learning to flag unusual API calls, suspicious login patterns, and potential compromise. It takes about five minutes to set up:

aws guardduty create-detector --enable --region us-east-1

Set up CloudTrail alerts for API calls that should rarely happen in normal operations:

aws cloudwatch put-metric-alarm \
  --alarm-name "SuspiciousIAMActivity" \
  --metric-name "ErrorCount" \
  --namespace "CloudTrailMetrics" \
  --statistic Sum \
  --period 300 \
  --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --alarm-actions YOUR_SNS_TOPIC_ARN

The API calls worth watching closely are CreateAccessKey (new credentials being generated), AssumeRole (identity switching), GetSecretValue (someone reading your secrets), and DeleteTrail (an attacker trying to cover their tracks).

A Practical Checklist to Start With
SECRETS
☐ No secrets in .env on production servers
☐ AWS Secrets Manager / Vault / GCP Secret Manager in use
☐ .env is in .gitignore
☐ .env.example committed with placeholder values
☐ detect-secrets pre-commit hook installed
☐ Git history scanned with TruffleHog

OAUTH AND THIRD-PARTY TOOLS
☐ All OAuth connections audited (Google, GitHub, AWS)
☐ Unused OAuth apps revoked
☐ Least privilege enforced on all integrations
☐ Enterprise accounts never used for personal tool signups
☐ MFA enabled on all accounts

DEPENDENCIES
☐ pip-audit and npm audit running in CI/CD
☐ Dependency versions pinned
☐ GitHub Dependabot enabled
☐ TruffleHog scanning in GitHub Actions

DETECTION
☐ AWS GuardDuty enabled
☐ CloudTrail enabled with alerts configured
☐ Secret rotation runbook documented
☐ Incident response plan exists

The Bigger Picture
No application is unhackable. Vercel is a serious engineering organization with real security investment, and they still got breached — not through their own code, but through a tool a single employee connected to their account.
The developers who get hurt the most aren't always the ones who made the most mistakes. They're often the ones who were too trusting of the ecosystem around them.
That AI productivity tool someone on your team installed last month — it has OAuth access to your Google Workspace. That npm package with millions of weekly downloads it might be maintained by one person who just had their credentials stolen. That CI/CD integration you set up six months ago when did you last check what it can access?

Security isn't something you set up once and forget. It's something you maintain, review, and take seriously on an ongoing basis. Audit your OAuth connections. Add a pre-commit hook. Enable GuardDuty. Pull secrets from a secrets manager.
You can't guarantee you won't be targeted. But you can make sure that when someone tries, they don't get very far.

If you found this useful, follow me for more content on cloud infrastructure, DevOps, and practical security. Drop any questions in the comments — happy to go deeper on any of this.

Connect with me on LinkedIn | X | GitHub

Tags: #security #devops #webdev #cloud #aws #python #javascript #opensource #vercel #supplychainsecurity

Claude Code and the Future of Building: A Developer's Honest Take

shittuay — Thu, 16 Apr 2026 21:07:22 +0000

I want to talk about something I keep seeing, and I think it is time someone said it plainly.
Every time a new wave of AI capability drops, a certain group of people shows up like clockwork. They say the tools are just for vibe coding. They say the apps people are building with them have no security. They say it is all going to collapse. They say you are going to get hacked. They sit on the sidelines cataloguing reasons why this is all going to fail, and they say it with a confidence that you would think they had already watched it happen.
I have seen this before. I remember when ChatGPT was first released.
When it launched, it could not do everything. It was far from perfect. But it was smart enough to cut a 15-day debugging session down to 4 or 5 days for early adopters who gave it a chance. I remember vividly how people would respond when someone shared a fix they got from the model. They would ask for the source. Even if the fix worked, they wanted a citation. And honestly, that skepticism was fair. Nobody wants to inject new problems into their codebase chasing an unverified solution. That caution made sense then.
But fast forward to today, and the landscape is completely different.
I use Claude Code inside VS Code to write production code, catch and fix vulnerabilities, ship new features, and handle tasks I never imagined being able to automate from inside my editor. I run a platform called devopsagent.io, and I can tell you firsthand that the things these models can help you build today, compared to what was possible when they were first released, is not even the same conversation.
That shift changes everything for builders.
If you have an idea, a real one, something that solves a problem and can generate revenue, you now have the tools to turn that idea into a working application without needing an army of engineers. From your VS Code environment, working alongside Claude Code or whichever model fits your stack, you can reach a level of execution that would have seemed out of reach before.
But I want to be honest with you, because this is not a motivational poster. There are nights you will not sleep. There are sessions where you will hit a rate limit and just have to sit there and wait, because you cannot afford the max plan. I know that feeling well. When the wait becomes impossible, I switch over to the API directly to keep moving. You find a way. That is what building looks like.
The reason I am writing all of this is simple. If you are not learning how to work with artificial intelligence right now, you are going to be behind in a way that is very hard to recover from. The people who are building with these tools today, treating them seriously, learning their limitations and their strengths, those people are going to look back in a few years and be grateful they started when they did.
Some will say AI is not safe. Sure. There are a lot of things in this world that are not entirely safe, including some of the food on our plates. That reality does not mean you disengage. It means you get involved early enough to understand how to protect yourself when problems arise. The builders who are in the room when things break are the ones who learn how to fix them.
Start now. Build the thing. Learn as you go.
The models are only getting better, and the window where getting in early still counts for something is still open. Do not wait for someone else to build your dream.

CloudEngineering #DevOps #ArtificialIntelligence #ClaudeCode #BuildInPublic #werisebyliftingothers

Building an AI DevOps Agent: Architecture and Live Infrastructure Automation

shittuay — Tue, 14 Apr 2026 19:02:45 +0000

The DevOps Agent started with a simple question: what if you could talk to your infrastructure like you talk to an AI — and have it actually execute safely?

Not plan generation. Not policy explanations. Real, audited, reversible infrastructure changes. Stop an EC2 instance. Resize it for cost. Tighten a security group. All from chat. All with human approval gates.

This post walks through how we built it, why the safety model matters, and what took us by surprise.

The Problem We Solved

DevOps teams live in a purgatory:

Terraform is powerful but requires code review, planning cycles, apply ceremonies
Cloud consoles are click-heavy and error-prone at scale
Scripts work but lack audit trails
LLMs can suggest infrastructure but can't execute

The gap: between recommendation and action. A model can tell you "your t3.large is underutilized, downsize to t3.medium, save $X/month." But you still have to:

SSH into the console
Stop the instance
Modify the instance type
Start it
Monitor for issues
Log the change somewhere

That's 10 minutes of manual work. For one instance. In a fleet of 200.

We built an agent that bridges this gap. You say "downsize my underutilized instances," it:

Lists them
Queries CloudWatch metrics
Generates a change plan
Waits for your approval
Executes (stop → modify → start → monitor)
Creates an audit record
Offers rollback if something goes wrong

One message. One approval. Done.

The Architecture: Three Layers

The agent isn't a single model. It's a three-layer stack with safety as the load-bearing wall.

Layer 1: The Orchestrator (`src/agent/core.py`)

The DevOpsAgent class is the conductor. It:

Takes a user message
Decides which tools to call (from a registry of 728 tools across AWS, K8s, Azure, GCP, Terraform, Docker, Git, etc.)
Streams the response back to the user
Logs every decision

Here's the critical part: **it doesn't make decisions in a vacuum. Every tool call is validated before execution.

class DevOpsAgent:
    def _call_tool(self, tool_name, tool_input):
        # Before execution: validate
        validation = self.safety_validator.validate_tool_call(
            tool_name, tool_input
        )

        if not validation.is_safe:
            # Return to user with explanation
            return f"Cannot execute {tool_name}: {validation.reason}"

        if validation.requires_confirmation:
            # Queue for manual approval
            return self._create_change_request(tool_name, tool_input)

        # Safe to execute immediately
        result = execute_tool(tool_name, tool_input)
        self._audit_execution(tool_name, tool_input, result)
        return result

This is the whole game. Safety is checked before, not after.

Layer 2: The Safety Validator (`src/agent/safety.py`)

This is where we say no.

Every tool is classified into three risk levels:

LOW: Read operations, creates that don't touch production. No confirmation needed.
MEDIUM: Updates, restarts, IAM changes, scale operations. Requires confirmation.
HIGH: Deletes, terminates, destructive operations. Requires confirmation + escalates to CRITICAL on production workloads.

The validator does pattern matching on tool parameters. Same tool, different inputs, different risk:

# manage_ec2_instance with action='start' → LOW (safe)
# manage_ec2_instance with action='stop' → MEDIUM (requires confirmation)
# manage_ec2_instance with action='terminate' → HIGH (critical)

This action-aware classification is the difference between a helpful tool and a footgun. Early versions classified the entire tool as HIGH risk, so every operation needed approval. Useless. Now, start/stop are gated appropriately.

Current classification:

728 total tools registered
42 tools explicitly classified as HIGH risk (delete/terminate operations)
36 tools explicitly classified as MEDIUM risk (updates/IAM/scale)
650 tools default to LOW risk (reads/creates)
Auto-detection fallback: any tool with prefix delete_, terminate_, destroy_, remove_, drop_, or purge_ is auto-classified as HIGH, even if not explicitly listed

The Tool Lifecycle: From Chat to Execution

When you type "downsize my ec2 instances that are under 20% CPU," here's what happens:

Step 1: Message Validation (20ms)

Three gatekeepers run in parallel before any credit is deducted:

Input Sanitizer — 10 regex rules checking for injection patterns. Blocks ~5% of messages from probing attempts.
Haiku Pre-Screen — Claude Haiku in structured mode, classifies: prompt injection, malware request, data exfiltration, etc. ~99% accuracy, fails open on error.
System Prompt Hardening — Fixed identity, no self-disclosure, refuse malware generation.

Violate any of these 3 times in an hour, your account locks for 60 minutes.

Step 2: Tool Selection (100-500ms)

The agent (Claude Sonnet or Claude Opus, depending on tier) reads:

Your message
Your infrastructure context (what you have deployed, learned from prior syncs)
The full tool registry (728 tools with descriptions + input schemas)
Active skills (domain expertise injected based on message keywords)

It decides: which tools are relevant? In what order?

This is where the agent is creative and where it can hallucinate.

We catch this with tool schema validation. Every parameter is validated against the registered input schema before execution. Mismatches are returned to the user with a correction prompt.

Step 3: Safety Classification (10ms per tool)

For each tool call:

tool_name='resize_ec2_instance'
tool_input={'instance_id': 'i-0123456789abcdef0', 'new_instance_type': 't3.medium'}

→ SafetyValidator.validate_tool_call(tool_name, tool_input)
→ Classification: HIGH RISK (stops and modifies running instance)
→ Action: Queue for approval + create InfrastructureChange record

Step 4: Change Request Creation (if needed)

For MEDIUM or HIGH risk operations, we don't execute. We create an InfrastructureChange record:

change = InfrastructureChange(
    user_id=current_user.id,
    resource_type='ec2',
    resource_id='i-0123...',
    action_type='resize',
    status='pending_approval',
    change_details={
        'current_type': 't3.large',
        'new_type': 't3.medium',
        'estimated_savings': '$45/month'
    },
    created_by='agent'
)
db.session.add(change)

This record is immutable (audit trail), has rollback data captured before execution, and awaits human approval.

Step 5: Execution & Audit

When approved:

ec2_client = get_user_boto_client(user_id, 'ec2', region)
ec2_client.stop_instances(InstanceIds=[instance_id])
ec2_client.get_waiter('instance_stopped').wait(InstanceIds=[instance_id])
ec2_client.modify_instance_attribute(
    InstanceId=instance_id,
    InstanceType={'Value': 't3.medium'}
)
ec2_client.start_instances(InstanceIds=[instance_id])

And we log every step:

[2026-04-15 14:32] EXECUTION_START - resize_ec2_instance
[2026-04-15 14:32] CHECKPOINT: instance stopped (i-0123...)
[2026-04-15 14:33] CHECKPOINT: instance type modified (t3.large → t3.medium)
[2026-04-15 14:35] CHECKPOINT: instance started
[2026-04-15 14:35] EXECUTION_SUCCESS - savings: $45/month

Step 6: Rollback

Rollback data is captured before execution:

rollback_data = {
    'instance_id': 'i-0123...',
    'original_type': 't3.large',
    'original_state': 'running'
}

One click reverts everything. All logged.

What We Got Wrong (And Fixed)

v1: manage_ec2_instance Was LOW Risk Across All Actions

The bug: All actions (start, stop, terminate) were classified LOW risk. User could terminate a prod instance without confirmation.

The fix: Action-aware validation. Same tool, different risk based on parameters.

v2: Sandbox Mode Fabricated Resources

The bug: Users in sandbox could see fake resources that didn't match reality. Confusion when they tried to execute the same operation in production.

The fix: Sandbox returns mock results with realistic data from their actual infrastructure. When in sandbox, real API calls still happen for reads, but writes are simulated.

v3: No Rollback Data Captured

The bug: After execution, if something went wrong, we had no way to revert.

The fix: Every change request captures rollback data before execution. Rollback creates a new change request that reverses the previous one.

v4: Model Hallucinating Tool Results

The bug: Free-tier model (Kimi K2.5) would occasionally write [Executing tool...] in plain text instead of using function calling — fabricating results.

The fix: Two-tier hallucination detection. Strong patterns (fake execution markers) always blocked. Streaming buffer: accumulate response, check before yielding. Dedicated regex to detect fake markers vs real tool output.

The Numbers

728 tools across 17 cloud/DevOps platforms
248 tests covering auth, billing, tools, safety, user isolation
3-layer guardrails — input sanitizer, Haiku pre-screen, system prompt hardening
0 unintended deletions in production use
0 credential leaks (Fernet encryption, per-user keys)
100% change audit trail (immutable journal entries)

The Moment It Clicked

Six months in, a user executed this from chat:

Me: "My database is at 85% CPU and costs $400/month. What can I do?"

Agent: "I found 3 options:
1. Upgrade from db.t3.xlarge to db.r5.2xlarge (better performance, higher cost)
2. Reduce IOPS from 3000 to 1000 (saves $120/month, risk of throttling)
3. Enable read replicas and route read traffic there (saves $200/month, requires app changes)"

Me: "Let's do #3 and also downsize the original instance"

Agent: [Creates change request #1: Create read replica]
       [Creates change request #2: Reduce primary IOPS]
       [Creates change request #3: Downsize to db.t3.large]

Me: [Approves all 3]

Agent: "Complete. Your database now costs $180/month (was $400)."

That user went from "we need a DBA to optimize this" to "I did it in 5 minutes from chat."

That's the win.

Try It

DevOps Agent is live at devopsagent.io.

Free tier: 20 credits/month, full agent access. No credit card needed.

You get:

Chat with your infrastructure
Change request approval workflow
Full audit trail
Sandbox mode for learning
Rollback for any change

Questions or feedback? Open an issue or email hello@devopsagent.io.

# log-insight This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Crushing the Command Line

shittuay — Fri, 09 May 2025 23:59:58 +0000

log-insight

This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Crushing the Command Line

What I Built

I built a command-line tool called log-insight to help developers and system administrators quickly analyze log files. Log files are like digital diaries that record what a computer system or application is doing, and they can become very large and difficult to navigate. log-insight automates extracting key information from these files, making it easier to identify errors, understand system behavior, and troubleshoot problems.

Specifically, log-insight provides the following functionalities:

Keyword Counting: Counts the number of times a specific keyword appears in the log file. This is useful for quickly assessing the frequency of particular events, such as errors, warnings, or successful operations. Users can specify the keyword they want to search for.
Top Entry Analysis: Identifies and displays the most frequent log entries. This helps in understanding recurring patterns and identifying the most common activities or issues within the system. The number of top entries to display is configurable.
Error Timestamp Extraction: Extracts and lists the timestamps of log entries that contain a specific error keyword (e.g., "error", but this can be customized). This allows users to quickly pinpoint when errors occurred, which is crucial for debugging and incident analysis.

log-insight aims to "crush the command line" by providing a simple, efficient, and scriptable way to gain valuable insights from log data, directly from the terminal. It eliminates the need for manual searching and complex text processing, saving time and improving productivity.

Demo

[Demo Video Link]

Code Repository

https://github.com/shittuay/log-insight.git

How I Used Amazon Q Developer

I used Amazon Q Developer as an invaluable assistant throughout the development process of Log-InSight. It significantly accelerated my workflow in several ways:

Code Snippet Generation: Amazon Q helped me quickly generate efficient code snippets for core functionality like file reading, text processing, and handling command-line arguments, saving me substantial development time.
Library Implementation: When implementing the argparse library for command-line arguments and regex patterns for log parsing, Amazon Q provided clear examples and implementation guidance that streamlined the development process.
Debugging Support: Whenever I encountered errors or unexpected behavior, Amazon Q helped identify root causes and suggested practical solutions, particularly for complex regex patterns and file handling edge cases.
Best Practices: Amazon Q provided guidance on writing clean, Pythonic code, ensuring that log-insight remains maintainable and follows industry standards.

Specifically, Amazon Q was instrumental in:

Developing efficient file reading mechanisms that could handle large log files
Creating optimized keyword searching algorithms that maintain performance even with massive logs
Structuring the command-line interface to be intuitive and following Unix philosophy
Implementing proper error handling for various edge cases

Amazon Q Developer proved to be like having an expert pair programmer available 24/7, offering suggestions, improvements, and solutions throughout the entire development cycle.