Forem: Shintaro Fukatsu

I ran AWS Security Agent's full pipeline on my personal project: Design Review, Code Review, and Pentest

Shintaro Fukatsu — Tue, 05 May 2026 15:56:40 +0000

A quick note before you read: I'm still learning English. I rewrote sections I wasn't confident about, so there may be places that don't read naturally. Sorry about that. The Japanese version has full step-by-step setup instructions if you want the complete picture: Japanese article on Qiita

After You Ship, Then What?

A lot of energy goes into security before a release: threat modeling, code reviews, maybe a pentest if budget allows. Then the app ships, and that security energy quietly dissipates. The system keeps running. No one re-checks the design against updated policies. Code keeps getting merged without systematic security review. The pentest from launch day grows stale.

This isn't negligence. It's just the reality of how operational security gets deprioritized once a product is live.

AWS Security Agent caught my attention precisely because it covers all three security checkpoints, design review, code review, and penetration testing, and they're available anytime, not just at launch. That includes the operational phase after release, which is exactly where the security gap tends to open up. The penetration testing feature went GA on March 31, 2026, the first of the three to move beyond Preview. I decided to run the full pipeline on my own personal project and see what actually came out the other end.

What Is AWS Security Agent?

AWS Security Agent is an AI-driven security tool that covers three distinct phases of the development lifecycle:

Feature	Status (as of April 2026)
Design Review	Preview
Code Review	Preview
Penetration Test	GA (March 31, 2026)

Design Review analyzes your architecture documents, requirements, and design specs against AWS security best practices. You can also define custom security requirements in natural language, which is useful for organizations with internal compliance policies that go beyond AWS defaults.

Code Review integrates with GitHub and automatically posts security-focused review comments on pull requests.

Penetration Test actively attacks your running application, combining static code analysis (SAST) with dynamic testing (DAST), and even generates fix PRs automatically for detected vulnerabilities.

All three features live under a single "Agent Space," which is the container you set up once and then operate from.

Note: As of April 2026, Design Review and Code Review are still in Preview. Verify current status in the official docs before using.

The App Under Test

Rather than running this on a purpose-built vulnerable app like OWASP Juice Shop, I used a real side project: a Kahoot!-style real-time quiz app.

The architecture is a standard serverless stack:

Frontend: CloudFront + S3
API: API Gateway (REST + WebSocket)
Backend: Lambda (Python)
Database: DynamoDB
Auth: Cognito (admin), 6-digit access code (participants)

Scale: about 3,500 lines of code, 22 API endpoints. Participants join via a rotating 6-digit code and receive quiz results over WebSocket. Admins authenticate via Cognito to manage questions and control the flow.

Not a toy app, but not enterprise-scale either. A realistic target for a solo developer running a serious security check.

Design Review

What You Upload

Design Review accepts DOC, DOCX, JPEG, MD, PDF, PNG, and TXT files, up to 5 files per review (2 MB each, 6 MB total). I uploaded four Markdown documents:

architecture.md: system architecture
requirements.md: functional requirements
security-design.md: security design
sequence-diagram.md: authentication and data flows

One thing that wasn't obvious: the agent reads across all uploaded files and synthesizes them together. I was half-expecting isolated per-file analysis, but it correctly interpreted relationships between documents.

Custom Rules in Plain Language

Beyond AWS managed rules, you can add your own security requirements as free-form text. The agent evaluates uploaded documents against your custom rules just as it does against the managed ones.

This is more useful than it sounds. In many organizations, security policies exist as internal documents written in plain language. Being able to paste those directly, without translation, reformatting, or mapping to a formal schema, lowers the barrier significantly for non-security-specialist teams.

I added two custom rules: a log retention requirement (365 days minimum) and a data residency requirement (specific region only) in Japanese. Both worked as expected.

The Probabilistic Finding

Here's the part that surprised me most.

My first design review came back clean with no non-compliant findings. On the second run after adding the custom rules, two things happened: the custom rules correctly flagged violations, but a third finding appeared that had nothing to do with my additions. "Secret Protection Best Practices" flipped from compliant to non-compliant, with the same documents.

The agent missed something on the first pass and caught it on the second.

This isn't a bug. It's a property of probabilistic AI systems. The practical takeaway: a single design review run giving you "all clear" doesn't mean you're clean. Multiple runs on important documents increase coverage. It's the same reason human security reviewers do multiple passes or pair reviews.

The design also makes re-running easy: you can clone an existing review and modify only the rules, which fits naturally into an iterative workflow.

Code Review

GitHub Integration

Code Review requires connecting a GitHub account to your Agent Space. One important constraint: a single AWS account can only be linked to one GitHub account. In multi-account AWS Organizations environments, you need to decide upfront which account owns the code review integration, as you can't spread it across accounts.

Once connected, aws-security-agent[bot] automatically posts review comments on pull requests. Comments arrive in two phases: an initial acknowledgment, followed by detailed findings. The experience feels similar to having a dedicated security reviewer on every PR.

I received an XSS finding on a feature PR, fixed it with the help of Claude, opened a new PR, and ran the review again. The cycle was smooth.

The Scope Gap (and How It Matters)

Code Review operates on PR diffs, meaning the delta between the head branch and the base branch. It does not scan your entire repository.

This became important later. I had intentionally left dummy credentials in the repo (admin email, password, Cognito User Pool ID, Client ID) to see if the system would catch them. The code review never flagged them, because those files weren't part of any PR diff.

More on this in the pentest section. The short version: code review and penetration testing have complementary, non-overlapping coverage. You need both.

You can reduce false positives by adding a filtering.md file to your repository. The agent's own comment explains it:

"Customize which findings are reported by adding a filtering.md file to your repository. Use it to exclude files from analysis and provide context hints that reduce false positives."

Penetration Test

Setup: Domain Verification

Before the agent can attack your app, it needs to verify you own the target domain. You register your domain in the Agent Space console, get a DNS TXT record token, add it to your DNS configuration, and then hit "Verify." Standard domain verification with no surprises.

One configuration detail that matters: if your app uses Cognito for authentication, you need to explicitly add the Cognito endpoint (https://cognito-idp.<region>.amazonaws.com) to the "accessible URLs" list. The agent needs to reach it during authentication flows. I missed this on my first run.

The First Run Failed

First run: 2.33 task hours, $116.50 in cost (about $50 per task-hour), and a failure after 2.5 hours.

The agent repeatedly failed to log in because the Cognito endpoint wasn't in the allowed list. It kept retrying, burned through task time, and eventually timed out. I came back from stepping away to find it had already failed.

After digging through the logs, I found the issue, updated the config, and ran it again.

78 Actions, 7 Hours, 44 Minutes

The second run succeeded. 7 hours 44 minutes of wall time, 34.41 task hours (the agent runs parallel workers, so task time accumulates faster than wall time), at $1,720.50 total.

The agent ran 78 distinct actions across multiple attack categories:

A few observations worth noting:

Code scanner (16 actions) is the largest category. Penetration testing typically evokes images of active exploits against a running system. But Security Agent starts with static analysis, reading the source code before touching the app. The sequence appears to be: network scan to understand structure, static code analysis to find attack surface, then dynamic testing to exploit.

IDOR (10 actions) gets serious coverage. Insecure Direct Object Reference is about changing /api/user/123 to /api/user/124 and seeing if you get someone else's data. It's notoriously difficult for automated scanners because it requires understanding authorization logic. Running 10 variations suggests the agent is trying multiple approaches: sequential IDs, predictable patterns, and authorization boundary probing.

Privilege escalation (9 actions) suggests chained attack simulation, not just finding one vulnerability, but testing whether that vulnerability can be compounded into escalated access across multiple steps.

Cleanup (5 actions) is worth mentioning. AWS Security Agent recommends running the pentest against a non-production environment, and these actions restore that environment after testing is complete.

What Was Found

After 7 hours 44 minutes, the results:

Severity	Count
Critical	1
High	5
Medium	7
Low	2
Info	2
Total	17

Risk types included: Business Logic Vulnerabilities, Server Error 500, Information Disclosure, Security Misconfiguration, Insecure Direct Object Reference, Default Credentials, Authentication Bypass, Unrestricted Resource Consumption, and Privilege Escalation.

The Critical Finding: What Code Review Missed

The most significant finding was also the most instructive about how these tools complement each other.

The finding was: "Plaintext Admin Credentials and Live Infrastructure Identifiers Committed to Repository"

Those dummy credentials I mentioned (admin email, password, Cognito User Pool ID, Client ID) had been sitting in the repository the whole time. Code review never caught them because they were never part of a PR diff. The pentest's static analysis scanned the entire repository, found them immediately, and flagged them as Critical.

What happened next was also noteworthy. The agent automatically created a PR titled "Security Fix: Remove Hardcoded Credentials and Enable MFA (CWE-798, CWE-312)". The code review bot then reviewed that PR and returned "No issues identified." The full loop of detect, remediate, and verify ran without any human intervention. That said, I reviewed the PR contents myself and did the final merge manually.

This made the boundary between code review and penetration testing concrete: code review covers what changed, and pentest covers what exists. Neither replaces the other.

The Report

Results are exportable as a PDF pentest report in the same format you'd receive from a security consulting firm: Executive Summary, per-finding Descriptions, Steps to Reproduce, CVSS v3.1 vectors, and recommended remediation. The information density is high.

The report is currently English-only. Output in local languages would make the tool more accessible to a broader range of organizations, and I hope that's something that comes in a future update.

Operational and Organizational Considerations

The Agent's Output Is Not a Final Verdict

This is the most important thing to understand before deploying this in an organization. When the agent says "compliant" or "no issues," that's input to a human decision, not the decision itself. The workflow needs to include human review of findings before any conclusions are acted on. "AWS Security Agent said it was fine" is not a sufficient audit trail.

Cost Planning Is Genuinely Hard

Pentest cost is usage-based at $50 per task-hour. The problem: you can't accurately predict task hours before running the test. It depends on application complexity, endpoint count, authentication flows, and how many vulnerabilities the agent finds to pursue.

My 22-endpoint, 3.5K-line app consumed 34.41 task hours. That number shouldn't be extrapolated to other apps. It's just a data point.

There's a free trial of up to 200 task hours, valid for 2 months from your first pentest run. The practical advice: run a test on a smaller surface during the trial period to develop a cost baseline before committing to a budget line item. Annual IT budget planning doesn't pair well with "we'll know what it costs after we run it."

AI Is Broad, Humans Go Deep

AWS Security Agent doesn't replace professional penetration testing. For regulatory compliance in financial services, healthcare, or other regulated industries, certified vendor pentests remain necessary.

What it does replace is the gap: the period after release when continuous security validation isn't happening because it's expensive and slow. The framing that makes sense organizationally is that Security Agent covers ongoing monitoring while human experts cover compliance certifications.

Multi-Account Organizations

If you're running AWS Organizations with multiple accounts, design the Security Agent topology upfront. Custom security requirements are managed per Agent Space, so decide early which account centralizes them. Otherwise you'll be updating the same policies across every account every time your security policy changes.

Code review has an additional constraint: one AWS account per GitHub account. This affects where you anchor the integration in a multi-account setup.

Conclusion

Running all three features in sequence on the same application made one thing clear: they're not independent tools. They're stages in a pipeline, and each stage catches different things.

Design Review found documentation gaps and policy violations. Code Review caught issues in active development. Penetration Test surfaced what neither had found, because it operates on a different scope, covering the full repository and the running application together.

The thing I took away most is that this pipeline addresses a real operational gap, not just a launch-time checklist. Security requirements change. Code accumulates. Operations staff turn over. Legacy systems that have been running for years without a formal security review are perhaps the clearest example of this gap. The cost of bringing in specialists often makes regular assessments impractical for those systems, and this kind of tooling changes that equation. Running this periodically on live systems, on a regular cadence rather than just at deployment, is exactly the use case it was built for.

Shipping an app isn't the end of your security work. It's closer to the beginning of the next cycle.

I also did a lightning talk on this topic in Japanese at Ops-JAWS Meetup 40. The slides and recording are available if you are interested.

Slides: SpeakerDeck
Recording (Japanese): YouTube

Connect My Local Kiro to the World - Setting Up Tavily Remote MCP Server

Shintaro Fukatsu — Tue, 12 Aug 2025 15:08:47 +0000

Introduction

We have a new member of the family: Kiro! My daughter made Kiro for a summer homework project(Our Kiro is made with UV resin). My daughter gave it to me as a gift! So, I'm going to turn it into a keychain and take it with me to re:Invent 2025!

Now let's get to the main topic. While using Kiro, I recently wanted to check for update information. When I asked via chat, I realized that web search functionality wasn’t available (I'm not an AI engineer, so I'm catching up very slowly).

Previously, I had the opportunity to use "Tavily" during a hands-on session hosted by Minorun. This time, I decided to set it up myself and document the process for future reference.

What is Tavily?

Tavily is an API service designed to allow AI agents and app developers to perform external information searches. Since queries can be made using natural language, it's a valuable tool for chatbots when processing prompts with external data. (This is how I understand it, at least.)
You can start using it for free.

Setup

1. Sign Up on the Tavily Website

Visit the Tavily website and click the "Sign Up" button at the top right.

I registered using my Google account.

After registration, you’ll see the screen where you can obtain the API key. Make a copy to local for late use.

2. Configure MCP Server in Kiro

I performed the setup on an M4 Mac mini.

In the Kiro interface, click “Open MCP Config” under MCP SERVERS in the lower-left corner.

Add the following content to the mcp.json file:
Set the API key obtained earlier after tavilyApiKey= and save.

{
  "mcpServers": {
    "tavily-remote-mcp": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://mcp.tavily.com/mcp/?tavilyApiKey=`Tavily API Key`"
      ],
      "env": {},
      "disabled": false,
      "autoApprove": [
        "tavily_search"
      ]
    }
  }
}

Since I wanted Tavily available for all projects, I set it in User Config. If you only need it for specific projects, it’s better to configure it in Workspace Config.

If “tavily-remote-mcp” shows as Connected under MCP SERVERS in the lower-left corner, the setup was successful.

Connecting my Kiro to the World

Finally, it’s time to search. When I made a search request, the Documentation MCP Server was used instead. Communicate can be tricky sometimes...

This time, I tried phrasing it as "Web search".

And then, the results came up beautifully. This was the moment my Kiro connected to the world. Also, I learned that AWS signed a wind power contract in India, showcasing its investment in renewable energy.

Conclusion

This is a simple guide to setting up the Tavily Remote MCP Server. Most other MCP Servers can be set up similarly. If you’re unsure about the configuration details (what to write in the mcp.json), asking Kiro might be the quickest way.

Although I wrote this for my own reference, I hope it will help anyone else starting out.

Kiro

Shintaro Fukatsu — Tue, 12 Aug 2025 15:06:29 +0000

Which is your favorite logo? Creating JAWS-UG games with Amazon Q CLI.

Shintaro Fukatsu — Mon, 09 Jun 2025 17:05:06 +0000

💡Introduction

There are nearly 600 AWS community groups in the world. There is a community group called JAWS-UG in Japan. There are more than 60 of them. They account for 10% of the community groups in the world. In addition to chapters in specialized fields such as networks and AI/ML, there are also regional chapters.

In order to contribute to regional revitalization, regional chapters are making efforts such as holding study sessions in conjunction with local events. I am involved in the management of one of the regional chapters.

Each branch creates a logo with its own unique characteristics, so there are many logos. I wanted to make it known more, so I decided to make it into a game with Amazon Q CLI.

I hope that you will be interested in JAWS-UG and participate in it.

🦈About JAWS-UG

The event is generally held in Japanese. There are also multinational events hosted by members who are fluent in English.
We also stream live using Zoom, MS Teams, etc., and record the events on YouTube.

AWS User Group Japan JAWS-UG

If you can't attend online or offline, how about YouTube?Prefer asynchronous viewing? Cath past sessions on Youtube.
If you're an engineer who wants to learn Japanese while learning AWS, please join JAWS-UG.

Let's coding with Amazon Q CLI

Now, let's move on to the main topic of game creation. I iteratively developed it using Amazon Q CLI. The first instruction is as follows.

>Please create a game in the folder "jaws-ug-slot-games-web".
This is a slot game that uses a logo.
Press the space bar to stop one lane at a time.

※The actual instructions were given in Japanese

And here is the first screen that was completed.
It came out in a different form than expected. It can't be helped since the instructions were vague.
I asked for corrections one by one and improved it.
Then, I was able to import the image into the game.

Points of ingenuity

Give specific instructions

In order to use the JAWS-UG logo, I asked Amazon Q CLI to use the image file on JAWS-UG's Github. I specified the URL of the relevant repository and instructed it to download and use it, and Amazon Q CLI cloned it locally using the git command and extracted the file.

Fix the small details later

While adding features, there were times when unintended changes were made to the small details, which required further corrections. Therefore, we prioritized the implementation of the main features over the small details.

Branch experimental features to preserve a clean main build.

A vertical loop would have been fine, but I suddenly thought of a reel that slides horizontally. Since you can't know until you try it, I instructed Amazon Q CLI to make changes after making a backup and creating a situation where you can revert to it.

For example, I asked them to create a pattern where the reels flow vertically and one where they flow horizontally.
The result of making the reels flow horizontally is as follows.

How to play the game

Open the game: https://sh-fk2.github.io/jaws-ug-slot-games/
Press Space to start the reels.
Press Space again—timed right—to stop each reel.
Land three matching strips to win!
Hit R to reshuffle logos.

Code is on GitHub too: https://github.com/sh-fk2/jaws-ug-slot-games

Conclusion

By using Amazon Q CLI, I was able to create a game with a screen that utilizes HTML and CSS.

Please try playing the game. Did you find your favorite logo? I would be happy if you found a logo you like. Please join the JAWS-UG of your favorite logo.