Forem: Lei Tang

AWS Account Security Restricted? Here's My 56-Hour Unblock Journey

Lei Tang — Tue, 05 May 2026 03:04:53 +0000

AWS Account Security Restricted? Here's My 56-Hour Unblock Journey

A production incident triggered by leaked Access Keys — the complete process from receiving the AWS security notice to full account recovery

How It Started: A Chilling Email

It was Saturday afternoon when I received an email from AWS — and it was the second day of China's Labor Day holiday.

"We are reaching out to you because your AWS Account may have been inappropriately accessed by a third-party."

Translation: Your AWS account might have been compromised.

Along with the notice came account restrictions — I couldn't start EC2 instances, couldn't use certain AWS services. And to make things worse, we had a production EC2 instance running in Singapore serving overseas users.

The feeling at that moment? Only those who've been through it can truly understand.

My carefully planned holiday vanished the moment I opened that email. For the next three days, I basically lived inside the AWS Console and Support Center.

Phase 1: Initial Response (Hours 0-6)

AWS provided clear recovery steps:

Change the Root account password
Enable MFA for the Root user
Review CloudTrail logs for suspicious activity
Clean up all unauthorized IAM users, roles, and policies
Reply to the support case confirming completion

I immediately logged into the console and followed each step:

Changed the Root password
Enabled MFA
Checked CloudTrail logs — there were indeed unusual API calls
Removed all suspicious IAM resources

After completing everything, I replied to the case: "I've completed all required steps. Please restore my account."

I thought this would be resolved quickly. Little did I know, it was just the beginning.

Phase 2: Root Cause Found, Escalation Requested (Hours 6-18)

During the investigation, our dev team finally identified the root cause of the key leakage — a code vulnerability in our project.

We confirmed that all related code had been fixed. I urgently requested AWS to start the Singapore EC2 instance, which had been down for over 4 hours — overseas business was completely paralyzed and users were complaining non-stop.

The error message when trying to start the instance was devastating:

This account is currently blocked and not recognized as a valid account.
Please contact AWS Support if you have questions.

At this point I realized: Security restrictions aren't automatically lifted once you complete the steps. There's a review process behind it.

Phase 3: Security Review, Answering Questions (Hours 18-24)

On Sunday morning, the AWS security team responded with four questions:

Which location (country/city) are you accessing the account from?
Does anyone else have access to your account (e.g., IT staff)?
If yes, where are they accessing from?
Are all active IAM users and roles authorized by you?

I answered truthfully:

I use a VPN to access my account — common nodes include the US, Singapore, and Hong Kong
No one else has access to my account
All IAM users and roles are authorized by me

The security team said they'd forwarded my case for review and asked me to wait.

I kept refreshing the case page. Every minute felt like an hour.

Phase 4: Cleaning Up Suspicious Resources (Hours 24-30)

Monday morning, the security team found a suspicious S3 bucket:

Bucket name: logs-94591786-wmn5rp

They asked me to confirm whether this bucket was needed for my business — if not, delete it.

I checked and confirmed it wasn't ours (likely created by the attacker), deleted it immediately, and replied to the case.

I thought this would be the last step. I was wrong again.

Phase 5: 48 Hours Passed, at My Wit's End (Hours 30-48)

It had been over 48 hours since the initial notification. The account was still restricted. Production was down. Overseas business was paralyzed.

I sent another strong message through the case:

"We've completed all security measures, but our account remains restricted. If this can't be resolved soon, we'll have no choice but to consider migrating our production services to Alibaba Cloud or other providers."

This wasn't an empty threat — every minute of downtime was causing real damage.

AWS replied that they were working with their service team.

While others were posting vacation photos on social media, I was writing essays in AWS support tickets.

Phase 6: Missed Details, Chat to the Rescue (Hours 48-52)

Monday evening, I received an unexpected email from AWS:

"The team was unable to perform the required remediation steps. There are some actions needed from your side."

They needed me to:

Step 1: Delete unauthorized IAM resources

IAM user "Server": Delete the role or add MFA

Step 2: Reset Root password and update MFA

But I'd already done all of this! Looking closer, the MFA for IAM user "Server" hadn't been properly recognized.

Instead of replying through the case, I chose Chat support this time.

The agent, Melissa, was very professional:

Confirmed my MFA setup for IAM user "Server"
Asked me to reset the Root password and update MFA again
Confirmed everything was in order and updated the service team immediately

Key takeaway: In urgent situations, Chat is far more efficient than case replies. Use it.

Final Chapter: Unblocked! (Hours 52-56)

Monday at 21:48, I finally received the email I'd been waiting for:

*"We've verified that you've taken the required steps, and we've reinstated your AWS account."

The account was finally unblocked! All restrictions were lifted.

From Saturday 14:06 to Monday 21:48 — 56 hours in total. My entire Labor Day holiday was consumed by AWS support tickets.
And the production environment was down for roughly the same duration — an entire weekend plus Monday.

Complete Timeline

Note: All times below occurred during China's Labor Day holiday.

Sat 14:06   — Received AWS security notice, account restricted
Sun 01:08   — Completed initial security steps, requested restoration
Sun 04:14   — Found root cause (code vulnerability), requested escalation
Sun 04:22   — EC2 down for 4h+, urgent request to start instance
Sun 10:38   — Security team asked questions (location, personnel, etc.)
Sun 10:45   — Answered, case forwarded for review
Mon 09:54   — Asked to delete suspicious S3 bucket
Mon 11:06   — Bucket deleted, requested restoration + escalation
Mon 12:11   — 48h+ still restricted, filed strong complaint
Mon 15:07   — AWS confirmed, asked service team to remove restrictions
Mon 18:02   — New requirements: handle IAM Server + reset password
Mon 18:07   — Chat support, completed operations with guidance
Mon 21:48   — Account restored!

Lessons Learned & Recommendations

1. Security First, Prevention Over Cure

The root cause was a code vulnerability that leaked Access Keys. Improvements we made afterward:

Removed all hardcoded credentials from the codebase
Started using AWS Secrets Manager or similar services for key management
Configured git-secrets in git repos to prevent key commits
Set up regular Access Key rotation

2. Understand the AWS Unblock Process

The full process looks like this:

Notice → Remediation Steps → Security Review → Follow-up Actions → Final Review → Unblocked

The entire process can take over 48 hours. If production is involved, have a contingency plan ready.

3. Choose the Right Support Channel

Channel	Use Case	Response Speed
Case Reply	Routine communication, providing info	Slow (hours)
Chat	Urgent issues, need guidance	Fast (instant)
Phone	Critical emergencies	Fastest

For production outages, go directly to Chat or Phone. Don't wait for case replies.

Also, always communicate in English — most AWS security team members don't read Chinese, so using Chinese means it'll go through a translation layer first, adding unnecessary delay.

4. Configure Security Contacts in Advance

AWS strongly recommends setting up alternate security contacts. This way, if something goes wrong with your account, AWS can reach you through multiple channels.

5. Set Up Budget Alerts and Cost Anomaly Detection

One thing I discovered through this incident — AWS Cost Anomaly Detection and budget alerts can notify you of unusual spending as soon as it happens. If we'd set these up, we might have detected the anomaly before AWS did.

Final Thoughts

Technically: Key leakage sounds like something that happens to "other people," but a single code vulnerability can make it your reality. Security isn't a switch — it's a continuous process.

Procedurally: AWS's security review process can be frustrating, but on the flip side, it's protecting your account. If unblocking were trivial, the security防线 would be meaningless.

Mentally: 56 hours of downtime is painful, but instead of complaining about the process, focus on how to accelerate communication and prepare thorough materials.

The bottom line — configure your security contacts, budget alerts, and key management before production goes down. These things always feel like "not urgent" when things are calm. By the time the storm hits, it's already too late.

Have you ever experienced a cloud provider security restriction? Share your story in the comments — I'd love to hear about your experience and what you learned.

Your Go Binary Won't Run on macOS? syspolicyd Is the Culprit

Lei Tang — Tue, 05 May 2026 03:00:57 +0000

A bizarre Go project debugging saga — full investigation assisted by Claude Code

The Strange Bug

It was an ordinary afternoon. I was developing a Go project on my Mac as usual. Code written, go build succeeded, everything looked normal — then ./server started and nothing happened.

No log output. No port listening. No error messages. The process was created, but it was like frozen in place — RSS stuck at 32 bytes.

Even stranger, air (my hot-reload tool) ran perfectly fine — and it's also a Go-compiled binary. Only newly compiled binaries refused to run.

If you've ever run into something similar, you know the drill: first you suspect your code, then the Go version, then you start questioning your life choices.

Fortunately, I used Claude Code (Anthropic's AI coding assistant) throughout this investigation. It was like having an experienced SRE sitting next to me, helping narrow things down step by step. Here's the complete AI-assisted debugging process.

The Investigation: One Elimination at a Time

Step 1: Verify the Build

First, I confirmed it wasn't a code issue:

go build ./...    # Passed, zero errors
go run ./cmd/server/  # Process created, but… no output

Checking the process state:

PID   STAT  RSS  COMMAND
92672 SN      32  server

RSS of 32 bytes basically means the process was created but never actually executed any code.

Step 2: Check Logs (Claude Code's Auto-Discovery)

I asked Claude Code to check the project startup. It automatically read logs/debug.log and noticed the log file timestamps hadn't been updated — meaning InitLogger() was never reached.

The code was stuck somewhere, very early in initialization. Claude Code quickly narrowed the problem to the process initialization phase by analyzing log timestamps and process memory.

Step 3: Confirm Binary Integrity (Claude Code's Automatic Check)

Claude Code ran file and otool -l to analyze the binary:

The binary was compiled correctly, the Mach-O structure was intact — not a corrupted file.

Step 4: Rule Out CGO

I suspected a CGO dynamic linking issue:

CGO_ENABLED=0 go build -o /tmp/testhello ./cmd/server/

Result: Same failure. CGO ruled out.

Step 5: Rule Out Code Signing

I tried signing the binary manually:

codesign -s - --force /tmp/testhello

Result: Same failure. Signing ruled out.

Step 6: Rule Out Quarantine Attributes

macOS tags files downloaded from the internet with com.apple.quarantine or com.apple.provenance attributes. I checked and the newly compiled binary did have com.apple.provenance, but the working air binary had the same attribute — so that wasn't the cause.

Step 7: The Breakthrough — syspolicyd (Claude Code's Key Finding)

After six steps ruled out all the usual suspects, I started suspecting a system-level issue. Claude Code suggested checking system processes and ran:

ps aux | grep syspolicyd

The result was shocking:

root 478  98.7%  /usr/libexec/syspolicyd

CPU usage at 98.7%. This process had been running for 6 days and 16 hours, essentially stuck.

Step 8: Connecting the Dots (Claude Code's Root Cause Analysis)

After discovering the syspolicyd anomaly, Claude Code immediately laid out the complete causal chain:

syspolicyd is the System Policy Daemon. One of its responsibilities is verifying code signatures for new binaries. When the system launches a new process, the kernel calls amfid (Apple Mobile File Integrity), which in turn calls syspolicyd for security validation. If syspolicyd is stuck, the verification request never completes, and the process remains in a loading state, never executing any code.

This also explained why RSS was only 32 bytes — the binary was mmap'd into memory (accounting for those 32 bytes), but the code was never executed.

This also explained why air worked fine — it was already loaded and didn't need re-verification.

Root Cause Analysis

Frequent go build of new binaries
    ↓
amfid frequently calls syspolicyd for verification
    ↓
Verification requests pile up, queue gets blocked
    ↓
syspolicyd CPU at 100%, enters a deadlock
    ↓
New binaries get stuck at the verification stage
    ↓
Process RSS=32, code never executes

Trigger conditions:

Long uptime without reboot — my Mac had been running for 6 days 16 hours
Frequent compilation of new binaries — repeated go build during development generates many new Mach-O files
Possible compounding factor — macOS notarization checks timing out and retrying, further exacerbating the issue

Solutions

Method 1: Reboot (Simple and Effective)

sudo shutdown -r now

After rebooting, syspolicyd resets naturally and everything returns to normal.

Method 2: Just Restart syspolicyd (No Reboot Needed)

sudo killall syspolicyd

The system automatically restarts it, and CPU usage drops back to normal levels. This is the fastest fix.

Method 3: Verify the Issue Is Resolved

# Check syspolicyd CPU usage
ps aux | grep syspolicyd

# Normal operation should be 0-1% CPU

How to Prevent It

Reboot your Mac weekly — macOS syspolicyd tends to misbehave after long uptime. Regular reboots are the simplest prevention.
Watch your CPU during development — glance at Activity Monitor occasionally. If syspolicyd spikes, deal with it immediately.
Keep macOS updated — Apple has fixed similar issues in later releases.
Don't hesitate to kill it — if you notice abnormal CPU usage, run sudo killall syspolicyd right away. Don't wait for it to lock up.

Reflections

AI-Assisted Debugging Experience

This investigation was conducted entirely with Claude Code. The experience was like pair-programming with a seasoned SRE:

I just said "the project won't start," and it automatically ran go build, go run, and analyzed logs
When it found the binary process with RSS of 32 bytes, I didn't even need to do the math — it told me directly "the process was created but never executed any code"
After systematically ruling out CGO, code signing, and quarantine attributes, it proactively suggested checking system process status and eventually pinpointed syspolicyd
From problem discovery to solution (sudo killall syspolicyd), it took less than 20 minutes

This really drove home the point: AI-assisted debugging doesn't replace human experience — it amplifies it. An experienced engineer might figure this out faster, but having AI handle all the repetitive checks, cross-references, and log analysis dramatically accelerates the process.

Back to the Tech

What this investigation taught me: sometimes the problem isn't in your code — it's in your operating system.

When developing Go projects, if compilation succeeds but execution fails, most people's first instinct is to check the code, configuration, and dependencies. But if none of those pan out, consider the system level — a stuck system daemon is all it takes to keep your binary frozen at the starting line.

macOS is generally stable, but syspolicyd seems to be a weak link. Especially for Go developers, frequent compilation triggers its verification mechanism more often than you'd think, making it more prone to breaking down.

If you ever encounter a Go binary that compiles but won't run, with RSS stuck at 32 bytes — don't rush to reinstall your system. Check syspolicyd first. Chances are, that's your culprit.

Have you encountered similar macOS development environment issues? Share your debugging stories in the comments. And if you've used AI-assisted debugging tools, I'd love to hear about your experience too.