<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>Forem: Tony Green</title>
    <description>The latest articles on Forem by Tony Green (@albatrossflavour).</description>
    <link>https://forem.com/albatrossflavour</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1055806%2F402f4836-b395-4931-8b37-ffc6e9f0e73c.jpeg</url>
      <title>Forem: Tony Green</title>
      <link>https://forem.com/albatrossflavour</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://forem.com/feed/albatrossflavour"/>
    <language>en</language>
    <item>
      <title>Creating Successful Migration Workflows with Puppet</title>
      <dc:creator>Tony Green</dc:creator>
      <pubDate>Mon, 18 May 2026 10:11:59 +0000</pubDate>
      <link>https://forem.com/puppet/creating-successful-migration-workflows-with-puppet-4b8n</link>
      <guid>https://forem.com/puppet/creating-successful-migration-workflows-with-puppet-4b8n</guid>
      <description>&lt;h2&gt;
  
  
  The goal isn't to move things. It's to only move them once.
&lt;/h2&gt;

&lt;p&gt;I've been doing this for over thirty years.&lt;/p&gt;

&lt;p&gt;Sysadmin, ops lead, global teams, and more data centre migrations than I'd like to admit. Site to site, P2V, V2V, cloud, hybrid, all of it.&lt;/p&gt;

&lt;p&gt;Every migration gets sold as a clean, well-planned transition. None of them are.&lt;/p&gt;

&lt;p&gt;They go wrong in very predictable ways. Not because moving infrastructure is especially difficult, but because nobody ever has a clear, current view of what's actually running, what's changed, and what still matters. So people fall back to spreadsheets, SSH sessions, scripts written at 2am, and a lot of "we think this is right".&lt;/p&gt;

&lt;p&gt;That's where migrations fail. Not in the move itself, but in the loss of control around it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why migrations go wrong
&lt;/h3&gt;

&lt;p&gt;After you've done a few of these, the pattern is obvious. Three things show up every time.&lt;/p&gt;

&lt;p&gt;The first is drift. Most environments look well understood on paper. In reality, versions don't match, configurations have wandered off in their own directions, "identical" servers aren't, and there are systems no one owns but everyone is afraid to touch. I once worked on an estate where three "identical" app servers were running three different JVM versions, and nobody could tell me which one production traffic was actually landing on. You start the migration with an incomplete picture and everything after that is guesswork. It gets worse once the migration is underway, because people make manual fixes to keep things moving, and nothing lasts as long as a temporary solution.&lt;/p&gt;

&lt;p&gt;The second is manual execution. No matter how good the plan looks in the slide deck, at some point it turns into a human running commands on a list of hosts they pasted out of a spreadsheet. Sequencing becomes tribal knowledge. Parallelisation becomes guesswork. The rollback plan, if anyone wrote one down, is usually a single line that says "restore from backup".&lt;/p&gt;

&lt;p&gt;The third is lost accountability. During a migration more people have access, more changes are happening, and less of it gets tracked properly. So when something breaks at 3am, nobody is quite sure what changed, or who changed it, or how to undo it. By the time you've worked it out, the on-call engineer has lost an evening and the project has lost a week of trust.&lt;/p&gt;

&lt;p&gt;None of this is new. What's interesting is that all three problems have the same root cause. There's no consistent way to know what your estate looks like and operate it predictably while it's moving.&lt;/p&gt;

&lt;h3&gt;
  
  
  The four things you need to stay in control
&lt;/h3&gt;

&lt;p&gt;Fixing this comes down to four things. You need to know what's actually out there. You need a way to say what it should look like. You need to be able to operate it safely at scale. And you need to know who's allowed to do what. Most teams have bits of all four scattered across different tools, which is why nothing quite lines up when it matters. &lt;a href="https://www.puppet.com/products/puppet-enterprise" rel="noopener noreferrer"&gt;Puppet Enterprise&lt;/a&gt; gives you all four in one place. One model, one set of controls, one audit trail. Not four tools you have to glue together yourself and pray they agree with each other.&lt;/p&gt;

&lt;h4&gt;
  
  
  Visibility
&lt;/h4&gt;

&lt;p&gt;Most migrations start with a discovery phase that's stale before the spreadsheet is finished.&lt;/p&gt;

&lt;p&gt;What you actually want is continuous visibility, and that's what &lt;code&gt;facter&lt;/code&gt; does. &lt;code&gt;facter&lt;/code&gt; runs on every node, collects system information, and reports it back centrally dozens of times a day. Out of the box you get the obvious things (OS, kernel, memory, network) but the part that matters during a migration is &lt;a href="https://help.puppet.com/core/current/Content/PuppetCore/custom_facts.htm" rel="noopener noreferrer"&gt;custom facts&lt;/a&gt;. You can teach &lt;code&gt;facter&lt;/code&gt; to collect anything you care about.&lt;/p&gt;

&lt;p&gt;Here's a small one that reports the version of an in-house payments app by reading a file the deploy pipeline drops on disk:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="no"&gt;Facter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:payments_version&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="n"&gt;setcode&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="no"&gt;File&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exist?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'/opt/payments/VERSION'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="no"&gt;File&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'/opt/payments/VERSION'&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;
    &lt;span class="k"&gt;end&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once that fact exists, every node running the app reports its version on every Puppet run. Five minutes later you can ask Puppet Enterprise "show me every node where payments_version is older than 4.2" and get a real answer, not a guess. Multiply that across the dozen things you actually care about (JVM version, storage layout, mount points, whether a vendor agent is running, whether a system is genuinely in use) and you've replaced your discovery spreadsheet with something that updates itself.&lt;/p&gt;

&lt;p&gt;That's the bit most teams never quite get to. Half the battle in a migration is just knowing what you've got.&lt;/p&gt;

&lt;h4&gt;
  
  
  State
&lt;/h4&gt;

&lt;p&gt;Once you know what you've got, the next problem is making sure it behaves the way you expect.&lt;/p&gt;

&lt;p&gt;This is the part Puppet has been doing for years. You define the desired state in code, and that code gets applied consistently across the old data centre, the new one, and any cloud environment you're bringing into the mix. No golden images quietly drifting. No "we rebuilt it by hand and it's nearly the same".&lt;/p&gt;

&lt;p&gt;The reason this matters during a migration is that it lets you stand up the new environment alongside the old one and keep them honest. Same roles, same profiles, same definitions. The question stops being "did we build this correctly?" and becomes "does it match the defined state?". The second question has an answer. The first one rarely does.&lt;/p&gt;

&lt;h4&gt;
  
  
  Execution
&lt;/h4&gt;

&lt;p&gt;At some point you actually have to do things. Stop services, drain load balancers, run pre-flight checks, kick off data sync, validate the result, move on. This is the bit that usually collapses into SSH and good intentions.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.puppet.com/docs/pe/2025.0/running_jobs_with_puppet_orchestrator_overview.html" rel="noopener noreferrer"&gt;Puppet Tasks and Plans&lt;/a&gt; give you a way to do it properly. Tasks run actions across large numbers of nodes. Plans sequence those actions and react to what they return. You target nodes by what Puppet already knows about them, not by a hand-built host list, which means the targeting stays correct as the estate changes underneath you.&lt;/p&gt;

&lt;p&gt;Here's roughly what a cutover step looks like as a Plan:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight puppet"&gt;&lt;code&gt;&lt;span class="n"&gt;plan&lt;/span&gt; &lt;span class="nf"&gt;migration::drain_web&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nc"&gt;TargetSpec&lt;/span&gt; &lt;span class="nv"&gt;$nodes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nv"&gt;$targets&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get_targets&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$nodes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nf"&gt;run_command&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'systemctl stop nginx'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$targets&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nf"&gt;run_task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'healthcheck::wait_for_drain'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$targets&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="py"&gt;timeout&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="m"&gt;300&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="n"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"drained &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="s2"&gt;{targets.size} nodes"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing clever. The point is that it's the same plan whether you're draining four nodes or four hundred, it runs through the orchestrator with the same RBAC and the same logging as everything else, and you can call it from another plan that drains the web tier, then the app tier, then flips DNS. The chaos of cutover night turns into something you can rehearse.&lt;/p&gt;

&lt;h4&gt;
  
  
  Control
&lt;/h4&gt;

&lt;p&gt;The last piece is the one most people only think about after something has already broken.&lt;/p&gt;

&lt;p&gt;During a migration more people have access, more changes are happening, and the blast radius of a mistake is bigger. You need to know who can define behaviour, who can execute it, and who can see what's going on. Puppet Enterprise handles that through &lt;a href="https://docs.puppet.com/pe/latest/rbac_intro.html" rel="noopener noreferrer"&gt;RBAC&lt;/a&gt; and its reporting layer. You can let a team run specific tasks against specific systems without handing them the keys to the kingdom, and you can give read-only visibility to the people who need to know what's happening without putting them in the room.&lt;/p&gt;

&lt;p&gt;The same controls apply no matter how the work is triggered. A change ticket in ServiceNow, a step in a CI/CD pipeline, an approval workflow, or someone clicking a button in the console all run the same plans against the same nodes. Same RBAC checks, same audit trail. That matters during a migration because cutover work doesn't all come from one place. Some of it is planned change, some is automated, some is "the network team needs us to drain that rack in the next ten minutes", and you don't want each route to have its own access model and its own log.&lt;/p&gt;

&lt;p&gt;If you're running PE Advanced, the same model extends into &lt;a href="https://www.puppet.com/products/security-compliance-enforcement" rel="noopener noreferrer"&gt;compliance&lt;/a&gt;. Continuous reporting, CIS benchmarks running on every node, and the ability to show an auditor that the new environment was in the expected state on the day you cut over. During a migration that evidence is the difference between "we think it went well" and "here's the report".&lt;/p&gt;

&lt;h4&gt;
  
  
  A few things that come along for the ride
&lt;/h4&gt;

&lt;p&gt;If part of your migration is moving workloads into the cloud, PE Advanced ships with CloudOps and FinOps capabilities that are genuinely useful in flight, not just once you've landed.&lt;/p&gt;

&lt;p&gt;Once your cloud accounts are connected, you get visibility of what's actually running, what it's costing, and what's been left switched on by mistake. Migrations are very good at producing all three: orphaned instances, oversized VMs that someone picked "to be safe", and test environments that nobody remembered to turn off.&lt;/p&gt;

&lt;p&gt;The bit that makes this useful rather than just another dashboard is that it's tied to the same node inventory you're already managing with Puppet. So the cost story doesn't end up as a separate project six months after cutover, run by people who don't know which servers were meant to be there in the first place.&lt;/p&gt;

&lt;p&gt;A couple of other things in PE Advanced are worth knowing about for the same reason. The &lt;a href="https://www.puppet.com/products/observability" rel="noopener noreferrer"&gt;data connector&lt;/a&gt; pushes Puppet's facts, reports, and events into whatever SIEM or observability platform you already run, so the migration shows up in the same dashboards your SOC and SREs are already watching, instead of being a parallel universe nobody looks at. And the &lt;a href="https://www.puppet.com/integrations/servicenow" rel="noopener noreferrer"&gt;ServiceNow integration&lt;/a&gt; syncs bi-directionally with your CMDB, which means the live view Puppet has of the estate stops drifting away from the system of record the rest of the business is using. Both of those matter more during a migration than at any other point in the life of an environment, because both are usually where the wheels come off after cutover.&lt;/p&gt;

&lt;h3&gt;
  
  
  You don't have to throw away what you've got
&lt;/h3&gt;

&lt;p&gt;One thing worth saying clearly, because it comes up in every conversation I have about this. You don't need to rip out your existing tooling to do any of it.&lt;/p&gt;

&lt;p&gt;Most environments I see are a mix of on-prem and cloud, multiple operating systems, and whatever automation has grown over the years. Usually that includes Ansible, a pile of homegrown scripts, and a fair bit of "don't touch that, it works". Trying to standardise all of that during a migration is a mistake. It adds risk at exactly the moment you want less of it.&lt;/p&gt;

&lt;p&gt;A more practical approach is to orchestrate what you already have. Puppet Tasks can call existing scripts and Ansible playbooks, sequence them alongside Puppet-managed actions, and target the right systems based on facts and roles. Instead of maintaining multiple inventories and disconnected workflows, you get one control layer driving everything you already run.&lt;/p&gt;

&lt;p&gt;Puppet isn't replacing those tools. It's coordinating them. That lets you keep what works, cut the duplication, and converge on a more consistent model over time, which is a much safer place to be in the middle of a migration than mid-rewrite.&lt;/p&gt;

&lt;h3&gt;
  
  
  Final thought
&lt;/h3&gt;

&lt;p&gt;A data centre migration isn't really a logistics problem. It's a control problem.&lt;/p&gt;

&lt;p&gt;If you know what's out there, what state it's in, and who's changing it, you can move systems safely, validate them properly, and recover when things don't go to plan. If you can't, it doesn't matter how good the plan looked at kickoff. You'll be discovering the gaps the hard way, one outage at a time.&lt;/p&gt;

&lt;p&gt;The goal of a migration isn't to move things. It's to only have to move them once.&lt;/p&gt;

</description>
      <category>puppet</category>
      <category>automation</category>
      <category>cloud</category>
    </item>
    <item>
      <title>Handling Dirty Frag and Copy Fail with Puppet</title>
      <dc:creator>Tony Green</dc:creator>
      <pubDate>Wed, 13 May 2026 21:00:55 +0000</pubDate>
      <link>https://forem.com/puppet/handling-dirty-frag-and-copy-fail-with-puppet-6ff</link>
      <guid>https://forem.com/puppet/handling-dirty-frag-and-copy-fail-with-puppet-6ff</guid>
      <description>&lt;h2&gt;
  
  
  Do you know which of your Linux servers are vulnerable right now?
&lt;/h2&gt;

&lt;p&gt;Two critical Linux kernel vulnerabilities are being actively exploited. Dirty Frag targets the IP fragment reassembly modules that almost every Linux server has loaded by default. Copy Fail targets the AF_ALG cryptographic subsystem's AEAD interface. Both can allow an attacker to gain escalated local privileges, and both have interim mitigations you can deploy right now, before the kernel patches land.&lt;/p&gt;

&lt;p&gt;The first question every team needs to answer is not "how do we fix it?" but "how many of our systems are actually exposed?" Not tomorrow, after someone runs a scanner. Not next week, when the security team finishes their audit. Right now.&lt;/p&gt;

&lt;p&gt;If you are running Puppet, the answer is already within reach. You just need a fact.&lt;/p&gt;

&lt;h3&gt;
  
  
  Add these to your Puppetfile. That is it.
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;mod&lt;/span&gt; &lt;span class="s1"&gt;'albatrossflavour-dirty_frag'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'1.0.1'&lt;/span&gt;
&lt;span class="n"&gt;mod&lt;/span&gt; &lt;span class="s1"&gt;'albatrossflavour-copy_fail'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'1.0.0'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each of these open-source modules, that I have published to the Forge, ship a custom structured fact that reports vulnerability exposure on every node. No class to include. No Hiera changes. No manifest edits. Next Puppet run, the data is there.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://github.com/albatrossflavour/dirty_frag" rel="noopener noreferrer"&gt;dirty_frag&lt;/a&gt; reports exposure to Dirty Frag (&lt;a href="https://nvd.nist.gov/vuln/detail/CVE-2026-43284" rel="noopener noreferrer"&gt;CVE-2026-43284&lt;/a&gt; and &lt;a href="https://nvd.nist.gov/vuln/detail/CVE-2026-43500" rel="noopener noreferrer"&gt;CVE-2026-43500&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/albatrossflavour/copy_fail" rel="noopener noreferrer"&gt;copy_fail&lt;/a&gt; reports exposure to Copy Fail (&lt;a href="https://nvd.nist.gov/vuln/detail/CVE-2026-31431" rel="noopener noreferrer"&gt;CVE-2026-31431&lt;/a&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  See where you stand
&lt;/h3&gt;

&lt;p&gt;Once deployed, every node reports its exposure state as structured data. Here is what the output looks like.&lt;/p&gt;

&lt;h4&gt;
  
  
  dirty_frag
&lt;/h4&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"esp4"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"loaded"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"blocked"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"available"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"esp6"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"loaded"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"blocked"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"available"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rxrpc"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"loaded"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"blocked"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"available"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"vulnerable"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"reboot_required"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h4&gt;
  
  
  copy_fail
&lt;/h4&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"algif_aead"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"builtin"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"loaded"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"active"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"blocked"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"available"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"initcall_blacklisted"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"vulnerable"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"mitigated"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"reboot_required"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both facts surface the same two summary keys that give you the answers you actually care about. &lt;code&gt;vulnerable&lt;/code&gt; is &lt;code&gt;true&lt;/code&gt; if the exploitable module is active. &lt;code&gt;reboot_required&lt;/code&gt; is &lt;code&gt;true&lt;/code&gt; if you have applied the mitigation but the module is still sitting in kernel memory.&lt;/p&gt;

&lt;p&gt;That &lt;code&gt;reboot_required&lt;/code&gt; flag is the one that catches people. You apply the mitigation, see the block in place, and assume you are safe. But the module is still loaded and exploitable until the machine restarts. The facts make this gap visible.&lt;/p&gt;

&lt;h3&gt;
  
  
  Query your entire fleet in one line
&lt;/h3&gt;

&lt;p&gt;The facts land in PuppetDB like any other structured fact. No extra tooling, no scheduled scans, no agents phoning home to a separate platform. It is just data, maintained automatically every Puppet run.&lt;/p&gt;

&lt;p&gt;Show me every node vulnerable to Dirty Frag:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;puppet query &lt;span class="s1"&gt;'facts[certname, value] { name = "dirty_frag" and value.vulnerable = true }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Show me every node vulnerable to Copy Fail:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;puppet query &lt;span class="s1"&gt;'facts[certname, value] { name = "copy_fail" and value.vulnerable = true }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Show me nodes where the Dirty Frag block is applied but a reboot is still needed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;puppet query &lt;span class="s1"&gt;'facts[certname, value] { name = "dirty_frag" and value.reboot_required = true }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Show me nodes where &lt;code&gt;algif_aead&lt;/code&gt; is built into the kernel (the harder mitigation path):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;puppet query &lt;span class="s1"&gt;'facts[certname, value] { name = "copy_fail" and value.algif_aead.type = "builtin" }'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is the bit that makes Puppet's model shine. You are not running a point-in-time scan. The data updates every run, automatically. As nodes get patched, rebooted, or have blocks applied, the numbers go down on their own. You can watch your exposure shrink in real time without maintaining anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  What makes a system vulnerable?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Dirty Frag
&lt;/h3&gt;

&lt;p&gt;The Dirty Frag vulnerability sits in three Linux kernel modules used for IP fragment reassembly: &lt;code&gt;esp4&lt;/code&gt;, &lt;code&gt;esp6&lt;/code&gt;, and &lt;code&gt;rxrpc&lt;/code&gt;. If any of those modules is loaded, the system is exposed. The two CVEs chain together to allow privilege escalation and remote code execution through these modules.&lt;/p&gt;

&lt;p&gt;These modules are almost universally loaded on production Linux servers. If you are running iptables, firewalld, Docker, Kubernetes, or any network policy enforcement, odds are at least &lt;code&gt;esp4&lt;/code&gt; is there. This one affects nearly everything.&lt;/p&gt;

&lt;h3&gt;
  
  
  Copy Fail
&lt;/h3&gt;

&lt;p&gt;The vulnerability is a use-after-free in the AF_ALG subsystem's AEAD interface (&lt;code&gt;algif_aead&lt;/code&gt;). An unprivileged user can trigger it by opening an AF_ALG socket and exercising the AEAD code path in a specific way.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;algif_aead&lt;/code&gt; is present on most stock kernels but rarely used for legitimate work. The main consumers are some VPN implementations that offload crypto to the kernel, &lt;code&gt;kcapi-tools&lt;/code&gt;, and certain FIPS-certified configurations. On systems where the module is loadable, an attacker can force-load it themselves just by opening the socket. On systems where it is built-in, it may already be active regardless of use.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why module presence, not kernel version?
&lt;/h3&gt;

&lt;p&gt;The real fix for both vulnerabilities is a kernel patch, but every distribution ships their own patched versions on their own schedules with their own version numbers. Tracking "which kernel version is safe" across Red Hat, Ubuntu, SUSE, Amazon Linux, and the rest is a moving target that goes stale the day you publish it. Module presence is the reliable signal: if the vulnerable module is active, you are exposed, regardless of kernel version.&lt;/p&gt;

&lt;p&gt;This is also what the vendor advisories recommend. Red Hat's &lt;a href="https://access.redhat.com/security/vulnerabilities/RHSB-2026-003" rel="noopener noreferrer"&gt;RHSB-2026-003&lt;/a&gt; and the equivalent Ubuntu and SUSE bulletins all point to the same interim mitigation: prevent the modules from loading using &lt;code&gt;install /bin/false&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  When you want Puppet to enforce the block
&lt;/h2&gt;

&lt;p&gt;The facts give you visibility. If you also want Puppet to actively prevent the modules from loading, include the classes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Dirty Frag
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight puppet"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'dirty_frag'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="py"&gt;mitigate_esp4&lt;/span&gt;  &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="py"&gt;mitigate_esp6&lt;/span&gt;  &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="py"&gt;mitigate_rxrpc&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Copy Fail
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight puppet"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'copy_fail'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="py"&gt;mitigate_algif_aead&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both classes write &lt;code&gt;install &amp;lt;module&amp;gt; /bin/false&lt;/code&gt; directives to modprobe.d configuration files. From that point on, any attempt to load those modules (whether by autoloading, a dependency chain, or an explicit &lt;code&gt;modprobe&lt;/code&gt;) will run &lt;code&gt;/bin/false&lt;/code&gt; instead of loading the actual module code. The module simply cannot get into the kernel.&lt;/p&gt;

&lt;p&gt;This is worth understanding because it is stronger than the kernel's &lt;code&gt;blacklist&lt;/code&gt; directive. A &lt;code&gt;blacklist&lt;/code&gt; entry only prevents autoloading, it does not stop explicit &lt;code&gt;modprobe&lt;/code&gt; calls. &lt;code&gt;install /bin/false&lt;/code&gt; intercepts the load at every entry point.&lt;/p&gt;

&lt;p&gt;If the module was already loaded before the block was applied, it stays in memory until the next reboot. The &lt;code&gt;reboot_required&lt;/code&gt; flag in both facts makes this gap visible.&lt;/p&gt;

&lt;p&gt;This is declarative and safe. Puppet manages the config file, ensures the desired state on every run, and the classes are entirely opt-in. If your patching process handles the fix, or you are rolling out kernel updates, you might not need them at all. The facts alone give you the visibility to track progress.&lt;/p&gt;

&lt;p&gt;Both modules ship with Hiera data defaults, so if you prefer to drive parameters through your hierarchy, see each module's README for examples.&lt;/p&gt;

&lt;h2&gt;
  
  
  The built-in module problem (Copy Fail only)
&lt;/h2&gt;

&lt;p&gt;There is a wrinkle with Copy Fail that Dirty Frag does not have. On most stock distribution kernels, &lt;code&gt;algif_aead&lt;/code&gt; is compiled directly into the kernel image as a built-in module. It is not a loadable &lt;code&gt;.ko&lt;/code&gt; file, so modprobe.d has no effect on it.&lt;/p&gt;

&lt;p&gt;For built-in modules, the mitigation is &lt;code&gt;initcall_blacklist=algif_aead_init&lt;/code&gt; on the kernel command line (via GRUB configuration). This prevents the module's init function from running on boot. It requires a reboot to take effect.&lt;/p&gt;

&lt;p&gt;The copy_fail module does not automate GRUB changes. Getting boot configuration wrong can render a system unbootable. You can manage GRUB with Puppet (using &lt;code&gt;file_line&lt;/code&gt; or &lt;code&gt;augeas&lt;/code&gt;), but that is out of scope for this module. The fact will report &lt;code&gt;initcall_blacklisted: true&lt;/code&gt; once the parameter is in place, and &lt;code&gt;reboot_required: true&lt;/code&gt; until the reboot completes.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;type&lt;/code&gt; field in the fact output tells you which mitigation path applies to each node:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;builtin&lt;/code&gt;&lt;/strong&gt;: needs &lt;code&gt;initcall_blacklist&lt;/code&gt;, modprobe.d has no effect&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;loadable&lt;/code&gt;&lt;/strong&gt;: the class and modprobe.d approach works&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;absent&lt;/code&gt;&lt;/strong&gt;: not vulnerable, nothing to do&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Force-unload a module right now (with care)
&lt;/h2&gt;

&lt;p&gt;Sometimes you cannot wait for a reboot. Both modules ship a Bolt task that removes the module from the running kernel immediately:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;bolt task run dirty_frag::unload &lt;span class="nv"&gt;module&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;esp4 &lt;span class="nt"&gt;--targets&lt;/span&gt; servers
bolt task run copy_fail::unload &lt;span class="nv"&gt;module&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;algif_aead &lt;span class="nt"&gt;--targets&lt;/span&gt; servers
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A word of caution. Unloading a kernel module is not the same as flipping a config switch. If the module is actively in use (for example, &lt;code&gt;esp4&lt;/code&gt; underpins IPsec tunnels, &lt;code&gt;esp6&lt;/code&gt; handles IPv6 ESP, and &lt;code&gt;rxrpc&lt;/code&gt; is used by AFS), pulling it out from under a running service can drop connections, crash VPN tunnels, or cause dependent subsystems to fail. This is exactly why the Puppet classes do not do it automatically. The classes apply the block and wait for a reboot. The Bolt tasks give you the lever to pull when you have decided the tradeoff is worth it.&lt;/p&gt;

&lt;p&gt;Each task validates the module name, checks it is actually loaded, and gives you structured output. If the module is in use by another kernel subsystem and cannot be unloaded, the task will tell you, and you are back to "apply the block and schedule a reboot".&lt;/p&gt;

&lt;p&gt;Test on non-production nodes first. Know what the module is doing on that system before you yank it. If in doubt, the block-and-reboot path is always the safer option.&lt;/p&gt;

&lt;p&gt;The unload task only works for loadable modules. Built-in modules cannot be unloaded.&lt;/p&gt;

&lt;h2&gt;
  
  
  The workflow
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Add the modules to your Puppetfile.&lt;/strong&gt; Deploy. Done. Every node now reports its exposure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Query PuppetDB&lt;/strong&gt; to see where you stand. One-liner, fleet-wide answer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Decide on mitigation.&lt;/strong&gt; Apply the classes via Hiera for persistent blocking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Use Tasks&lt;/strong&gt; to immediately unload modules on critical systems that cannot wait.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Watch the numbers drop.&lt;/strong&gt; The facts update every run. No manual tracking, no spreadsheets, no scheduled scans.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The whole approach is declarative. You tell Puppet what state you want, and it converges towards it. The facts keep reporting reality. The gap between the two is your exposure, and it is always visible.&lt;/p&gt;

&lt;h2&gt;
  
  
  Get started
&lt;/h2&gt;

&lt;p&gt;Both of these open source modules are available on GitHub:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://github.com/albatrossflavour/dirty_frag" rel="noopener noreferrer"&gt;albatrossflavour/dirty_frag&lt;/a&gt; (Puppet 7/8, no dependencies)&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/albatrossflavour/copy_fail" rel="noopener noreferrer"&gt;albatrossflavour/copy_fail&lt;/a&gt; (Puppet 7/8, no dependencies)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Both modules have been built for RedHat, CentOS, Ubuntu, Debian, Amazon Linux, and SLES.&lt;/p&gt;

</description>
      <category>puppet</category>
      <category>vulnerabilities</category>
      <category>automation</category>
      <category>devops</category>
    </item>
    <item>
      <title>Using Let's Encrypt with the Puppet Enterprise console</title>
      <dc:creator>Tony Green</dc:creator>
      <pubDate>Thu, 30 Mar 2023 22:28:40 +0000</pubDate>
      <link>https://forem.com/puppet/using-lets-encrypt-with-the-puppet-enterprise-console-21l4</link>
      <guid>https://forem.com/puppet/using-lets-encrypt-with-the-puppet-enterprise-console-21l4</guid>
      <description>&lt;p&gt;Had an itch I've been meaning to scratch for a while.  I build my &lt;a href="https://puppet.com" rel="noopener noreferrer"&gt;Puppet&lt;/a&gt; environment using &lt;a href="https://terraform.io" rel="noopener noreferrer"&gt;Terraform&lt;/a&gt;, which makes it nice and easy to tear things down and rebuild them.  That is great, but it does leave me with an issue when it comes to the console SSL certificates.&lt;/p&gt;

&lt;p&gt;Puppet will generate self-signed certs for the console, which work &lt;em&gt;fine&lt;/em&gt;, but it was always a niggle that the certs couldn't be automagically coaxed into being valid.&lt;/p&gt;

&lt;p&gt;Since moving over to &lt;a href="https://github.com/albatrossflavour/k3s-cluster" rel="noopener noreferrer"&gt;kubernetes&lt;/a&gt; for my home lab, I've come to expect managed SSL certificates for any public facing services, without me having to do anything.&lt;/p&gt;

&lt;p&gt;Finally set some time aside to look at the options and thought I'd publish the details of the journey as well as where I ended up.&lt;/p&gt;

&lt;h1&gt;
  
  
  Step 1 - Let's Encrypt
&lt;/h1&gt;

&lt;p&gt;Obviously I wasn't going to reinvent the wheel.  If I wanted to manage and use Let's Encrypt on a Puppet Enterprise server, I'd be using the &lt;a href="https://forge.puppet.com/modules/puppet/letsencrypt" rel="noopener noreferrer"&gt;Let's Encrypt module&lt;/a&gt;.  The module makes it very simple to get the relevant packages installed and configured.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight puppet"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'letsencrypt'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="py"&gt;config&lt;/span&gt;     &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="py"&gt;email&lt;/span&gt;  &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'certs@albatrossflavour.com'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="py"&gt;config_dir&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'/etc/letsencrypt'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One of the beauties of this module is that it also sets up a &lt;code&gt;cron&lt;/code&gt; job to renew the generated certs, so you don't need to keep an eye on it.&lt;/p&gt;

&lt;p&gt;Classify your server with that and you'll end up with &lt;code&gt;certbot&lt;/code&gt; and it's dependencies.  Great start and feeling confident about the future!&lt;/p&gt;

&lt;p&gt;Then we need to generate a certificates:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight puppet"&gt;&lt;code&gt;&lt;span class="n"&gt;letsencrypt::certonly&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'puppet.gcp.albatrossflavour.com'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="py"&gt;domains&lt;/span&gt;       &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'puppet.gcp.albatrossflavour.com'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
  &lt;span class="py"&gt;manage_cron&lt;/span&gt;   &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="py"&gt;plugin&lt;/span&gt;        &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'webroot'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="py"&gt;webroot_paths&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="err"&gt;'&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;www&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Annnnnnnd that's where things start to go south.&lt;/p&gt;

&lt;p&gt;When you request a cert, the most simple method of validating you are who you say you are is to have a web server on the host respond to a query sent to port &lt;code&gt;80&lt;/code&gt;.  Sure we say, easy, couldn't take much to do!&lt;/p&gt;

&lt;p&gt;By default, the Puppet Enterprise &lt;code&gt;nginx&lt;/code&gt; config does a &lt;code&gt;301&lt;/code&gt; redirect for &lt;code&gt;http&lt;/code&gt; requests to &lt;code&gt;https&lt;/code&gt;.  This means any queries from Let's Encrypt to validate the requests will end up as &lt;code&gt;https&lt;/code&gt; requests and the cert request will fail.&lt;/p&gt;

&lt;h1&gt;
  
  
  Step 2 - &lt;code&gt;nginx&lt;/code&gt;
&lt;/h1&gt;

&lt;p&gt;Let me introduce you to &lt;code&gt;puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect&lt;/code&gt;.  This parameter controls if that redirect is in place.  So step .... 6(?) was to use &lt;code&gt;hiera&lt;/code&gt; to disable the &lt;code&gt;http&lt;/code&gt; redirect:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="s"&gt;❯ cat data/role/role::pe::master.yaml&lt;/span&gt;
&lt;span class="na"&gt;puppet_enterprise::profile::console::proxy::http_redirect::enable_http_redirect&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once we run this through a &lt;code&gt;puppet&lt;/code&gt; run, the redirect gets removed!  Score!&lt;/p&gt;

&lt;p&gt;Only problem is, without the redirect, the &lt;code&gt;nginx&lt;/code&gt; server doesn't listen on port &lt;code&gt;80&lt;/code&gt;.  OK, we can fix that easily.  We could use the &lt;code&gt;pe_nginx::directive&lt;/code&gt; type, but I found it to be a bit of an overkill for what I needed.  Instead I opted for a simple template:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight puppet"&gt;&lt;code&gt;  &lt;span class="n"&gt;file&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'/etc/puppetlabs/nginx/conf.d/certs.conf'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="py"&gt;ensure&lt;/span&gt;  &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;owner&lt;/span&gt;   &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'root'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;group&lt;/span&gt;   &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'root'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;mode&lt;/span&gt;    &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'0644'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;content&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;template&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;${module_name}&lt;/span&gt;&lt;span class="s2"&gt;/cert_vhost.conf.erb"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="kp"&gt;notify&lt;/span&gt;  &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Exec&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'pe_nginx'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and the template is just:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight erb"&gt;&lt;code&gt;server {
  listen       80;
  server_name  &lt;span class="cp"&gt;&amp;lt;%=&lt;/span&gt; &lt;span class="vi"&gt;@fqdn&lt;/span&gt; &lt;span class="cp"&gt;%&amp;gt;&lt;/span&gt;;
  index        index.html;
  location /.well-known {
    root /var/www;
  }
  location / {
    return 301 https://$server_name$request_uri;
  }
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(Sorry, still using &lt;code&gt;erb&lt;/code&gt;, I will &lt;strong&gt;at some point&lt;/strong&gt; rewrite everthing in &lt;code&gt;epp&lt;/code&gt;)&lt;/p&gt;

&lt;p&gt;The template keeps the &lt;code&gt;301&lt;/code&gt; redirect in place for anything &lt;em&gt;other&lt;/em&gt; than a request to &lt;code&gt;/.well-known&lt;/code&gt;, which is where Let's Encrypt looks for the validation info.&lt;/p&gt;

&lt;p&gt;Run this through and the &lt;code&gt;nginx&lt;/code&gt; vhost gets created.  However the &lt;code&gt;letsencrypt::certonly&lt;/code&gt; call still fails on the first run.  The &lt;code&gt;notify&lt;/code&gt; to the &lt;code&gt;pe_nginx&lt;/code&gt; service, which is done when we create the &lt;code&gt;cert_vhost.conf&lt;/code&gt;, doesn't happen in the order we need.  The Let's Encrypt module is trying to get a response &lt;em&gt;before&lt;/em&gt; we've setup the vhost.  Now this would work on subsequent runs, but &lt;strong&gt;Golden Rule #1&lt;/strong&gt; is to make sure, whenever possible, that you get a clean puppet run in one pass.  Plus this was a challenge.&lt;/p&gt;

&lt;p&gt;Before I worked on fixing that, I wanted to make sure I could make the rest of it work.  Let's sum up where we are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;I've got the Let's Encrypt client installed and configured&lt;/li&gt;
&lt;li&gt;I've got a new &lt;code&gt;nginx&lt;/code&gt; vhost running that allows the Let's Encrypt web validation queries through and redirects any other &lt;code&gt;http&lt;/code&gt; traffic.&lt;/li&gt;
&lt;li&gt;After a couple of puppet runs, I've got valid SSL certs in &lt;code&gt;/etc/letsencrypt&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h1&gt;
  
  
  Step 3 - The console certs
&lt;/h1&gt;

&lt;p&gt;It'd been a while since I played with the Puppet console SSL certs, so I checked in with &lt;a href="https://www.puppet.com/docs/pe/latest/use_a_custom_ssl_cert_for_the_console.html" rel="noopener noreferrer"&gt;the source of truth&lt;/a&gt;, which outlines the steps we need to go through to use custom SSL certs with the console:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Retrieve the custom certificate and private key.&lt;/li&gt;
&lt;li&gt;Move the certificate to &lt;code&gt;/etc/puppetlabs/puppet/ssl/certs/console-cert.pem&lt;/code&gt;, replacing any existing file named &lt;code&gt;console-cert.pem&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Move the private key to &lt;code&gt;/etc/puppetlabs/puppet/ssl/private_keys/console-cert.pem&lt;/code&gt;, replacing any existing file named &lt;code&gt;console-cert.pem&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So we can just create a &lt;code&gt;file&lt;/code&gt; resource that takes the Let's Encrypt cert/key and places them into the console SSL directory structure.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight puppet"&gt;&lt;code&gt;  &lt;span class="n"&gt;file&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'/etc/puppetlabs/puppet/ssl/certs/console-cert.pem'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="py"&gt;ensure&lt;/span&gt;    &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;owner&lt;/span&gt;     &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'pe-puppet'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;group&lt;/span&gt;     &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'pe-puppet'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;links&lt;/span&gt;     &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'follow'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;mode&lt;/span&gt;      &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'0640'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;source&lt;/span&gt;    &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"/etc/letsencrypt/live/&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="s2"&gt;{facts['puppet_server']}/cert.pem"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;backup&lt;/span&gt;    &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'.puppet_bak'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="kp"&gt;notify&lt;/span&gt;    &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Service&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'pe-nginx'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="kp"&gt;subscribe&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Letsencrypt&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Certonly&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;$facts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'puppet_server'&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="n"&gt;file&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'/etc/puppetlabs/puppet/ssl/private_keys/console-cert.pem'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="py"&gt;ensure&lt;/span&gt;    &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;owner&lt;/span&gt;     &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'pe-puppet'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;group&lt;/span&gt;     &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'pe-puppet'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;links&lt;/span&gt;     &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'follow'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;mode&lt;/span&gt;      &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'0644'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;source&lt;/span&gt;    &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"/etc/letsencrypt/live/&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="s2"&gt;{facts['puppet_server']}/privkey.pem"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="py"&gt;backup&lt;/span&gt;    &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'.puppet_bak'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="kp"&gt;notify&lt;/span&gt;    &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Service&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'pe-nginx'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="kp"&gt;subscribe&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Letsencrypt&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Certonly&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;$facts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'puppet_server'&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sure enough, when I try this on the Puppet server, life is good and we have a console with valid SSL certs.&lt;/p&gt;

&lt;p&gt;Time to crack open a bottle of red and tick an item off my to do list.&lt;/p&gt;

&lt;p&gt;Damn it, &lt;strong&gt;Golden Rule #2&lt;/strong&gt; raises it's head.  It's not finished until you know it works fine from scratch... &lt;strong&gt;&lt;em&gt;WITHOUT&lt;/em&gt;&lt;/strong&gt; breaking &lt;strong&gt;Golden Rule #1&lt;/strong&gt;.&lt;/p&gt;

&lt;h1&gt;
  
  
  Step 4 - Rebuild
&lt;/h1&gt;

&lt;p&gt;Nuked the Puppet server and rebuilt it.  Lots of failures on the first run (we kinda expected that), however they didn't go away.  The &lt;code&gt;nginx&lt;/code&gt; service never gets restarted as the various dependencies can be resolved, but not in the way we need.&lt;/p&gt;

&lt;p&gt;This is the &lt;em&gt;fun&lt;/em&gt; (?) of declarative configuration management.  I can only &lt;strong&gt;&lt;em&gt;manage&lt;/em&gt;&lt;/strong&gt; things once, which includes only being able to &lt;code&gt;notify&lt;/code&gt; the &lt;code&gt;pe-nginx&lt;/code&gt; service once.  Then the compiler will figure out when the service is restarted, based on all of the dependencies in the catalog. &lt;/p&gt;

&lt;p&gt;I played around with a lot of options, some &lt;em&gt;waaaaay&lt;/em&gt; hackier than I wanted to go with.  Plus this was becoming a fun exercise.&lt;/p&gt;

&lt;p&gt;All I needed to be able to do was to inject a restart of the &lt;code&gt;pe-nginx&lt;/code&gt; service twice in a single run.&lt;/p&gt;

&lt;p&gt;To get there, I bent a few of the future &lt;strong&gt;Golden Rules&lt;/strong&gt; and came up with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight puppet"&gt;&lt;code&gt;&lt;span class="n"&gt;exec&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'restart_nginx'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="py"&gt;command&lt;/span&gt;     &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'/bin/systemctl restart pe-nginx'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="py"&gt;refreshonly&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This &lt;code&gt;exec&lt;/code&gt; resource will allow me to do a restart of &lt;code&gt;pe-nginx&lt;/code&gt; outside of, and before, the &lt;code&gt;notify =&amp;gt; Service['pe-nginx']&lt;/code&gt; parameters.&lt;/p&gt;

&lt;p&gt;Yes, it's a hack, but it's a hack on the side of the angels.  Not only will it solve the issue of the process not working at all, but it will also allow the certs to be generated, and installed, in a single run.&lt;/p&gt;

&lt;h1&gt;
  
  
  Step 5 - Quick Robin, to the &lt;code&gt;pdk&lt;/code&gt; mobile
&lt;/h1&gt;

&lt;p&gt;I did a fair bit of testing and found the end result to be far more reliable and useful than I thought it would be.&lt;/p&gt;

&lt;p&gt;I first created a fully parameterised profile to manage the certs.  Once I got that working, it was a no-brainer to create a module to share the love.&lt;/p&gt;

&lt;p&gt;That's how I ended up with the &lt;a href="https://forge.puppet.com/modules/albatrossflavour/pe_console_letsencrypt/readme" rel="noopener noreferrer"&gt;pe_console_letsencrypt&lt;/a&gt; module.  You can also just &lt;a href="https://github.com/albatrossflavour/pe_console_letsencrypt/blob/main/manifests/init.pp" rel="noopener noreferrer"&gt;check out the code&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I've done a fair bit of testing, but I'm certainly not saying it's bulletproof.&lt;/p&gt;

&lt;h1&gt;
  
  
  Step 6 - &lt;a href="https://en.wikipedia.org/wiki/Gnomes_(South_Park)" rel="noopener noreferrer"&gt;Profit&lt;/a&gt;?
&lt;/h1&gt;

&lt;p&gt;I'm still scratching my head to think of another way I could get the certs generated and in place in a single run &lt;strong&gt;without&lt;/strong&gt; the &lt;code&gt;exec&lt;/code&gt; hack.  That being said, I've had hacks a lot worse go live in the past!&lt;/p&gt;

&lt;h1&gt;
  
  
  Note
&lt;/h1&gt;

&lt;p&gt;While researching the links for this post, I came across a &lt;a href="http://blog.graypockets.com/2017/10/lets-encrypt-puppet-enterprise-console.html" rel="noopener noreferrer"&gt;blog less frequently updated than mine&lt;/a&gt; which has a post from 2017 showing a similar way of achieving the same thing, but without the hack to make it work in a single run.&lt;/p&gt;

</description>
      <category>puppet</category>
      <category>ssl</category>
      <category>nginx</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
