Forem: Allen Helton

Your AI agents are a security nightmare

Allen Helton — Fri, 27 Mar 2026 15:04:48 +0000

I've noticed something about the tech industry over the past couple of months. Through the use of agentic code editors, developers are building small, custom solutions to problems they're having. Recently, there's been a surge in developers writing their own content platforms, abandoning bigger platforms like Hashnode and Medium. While I think this is genuinely amazing, we're creating a gap that we have to address sooner rather than later.

That gap, of course, is maintenance. Even though AI coding is genuinely good now, new development is only a fraction of the software lifecycle. What happens when you get your app to prod and it suddenly breaks?

Do you know the codebase like you did back when everything was hand coded? Would you even know where to begin troubleshooting if your API started returning a bunch of 500s?

Let's assume the answer is yes to both of those. Now let me ask the hard-hitting one: do you have time to troubleshoot and fix issues? This is a side project, not your day job. Many of us don't have the luxury of dropping the “real work” tasks for a few hours while we figure out what's going on.

I'm no exception. I've been building Good Roots Network, an app that connects local gardeners with people in their communities who need food. Growers list produce, gatherers like social workers and non-profits request it, and the system handles the coordination from availability through pickup. It's API-driven from the ground up. State machines, role-based access, real-time inventory transitions. Real production infrastructure running in my personal AWS account.

There's a lot of surface area to cover for a solo project. And when something goes wrong, I don't have the bandwidth to dig in and figure out how to get it back on its feet - which means I'm potentially blocking people from getting access to fresh produce for themselves or others.

So I built an agent to be on call.

Before I continue, thank you to Teleport for sponsoring this post. All opinions are my own.

Building an agent that runs locally

The oncall-agent is a local, persistent process that automatically responds to production incidents. It subscribes to a Momento topic for real-time notifications and when something fires, it gets to work on its own. No need to wait for me to react.

The agent has an internal state machine: receive the incident signal, authenticate against AWS, investigate, then post a summary to Slack. During investigation, the agent loop runs on Amazon Bedrock, using the AWS CLI to query CloudWatch logs and metrics, inspect Lambda functions and DynamoDB tables. It also has access to my source code and deployments via a GitHub app to diagnose root cause and open a PR with the recommended code change.

I run it locally (rather than as a deployed service) for a couple of reasons. First, I specifically want it to make code changes if they're needed, and pulling the source into a Lambda function in my account doesn't seem like a great use of resources. Second, and more importantly, it touches production infrastructure directly. I want full control over the process, and I want the blast radius of any security issue limited to my machine, not a cloud-hosted process that's running 24/7.

By the time I check Slack in the morning, the investigation and proposed solution is done and awaiting my approval.

Keeping the agent secure for a change

It's tempting to just open up the doors on your machine and give the agent everything it needs to be successful (and then some). But as we've seen throughout 2026, doing so is a security nightmare. I don't want to be another statistic who let an agent wreak havoc on my system because it was easy.

When it came time to give the agent AWS access, my first instinct was to put an access key in the .env file. Long-lived credentials. It's the familiar pattern, it takes 30 seconds, and it works. But the more I sat with that approach, the more it felt like I was solving the wrong problem.

My issue with that approach was that the agent didn't really exist as an identity in my system. It was just a privileged background process acting on my behalf with standing access.

So, I reconsidered the agent as a first-class principal. If it was going to investigate production issues autonomously, it needed to prove who it was at runtime, operate within a scoped session, and leave behind a traceable chain of actions.

Once I started thinking about the design this way, long-lived keys stopped making sense entirely. 👎

Identity is not the same thing as access

With this in mind, I needed something more than a vault. I needed a way for the agent to establish its identity at runtime.

That's where Teleport fits into the design. Instead of access tokens being defined in a config file, the agent starts every investigation by authenticating into a new session. From there, it can request scoped AWS access and perform specific actions with clear attribution.

What I can now see in CloudTrail is a stable GUID that uniquely identifies the agent as its own principal. I can filter to that identity and see everything the agent has ever touched.

If something unexpected shows up, I know immediately it was the agent, and not a human or another process. That's already a huge improvement over a shared access key where attribution is basically impossible.

Now, I have a legitimate session-bound identity operating on my system. Something that only has the permissions I gave it, but more importantly, is independently auditable.

When the agent runs tsh login, I set it up so I have to approve the MFA challenge myself. Fully autonomous credential rotation via Teleport Machine ID is available–and it's the natural next step–but I'm not there yet. I want a few more practice reps with the agent before I remove myself from the auth flow entirely. Trust has to be earned, and I'd rather manually approve an MFA prompt than wake up to an agent that went sideways with no human checkpoint anywhere in the chain.

Agent security posture is something I'm really honing in on. I'm changing my mental approach from "does this agent have the right key?" to "was this agent authorized to do this specific thing, in this specific context, at this specific moment?" It might feel like a subtle difference, but it's how we need to be thinking before we give agents the keys to our production systems. And a static credential sitting in an env file isn't the way to answer it.

This isn't a side project problem

The most common pattern I see these days is the same one I almost fell into: drop a long-lived token into the environment, grant it the permissions it needs, and ship it. Even if you do it as a “temporary stop gap” and swear you'll come back to fix it, you won't.

It always feels like it's fine until something unexpected happens. The problem with unexpected behavior in autonomous systems is that you need more audit resolution, not less. Imagine an agent made unusual calls to your production services. Was it a bug? A prompt injection? A legitimate investigation path you just didn't anticipate? With static credentials you can't tell. The trail goes cold at "this IAM principal made these calls."

Agentic AI is exposing some serious issues with the access model many of us are running. IAM was originally designed around humans doing episodic work. Even automated pipelines are episodic in that they run, finish, and stop. But that doesn't map cleanly to processes that persist indefinitely, reason autonomously, and act faster than the blink of an eye.

When I think about what agents need to operate safely, I land on a few things. They need a cryptographic identity of their own, meaning something that can be uniquely identified (much like an individual developer). They need access that's scoped and enforced at runtime–not granted once like an API key, and left open. They need some way to detect when they're behaving outside expected parameters, or when the operating context has been tampered with. And if they're coordinating with other agents or MCP servers, those connections need to be governed, too (you can't have a secured agent talking to an unsecured tool).

These are all things Teleport is building with their Agentic Identity Framework. My agent only scratches the surface of all this with its short-lived credentials and isolated identity. But starting there forces you to think about the agent as a principal that needs its own security model, which is the right mental shift we need heading into mid-2026.

This probably sounds a little familiar

We've all seen this play out before in different forms. When containerization became mainstream, teams that had good practices with VM security often skipped the basics on containers because it felt like a "smaller" problem. When Lambda came out, teams that carefully managed server permissions dropped service roles for wildcard policies because it was a new execution model and they were moving fast (hopefully I'm not the only one who did this 😬).

Agentic AI is the same thing happening again. A new execution model, a lot of excitement, and a temptation to skip the identity and access fundamentals–because it feels like a side project, a POC, or just something you're trying out.

The lesson is the same every time: the important components in production are still important even if the thing running is new, or small, or experimental. An agent with standing access and no attribution is a liability, whether it's running inside a Fortune 500 or on your laptop handling incidents for a community gardening app.

The oncall-agent repo is open source. The Teleport integration is in the state machine if you want to see how it fits together in practice. And if you're experimenting with agents at all, it's worth triple-checking how they establish identity and access production systems. Move a small workflow to session-bound credentials to help you make better architectural decisions and make your automations easier to trust.

Tools like Teleport make that change practical without completely reworking your stack. It's a small change in implementation, but a big change in how much autonomy you're willing to give your systems.

Happy coding!

When serving images from S3 stopped being good enough

Allen Helton — Wed, 07 Jan 2026 15:12:39 +0000

I published my first blog post 7 years ago. I wrote on Medium for about a year before I built Ready, Set, Cloud. For most of its life, the site hasn't had much of a facelift or performance updates. It's primarily served as a home for my writing, newsletter, and podcast.

I've been running this site for six years. I'm usually the kind of person who can't leave things alone for long, yet this has been largely unchanged for years. It worked. Until it didn't.

When I finally took a look at site performance, something was painfully obvious. Images. PageSpeed Insights showed that large, unoptimized images were dominating load time. Single file sizes served directly from S3, no format negotiation, and no caching were starting to add up.

That setup wasn't wrong when I built it. It was a perfectly reasonable tradeoff at the time. It was simple, low maintenance, and honestly, good enough. The site had a couple dozen readers, and small-scale simplicity won. But as the site grew, both in content and audience, performance expectations changed. Serving raw images straight from S3 no longer cut it. The site outgrew its initial build.

What I needed from image delivery

The knee-jerk reaction is to jump into S3 and manually optimize a bunch of images. That fixes a symptom, but it doesn't solve the underlying problem. I needed to step back and rethink what image delivery should look like for a content-heavy site serving tens of thousands of monthly visitors.

Manually resizing images, converting them to web-friendly formats, and deciding which size to upload per post was never going to scale. More importantly, I didn't want to change my workflow. I wanted to continue to write, publish, and move on without adding in layers of performance checking.

A good system should make that possible. It should optimize images automatically in the background. It should serve modern formats like WebP when the browser supports them. It should provide multiple image sizes so mobile devices aren't downloading desktop-sized assets. And it should be aggressively cacheable, because image delivery is a solved problem, and CDNs already do this exceptionally well.

Thinking about these requirements, I realized I wasn't really looking to build a new image optimization tool (like I would have done in the past). Instead, I was updating the site's image handling capabilities so these decisions were made once, centrally, in an industry-standard way.

Fitting a CDN into the workflow

I upload images to Ready, Set, Cloud using a small JavaScript script that lives in the repo. It takes a file prefix, scans a local folder for matching files, and uploads them directly to S3. There's no web UI and no automation that tries to interpret content. It's simple by design, and I wanted to keep it that way.

That simplicity turns out to be a feature. Because images already flow through S3, the system can react to uploads without changing the workflow at all.

Image uploads automatically trigger a Rust Lambda function (through EventBridge) that converts the original image to WebP and generates a handful of standard sizes. Those optimized versions are written back to S3 alongside the original.

This does a few important things. First, it makes image optimization deterministic - meaning every image goes through the same process every time. Second, it keeps work off the critical path. All of this happens asynchronously and doesn't interfere with the rest of the publishing workflow like cross-posting, writing analytics, or scheduling.

Finally, I added a CloudFront distribution in front of the S3 bucket. That keeps delivery fast and predictable by serving cached assets from points of presence around the world. Because it points at the existing bucket, there's no data migration involved. Image delivery improves without changing where anything lives.

Serving the right format without changing URLs

Once image processing was handled, delivery became the next concern. All optimized assets were already in S3, but I wasn't interested in changing 2,000+ image links across existing content just to take advantage of them.

The answer was to push that behavior into the CDN. By updating the distribution to look for WebP support in the Accept header, requests for images could be routed to the optimized versions without changing a single URL. Modern browsers already advertise their supported formats, so content negotiation becomes a routing concern rather than a frontend one.

A CloudFront function runs on the viewer request event and rewrites the request path whenever the browser advertises WebP support. It doesn't check whether the WebP file exists - that's resolved later when CloudFront fetches from the origin (the S3 bucket). The important part is that the rewrite happens consistently at the edge, and the resulting response is cached, so subsequent requests follow the same path.

Letting the browser choose the right size

Using WebP was only one part of the solution. Once multiple sizes of every image existed automatically, the next question was which one to serve. Mobile screens don't need desktop-sized images, and high-DPI displays can benefit from larger ones. Not to mention saving on load times for thumbnails on the home page.

Using srcset allows the browser to make that choice on its own. The markup stays the same, there's no runtime logic, and each client downloads only what it actually needs. The responsibility for choosing the right size moves to the client, where that decision can be made with full context.

<img
  src="/images/diagram.png"
  srcset="
    /images/diagram-480.webp 480w,
    /images/diagram-960.webp 960w,
    /images/diagram-1600.webp 1600w"
  sizes="(max-width: 768px) 100vw, 768px"
  alt="..."
/>

Ready, Set, Cloud is a static site generated by Hugo. In my case, adding srcset support meant overriding the render-image hook and adding a small amount of logic to create the additional attribute.

Performance improvements and an unexpected bonus

My primary goal with all this was to decrease load times so my site felt snappy without adding anything to my existing workflow. With the image optimization component plus the CDN for cached delivery (and the client-side srcset addition), I'm pleased to say page payloads dropped significantly. Pages render faster, Largest Contentful Paint (LCP) improved, and overall performance is more predictable.

On my homepage, the First Contentful Paint (FCP) dropped from 4.5 seconds to under a second. Load times are much more consistent across desktop and mobile as well, which has been an ongoing challenge.

The unexpected bonus was that these same changes also helped with search engine rankings. Faster pages, smaller downloads, and aggressively cached assets all feed into Core Web Vitals. So without targeting SEO explicitly, the site became easier to crawl, faster to index, and rank higher in search results simply by prioritizing user experience.

Making this easy to reuse

If you've built your own blog, you've probably run into these performance bottlenecks just like I have, and that's okay. Serving assets out of S3 is a valid solution and has worked well for me for six years as the site grew.

When my constraints changed, I wanted to extend the system without changing the deployment workflow. If you're interested in the same improvements I've described here, this setup is available in the Serverless Application Repository as a drop-in addition to an existing system. The full source is also available in GitHub if you want to dig into the details or adapt it further.

Above all else, this was about taking an entire class of decisions around image formats, sizes, and caching, and keeping them out of the day-to-day workflow. With everything in place, performance simply happens by default.

This was as fun as it was important for Ready, Set, Cloud. I wanted something easy to adopt, easy to remove, and something that quietly does the right thing as everything else moves around it.

Happy coding!

The Real Cost of Swapping Infrastructure

Allen Helton — Mon, 15 Dec 2025 20:43:40 +0000

I've gone through enough infrastructure evaluations as an architect to recognize the moment when the energy leaves the room. It's not when someone questions the performance numbers or the cost model. It's when someone pulls up the codebase and starts counting how many services need to change.

The infrastructure might be more reliable, easier to operate, or have better economics, but it doesn't matter if getting there means touching stable production code across dozens of services. The conversation shifts from "should we do this?" to "can we afford to do this?" and the answer is usually no.

That gap between "this is better" and "we can actually adopt this" is where many decisions stall or get turned down.

The real cost of infrastructure change

Architecture discussions tend to follow a familiar pattern. The whiteboard fills up with boxes and arrows, the tradeoffs look reasonable, everyone agrees that you'll come out the other end better. Then someone asks: how much code do we have to touch?

That question isn't about features or benchmarks. It's about risk. Architects evaluate the blast radius of change alongside performance and reliability. Every line of application code that needs to move, every client library that needs to be swapped, every behavior that needs to be re-learned increases the cost before you can even run a proof of concept.

For systems already in production, touching stable code introduces uncertainty. It stretches review cycles, kicks off regression testing, and makes rollback complicated. Good ideas often don't make it past this point because weaving them into existing applications costs too much.

This is especially relevant for infrastructure on the hot path. When caching misbehaves, it takes other systems down with it. Teams are rightfully cautious about changes here, even when the infrastructure side of the proposal is compelling.

What teams actually trust

Teams trust behavior they've observed in production, like how commands serialize, how errors surface, or how retries behave under load. That behavior has been exercised millions of times. It's hardened by real traffic, load testing, and years of incremental fixes. In practice, this behavior acts as a contract between the app code and infrastructure.

This is why client changes feel expensive even when two libraries look similar on the surface. Timeouts, connection handling, pipelining behavior, and edge cases around failures all shape how systems respond when stressed. At scale, subtle differences show up as tail latency spikes or incident tickets that are hard to explain.

For cache-heavy systems built on Redis or Valkey, this contract is often the wire protocol itself – RESP, the wire format the client already speaks. The application doesn't depend on “a cache,” it depends on this specific way of talking to one.

Changing what sits behind the contract

When you hold the contract constant and change what sits behind it, the risk potential drops dramatically.

Instead of rewriting cache layers or swapping SDKs across services, teams can point existing Redis or Valkey clients at Momento, authenticate, and issue the same commands they already use. The infrastructure changes. The operational model changes. The application code largely does not.

That distinction turns evaluation from a refactor into a configuration change. It lets teams observe real production behavior without committing to a rewrite upfront. More importantly, it makes rollbacks boring. Simply change an endpoint back and that's the extent of it.

This doesn't eliminate all risk. RESP compatibility has edges and limitations worth understanding. Not every Redis command is supported, but it shifts the evaluation risk from application code to infrastructure, where it's far easier to observe and reason about.

Lowering the cost of evaluation

I've noticed the infrastructure platforms that gain real adoption share a common trait: they meet teams where they already are. They respect existing contracts, existing mental models, and the realities of systems that have been running in production for years.

"Better" isn't enough if getting there requires destabilizing code no one wants to touch. The platforms that succeed make it easy to start small, observe real behavior, and back out safely when something doesn't line up. When evaluation feels reversible, teams engage honestly with the tradeoffs instead of inventing reasons to stay put.

RESP compatibility fits this philosophy. It doesn't ask teams to abandon the clients or patterns they rely on. It allows them to keep the contract they trust while changing the parts that benefit most from being managed: scaling, availability, and operational complexity.

In practice, that's often what separates interesting technology from technology that actually gets adopted.

Breakthroughs are just boring improvements that pile up

Allen Helton — Mon, 24 Nov 2025 21:04:05 +0000

We romanticize big engineering wins.

The performance chart that hockey-sticks up. The keynote slide with the impossible number. The "we hit a billion requests per second" humble brag.

The truth behind those moments is never one big breakthrough. It's dozens, possibly even hundreds, of small changes and enhancements. The kind that look ordinary when they land in a PR. The kind that your eyes casually skip over in release notes. The kind nobody celebrates… until they compound into something revolutionary.

I was recently reminded of this while listening to the Cache It podcast. Valkey project maintainer Harkrishn Patro described how his team scaled their system to handle a billion requests per second. He wasn't describing a feature launch, but rather the result of continuous refinement across every corner of the system.

Engineers, myself included, often hone in on small pieces of a system and lose sight of the forest for the trees. But with bold headlines and sensational marketing, we forget the forest is made from trees. As a result, major innovations overshadow the hard work that was done in the years and months leading up to "the big reveal."

So what happens if we take a stroll through that forest to see what the big announcement was really about?

Innovation removes friction

We tend to think innovation shows up wearing a cape. A disruptive new framework. A patented algorithm. A major architectural redesign. But most real innovation is far less glamorous – it's the process of shaving off tiny sources of drag until the entire system starts to glide. The "death by a thousand paper cuts" phenomenon is real.

Distributed systems make this painfully clear. The moment you push scale beyond "normal," you reveal dozens of inefficiencies that rarely show up in happy-path benchmarks. And this doesn't just refer to hyper scale! Problems change at every tier of scale. From POC to startup. From startup to regional. From regional to global. Suddenly, small issues aren't small anymore, they're multiplied across thousands of functions, nodes, and containers processing millions of requests per second.

That's exactly what the Valkey team discovered while pushing cluster sizes past limits that were never originally intended. Instead of rewriting the system from scratch, they focused on identifying and removing friction one piece at a time. For example, early pub/sub workloads broadcast messages to every node in the cluster, regardless of who needed the data. Reducing that via sharded messaging meant drastically less traffic flowing through the cluster bus, which freed up capacity for the work that mattered.

Or the size and frequency of internal coordination messages. By making those packets smaller and more efficient, Valkey stopped wasting cycles on communication overhead, freeing up capacity for serving actual requests.

Individually, neither of these changes would get a developer on stage at a conference. Nobody ships a "lighter gossip protocol!" feature release. Stack those improvements long enough, and the system starts performing at a level that once felt impossible.

Innovation isn't about adding more. It's about taking away.

When you remove enough drag, the system suddenly feels like it's leaping forward… even though it got there one subtle change at a time.

Reliability is a force multiplier

A fast system under perfect conditions is just a prototype. The moment you introduce chaos, like what happens regularly in production, performance either collapses or compounds. Reliability determines which way it goes.

Just like removing friction creates capacity, improving reliability creates opportunity. In distributed systems, fragility multiplies just as quickly as performance. One node degrades, then another, and suddenly a perfectly healthy cluster wastes capacity on recovery instead of serving requests. Failover becomes a performance bottleneck waiting to happen if handled poorly.

Back to the small wins with Valkey – when multiple primaries failed simultaneously (like during an Availability Zone outage), the system would struggle to elect new leaders because candidate requests flooded in all at once. Leadership battles caused downstream stalls that made the whole cluster feel unstable under pressure. The solution was introducing strategic delays to election behavior, allowing replicas to take over cleanly and confidently without overwhelming the system.

But that's the pattern – resilience opens up throughput. A cluster that can route around failure without spiking CPU or tail latency stays fast when users need it most. In Valkey's case, the additional throughput in unfavorable situations kept cycles free to handle more requests per second. Another blip on the release notes that continued to add up to the breakthrough.

Reliability amplifies every other improvement in the system. It creates breathing room. It enables aggressive scaling. It gives teams the confidence to grow beyond today's limits. When recovery becomes routine and uneventful, progress accelerates.

Scaling is an operational behavior

Contrary to what you might think, scaling isn't a hardware problem. It's a behavior problem. If changing scale feels risky, disruptive, or expensive, teams simply don't do it – even when they need to. They take the safe route: over-provisioning up front and praying they don't get surprised later.

There's nothing innovative about survival mode.

Real scalability emerges when the cost of adjusting capacity falls so low that it becomes a non-event (that's one of the reasons people like serverless so much). The faster and safer adding nodes, redistributing data, or shifting workload patterns feel, the more frequently teams perform them. And the more frequently teams perform them, the closer infrastructure can track real-world demand without waste, stress, and cost.

Once again, the Valkey team learned this firsthand. Since Redis 1.0, data migration has been a scary operation. Moving slots between nodes meant risking performance hits, stalled operations, or complicated (and error prone) manual intervention. So people avoided it, and clusters either stayed small or stayed oversized. Innovation would stall because operating it at its potential felt like walking a tightrope.

Atomic slot migration changed the behavior. Now, moving data around a cluster is predictable, controlled, and boring – exactly what operators need when managing production systems. This feature opened up capabilities that already existed but weren't safe to reach for.

When scaling becomes effortless, teams stop treating it like a last resort and start doing it as a reflex. Progress begins to compound. Capacity and confidence reinforce one another. And suddenly, what once felt like a milestone becomes just another Tuesday.

Breakthroughs happen gradually… then suddenly

From the outside, breakthroughs look like a leap. One day the headline appears. The graph turns vertical. Everyone wonders how a system could possibly jump from "pretty good" to "borderline unbelievable" in such a short time.

But from the inside, it looks like months, sometimes years, of sanding down rough edges. Fixing the tiny things that don't feel worthy of a celebration. Optimizing behavior that only shows up in profiling. Eliminating friction one pull request at a time.

That's exactly what happened with Valkey. A billion RPS was never the goal, but it was the receipt of countless invisible improvements. The milestone only looks magical because we didn't see the work it took to make it boring.

And this isn't unique to distributed systems or caching engines. It's universal. Every major engineering accomplishment is built from the moments where someone looked at a sharp corner and decided it didn't have to stay sharp.

Progress compounds. Confidence compounds. Capability compounds. Breakthroughs happen gradually… then suddenly.

So if you're in the middle of the grind, whether you're shipping incremental changes, cleaning up edge cases, or improving reliability and operability in small ways, take a breath of solace. You're stacking. You're creating the conditions where something extraordinary is going to look effortless.

The day is coming when the big announcement arrives, the system holds steady, and people assume it was easy. But you'll know better. You'll know it was earned one small improvement at a time.

Happy coding!

Cache Rebalancing Was Broken. Here's How They Fixed It.

Allen Helton — Tue, 18 Nov 2025 22:16:54 +0000

Few things make SREs more nervous than rebalancing a cache cluster.

You know the feeling. You add a node, trigger a rebalance, and suddenly latency graphs start jumping. It's a familiar risk of the job, especially when your cache sits between your users and your database. A small configuration mistake here can suddenly unleash a storm of GET requests on your primary data store.

I admit, I never really understood the concept of a slot or why it was needed. But after listening to a recent episode of the Cache It podcast on the new atomic slot migration feature in Valkey 9.0, I finally decided to dig in. The deeper I went (and the more times I replayed the episode), the more it clicked.

My learning adventure led me to ask, and finally answer, four important questions regarding slots and what happens to them when a cluster resizes. Let's dive in.

What are slots?

Most caching systems rely on consistent hashing to decide where data lives. It keeps keys evenly balanced across nodes while allowing clusters to grow or shrink with minimal reshuffling.

Valkey uses a fixed hash-slot model, specifically 16,384 slots, that together represent the entire keyspace. While 16,384 slots might seem random, it's actually 2¹⁴, a power of two chosen because it balances just the right amount of precision and efficiency. With that many slots, data can be distributed across large clusters without adding unnecessary overhead to routing or metadata.

Every key hashes to one of those slots, and each node in a cluster owns a subset of them. Rather than mapping every key directly to a node, Valkey maps slots to nodes. Since keys are deterministically hashed to slots, this makes scaling predictable. When you add or remove a node, Valkey only has to move the slots it owns, not millions of individual keys.

When the Valkey client library hashes a key, it already knows which node owns the corresponding slot. If the topology changes (aka slots being assigned to a different node due to a scaling event), Valkey issues a quick redirect so the client can retry against the correct node.

What was broken about the old migration model?

The old migration model moved one key at a time between nodes, triggering a flurry of redirects and topology changes. Clients were constantly told, "Sorry, this key moved, try over there."

It was essentially a brute-force way of moving slots from one node to another. It worked, but it wasn't elegant – and it definitely wasn't fast.

Each key transfer required multiple round trips, and every slot migration forced clients to refresh cluster topology. Large values could even block threads during serialization.

When you're working with millions of keys across your cluster, that adds up to a resource-intensive process that can take minutes to complete, all while the cluster remains live and serving traffic.

The result was instability and slower tail latencies during migration. Which means it was something you'd postpone unless you absolutely had to run it.

How does atomic slot migration fix it?

Valkey 9.0 redesigned the slot migration process using the same principles that power replication. Instead of moving keys one by one, atomic slot migration runs in three distinct phases:

Snapshot – The source node forks a background process and captures a point-in-time snapshot of the slots being migrated while continuing to serve live traffic.

Streaming – Any writes that happen during the snapshot are captured in a buffer and streamed incrementally to the target node.

Finalization – Once all data is synchronized, Valkey briefly pauses new writes, sends a final marker, and performs a single, atomic handover.

This three-phase approach eliminates the multiple round trips, client redirects, and fragile user experience that used to come with slot migration. Because the process runs in the background and atomically switches ownership when complete, there is no risky in-between state. Once it finishes, your slots are already assigned to the target nodes without interruption or confusion.

Why is this better?

For operators, this is a big deal.

Fewer round trips, fewer topology changes, and no split-brain state during migration. You can rebalance entire clusters without disturbing workloads or waking up the on-call engineer (hooray!).

Both models still exist today, but atomic slot migration represents a new standard for reliability. It shows how thoughtful engineering can make the hardest operational tasks feel invisible.

As Khawaja said in the latest Cache It podcast episode with Jacob Murphy, "This moves us closer to a world where scaling a cache never means taking it offline."

And that's a world every SRE wants to live in.

Happy coding!

Designing smarter caches with Valkey 9.0's numbered databases

Allen Helton — Fri, 07 Nov 2025 16:54:32 +0000

I recently watched a podcast episode where Khawaja Shams, CEO of Momento, and Kyle Davis, Developer Advocate at Valkey, talked about everything BUT performance of Valkey 9.0.

"That's the first time anybody's asked me to talk about Valkey and not talk about performance," Kyle laughed when Khawaja asked him to skip the speed benchmarks. What followed was one of those conversations where you realize a seemingly simple feature opens up architectural patterns you didn't even know were possible.

I didn't quite understand it at first. Kyle mentioned numbered databases in Valkey 9.0. Then he mentioned clustering and how numbered databases get a boost when used together. The terminology was throwing me off until I heard it explained a different way.

Kyle referred to numbered databases as "namespaces" (which makes way more sense), and explained that they let you logically separate your keys. The same key name can exist in database 0 and database 42 with completely different data, and they'll never collide.

"This is a feature that goes way back to Redis one," Kyle explained. "They had this concept of numbered databases… but it allows you to take the keys that you have and separate them up logically into, by default, 16 databases."

Up until v9, this only worked in standalone mode. The moment you needed cluster mode (which basically every production workload does), you were stuck with just database 0. For over a decade. An update has been due for a long time.

When Valkey calculates which slot a key belongs to, it only looks at the key name. The database number doesn't matter. So if you have a key called user:100 in database 0, and another key called user:100 in database 5000, they both live in the same slot.

On the surface, this might not seem like a big deal. But digging into what that means – you can move data between databases almost for free. Moving keys between databases is literally a pointer change, not a data migration. You can move a sorted set with 3 million elements between databases with no network transfer. Just… instant.

Problems this solves

I immediately started thinking about real problems this solves. And I found a few that made me really excited.

No more key prefixing

The obvious use case we jump to when we hear "multiple databases" is multi-tenancy. In a multi-tenant SaaS app, you probably build your keys like this (if you're anything like me):

cache.set("tenant_12345:user:100", user_data)
cache.set("tenant_12345:session:abc", session_data)
cache.set("tenant_12345:cart:xyz", cart_data)

Every key you get, set, and delete has the tenant id prefixed on it. This is a (generally) safe way to make sure you logically separate customer data. If you have 100 million keys with a 15-byte prefix, that's 1.5GB of RAM just storing the same string over and over. It's an incredibly wasteful pattern!

But now, we can save tenant data in a tenant-specific database for a set of keys. It not only makes the code cleaner, it also frees up tons of memory to be used on… more keys!

cache.select(12345)  // Tenant 12345's namespace
cache.set("user:100", user_data)
cache.set("session:abc", session_data)
cache.set("cart:xyz", cart_data)

In other words, we've effectively expanded our potential working set size without any hardware changes. Same isolation. Zero prefix overhead. And remember, Valkey 8.0 made headlines for saving 8 bytes per key. We're talking about eliminating 10-20 bytes here.

Black Friday-scale price switches

Kyle described a use case that made me immediately think of every e-commerce system I've ever seen:

"Let's say you have something that you want to instantly switch over, atomically switch over from one set of data to another. So you've said, here it has a million elements and then at 12:01 AM you want to have a different set of million elements."

Sounds exactly like holiday pricing to me. Businesses have their entire catalog of products and prices being served out of their cache cluster. The midnight after Thanksgiving, they need to flip to sale prices. All of them. Instantly.

Typically, you'd have a batch job that updates prices in bulk or update your data with feature flags, or some other process equally prone to error that results in customers seeing random mixes of old and new prices.

But with multiple databases, you can build your catalog in a different database and instantly move it over with a single command

cache.select(2)  // holiday pricing database
cache.move("product:prices", 1) // moved to production database

No data copying. No network overhead. Really, just a configuration change and suddenly your entire pricing model flips. This is the kind of thing that sounds impossible until someone shows you it's not.

Parallel test environments

This one hit home for me. If you're running tests in CI/CD, you've probably tried to run multiple parallel test jobs, all needing isolated data – but all stepping on each other's toes.

Of course, you could get around this by not running your tests in parallel and resetting your test data after every job, but nobody has time for that anymore. Multiple databases makes this one a trivial problem.

test_db = int(os.getenv("CI_JOB_ID")) % 10000 
cache = Valkey(host='valkey-cluster') 
cache.select(test_db) 
cache.set("user:test@example.com", user_json) 
assert cache.get("user:test@example.com") == user_json 

cache.flushdb()

And it scales, too! Kind of. The podcast talked about running a cluster with 10 million databases just to see what would happen. Turns out, it worked fine! Valkey does lazy allocation on databases, so they only consume resources when you actually put data in it. So you can assign unique database numbers to thousands of concurrent test runs and it costs you nothing until you write data.

This is the kind of engineering that makes you want to high-five someone through the internet 🙌

More is (probably) coming

In the podcast, Kyle hinted at – but didn't commit – to some cool feature enhancements for numbered databases that make it feel like a first-class citizen in Valkey.

Named databases – Instead of SELECT 42, imagine SELECT "production" or SELECT "tenant_acme_corp". This is a huge boost for developer experience for obvious reasons, but also makes it easier in multi-tenant scenarios so you don't have to track KV mappings of your tenant ids to numbers.

Per-database memory limits – Prevent your test database from eating all your RAM. Sandbox test environments so they don't take up all the resources. It also fights noisy neighbors in multi-tenant production scenarios.

Per-database rate limiting - True multi-tenant resource isolation at the namespace level.

Numbered databases in cluster mode give us something new – predictable, lightweight isolation built right into the cache layer. That's a big deal for enterprise workloads. Multi-tenant apps get simpler. A/B testing and CI environments get safer. Rollouts and migrations get less risky.

Kyle summed it up perfectly:

Somebody can roll in and just take something, start saving memory with it, or have new use cases entirely by going SELECT 4 instead of not worrying about a database at all.

It's a subtle-yet-powerful new feature that might end up shaping how we architect distributed caching in the future.

Happy coding!

This is the best new feature from POST/CON

Allen Helton — Wed, 01 May 2024 19:05:16 +0000

I make it no secret that I think APIs will literally shape the future of tech. APIs give us access to everything - the lights in our homes, our favorite books, the weather, inventory of the grocery store down the street. Being able to use these APIs as tools enables us to build automations and abstractions that take care of everything in our daily lives without lifting a finger.

My daily routine starts with an alarm that wakes me up within a 20-minute window at the best possible time based on my sleep cycle. I get out of bed and start brushing my teeth to some energizing music that turned on automatically after my alarm went off.

I work my way into the kitchen where the lights turn on when my presence is detected. I drink my pre-workout as an email hits my phone with the generated workout of the day. I make it over to my home gym where a catered workout playlist is already playing. I have all these conveniences follow me around my house without having to do anything. It just does it.

How does all this happen? With APIs. Not just APIs though, I've built this particular set of capabilities as a workflow. This happens, then that happens, and as a result, these other two things kick off.

Chaining actions together to create a workflow is where the power of APIs really shines.

Think of APIs as LEGO bricks. An individual brick doesn't do much. But when you put them together in a meaningful way, you quickly go from a bunch of random squares to the Taj Mahal.

This is why I'm so excited about related requests inside of Postman. There were a bunch of cool announcements at POST/CON, but related requests has the potential for some serious disruption and incredible opportunities.

What are related requests?

Inside of Postman, when you add a request from a verified API to your collection, you'll see a new little icon on right-hand toolbar. If you click it, you'll see some recommendations of other requests to use alongside it. You can click on one of these recommendations and have it brought into your collection automatically.

I've built so many workflows in Postman where I add a request and think to myself, "now what?" I have to figure out what to do next to accomplish my tast end-to-end. Usually this means going back to the API spec - if there is one - and trying to identify the next endpoint to add to my workflow. But with related requests, Postman will identify what you're working on and offer intelligent recommendations to move you along faster. These recommendations show you full documentation so you can make informed decisions on what you're adding to your workflow.

Sometimes when I'm building a workflow I know exactly what I want to do. The hard part isn't knowing what I want the outcome to be, but rather the steps to get there. Chances are high the workflow I'm building is not reinventing the wheel - meaning I'm probably trying to create something that has been done before.

API vendors (usually) know how customers use their products. In addition to an API reference, they regularly include tried and true patterns and best practices for using their services in the documentation. It's often up to developers to read the docs, learn the pattern, and implement it themselves. There's nothing wrong with this at all - in fact, it's been a standard practice in the tech industry for years. I often refer to it as the "copy, paste, replace" method.

Related requests are taking that a step further. It sees what you're building and offers recommendations from the vendor to get you to your end goal faster. Not only is it offering suggestions, but it's also adding functional requests to your collections. Think of this almost like Amazon CodeWhisperer or GitHub Copilot but for Postman. Except that it's a version trained on exclusively reputable sources with training data aimed at doing what you're already trying to do.

Where I can see this going

I wish I had insider information on the roadmap for Postman. But I don't. So I'm going to offer some speculation about what I see this turning into.

This initial release is step one of three:

Create a searchable index of requests from reputable API vendors
Update the index with cross-vendor related requests
Unleash PostBot on the index

These are pretty big steps with a lot of implications, so let's quickly touch on each one.

Searchable vendor-specific index

This is what we have right now. Postman has indexed collection data from verified APIs and is using it to offer recommendations for the next request to use. These collections are maintained by individual companies and generally exclusively include requests specific to them - which is a totally acceptable and understandable thing to do!

But because of this, you likely won't be getting recommendations of how to turn on your lights after adding a request to play a specific playlist on Spotify. To do that, we need to progress to step two.

Cross-vendor collections from trusted sources

Right now the only data being indexed for related requests is coming from verified teams. These teams represent individual companies like Mastercard, HubSpot, and Momento. But what if Postman started indexing data from verified builders, like the individuals in the Supernova program?

You can't reasonably expect companies to build and maintain collections that contain requests from other companies. But you can expect that behavior from builders. If open-source software has taught us anything, it's that builders love finding and sharing creative ways to do something.

So if I was to build a public collection that contained all the requests from my morning routine, it could potentially be added to the related requests index. Then when you start building a collection that turns your lights on with the Kasa API, you might see a recommendation of how to use the Spotify API to turn on a playlist of your choice. This would start bringing to light clever use cases and possibly sparking new, innovative ideas in the process.

Doing this involves a lot of trust-building. Trust from Postman to individual builders. Trust from Postman consumers to the related request functionality. Open-source projects have also taught us that not everything you see on the Internet is trustworthy. So establishing the trust early with reputable builders is going to be pre-req #1.

You also need to balance priorities. Prioritize vendor collections for recommendations over community collections. Maybe the user building a collection wants to turn their lights off instead of playing a playlist next. They'd be able to do this easily if the vendor specific recommendations were at the top.

AI recommendations

Postman has been heavily investing in generative AI with PostBot. It's a built-in companion that can help you debug issues, generate documentation, write tests, and much more. But what if we started training it on the recommended requests index?

Assuming we have verified builders contributing to related requests in addition to the vendors themselves, there's a lot of data to train on. By then, maybe the public API network has also evolved to the point where vendors can classify themselves as different categories, like Spotify being classified as a music and podcast API, Stripe as a payment processor, and Momento as a serverless cache and storage provider. This type of context can provide an LLM with everything it needs to offer next-gen recommendations.

And why stop at recommendations? If PostBot is trained on thousands of verified collections and workflows and knows what each set of APIs are useful for, it could absolutely generate and build entire workflows automatically. We know PostBot has this capability, as it was announced at POST/CON that it can now help you build flows.

But with the power of the entire public API network behind it, imagine what would happen if you gave it a prompt of "Build me a payment processing app that sends confirmation emails to the recipient, updates a spreadsheet, posts the transaction to Slack, and plays a 'tada' sound every time a transaction is successful."

In seconds, you could have a flow that uses the Stripe API to post a payment, the Google Sheets API to add a row for internal bookkeeping, SendGrid to send emails, Slack for internal messaging, and Spotify for the fun sound. All you have to do is authenticate (which just got easier as well)!

Final thoughts

The game has just been changed. Related requests are a huge step in the right direction for developer experience and time to market. Knowing what to do next as instructed by the vendor is a big deal.

APIs can be intimidating - especially when it has hundreds of calls you can make. Knowing what to do next might only be intuitive to the developers of the API, which makes it hard to consume. Back to the LEGO analogy, if you dumped all the bricks out from a set into a big pile and told someone to go build it without any instructions, they might look at you in disbelief. They'd try, fail a few times, try again, and maybe get it eventually. But if you give them the instructions, chances are good they'll come out the other side of it with what they wanted.

This is what related requests are doing for builders! They're providing clear instructions on what to do next. Once we get additional data in there for building workflows across multiple APIs and the help of an LLM, software might actually be easy to build 🤔

I envision a future where capabilities like this enable both non-technical and technical people alike to build what they want with minimal effort. What a time to be alive!

Happy coding!

Serverless Postgres with Neon - My first impression

Allen Helton — Wed, 24 Apr 2024 13:04:40 +0000

When you decide to go serverless, be it a personal decision or enterprise-wide, you're signing up to be a forever student. Modern technology moves fast and keeping up with all the new features, services, and offerings week after week is something that you need to do to stay effective. Cloud vendors are continuously releasing higher and higher abstractions and integrations that make your job as a builder easier. So rather than reinventing the wheel and building something you'll have to maintain, if you keep up with the new releases, someone may have already done that for you and is offering to maintain it themselves.

Such is the case with Neon, a serverless Postgres service, that went generally available on April 15. Congrats Nikita Shamgunov and team on the launch. When I saw the announcement, I knew I had to try it out for myself and report back with my findings.

I host a weekly show with Andres Moreno for the Believe in Serverless community called Null Check. Andres and I find newly released services or buzzing features and try them out ourselves live on-air for a true "this is what it is" experience. We try to build some silly stuff for a little bit of fun while we're at it.

Last week, we did a full assessment on Neon, looking at pricing, developer experience, elasticity, and serverless-ness (we're going to pretend that's a word). Let's go over what we found.

Pricing

Neon has two pricing metrics - storage and compute, which feels spot-on for a managed database service. On top of the two pricing metrics, there are four plans to choose from ranging from free tier to enterprise.

Storage is a metric we should all be familiar with. It refers to the amount of data you're storing inside of Neon. Each plan includes a certain amount of data, and if you exceed it you pay for the overage. The overage cost decreases the higher the plan. Meaning it's more per GB of data on the Free tier than it is at the Scale tier.

Compute, on the other hand, is similar to what we see in other serverless services, but also kind of different. It's the same in that you're charged for the amount of compute in hours x vCPUs consumed, but it's different from other managed DB services in that you're charged for this time while queries/operations are running. With something like DynamoDB, you're charged for read and write capacity units - which are determined by the size of the data you're handling. With Neon, it's all about the effort the machines are putting in with seemingly no direct charge for the amount of data handled.

Developer experience

Developer experience (DX) refers to a lot of things. Onboarding, ease of use, clearness of documentation, intuitiveness, etc... all play a role when talking about how a developer uses your service. Andres and I tried to see how far we could get without needing to dive into the docs - which is a great indicator of how easy and intuitive the experience is.

We started in the console, where a wizard guided us through setting up our first project and database. This was a simple form that required us to come up with a name for everything and select a region to use. After that, it was set up and ready to use! We were even presented with a quickstart for getting connected to the DB we just created.

In this popup, there was a language picker that allowed us to select from psql, Next.js, Primsa, Node.js, Django, Go, and several others. As far as quickstarts go, this one was fast. I put the provided code as-is in a JavaScript file and was able to connect and query data within seconds.

Since the database was created void of tables and data, my query didn't do anything 😅 so we used the integrated SQL editor in the console to create tables and the Neon CLI to insert data in bulk from a csv. Overall, we went from idea to queryable data inside of tables in less than 5 minutes!

The Neon SDK was easy to use as well. Granted all I did was use raw SQL with the SDK, which is ripe for a SQL injection attack, it still did exactly what I needed it to do without needing to go digging through docs for 30 minutes. In fact, I could have easily used the industry standard node-postgres package instead of the one from Neon and communicated with my database the same way I've always done. This means if I migrate from another service to this one, all I'd need to do is update the connection string!

Overall, I was delighted at how easy the developer experience was to get started with this service. It also feels like a breeze to maintain.

Elasticity

A service is only as good as its elasticity. If your app has more traffic than a service can handle you're forced to either throttle requests or go with a service that can handle it. So we ran some tests to see how Neon could scale with traffic spikes.

For this benchmark, we ran two tests: one for heavy reads and another for heavy writes. To run the benchmark, we built a small web server inside of a Lambda function and created a function url for public access to the internet. We created an endpoint that would do a large read, joining across several tables and returning several hundred results, and another for a write operation which created a single row in one table.

Then, we used the Postman load generator to push load onto each of these endpoints, running at a sustained 75 requests per second (RPS) for two minutes.

Granted this isn't a ridiculously high load, but keep in mind I was testing out the free tier that has limited compute usage. So 7,500 requests over 2 minutes will have to do. And it did well!

Funnily enough, the only time we were getting errors in this test was when the Lambda function was trying to keep up with scaling. When we sent burst requests out of nowhere, we'd get back 429 - Too Many Request status codes from Lambda as it was scaling up. But neither the write nor the read tests overwhelmed Neon - even at the lowest tier.

From what I can tell, there is a little bit of a cold start when Neon is initiating compute. The documentation says the compute instances wind down after 5 minutes of inactivity in the free tier. Running a large read query on an inactive database resulted in about 800ms round-trip time (RTT) for the endpoint. This included the Lambda cold start as well, so overall not a terrible latency. Subsequent runs dropped to about 350ms.

It appears to me that Neon will cache reads for about 1-2 seconds. After a warm start, running the same query resulted in an 80ms RTT for a couple of seconds, then would spike up to 300ms for a single request, then back down to 80ms again. Given that pattern, I imagine there's built-in caching with a short time-to-live (TTL).

Serverless-ness

Any time something is branded as serverless, I give it a bit of a skeptical eye roll. Depending on who you believe, serverless has no meaning. Rather than thinking about serverless as a "thing" I like to approach it as a set of capabilities:

Usage-based pricing with no minimums
No instances to provision or manage
Always available and reliable
Instantly ready with a single API call

Given what we've already discussed about Neon, I feel like it checks the boxes on the serverless test. For pricing, we're only paying for what we're using - the amount of data stored in the service and the amount of compute time we're consuming in our read and write operations. There is no minimum cost and it scales linearly with the amount I use it.

It took me a few days to land on my opinion for no instances to provision or manage. There definitely is compute with an on/off status and it's made clearly visible to end users. However, users can't do anything with it. It's managed completely by Neon. If more compute is needed, it is automatically added. I'm not responsible for configuring how it scales, it just does it. I initially didn't like that I was given a peek behind the curtain, but that's really all it is - a peek. I'm not managing anything, Neon is.

Always available and reliable is a big one for serverless. If I need to use it now - by golly I need it now. Even though compute instances start idling after 5 minutes, they're still available at a moments notice when traffic comes in. There are no maintenance windows or planned downtime, so this one is checked as another win in my book.

The onboarding experience for Neon was minimal. I typed in a name for my database and hit the "Go" button and it was immediately available. This feels serverless to me. I don't have to know the amount of traffic I expect or configure auto-scaling groups or decide what operating system I want the compute to run on. Again, this is all managed for me by Neon.

By my count, that makes this service definitely serverless!

Final thoughts

This was a great test of a super cool service. It's nice to see SQL catching up a bit in the serverless game. One of the really cool features I liked about Neon was data branching. They treat database data similar to code in a GitHub repository. You can branch your data and use it in a sandbox environment, like a CI/CD pipeline or a trial run for an ETL job. When you're done with the branch, you can either discard it or merge it back into its parent.

This is a huge capability for any managed database. It simplifies many workflows and provides an easy and instant way to get a snapshot of data. It follows a copy-on-write principle, meaning the data is copied whenever it's modified in your branch. If you're just doing reads it uses a pointer to look at the parent branch data. This helps reduce storage costs and increase availability of data when you branch.

Overall, I think this is a great product and I will be building more with it. It's a nice alternative to something like Aurora Serverless, which has a usage minimum and requires a VPC 😬

If you're into relational databases and are a fan of Postgres, it's worth a shot.

Happy coding!

Show your personality

Allen Helton — Wed, 17 Apr 2024 14:37:54 +0000

Yesterday was my birthday. I just turned 34, landing me on the soft side of the "mid-30s". I've always had a hard time with birthdays because in my head I have that feeling of "I'm going to be young forever." But aches, pains, and hangovers are beginning to tell me otherwise.

When I turned 30, I had someone tell me that your 30's are much more fulfilling than your 20's. I didn't know what that meant at the time, but 4 years into it, I'm starting to understand what he meant. When I was in my 20's, believe it or not, I was very shy. I fully identified as an introvert and wouldn't give you an opinion unless I was specifically asked for it.

I didn't know much about myself, and I would constantly watch others to see what they did and how they reacted to things. I was learning.

In my 30's, with my kids starting to walk, talk, and go to school, plus a few nudges from some enablers, I stopped watching all the time. I knew what I liked and to be honest, I cared a lot less about what other people thought. Sounds like a harsh statement, but hear me out.

Once I fully realized the things that made me happy, I wanted to do those things. All the time.

Before, I would be nervous about being judged or ridiculed and back off doing some of the things that made me happy because I cared too much about what other people thought of me. I'd act reserved or not do something because I didn't want to stand out at the risk of being embarrassed. Now though, I do it anyway and try to get other people to have some fun with me because it doesn't matter.

A person passing you in the grocery store might hear you singing along to your favorite song that just came on over the speakers. Sing it! It doesn't matter if they laugh at you! You know what? You probably just brightened their day.

I like making people laugh. That's one of my strongest personality traits. For too long, I've held that back to just my friends and family because I was too shy to do it to strangers. But something about being in my 30's changed my outlook on life. I know what I like, and by golly I'm gonna do it.

How things changed by showing some personality

In addition to becoming more and more extroverted as I got older, other aspects of my life changed considerably as well. From how I produced content to who I was hanging out with, leaning into your personality has a huge ripple effect.

Content creation

The first time I ever published a blog post, I almost threw up with anxiety. The imposter syndrome was fierce. I wrote an article about the way I ran Scrum retrospectives for my team. I had come up with a way to get the other introverted developers on my team to talk and I was excited to share that with world.

But I was so nervous.

Classic imposter syndrome hit me like a bag of doorknobs. "Why should people listen to you?" "There are tons of articles on retrospectives out there, the world doesn't need another one." "Nobody is going to read this, it's just by some random dev."

But whether I knew it or not, I wrote something unique. Something with my personality embedded directly in it. Even if someone had written about my exact topic before, they didn't write it the way I did - because I showed my personality.

It was something I didn't know I was doing at first, then something I intentionally did and was extremely nervous about. The first time I ever wrote an opinion piece I remember telling my wife at the dinner table, "we're about to see how the community views my opinion." I was intimidated, to say the least, because it was very bluntly showing my opinions and personality.

Turns out it was received really well. As was the next opinion piece I wrote. And the next one. And the next one. With each post, my confidence went up as did my understanding of what I liked.

This led to my passion for content creation. I took a big leap of faith to leave enterprise architecture and become a developer advocate. I'm thankful every day for that decision. But if I hadn't started showing my personality and begun to build a portfolio of blog posts, newsletters, podcast episodes, and conference talks, I wouldn't have been qualified to make a move.

Now, I've discovered that personal touches make all the difference. I moved from generic vector graphics to expressive pictures of my face for blog post header images. I switched from generated text-to-speech to recording myself read my blog posts. I try to throw as much of myself into everything I create - not just for fun, but to show that there's a person behind the content, not some random LLM.

So many friends!

It's good to be different. People feel connected to other people they feel are genuine. I show lots of emotion and make strong claims in the content I create, and as a result I've had lots of people reach out to me for help, to meet up for coffee, or challenge my opinion. I take every chance I get to take someone up on offers like this.

I recently released an episode on my podcast about the value of meeting in person, where I have a discussion with my guest about the tremendous value of networking. Making friends and building connections not only helps you professionally, but personally as well.

I have friends all over the world that I try to visit any time I travel. Not to talk tech, but to chat, experience their culture, and build stronger friendships. If I hadn't shown my personality and started with trust-building in my content, I'd have a completely different experience when travelling - usually a much less fulfilling one!

Fun observation - because I share so much, many of my new-found friendships have been accelerated! People feel like they know me already when we meet in person, which is a fun and surreal feeling to say the least 😅

Growing into it

Showing your personality is all about comfort. Are you comfortable showing who you are to the "open internet?" Now, you don't have to go and tell your deepest darkest secrets, but throwing a little bit of your flair into the mix when writing or recording a video is absolutely the way to get started. Tell a joke! Share a personal story. Throw in something that makes you uniquely you.

Over time your comfort will grow. Like I said earlier, I like making people laugh. I started by telling funny little anecdotes in my blogs. Now I dress up like a farmer and post stuff like this on social media.

// Detect dark theme var iframe = document.getElementById('tweet-1778788914708332740-325'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1778788914708332740&theme=dark" }

But it was a journey to get here. A combination of age, confidence, and lots of practice have all led to how I create content and interact with people on social media. And let me tell you, I'm having a blast.

Start with what you're comfortable with, it doesn't have to be much. If you're anything like me, you're probably already showing your personality in whatever you're creating and you might not realize it. People like opinions and it's totally ok if they don't agree with yours. I often talk about diversity of thought, which is just another way to say getting a bunch of other opinions before you do something. Hearing the opinions of others and considering them in your decision makes you more balanced and generally leaves you with a more complete view of something.

So be funny. Or witty. Or terse. The point is to be you. You don't need to spend effort to abstract away your personality from the content you create. Share it! Over time, you'll really pick up on what you like and you'll want to do it more and more. And that's why we're all here anyway, isn't it?

Happy coding!

How to trigger events every 30 seconds in AWS

Allen Helton — Wed, 10 Apr 2024 13:09:18 +0000

Software engineering is one of those industries that likes to periodically humble you. You think you know exactly how to build something, come up with a plan and an estimate, sit down to start writing some code, and BOOM. It does not at all work like you thought it did.

That happened to me the other day.

I was building a basic multiplayer puzzle game that required all the players to start playing at the same time. The game took roughly 20 seconds to play, so I figured I'd send out a broadcast push notification every 30 seconds to start the next round. I've used the EventBridge Scheduler a bunch of times over the past year, this sounded like an easy task. I just needed to create a recurring schedule every 30 seconds that triggers a Lambda function to randomize some data and send the broadcast.

It wasn't that easy. The smallest interval you can set on a recurring job via the scheduler, or any cron trigger in AWS for that matter, is 1 minute.

I asked Amazon Q what my options were and had mixed feelings about what it said.

As I just mentioned, the first option Q gave me is wrong. If you put in a value of rate(30 seconds) into a schedule, you get an error saying the smallest rate it can do is 1 minute. So that option was out. Option 2 with Step Functions was intriguing and we'll talk more about that one in a second.

Option 3 where you recursively call a Lambda function seems like a terrible idea. Not only would you need to pay for the 30-second sleep time in the execution, but this method is also imprecise. You have your variable time of compute for the logic, then a sleep time to bring you up to 30 seconds. What's more, this puts you in a loop, which AWS detects and shuts down automatically. You can request to turn it off, but the docs make me feel like you really shouldn't.

The last option given to us by Q was triggering the Lambda function off a database stream - which in itself sounds like a reasonable idea, except you'd have to have a mechanism that writes to the database every 30 seconds, which is what we're trying to do in the first place. So that doesn't solve our problem at all. This leaves us with Step Functions as our only real option according to Amazon Q. Let's take a look at that.

Before we continue, if you're here looking for the code you can jump straight to GitHub and grab it there.

Using Step Functions for 30-second timers

Step Functions itself doesn't allow you to trigger things at sub-minute intervals. It is still constrained to the 1 minute minimum timer. However, what Step Functions has that other things do not is the wait state. You can tell Step functions to run your Lambda function, wait for 30 seconds, then run the Lambda function again.

This is what I did in my first iteration and received a wealth of mixed opinions on Twitter 😬

// Detect dark theme var iframe = document.getElementById('tweet-1761138903874928761-962'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1761138903874928761&theme=dark" }

The premise was simple. I would trigger this state machine to run every minute. It would immediately execute my Lambda function that did the broadcast, wait 30 seconds, then run the Lambda function again. Easy enough, right?

After being told time and time again that this solution seemed really expensive, I wanted to see how much it cost to run for a month. I was using standard Step Function workflows, which are billed by number of state transitions as opposed to execution time and memory, which is how Lambda and express workflows are billed.

Standard workflows cost $0.000025 per state transition. As far as costs go in AWS serverless services, this one is generally viewed as expensive. I don't disagree, especially if you make heavy use of Step Functions on a monthly basis. But what you're paying for here is much more than state transitions, you're also paying for top-of-the-line visibility into your orchestrated workflows and significantly easier troubleshooting. It's an easy trade-off for me.

My workflow above has three states, but five state transitions! We are always charged a transition from "start" to our first state and from our last state to "end". If we run our workflow every 30 seconds for 24 hours, we can calculate the math like this:

5 transitions x 60 minutes x 24 hours x 2 iterations per minute x $.000025 = $0.18

Ok $0.18 per day isn't that bad. That's $65 per year for the ability to run something every 30 seconds. But I'm confident we can do better.

Minimizing state transitions

We know that state transitions literally equal cost with Step Function standard workflows. We also know that every time you start a workflow you're charged 2 transitions for start and end states. So what if we just added more wait states and function calls?

I started down this path and very quickly realized it was not a good idea. It's probably not very hard to see why based on the picture below.

This workflow is a nightmare to maintain. It's the same function being repeated over and over again with the same wait state between each execution. If I wanted to extend it with more executions, I'd add more states. If I ever changed the name of the function, then I'm in for a lot of unnecessary work updating that definition in X number of spots.

However, this did cut down on the number of state transitions. Instead of triggering the workflow every 60 seconds, I could now trigger it every 2 minutes, eliminating half of the start/end transitions every day. I liked that.

Doing the math, it ended up saving $.02 per day. What I needed was to queue up something like that over the course of an hour, but I wasn't about to maintain a workflow that used the same function + wait combo 120 times in it. So I made it dynamic.

Dynamic loops

My goal here was to minimize the number of times I ran the workflow each day so I could keep the number of state transitions to a minimum.

Turns out I can use a Map state to loop over the function execution and wait states. Maps do not charge you a state transition when it goes back to the start. This is perfect! Exactly what we needed to reduce the number of start/end transitions. It also is a much, much simpler workflow to maintain.

But there are a few things we need to keep in mind.

Maps aren't "for loops"

As much as I'd like to say "run this loop 100 times," in a Map state, it doesn't work that way. These states iterate over an array of values, passing the current iteration value into the execution. Think of it like a foreach rather than a for. To get around this, we need to pass in a dummy array with N number of values in it, with N being the number of executions we want to run. For our workflow above, that means an initial input state like this:

{
  "iterations": [1,2,3,4,5,6,7,8,9,10]
}

This would result in 10 executions of the Map state. We can include as many iterations as we want here to reduce costs with one exception. More on that in a minute.

Threading

Map states are concurrent execution blocks. The Step Functions team recommends going up to 40 concurrent executions in your Map states for normal workflows. But we don't want multi-threaded execution in this workflow. We want the iterations to go one after another to give us that every 30 second behavior.

To do this, we must set the MaxConcurrency property to 1 so it behaves synchronously. Without this, our state machine would be starting games all over the place!

Execution event history service quota

Did you know there is a service quota for a state machine execution that limits the number of history events it can have? Me neither until I ran into it the first time.

You can have up to 25,000 entries in the execution event history for a single state machine execution. Before you ask, no that does not mean state transitions. These are actions the Step Function service is taking while executing your workflow. Take a look at the entries for our state machine.

You can see that each Map iteration adds 3 entries to the execution history. On top of that, each Lambda function execution adds 5 entries, and the wait state adds 2. So each Map iteration for our specific workflow adds 10 entries. There are also entries for starting and stopping the execution and the Map state itself. To figure out the maximum number of loops we can pass into our map, we need to take all of these things into consideration.

(25,000 max entries - 5 start/stop entries) / 10 loop entries per iteration = 2499 iterations

So we can safely loop 2499 times without exceeding the service quota. At 30 seconds an iteration, that will get us through a little over 20.5 hours. To give it a little buffer and to make nice, round numbers we can safely run this state machine for 12 hours. This would result in a starting payload like this:

{
  "iterations": [1,2,3,4,5...1440]
}

This will result in our Map state looping 1440 times over the course of 12 hours, putting us well within our execution event history service quota and reducing the number of start/stop state transitions by 2872 every day or roughly $.07, which is a 39% cost reduction!

Deployment and configuration

Now that we understand conceptually how this works, it would be nice to look at an example so we can understand the moving parts. If we want our state machine to run every twelve hours, we can add a ScheduleV2 trigger in our IaC along with the iterations array we defined earlier.

HeartbeatImplicitStateMachine:
  Type: AWS::Serverless::StateMachine
  Properties:
    Type: STANDARD
    DefinitionUri: state machines/heartbeat-implicit.asl.json
    DefinitionSubstitutions:
      LambdaInvoke: !Sub arn:${AWS::Partition}:states:::lambda:invoke
      HeartbeatFunction: !GetAtt HeartbeatFunction.Arn
    Events:
      StartExecution:
        Type: ScheduleV2
        Properties:
          ScheduleExpression: rate(12 hours)
          ScheduleExpressionTimezone: America/Chicago
          Input: "{\"iterations\": [1,2,3,4,5...1440]}"

In the real implementation, you would include all the numbers in the array and not use 5...1440 like I did. I did it that way for brevity. Nobody wants to read an array with 1440 items in it 😂

This type of trigger in SAM sets up an EventBridge Schedule with the rate you configure and passes in whatever stringified JSON you have for input to the execution. Depending on your use case, you can change the rate and iteration count to whatever you want - as long as it doesn't surpass the execution event history quota.

There is no magic with the state machine itself. It is looping over the function execution and wait state as many times as we tell it to.

Advanced usage of this pattern

This is a great pattern with straightforward implementation if you want to build something like a heartbeat that runs at a consistent interval around the clock. But what if you don't want it to run all the time?

The use case I had for my game fit this description. I wanted it to trigger every 30 seconds between 8 am and 6 pm every day, not 24/7. So I needed to come up with a way to turn the trigger on/off on a regular interval. This made the problem feel much harder.

To get the behavior we want, we need to turn on and off the recurring EventBridge schedule for our state machine at specific times. So I built another Step Function workflow that simply toggles the state of the schedule given an input.

This state machine is triggered by two EventBridge schedules:

Every day at 8 am that enables the 30-second timer state machine schedule
Every day at 6 pm that disables the 30-second timer state machine schedule

To change the behavior of each execution, I pass in either ENABLED or DISABLED into the input parameters in the schedule.

EnableDisableScheduleStateMachine:
  Type: AWS::Serverless::StateMachine
  Properties:
    Type: EXPRESS
    DefinitionUri: state machines/enable-disable-schedule.asl.json
    DefinitionSubstitutions:
      GetSchedule: !Sub arn:${AWS::Partition}:states:::aws-sdk:scheduler:getSchedule
      UpdateSchedule: !Sub arn:${AWS::Partition}:states:::aws-sdk:scheduler:updateSchedule
      ScheduleName: !Ref HeartbeatSchedule
    Events:
      Enable:
        Type: ScheduleV2
        Properties:
          ScheduleExpression: cron(0 8 * * ? *)
          ScheduleExpressionTimezone: America/Chicago
          Input: "{\"state\":\"ENABLED\"}"
      Disable:
        Type: ScheduleV2
        Properties:
          ScheduleExpression: cron(0 18 * * ? *)
          ScheduleExpressionTimezone: America/Chicago
          Input: "{\"state\":\"DISABLED\"}"

By doing this, I also need to change the iterations of my 30-second state machine since it isn't running 24/7 anymore. Since it's only running for 10 hours, I can update the interval to rate(5 hours) and update my input intervals from 1440 to 600 and I'm all set!

Final thoughts

There is more than one way to trigger tasks in sub-minute intervals. In fact, there is an entire chapter on the concept in the Serverless Architectures on AWS, Second Edition book by Peter Sbarski, Yan Cui, and Ajay Nair. I highly recommend that book in general, not just for the content on scheduling events.

Everything is a trade-off. You'll often find scenarios like this where cost and complexity become your trade-off. Figure out a balance to make things maintainable over time. Human time almost always costs more than the bill you receive at the end of the month. Time spent troubleshooting and debugging things that are overly complex results in lost opportunity costs, meaning you aren't innovating when you should be.

The solution I outlined above balances cost and "workarounds". I had to use a workaround to get the Step Functions Map state to behave like a for loop. I had to work around not being able to schedule tasks more frequently than 1 minute. I worked around scheduling tasks over different parts of the day. All of these things add up to make for an unintuitive design if you're looking at if for the first time. Somebody will look at all the components that went into it and say "this is just to trigger an event every 30 seconds? It's way too much!"

In the end, it worked out for me and my use case. I was able to save some money while getting the features I was looking for.

Do you have another way to do this? What have you found that works for you? Let me know on Twitter or LinkedIn, I'd love to hear about it!

Happy coding!

I didn't know it did that - Postman Postbot

Allen Helton — Wed, 03 Apr 2024 15:31:14 +0000

Last week I was updating the public workspace for Momento in Postman when I saw a little helmet icon in the corner of my screen. I hadn't seen that icon before, and being curious by nature, I stopped what I was doing and clicked on it. Much to my surprise, it opened up Postbot - the native AI helper inside of Postman.

I've been a Postman user for a long time and Postbot is relatively new, I always forget it exists. So I took this as an opportunity to procrastinate on my work task and do a bit of exploration. I wanted to see how good it actually was. I knew Postbot was supposed to help you build your tests and answer questions about your requests, but I didn't know how involved I could get or how much time it would save me in real life. I started off by asking it to visualize some data for me by turning a JSON list into a bar graph grouped by location. Then I followed up by asking it to create a JSON schema for the response and using it in the documentation and tests for validation.

It did surprisingly well on both tasks, so it was time to really put it through its paces. I gave Andres Moreno, my enabler friend who loves APIs as much as I do, and Sterling Chin, engineering manager over Postbot, a call and we hopped on a live stream in the Believe in Serverless community to figure out exactly how far we could push it.

Before I continue, I want to share this is **not* a paid endorsement. I'm just really excited about a new-ish feature in an app I know and use on a daily basis.*

What Postbot does well

In the live stream, we went through several key areas of Postman, putting Postbot to the test to see just how much time it can save developers and how much work you can offload to it.

Test writing

I can't tell you how much time I've spent in Postman over the years writing tests. Not necessarily difficult or tricky ones, but simple validation tests or tests that set chaining variables to flow into the next request. It's a lot of typing the same thing over and over again with minor changes in each request. With Postbot, that all changes.

I found great success typing phrases like "create a test that validates there is at least one record with an id of X" or "verify the response does not have any records where the person is older than Y." Postbot would look at the response of the request I'm on, get context of the schema, then write JavaScript and tests that do exactly what I asked. If there was a problem with the test or I needed to make an update to the criteria, conversation history is preserved so I could follow up with "fix that test to verify no people are older than Z." The test is updated in place without needing to touch anything 🔥

It also has autocomplete based on the name of your test! Similar to inline coding assistants you see today like Copilot and Code Whisperer, all you need to do is intuitively name your tests and the code for it will show up automatically. Hit Tab to accept the suggestion and you're done!

Schema creation and validation

While I'm a firm believer in API-first development and fully building your API spec before anything else, I do realize many people prefer a code-first approach and might not have a strongly defined specification at all. Turns out that might not be a problem anymore.

You can tell Postbot to "create a JSON schema for the response and add it to the documentation with meaningful definitions for each property" followed by "now add a test that validates the response against this schema" to fully document and validate your payloads. I was genuinely impressed by the quality of the produced schema and how quickly I could get a starting point for meaningful documentation. If you have flexible objects in your schemas defined by a oneOf, anyOf, or allOf, you might not get the best results right now, but I assume that will improve over time as LLMs continue to get better.

The main takeaway here is to use it as a starting point and make sure you double-check the schemas for minor details.

Visualizations

I always forget the visualizer exists. When I do remember it's a thing, I always struggle to get it working properly. It's super flexible in what it can do, but when something is flexible that usually means it's complex, too.

If you want to take your JSON payloads and turn them into meaningful graphs and charts, Postbot can do that for you in seconds. Not only can you ask it for exactly what you want, but you also have a wizard that will walk you through options if you don't know what you're looking for. Bar charts, line graphs, and pie charts are some of the options you can generate with a single command.

I did find it difficult to update the visualizations in chained commands. So if you ask it to "create a bar chart of the results" followed by "now group it by city/state" then "update the colors to various pastels", it struggles a bit. Your best bet is to figure out exactly what you want and ask it in a single, detailed command.

Documentation

Ah, documentation. The thing that developers hate writing yet are so angry if it doesn't exist. Meaningful documentation goes a long way with APIs and it's much more involved than simple schema definition. Explaining context and building story around use cases turn flat, code-generated documentation into engaging, effective tools for developers. This can be hard to do for many of us! While developers tend to be creative, it's not often the storytelling type of creativity.

Since Postbot uses generative AI to come up with responses, you can seed it with anything and have it come up with a consistent story across all of your requests. To me, this is probably the most significant yet underused capability of Postbot. Using it to elevate your documentation is a game-changer, and I expect it will open up a swath of possibilities down the road.

Fixing things

I'll be honest, when I saw "fix tests" as a suggestion in Postbot I glanced right over it. Seeing something as generic as "fix it" doesn't typically fill me with optimism. I see something like that and I usually look right past it. That's too vague to do anything meaningful.

But it works!

Per Sterling's suggestion, I intentionally wrote broken code and ran a request. After it broke, I told Postbot "fix the tests". It was able to analyze my code, find my error (I changed a variable from responseData to responseData2), and update it to use the correct variable name. So it worked for syntax/code errors!

I also tried another test that's all too real. I wrote a status code test to verify the response was a 201 Created. My endpoint actually returns a 200 upon success instead of a 201. When my test failed, I asked Postbot to fix it and it updated my test to check for a 200 OK instead of a 201. I've had so many copy/paste test failures like this over the years, it's really cool to see something that can find and fix them for me automatically.

What it does not do

One of the fun things I did on the live stream was simply see where my assumptions were wrong. I was tempted many times to ask Sterling "what would happen if I..." but decided to just try it myself. I learned real quick I had several assumptions that were flat out wrong.

Work with API definitions

Postbot doesn't do anything with API definitions. I tried asking it things like "Does this specification meet OAS best practices" and "Update the endpoint descriptions to be more meaningful" to no avail. It doesn't understand the spec as context. Seems like it might be limited to requests only for now.

Create or work with other resources

You can't ask Postbot to create a new collection for you. Or a request. It's intentionally not a creation tool. It's intended to be used as a pair programmer to work on a single request at a time. With that in mind, you also can't ask it to update a request you don't currently have focused on screen. The conversation history in Postbot is scoped to a specific request, so keep your conversations focused on the tab you have in front of you.

Undo

While I was trying to update my visualization to render bar graphs in various pastel colors for Easter, something went wrong and put my request in a bad state. My tests were full of syntax errors and the visualization completely broke. I had definitely stumbled across a bug, which is fine - everything has bugs. But the part that irked me was that there is no undo button. I pressed ctrl + Z (or command + Z for you Mac folk) to get back into a working state, but nothing happened. This led me to realize that everything Postbot does is destructive. Kinda scary phrased that way, but technically it's true. So be careful when prompting, because it's possible to lose your work.

Answer random questions

Postbot is designed to help you build things inside of Postman. It knows how to write and fix tests, create documentation, and build rich visualizations. It's not meant for answering things like "What is the weather today?" It will be the first to tell you that! If you ask it a random question, you'll get back a "I can only help you with Postman and API related queries" response. But you can ask it questions around APIs, like "What status code should I return on a POST" or "How should I implement idempotency on a PUT" you'll get meaningful answers and links to documentation or blog posts on Postman's website.

I had incorrectly assumed that Postbot only worked when asking it to change something on a request. Knowing it can also be a great reference for API-related questions is a huge win!

What I learned

Postbot is a big deal. If you haven't used it before, it's easy to shrug off as Postman's answer to the AI boom. Now having used it for real, I realize that this can fundamentally change how I use Postman. It not only has me moving faster when it comes to API creation, but it has me creating more complete, thorough tests and documentation.

Some "insider information" I discovered on the stream was that Postbot is powered by OpenAI. I wasn't able to get more info from Sterling on this, but I'm always willing to speculate. It appears to me that Postbot has a layer of Postman-specific context that it gathers and sends to OpenAI when someone types a message. The response from OpenAI is then processed by this layer and either turned into an action, like updating tests or other request information, or into a simple answer to your question. Based on the quality of the generated code and Postman-specific influence, I'd guess that they are using LangChain with GPT-3.5 and a Postman Knowledge Base for a RAG query. But again, it's just speculation.

I get periodic errors with extensive use of Postbot. Randomly I'll get a response back that tells me there was an unexpected error and that I should try again. When this happens, I send the same message again and usually get a working response. Remember, there's a ton of stuff going on when you hit that Enter button. Errors happen - especially in new tech. Try it again if you get an error. If you get an error again, try once more.

Postman seems to be investing a lot of time and energy into Postbot and I can see why. This is quickly becoming an enabler for both technical and non-technical users. I expect to see it doing things like fetching requests from the public API network and building workflows in the not too distant future. It's an exciting time to be a developer and an API builder.

Personally, I'm sold. While it's not perfect, it certainly beats doing everything yourself. So go give it a shot and tell me how you use it. I'm always looking for new, creative ways to build.

Happy coding!

Be an enabler

Allen Helton — Wed, 27 Mar 2024 13:08:24 +0000

I'm an enabler.

Let me explain. Some of you might see that word and think of it negatively. While it's true that being an enabler often refers to encouraging others to do bad or self-destructive things, like lending money to someone you know who has a gambling addiction, it doesn't always mean that.

You can enable people to empower themselves, too. Encourage them to take the risk they're too afraid to commit to or provide them with an opportunity they couldn't normally get on their own. Get them out of their comfort zone so they have a chance to learn and grow. This is the type of enabler I am.

Where it started

I had a really hard time mentally when I turned 30. On top of being a milestone birthday people generally associate with "starting to get old," we were two months into the COVID lockdown, meaning I wasn't able to get much reassurance from friends or family that life wasn't rapidly passing me by. Instead, my wife and I ate pizza in the yard watching cars drive by. I had a lot of time to think about what I was doing with my life and decide if I was happy with the diversity I had and the opportunities I'd fallen into.

To be honest, the answer was no.

Don't get me wrong, I've lived an absolutely blessed life. I've had the opportunity to do what I want, when I want, for most of my life. I've traveled all over the globe for both fun and work. I've competed on world stages for running, triathlons, and obstacle course racing. I've had content I created go viral. As far as the highlight reel goes, my life is awesome.

But it was safe. At 30, I had been at the same job for 9 years, climbing the corporate ladder in a large enterprise in the software engineering department. There was no risk. It was cushy. I rarely, if ever, woke up with a rush of adrenaline wondering what the day was going to bring and how I was going to be pushed out of my comfort zone. At the time, cushy felt nice. But 30 did something to me. I needed something else. Something more. But I didn't know what.

Where it needed to go

What I needed was a mentor. Someone to tell me, "What are you still doing here" or "Why don't you go try X, Y, and Z?"

At least, so I thought. Throughout my career, I've had two amazing mentors - Mike and Mark. I attribute a lot of who I am today to these two. They've steered me in the right direction time and time again and provided me with advice that I hold near to my heart. Mike and Mark… thank you.

Buuuuuuut that wasn't always what I needed. At a certain point, I felt like I hit a glass ceiling. I could see where I needed to go but there was no way for me to get there. The advice I was getting wasn't bad, but it wasn't helping me grow in the way I wanted to grow anymore. I was ready to take a risk.

Historically I've been a risk-averse guy. Everything I chose to do was a well-thought-out calculated risk. Once something passed a certain threshold, my internal voice would say "Don't do it. What if you fail? What if it leaves you without a job? You have a family that depends on you having a job." And I would listen to that voice. It made some good points.

What I actually needed was an enabler. Somebody to simply tell me "Yes. Do it, I'll go with you." I needed to be pushed past my doubts and into the unknown with someone I trusted who would go along with me, regardless of the risk. To me, this differs wildly from a mentor. A mentor gives you fantastic advice from their past experiences and helps you from a comfortable position. It's a tremendous way to grow and learn. An enabler is someone who understands there's risk or that what you need to do is going to be uncomfortable but says "Hell yeah, let's do it!"

They aren't just "yes men" though. Not every risk is worth taking. But they have your best interest at heart and are willing to do everything they can to back you up and get you there.

Be comfortable being uncomfortable

Andres Moreno was my first enabler. A close friend I've had for over 10 years and trust beyond measure. If you've met me at re:Invent before, chances are you've met Andres too. We're always together... for good reason! I can't tell you how many times he's told me "You should go talk to that person" or even something as simple as "Why not?" We always walk around enabling each other to be uncomfortable. Ultimately it was his nudges that gave me the courage to leave my cushy job for something completely different.

I was a full-time enterprise architect who did content creation as a hobby project. I had been named an AWS Hero as part of my fun content creation, but it scared me to death considering doing it as my job. In my head, moving to developer advocacy was a career change. Not only was I going to be not building production code or planning long-term technical strategies anymore, but I also wanted to go fast. I wanted startup. When I told Andres about Momento, he said "I will be sad to see you go, but you need to do it."

So I did.

And I love it. I took a tremendous leap of faith and couldn't be happier. I've learned so much in the past year and have built so many strong relationships that have been genuinely life-changing. Now I get the opportunity to be an enabler for hundreds of people in tech - AS PART OF MY JOB!!

To think, all this because I had someone help me recognize and realize I wasn't doing what I really wanted. And they gave me the courage to go out and do it.

Pay it forward

I am always willing to be an enabler. I love helping people get out of their comfort zones and encouraging them to live the life they want to live. This is about more than just your day job. It's about finding out who you are and what makes you happy. It's about learning and growing in completely new ways. It's about you.

It's hard to take risks by yourself. Self-doubt always rears its ugly head and dashes your confidence, leaving you questioning if something is the right move or not. There's power in numbers. Get an enabler to ride along or pave the way to an opportunity you couldn't reach yourself. You're helping them as much as they're helping you. Trust me.

Over the past year, I've surrounded myself with as many enablers as possible, and what a ride it has been. Find the people who encourage you to do hard things, say "why not", and are willing to ride along with you. You'll find the journey is just as fulfilling as the destination.

That's the end of my story. If you walk away from this with only one thing, I want you to take an inward look at yourself and figure out if what you're doing both in and out of work is making you happy. Are you getting the thrills you need? Do you even want thrills? Maybe long-term job security is what makes you the happiest. Find who you really want to be and find the people who are willing to go with you to get there.

Thank you to all my enablers, you know who you are. You really have made me a happier person.

Happy coding!