Forem: Coco

The Paradox of Simplicity and the Rise of Invisible Tech

Coco — Sun, 23 Mar 2025 03:17:15 +0000

Modern technology is supposed to make our lives easier, but sometimes it feels like it is actually making things inaccessible to those who need it most. Sometimes it hides the simple pleasures of life and distracts us from them. Modern tools demand a level of technical fluency that only engineers and enthusiasts will comfortably navigate, leaving the rest of the world to either struggle or opt out entirely.

But true innovation isn’t about adding complexity — it’s about making complexity invisible.

Take ChatGPT as a positive example. On the surface, it’s just a text box. You type words in, and it responds. But behind that simple interface is an absurd amount of machinery—vast networks of servers, AI models layered on top of each other, content filtering, and more. It’s hyper-complex, yet completely accessible. This is part of why it has taken the world by storm. Anyone, regardless of their technical background, can verbally ask for a pie recipe and instantly get one, without needing to understand a single thing about LLMs or cloud computing or why a nuclear power plant needs to spin up to power OpenAI’s energy needs (see footnote 1).

This is invisible tech—where complexity exists, but it never burdens the user. And this is where technology should be heading.

Serving the Revolution to Low-Tech Users

Right now, technology is designed by engineers, for technophiles. Even when companies claim to focus on “user experience,” they’re usually just smoothing out the edges of already complex systems rather than rethinking how those systems could serve people with minimal technical literacy. There is now a ton of opportunity to serve those people.

Take my grandma, for instance. She doesn’t care about AWS Lambda, API Gateways, or microservices. She doesn’t need to know what Spotify is. What she cares about is getting the right information or response when she needs it. She would love to just ask her phone to deliver her a pizza and pay for it. She does not want to know about Domino’s new iOS pizza app, because that would require knowing what an app is.
In a way, I have to learn about tech and serverless solutions to enable her not to have to. I don’t want her to have to! Yet most tech solutions assume a level of digital fluency she—and millions of others—don’t have.

If technology truly served people rather than just serving engineers’ impulse to tinker, we would see low-tech, low-barrier, high-impact interfaces—tools that feel as effortless as talking to another person. The best interfaces would be:

High fidelity – No clutter, no unnecessary features, just the essential functions
Low complexity – No complex setup, no steep learning curves
Low knowledge barrier – No manuals, no prerequisite skills needed
High utility – Actually useful in real-world, everyday situations

The Role of AWS and Serverless in Invisible Tech

The beauty of serverless computing is that it allows us to build high-powered applications with lightweight, accessible interfaces. AWS already provides many of the building blocks to create seamless, invisible tech experiences:

Amazon Lex & Polly – AI-driven voice interaction that can make services available through simple phone calls
AWS Lambda – Event-driven, scalable backend logic without infrastructure concerns
Amazon Connect – Cloud-based call centers that could act as automated helpers for people who struggle with traditional apps
API Gateway & Step Functions – A way to chain together complex logic without forcing the user to navigate it

With these tools, we could build systems that let people interact with powerful tech using only voice commands, phone calls, or simple text input—no apps, no accounts, no friction.

Invisible Tech Improvements for Underserved Communities

Drawing from my recent experiences training as an EMT, I’ve encountered various individuals—particularly older adults, people battling substance use disorders, those managing pain, and those suffering from chronic conditions. Many of these individuals have minimal familiarity with complex technologies, yet their need for accessible, intuitive tech is deep.

Imagine how invisible tech could make an impact in underserved communities:

1. Accessibility-First Interfaces for Low-Sight Users

In my local community, I’ve come across multiple individuals with sight accessibility issues. While I, as an engineer, can navigate complex user interfaces with multiple buttons, sliders, and options, these individuals struggle significantly with these designs. The tech world has made strides with voice-controlled tools like Amazon Alexa and Google Assistant, but there is room for improvement.

Empowering tool: Imagine a purpose-built interface specifically designed for individuals with low or no sight. This could involve voice-activated and haptic feedback systems that guide the user through an app’s functions without requiring any visual interaction. By leveraging AWS Lex for voice recognition, we could offer an interface where individuals can escalate issues, get notifications, and access essential services through simple voice commands or touch-based gestures.

2. Algorithmic Content Customization for Accessibility

Applications like TikTok, Spotify, and others use an algorithmic approach to generate content tailored to an individual’s interests. While this approach works well for people with full accessibility, it also holds potential for those with sight impairments. The algorithm doesn’t rely on the user interacting with a traditional interface, but rather on their interests and preferences over time, making it easier for users to engage without having to see every option.

Empowerment tool: For individuals with sight limitations, an algorithm-based content stream could be designed to serve them better. For example, an app that curates their own calendar and task list over time to generate streams of upcoming events and todo items combined to give them a verbal/audio map of the possible day ahead for based on their calendar, their task list, and also algorithmically adjusting to their preferences over time and individualized specifically to what is happening with them. I am considering solutions in the AWS space, and here AWS Personalize is an engineering and serverless way to turn thin user behavior provided data via a recommendation engine into individualized recommendations.

3. Emergency and Health Monitoring via Voice

In the context of healthcare, particularly in emergency medical situations, individual are often may be struggling with pain, confusion, or chronic conditions. We often get calls from life-alert necklaces in my area that patients never even knew they had triggered because the tech is silent about what it is doing. The need for tech that can respond to a voice command, especially when the person is in distress, becomes crucial.

Empowerment tool: A voice-activated system designed for individuals with health concerns could integrate with wearables and sensors to allow users to request help, check vitals, or even get emergency instructions without needing to touch anything. This could be paired with AWS Lambda and API Gateway to provide real-time feedback and monitoring through an easily understandable voice interface. Imagine an elderly person at home experiencing chest pain: they could simply ask, “What should I do if my chest hurts?” and the system would respond with step-by-step questions, as well as alerting emergency services.

4. Simple Communication Tools for Substance Recovery Patients

Many individuals in substance recovery face significant barriers when it comes to accessing complex healthcare technology. Even the traditional concept of “rehab“ requires a huge amount of effort and energy to go to a “rehab clinic“. What if we could bring rehab to the people who needed it, wherever they walked, wherever they drove, wherever they were? In situations where they may be struggling with their health or mental state, they need simple, intuitive tools to help them stay connected with their support systems.

Empowerment tool: A low-barrier communication platform, like a voice-activated chatbot or a very simplified text interface, could allow these individuals to easily check in with healthcare providers, support groups, or have scheduled check-ins with family members. This system could use AWS Polly for natural language processing and allow users to send messages or receive advice without dealing with complicated forms or settings. It could be as simple as saying, “I need to talk to my sponsor,” or even just “I need to talk to SOMEONE” and the system would connect them to the right person for that time and place.

Fragments of these tools are starting to exist in the world, but they remain locked behind some very substantial technical and societal barriers. Often one of the most substantial barriers is that they are being built by technophile engineers, for digitally fluent users. The challenge may not just be in building the technology, but in making the technology disappear.

Building for Outcomes, Not Just Implementation

As engineers, it’s easy to get obsessed with the underlying machinery—optimizing APIs, tweaking performance, debating monoliths vs. microservices. But none of that matters if the end result isn’t usable by real people.

The real question isn’t how something is built—it’s who it’s built for. The best technology serves people without them ever needing to think about it.

Invisible tech isn’t just a design goal. It’s a necessary shift if we want technology to truly serve humanity—not just the engineers building it and the younger generations being immersed in it.

Footnotes

Footnote 1: Yes, I understand there’s a danger to hiding all that complexity. When we hide the complexity of a thing we get perverse outcomes. Even the example of someone using complex neural network technology and artificial intelligence just to get a pecan pie recipe could be considered a counter example, and a negative use of resources. I still personally do do that sometimes, I ask ChatGPT for a pie recipe that I could google…

There is also the subtle side effect of how when you use AI and an LLM to get a pecan pie recipe, it decreases the incentive for someone out there on the Internet to write that pecan pie recipe in the first place.

I leave these two problems as out of scope, for others far smarter than I to solve.

How to quickly move from Exim4 server-ful email to AWS SES serverless email

Coco — Thu, 20 Mar 2025 19:22:29 +0000

Let's say you have Exim or another email app on a server that you have to maintain (and maintenance of email servers isn't easy). You want to just hand off the work to AWS Simple Email Service.

How many moving pieces do you need?

Well, I love the low complexity of event driven architecture. And in this case, since email will take a while to get to a person's inbox anyway, might as well make an event driven email.

First you'll want to configure SES and start getting it out of sandbox mode (see details here):
setup for SES

Ok, so assuming SES is on it's way, and you've added at least your own email to the whitelist on the sandbox, now we can get into the fun stuff:

Create a lambda with just a test event
Create an EventBridge rule to trigger the lambda on certain events of detail-type
Update the lambda contents to send out the email
Create the event that will trigger the email sendout... ...from whatever codebase or language you want

Ok, more specifics:

Lambda Hello World

Create a hello world lambda, any will do for now, just log the event and leave the SES stuff for later.

EventBridge Setup

Create an EventBus for your application:

Create a rule against that new bus:
Sample Event:

  {
    "version": "0",
    "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
    "detail-type": "CommandNWEmailRequest",
    "source": "aws.apigateway",
    "account": "123456789012",
    "time": "2017-12-22T18:43:48Z",
    "region": "us-west-1",
    "resources": ["arn:aws:apigateway:us-west-2::/domainnames/my.domain.com"],
    "detail": {
      "emailParams": {
        "to": "some-test-email@gmail.com",
        "from": "some-source-email@example.com",
        "subject": "Test event fired via lambda email sendout nwEmailSendout function",
        "text": "Raw text: of the email body that is sent in: Test event fired via lambda email sendout nwEmailSendout function",
        "html": "<h1>Hi You</h1>          <div>           <p>           Raw text: of the email body that is sent in: Test event fired via lambda email sendout nwEmailSendout function </p></div>"
      }
    }
  }

Create a matching pattern in the rule, I recommend on the detail-type for application events:

{
  "detail-type": ["CommandNWEmailRequest"]
}

Simple is better, so just matching on detail-type is usually quite good enough.

Run the [Test Pattern] button to make sure it matches what you want:

Finally just pick the EventBus for your application that you created earlier.

Back to the lambda, let's hook up the event rule to the lambda so that the lambda runs when the rule is triggered:

Here is the lambda with a trigger from the eventbridge:

Send a test event to your newly created bus:
https://us-east-1.console.aws.amazon.com/events/home?region=us-east-1#/eventbuses/sendevents?eventBus=NWEventBus

And see if your hello-world lambda logged anything.

Once you have got your lambda being triggered, you're going to have to give the lambda access to the aws-sdk. Use layers for this from the console (or cdk locally, which is out of scope of this).


aws lambda publish-layer-version --layer-name aws-sdk --zip-file fileb://aws-sdk.zip --compatible-runtimes nodejs18.x --description "NW AWS SDK" --profile=YourProfileToUploadHere

Ok, hopefully you have the layer available, you'll need to attach it to the aws function:

Now you should be able to just easily import aws-sdk in modern js module files .mjs

Here is a lambda for SES sending:


// Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0

import aws from 'aws-sdk'
const ses = new aws.SES({ region: "us-east-1" });

const validate = (params)=>{
    if(!params || params?.length === 0){
        throw new Error('No params passed to lambda function to send the email.')
    }
    const shouldExist = {
        to: true,
        from: true,
        subject: true,
        html: true,
        text: true,
    };
    Object.keys(shouldExist).map((key)=>{
        if(typeof params?.[key] !== 'string' || params?.[key] === ''){
            throw new Error('Cannot send out email with missing or blank parameters')
        }
    });

}

export const handler = async(event) => {
    console.info('Initial event', event);
    const incoming = event?.detail?.emailParams;
    validate(incoming)
    const {
        to,
        from,
        subject,
        html,
        text
    } = incoming;

  var params = {
    Destination: {
      ToAddresses: [to],
    },
    Message: {
      Body: {
        Html: {
              Charset: 'utf-8',
              Data: html,
        },
        Text: { Data: text},
      },

      Subject: { Data: subject },
    },
    Source: from,
  };

  return ses.sendEmail(params).promise()
};

And here is a test event to fire against the lambda:


{
  "version": "0",
  "id": "6a7e8feb-b491-4cf7-a9f1-bf3703467718",
  "detail-type": "CommandNWEmailRequest",
  "source": "aws.apigateway",
  "account": "123456789012",
  "time": "2017-12-22T18:43:48Z",
  "region": "us-west-1",
  "resources": [
    "arn:aws:apigateway:us-west-2::/domainnames/my.domain.com"
  ],
  "detail": {
    "emailParams": {
      "to": "roy.ronalds@gmail.com",
      "from": "tchalvak@ninjawars.net",
      "subject": "Test event fired via lambda email sendout nwEmailSendout function",
      "text": "Raw text: of the email body that is sent in: Test event fired via lambda email sendout nwEmailSendout function",
      "html": "<h1>Hi Roy</h1>          <div>           <p>           Raw text: of the email body that is sent in: Test event fired via lambda email sendout nwEmailSendout function </p></div>"
    }
  }
}

You will also need to allow the lambda to run calls against SES, follow the instructions here for that:
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-send-email-ses/

In broad strokes, you'll need to add a policy like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ses:SendEmail",
                "ses:SendRawEmail"
            ],
            "Resource": "*"
        }
    ]
}

And attach it to a role, at least to the role that the lambda is using to run:

Email Sent And Received

A Powerful System

You will have a full fledged system on serverless AWS with a ton of different powers coming from SES, including mail bounce measuring, trust measures of whether your email is being flagged as spam too often, and at many email volumes, low to no cost, all driven by event-based triggers.

The Subtle Case For and Against React

Coco — Fri, 19 Apr 2024 07:41:07 +0000

What are the hidden downsides of React?
What if React makes everything much harder and more complex?
Why do we bother to use it at all, even if we know there are these major disadvantages?

A friend in a Slack group that I'm part of recently posted an evaluation of React as a staple of many software teams today. Essentially they said:

"Why is React everywhere, when..."

It is extremely complex. It adds complexity on top of javascript, which is already a nightmare of learning cliffs. React now recommends using Next.js so you truly have to become an expert in html->js->react->nextjs to work in the space.
It is huge and probably slow compared to pure js and HTML, so why would we inflict that on ourselves?
It makes development slow because the thinking and planning require so much hard work to build things with it, so shouldn't we avoid it?
jQuery was the way to go before SPA frameworks, and it worked well, why trade down to something that's both slower and less stable?
Are people trying to get work done in React because it's standardized and common, not because it's effective?

Essentially we can summarize the negative experience with this ironic meme image:

(The image roughly sums up that friend's concerns, my position on the matter is different )

Fair Accusations give pause

This all gave me pause. These are not small concerns, and I agree with all of them and have the same concerns on the face of them. So how could I still like or recommend React? In fact I am even in the process of refactoring another old personal project away from jQuery and towards React. Why am I trying to do this if I can't articulate counterarguments to the above very apt points?

The Subtle Case for React

So let's run through the points and see if I still can recommend React with a straight face at all, despite generally agreeing with the underlying concerns 1 - 5 against it.

1 - React complexity weighed as if it was ballast

Lower complexity is usually a good argument in software. "That pull request will highly increase complexity." is a valid rationale to take another look and think through the cost. But complexity in the case of React is also a double-edged sword. We can look at React, nextjs and say "That is too complex to learn and maintain and work with" because we are actually silently weighing it against the strawman of how easy we imagine it would be to do on our own, solo, from scratch.

I have spent hours tweaking a single button. Not a calendar widget (god forbid) or a cool custom widget, just the button that comes in html, which by default has this beautiful look:

And here are the variations that you'll have to encounter of how a button is rendered in a few different browsers (just a button, one of the simpler building blocks): (Chrome, Edge, Old IE 11, Safari)

Over the course of my career I have found there to be similar or worse variations in <button>, <select>, <ul>, <dl>, <abbr>, <small>, <iframe>, <input> types, of course with <audio> and <video> not to mention functional differences (and sometimes things just don't work as needed in different browsers without polyfills, like how <abbr> need a polyfill on mobile browsers for definitions to be readable).

The foundations of the web are pretty shaky to create app-based UI on top of. Even with html5 we get the hint that it's really geared towards newspaper <article>s and blog posts with <asides>, it was not made for building web apps. Especially to fulfill invisible and nonfunctional requirements like a11y and consistency, we need all the help we can get. We can get that help by standing on the shoulders of giants; UI component libraries can crowdsource issues that we and others would encounter, and having composition via components helps avoid reinventing the wheel during the creation of each new widget of an app.

2 - HTML+JS vs React: Fight!

Html is a breeze. Javascript is fun. CSS is... ...well, I can barely think of a worse way to make UI, but with tailwind it's simple enough. React on the other hand is a pain. You build an application and it's thousands of lines of code and relies on hundreds or thousands of underlying npm packages as well. In the end, you may have an application that is prone to errors (components within components) and browser errors (rendering complexities in different browsers are real). So then here is my question:

If someone hired you to create a webpage (no framework involved, minimal js) today, would you take the job? Can you even find feasibly find work at your pay scale where companies are looking to hire people to create webpages?

For my personal answer to the above, if someone offered me my current salary to only work on web pages... simpler and easier work than what I currently do, I would... turn them down. That is not due to ego, it is because I have been down that road already. As the meme image hinted, you can work on web stuff, visual design, web pages, companies can use that to keep you out of an engineering space by pushing you into an infinite treadmill of pixel perfection and throwaway marketing sites. Being told that your passion for outcomes that require time to think through and create deep quality, because it would take too long, that sucks. The space that we call "engineering", complete with companies that want engineering done, while a ton harder to work within than the raw web page space, is also a very creatively rewarding space to be within. That is the space that React lives in, and the space that pure js apps can often exit from. The problematic expectations in that space are often for speed over engineering and "you're just creating a webpage, that should only take a day, right?".

[ lucille bluth how much is a banana file footage not found ]

I'll go a step further and say that because engineering is usually an entrance fee to React, you will often find yourself working with a team that share that engineering mindset. That can sometimes lend itself to overengineering the whole team you'd work with in a React space will often be in an engineering mindset as well.

As far as high paying work at companies, I haven't seen people looking for just webpage edits in a long time. That space has been eaten up by WordPress, squarespace, themeforest, free themes, 5 html templates for 0.000004 bitcoins, etc etc. The game is over there. The space is saturated. If you think it's difficult to compete with the tons of other people who also write React better than you and I, try competing with some of the people who make webpages out there. It's insane what beautiful stuff they can visually create and how fast flat html pages can be created now.

So React filters over from page creation space into the engineering space for people who don’t want to deal with React’s bull****, and that's good for me, because I don't want to compete with what's out there in the non-engineering space. There there be dragons.

3 - It makes things take longer to plan... ..isn't that bad?

I would say that it takes longer to plan and execute building things with React than without React. That's irony. That seems like a bad outcome for the metric of speed? Imagine the headlines:

However, reality takes so much more work and expertise to generate high quality than it seems. We think we can and should take shortcuts, but we're better off letting go of that expectation of an easy win early on and settling in.

It takes mastery to cut a hedge shrubbery correctly (I learned this to my dismay after dropping the hedge clippings into the hedge when they were green and having to manually painstakingly pull them out weeks later when the clippings turned brown). It takes months to grow a carrot in a garden, yet it's one of life's great pleasures to grow plants. Why do we want software to be done as fast as humanly possible? Sometimes that speed is to the detriment of long term quality. Good companies and good teams can absorb the stark reality that good stuff takes a long time and a lot of iterations. Are you sure you want to work for the companies that can't accept that things might take time to get done to a high level of quality?

4 - jQuery and JS vs React, why bother if the first one will get the job done?

I worked with jQuery back in the day, and I work with it on a legacy project today. jQuery still has a place in my heart, only because I have a deep-rooted mistrust of browser compatibility due to being burned so many times in the past. The main things that I needed jQuery for have been replaced by document.querySelectorAll() and a domready wrapper. So we have the best bits of jQuery in browsers today. However jQuery and still not powerful enough. Building the basic foundations is so difficult that people routinely think that they can rebuild Facebook over a weekend. We think we can build a simple carousel component on our own (but we forget about the invisible accessibility non-functional requirements (http://web-accessibility.carnegiemuseums.org/code/carousels/) ).
We need... dun dun dun, something as powerful as a framework to harness the full potential of what we could build for web UIs, we have to rely on other’s works as a baseline to build better outcomes.
So the formula becomes:
jQuery + js = fine for solo one-offs. React + ui framework = help long term consistency.

5 - Are people hiring for React because they want cogs in a machine?

A lot of companies hire with React as a skill keyword, and a lot of those teams at the companies are spending a lot of time on React projects, but is React just a lowest common denominator, a keyword matching mechanism? A person in the thread made the astute observation that if a role lists all the frameworks, such as saying "This role must know at least one of (React/Angular/Vue)" then the writers may effectively mean "must know javascript" because those frameworks are not the same at all anyway.
In a way that's close, but React/Angular/Vue are not just a technology, they are also a clique or community; a way of thinking about engineering on the web that didn't exist in the 90s. It also involves engineering for problems that didn't exist in the 90s. The web interface for the spotify player and other web apps with subtle tweaks and requirements cannot be aggregated from a collection of jQuery plugins. So React is not for solo work, it’s for team work, and the clique and the community methodologies are more than the technology.

Pure js = fine for solo development. React = assists with team consistency.
If you like to spend a weekend writing an amazing 100k pure javascript genius solution, but two months from now no-one else on your team will get to understand what the flip you built, React helps push back against that with the huge amount of community documentation and best practices that we might not be able to find in the work of a virtuoso solo javascript developer.

So for now I'm still moving that project to React, but let me know of any great alternatives I should check out

So for now I am planning to move my legacy project from jQuery+js to React. I yearn for the day when js is robust enough and consistent enough to get done what I want to get done (creating a lexicon of UI verbs to compose, and app widgets that are customized to my app, mainly). There may be other alternatives out there with good component systems, and I'm happy to hear of them, but the conceptual component is one crucial piece, and being able to build on the shoulders of (ui) giants who are better than me is the other crucial piece. I don't want to spend another day building a good button again. How about you? Where are you sourcing your grain-fed free-range buttons from these days?

Honorable Mentions

Alternatives to React to consider:

https://swup.js.org/ single-page-app but with minimal framework, still along for the feel of an SPA
HEC no-framework approach: https://medium.com/@metapgmr/hex-a-no-framework-approach-to-building-modern-web-apps-e43f74190b9c
Astro to use every framework at once instead of just react? https://astro.build/
What about a little boilerplate repo that I am working on myself as a result of this article, (WIP) called lazier: https://github.com/tchalvak/lazier/tree/main

The bane of AWS Lambda Coldstarts

Coco — Sun, 17 Mar 2024 17:55:21 +0000

Coldstarts of AWS lambdas are the bane of my existence, are they for you? I often find that the coldstart delay degrades the user experience, and while it seems like they should be very avoidable because the coldstarts should not be exposed to end users, they end up being a little difficult to see the way around.

When The User Encounters A Coldstart

Here is the scenario of interaction with coldstarts that you may encounter. You do an api call for some data, it involves a lambda, and it's fast. Like blazing fast, 12ms, 15ms, great response time. But every so often, the api call hangs... ...and hangs, and hangs, for seconds, not ms. And because it only happens infrequently, it's really hard to debug, but when it does happen it's a pretty bad user experience.
That is it. You've encountered the coldstart problem. Your lambda interaction that is usually blazing fast is sometimes really slow BUT ONLY INFREQUENTLY. The problem is that it's so slow that infrequently that it really degrades the user experience.

When you get a hot lambda instance it's like a good cup of fresh coffee, only blazing fast, taking just a few miliseconds to perk you right up.

When it's cold, it's like yesterday's coffee, you take a sip and grimace and the caffeine is going to take a while to kick in

So what can we do?

A Few Avoidance Techniques

I am going to give a few hacks that have worked for me first, because the real answer overall is going to be annoying; it requires a mindset change in how we deal with data (precomputation, which I will get to later). So here are a few good hacks to tide us both over.

Don't put synchronous lambdas at the end of your api gateways (in other words if you expect to wait on a response or to have to manipulate the response data, make it an async lambda)
- when writing input or "commands" through an api endpoint, try to immediately emit an event to eventBridge and return {'success':'OK'} respond, instead of doing everything all at once (this is generally a CQRS approach)
- when reading or querying, try to: read directly from database without a lambda, or turn on caching on apigateway, or precompute the resultset seperately
Warming: If you have an event driven architecture, you can set up an event scheduler that fires an event to warm up a lambda with whatever schedule you want, to wake it up at appropriate times (e.g. 7 am and 8 am in the morning is a good one before people come online for the day to warm it up for others) or use someone else's warmer library
Increase the memory of a lambda function (because in the background that causes AWS to increase the CPU class of a function), e.g. double the provisioned memory will probably increase the CPU class that AWS will give it and because you're charged by resources*time, faster invocations with more memory may not actually cost more in the end (your mileage may vary)
A Temporary Stopgap: Provisioned Concurrency: It's an option to increase the provisioned concurrency to 1 so that there's always a "warm" instance, but I don't recommend it and only include that here for completeness. It really doesn't solve the problem anyway (when enough load occurs, a new instance will spin up and you'll still have to incurr the cost of a coldstart)

AWS's Recommendations for Coldstarts

One of the concepts that you will encounter when reading about coldstarts and how to deal with them is as follows:

AWS will recommend ways to make coldstarts less slow
AWS will recommend not putting a lambda directly at the end of a user-facting interaction, because the coldstart will kill the user experience

What doesn't happen often enough is a recommendation for what to actually do instead of sticking lambdas at the end, instead of suffering from coldstarts at all.

...a glimpse of a model to determine where slowness can come from via lumigo where the problem of coldstarts is generally happening in the orange band

Any recommendation to decrease the time of the lambda itself or the lambda allocation where coldstarts occur is only going to get you so far. It's useful when you have already written lambda code and just want it to act faster, but the deeper solution brings us to precomputation...

The Deeper, Harder Precomputation Solution

The really deep solution is to develop systems with cold starts in mind by relying on precomputation. In this model, instead of calculating and manipulating data on the fly inside a lambda when an api call requests it, put the data into dynamodb in a triggered manner (whether triggered by writes or by a time-to-live or on a schedule) using a precomputational lambda and then just always pull right from that precomputed data set in api calls. So let's say there is a latest_statistics dynamodb table, the api calls and apigateway would pull directly from that dynamodb fast and simplistically with no computation at all, and then separately out of band a lambda would update what data gets in latest_statistics.
Precomputation is a major change from approaching things from a "pull, manipulate, respond" way of developing, so we may need helpful techniques like the avoidance techniques above to help us along the way.

Techniques for Faster Hot Lambda Invocations

This isn't applicable to coldstarts, but here are few techniques for speedier hot lambda invocations to round it out for good measure.

Cache variables in the global scope of the instance, so that the hot lambda instance can have them in memory, things like api clients and unchanging dependencies
Cache a response if a request happens close enough in time, in other words putting a json/data response into memory along with a timestamp, and if little enough time has elapsed that you are comfortable, just return the in-memory data without having to reach out again to the database
Predictive, 2-stage approach: Return a quick and dirty response fast, and then set the slower results to be collected and displayed with once updated timestamp or a specific command id becomes available

Overall

In the 1,000 foot view from above, there are two parts to mitigating the subtle effects of coldstarts: The first is an approach change to ensure that users cannot touch lambdas that will wait during coldstarts. For example, if you have to get a user profile, instead of manipulating the data in custom ways in the lambda compute, have apigateway pull directly from dynamodb. And secondly, deferring results, so that the lambda that is front-facing and user facing can be fast, and the slow compute work can happen behind the scenes.

Additional resources

A resource for more coldstart avoidance techniques and countermeasures: https://www.simform.com/blog/lambda-cold-starts/

Serverless warmer library: https://github.com/juanjoDiaz/serverless-plugin-warmup

Coldstart Stats Daily By Language/Environment:
https://maxday.github.io/lambda-perf/

An option to watch is LLRT, Low Latency Runtime: https://github.com/awslabs/llrt which is in the early stages, but generally bundles node into a single bundle

If connecting to an RDS database, you can consider using an RDS proxy for connection pooling, though it is an additional expense: https://docs.aws.amazon.com/lambda/latest/dg/configuration-database.html

The 1000 papercuts of AWS Serverless

Coco — Tue, 12 Mar 2024 07:16:13 +0000

AWS serverless is great for scaling up to heavy usage and down to zero. It is truly a space to do engineering at a true sense of the word, putting together systems and composing pieces of a puzzle in a way that servers and monoliths can't match. It's how I make my living. It's why I get paid the big bucks. :p

However, the more I use serverless, the more I come to intimately know some parts of it that I wish were better. From trying to tamp down runaway costs, to trying to "see" what these very complex interlacing systems are doing internally, to just avoiding having to constantly context switch, there are obstacles that tend to pile up. Individually, each obstacle - obstable to doing stuff easily and simply - doesn't amount to much on it's own, but like "death by a thousand (paper) cuts", the cumulative effects tend to weigh down attempts at good engineering.

One of the first things that I'll talk about is surprises lying in wait in terms of cost, then I will move on to structural and complexity issues and end with context switching and finding your place again. But costs first, because as AWS's Dr Werner says, costs is a proxy for sustainability, so if you can't control it well, things can get unsustainable fast.

Cost Boundaries

Scaling up Spikes

In many server-based services, it was common to just spin up a server and have a relatively stable expectation of cost. Years ago, if I spun up a virtual server on Heroku or DigitalOcean or Rackspace or EC2, I wouldn't worry much about how much it was going to cost me generally, because the most it would cost was 1 server's worth, and was pretty stable.

The scaling and performance was also pretty static is the problem of course. Essentially every time I was creating a server, I was way over-provisioning the server in the hope that I would never run up against it's upper limits. Or if I did, I would have time to adjust. It's not great to overprovision, but at least the cost had natural maximums that were usually stable and predictable.

... A Year Of An EC2 Server ...

On the serverless cloud, since it is often on-demand by default, the cost can go up and out of control if I don't take the time to set up limits up front. AWS ships with a 1000 concurrent lambda invocation limit on the account, but if someone (coughs we won't say who) writes an infinite loop in a lambda or a poison pill in an SQS queue, the system could run 10 invocations x 1000 concurrent instances for 10,000 invocations per second. Speaking from experience, this can rack up thousands of dollars in a day.

... Roughly the same timeframe of a serverless service ...

This is by design; you only pay for what you use. However, the dark side of that is that a spike in usage can rack up a major spike in cost.

Missing low-granularity budget controls

The granularity of budgets is either really long, like a month, and thus I will hear about a problem way too late (Finding out on the 15th of the month that there's a major cost spike can be too late and costly), or it is really noisy and thus the signal that I want to hear about get lost in the noise ("Oh no, a cost overrun is currently happening!" gets quickly lost in "Another daily budget adjustment as it swings back and forth")

Here is a monthly cost bump that I got in RDS (the maroon), that because it is viewed at a monthly granularity, is difficult to track down.

When viewed with a daily granularity, it becomes more clear that this was a service that was added:

So the more granular you can make your cost monitoring, the easier it can be to debug.

Unfortunately, for -budgets- the granularity min-maxes out at a day at the lowest, so for alerting and alarming against cost overruns budgets are not particularly effective.

Cost Anomaly detection

Ideally, automatic anomaly detection would fill the gap of weird cost overruns that may happen in unexpected areas not cost anomaly detection to be a helpful assistant in detecting cost spikes. It just responds to late, and to too regular of anomalies:

At a granularity of a day, by the time 24 hours has gone by, I can have already racked up thousands of dollars before ever getting an email that something has gone wrong.

Here is an example of some cost anomalies that get caught, but where the anomalies are often so minor that they're like an anomaly of 0.7 cents?

Architectural Design Simplicity

Automating code deployments has been made easy, automating Infrastructure-as-Code deployments is still rough

As I have gone in depth into using Infrastructure-as-Code (hereafter IaC) for work and personal projects, I commonly want to automate deployment of my infrastructure alongside my code. Code in the same repository, infrastucture in the same repository, data fixtures in the same repository, and I can build the whole application all at once. Guides that talk about code deployment often have great direct approaches. On the other hand, guides that talk about IaaC deployments are often another matter.
There seems to be a substantial lack of visualization tools with IaaC to make it easier to handle the complexity that it brings with it. I have spent many hours at work designing architectural diagrams at work... ...of already existing infrastructure, in an attempt to understand it and capture it's purpose. Conversely, if I am diagraming something that I am architecting from scratch, it would make a lot of sense if there were a way to map from a visual diagram to a baseline architecture and modify both in concert.

There are possibilities around this space:

AWS CodeStar
AWS Amplify Studio
AWS Codecatalyst
etc

But these generally seem to revolve around using a standard blueprint and spinning up a pre-generated set of infrastructure, as opposed to creating infrastructure and then splitting apart the pieces that are provisioned, in order to retroactively refactor as time goes on.

I find a gap with cloudformation and AWS CDK where it can be very difficult to see what has already been created holistically. For example, if there were a legacy architectural stack that was created with little institutional architectural documentation, it is quite difficult to go see what all the resources the app/ecosystem of that legacy stack are.

So if feels like there is a missing toolset around IaaC that code has more matured enough to have good tools for. Unfortunately, because the power of Iac is so strong and far reaching, architects and engineers need the tools all the more on to keep complexity under control. Just like the thoughts that early global explorers probably had while navigating new coastlines and navigating by the stars; "I really wish I had a good map to see where I am NOW."

Visibility

While we're near the topic of visibility, let's talk about the somewhat unrealized promise of observability on AWS cloud. With a complex system, it becomes important to "see" what each of the pieces are doing when each operates independently of each-other. With micro-microservices, each piece could be the failure point that I am looking for when debugging, so I need to be able to "see" each. But observability is pretty difficult to set up on AWS when not working directly in the console, and after-the-fact monitoring is just no longer enough when trying to track down which peice of a 30 link service chain is the one that is broken.

Take an SQS<-to->Lambda<-to->Dynamo stack.

If I look in dynamo, I can see the stored data. If I look at SQS, I may see the messages in flight if something is wrong or slow, but more likely is I will see an empty queue and no evidence of whether it is empty because messages have already flowed through it. Or because messages have not yet flowed through it.

With the lambda I will see the traces after the fact of events, the invocation metrics, and the logging entries that I have to manually create, but I actually want to see the events that are continually flowing through the lambda.

So serverless requires connecting together these many-link chains of services, and it stops being effective to just monitor the input end of the chain and expect certain output at the other end of the chain. The number of chain links where it could be breaking down is too high, real time observability of the behavior of the data through the chain becomes necessary.

Unfortunately, observability seems to have a high overhead setup, so much so that it is usually relegated to the domains of large corporations, and adds a lot of overhead for individual engineers. For example, I was interested in getting real time tracing and observability with AWS X-Ray, but even after setting it up for a reduced test case stack, it ended up not actually showing any live data, and I gave up on it because the overhead, not just the learning curve, seems to be too high to be an efficient use of time in standard use cases.

Cloudwatch Log Ease of Access

Ok, so if observability may not be easily accessible, we have to fall back to monitoring (e.g. monitoring lambdas, since most custom code will be in the lambdas), and for that we need Cloudwatch.

Now Cloudwatch logs have two problems. out-of-time-order, and difficult findability.

The out-of-time-order problem is generally an interface problem. It is common to want to do X behavior in your app, and check for output Y in the cloudwatch logs. However, because looking at a stream doesn't get you to the next stream when a new lambda instance is created, you may never see your output Y when looking at a single log. Since every lambda instance creates it's own stream, and you can't predict when a new lambda instance will spin up, whether for concurrency or because of an old instance being killed off, you can't predict where your logged information will appear.

The way that logwatch is configured in the AWS console is... ...like a jigsaw puzzle that has been split apart. Each stream is it's own separate peice, and (ironically), they don't flow into each-other.
I can't tell you how many times I have been looking for a log output message to come through in log stream A... ...but it never will because a new stream B has been spun up in the meantime, and the log output is actually in stream B.

Log output findability is low

The findability / search problem is just that searching cloudwatch log streams is not a very user friendly experience. It's case sensitive by default, for some reason, and the documentation on operators, well, there almost isn't any, so you're often limited to mostly knowing a priori the exact string that you need to find.
For an egregious example, to trivially find all "Error", "ERROR" and "error" instances, none of these will get you all you need:

error (1/3rd of results)
ERROR (1/3rd of results)
error ERROR Error (0 results because of exact matching)

The actual search terms and operators you need, after quite a bit of research, end up being along the lines of:

?error ?ERROR ?Error

And good luck if your error is actually like ERR or something, because it's case sensitive.

Frankly, even after all the time that I have been using cloudwatch logs, and getting used to the problems above,
I still have no idea how other engineers are getting good information out of lambda invocation logs. I am still searching for some realtime log viewer & aggregator tool, so that I could just see some output pop up, in real time, whichever log stream it ends up in.

These papercuts of cloudwatch are small, but because cloudwatch and monitoring are so core to everything, and how frequently you'll be looking at logs for so many scenarios - invocations, alarms, dynamo errors, even billing - those usability issues are going to be hitting constantly, probably daily.

Navigation

Now, in a hypothetical monolith, if you have a problem, you generally know where to go to debug, you go to the application (and it's server, and the infrastructure built by other people that runs that app), but crucially, because it's a monolith, it's generally all in the same place. When running serverless, though, you have to collect together the services and context switch between them.

Again, with a monolith, it's all there, all in the same place. The place is complex and intermingled, but hopefully you at least know where it is, it's on that server or whatever.
With serverless, you have to navigate and metaphorically travel back and forth, between contexts and between services. It can be disruptive to process to hop between Lambda, SQS, Alarms in cloudwatch, and Dynamo. So navigation becomes a key activity.
Navigation should be a smooth helper for moving between these things. Navigation, through a good information architecture, should highlight the things that are likely to be most important to the user, and deprioritize those things that the user rarely uses.
Here is what my current navigation in aws looks like, though:

Or on the homepage there is one other jumping-off point:

Unfortunately, while services are great for engineering and putting together applications, they don't really work for Navigation, for finding things.
For example, I really don't want to navigate to the base case of AWS Lambda, I want MY lambda functions that I was most recently working on. I don't want the general category of DynamoBD, I just want to get back to the actual Chat table that I was most recently working on. Because I had to context switch out to a different service, and now I am trying to get back into flow.

The current navigation is very "flat". There are 100s of AWS services, and each is presented as if they were equally important to the current user. Of course they aren't. They can't be all equally important. Manually favoriting and bookmarking services is one way to communicate a bit of priority, but the "services" really are never what I'm looking for anyway. The services are like a whole herd of similar zebra, and I am searching for that one that I previously shot a tranquilizer dart into.

UI Hierarchy

Traditional websites go to a ton of trouble to make it easy for me to get to the most useful thing that I want to do. Especially when they are selling a product. "Oh, you're ordering seeds online again? Let me make the [Buy This Seed] button 5 times bigger, with an image." By contrast, with AWS services you are switching betweens concepts that are treated a bit too equally in terms of concept and navigation. So that flat treatment of services makes serverless context switching that much more of a difficulty.

Fixing the UI Hierarchy

You can create an application and a resource group to gather up the pieces within a category that you are working on. That is a very manual process. What is really needed and desirable within the AWS UI is a landing page for people working in AWS where the most recently used resources (not just services) are continuously collected. Last 5 viewed lambdas. Last 5 viewed queues. Mix the services together into a "last 10", so that returning to work on a serverless pieces is just 1 click away, and getting back to work is easy.

Service Icons

Finally, an anecdote about the AWS service icons:

There is a joke that floats around the AWS community about a quiz you can take to try to show your knowledge of the 3d AWS service icons. It's a fun little 5 minutes of satire rooted in a reality of using all the services: https://news.ycombinator.com/item?id=17697366 I personally have memorized some of the icons as a way to quickly try to navigate, but is memorizing that really the best navigational aids we can have?

Building The ELF, a handcrafted-gift planning app, with AWS Bedrock

Coco — Thu, 14 Dec 2023 03:52:05 +0000

Are you as bad as me at coming up with gifts? Here's my contribution towards the holiday season. I call it The ELF, and briefly explain what led me to create and it how you can use it:

I really suck at coming up with gifts for my loved ones. I waaaannt to, because they're great, but I just never know what to do or how to do it. Does that sound like you to? It is me. So I created an app which just for fun I have named "The Elf" where you can describe a loved one's hobbies and likes and dislikes and such, and the AI language model suggests crafts and other homemade things to create for them.
Everyone, it has suggested some of the best gifts that I will have given in 30 years, things that are so obvious and yet so perfect for the people I put in; it blows New York Times gift recommendation posts out of the water, and it also blows my own personal gift brainstorming skills out of the water.
Why does it work well? Because you know the person and the AI knows lots of approaches to creating homeade items written about across the web. So it combines your personalized knowledge with the AI's broad knowledge into a personal result for your giftee.

Try it out:

Homemade Gift Planner Elf

Converting a custom EC2 websockets chat to use serverless

Coco — Thu, 20 Apr 2023 00:32:01 +0000

I have been maintaining a custom, on-server websockets chat for about 5 years as part of the broader ninjawars.net web app. Today, I got tired of maintaining the uptime of the chat server, and decided to move it over to some form of serverless backend. So I decided to go through converting my custom EC2 server-based websockets chat over to the new(ish) apigateway serverless websockets, so I wouldn't have to keep the chat server "up" and the chat could still work. Here is what I found.

Read time: 3 minutes
Development time: 3 hours (your mileage may vary, much of mine was spent debugging AWS arn permissions :) )

Findings first:

Don't handcode your websocket realtime chat, (except as a learning exercise for how to deal with other realtime websocket clients). There are quite a few subtle obstacles that make a custom coded websocket finicky, and require custom coding. The one I ran into was the “ack” system needed to deal with keep alive. Otherwise the chat may experienced a loss of connection that the serverless websocket api gateway does after a time of inactivity, disconnecting the client.

The Repo with the chat tools

I have not yet extracted the aws lambda code down into that repository, nor converted the quick prototype over to something like aws cdk. That may be a future exercise.

The path

Ok, so what overall pieces are there to create a serverless realtime chat with websockets?

AWS API Gateway's websockets api is relatively new, and due to the potential need to do "ack" (keepalive of the connection), it may be necessary to convert over to aws appsync in the future (see this article: full realtime websockets client via appsync ). However, in the meantime I just spun up some custom resources: A dynamodb table, three lambdas, and an apigatway. Oh, and I connected it up to my pre-existing domainname *.ninjawars.net. Websockets is weird. It's like an echo chamber, you send a message and it broadcasts it out to the various connected client connections.

Let's walk through a few of the necessary parts to get a quote unquote "simple" web chat up in a realtime way (without having to maintain a full EC2 server for the websocket)

Step 1: Create an API Gateway WebSocket API

The first step is to create an API Gateway WebSocket API. To do this, navigate to the API Gateway console and select "Create API." Then, select "WebSocket API" and follow the prompts to create your API.

Step 2: Create a Lambda Function or Three

The next step is to create a Lambda function that will handle the WebSocket connections and messages. In my case, I used Node.js and the AWS SDK to handle the WebSocket connections and messages. I am using the guide here: websockets-api for the general websockets mechanism.

Step 3: Configure the WebSocket Routes

Once you have created your Lambda function, you need to configure the WebSocket route to use the Lambda function. To do this, navigate to the API Gateway console and select your WebSocket API. Then, select "Routes" and assign each route ($connect, $disconnect, and the custom "onMessage", or perhaps just the fallback route of $default) to point at their respective lambdas. For the "Integration target," select your Lambda function.

Step 4: Deploy Your API

After configuring your WebSocket route, you need to deploy your API to make it available for use. To do this, navigate to the API Gateway console and select your WebSocket API. Then, select "Actions" and "Deploy API." Follow the prompts to deploy your API.

Step 5: Test Your API from the lambdas and externally

Once your API is deployed, you can test it using a WebSocket client. I used the wscat command-line tool to test my API. You connect to the wss: url, then just paste in any arbitrary text, though probably it should in the end be a json object.

I hear there are also browser-based clients such as the "Simple WebSocket Client" Chrome extension.

Test Event For AWS Lambda

{
  "requestContext": {
    "accountId": "123456789012",
    "resourceId": "123456",
    "domainName": "000000xxxx.execute-api.us-east-2.amazonaws.com",
    "stage": "production"
  },
  "body": "{\"message\":\"test from test event in aws console\",\"uname\":\"xxxxxxusername\",\"date\":\"1681948677627\",\"sender_id\":\"128274\"}"
}

Test message from client

{"message": "test from test event in aws console", "uname": "xxxxxxusername", "date": "1681948677627", "sender_id": "128274"}

Takeaways

Beware!
Because there is the assumption that websockets is the mechanism for the realtime, there's a pitfall that you don't get any long-term storage out of it! So if you want to show a "history" for the chat (and you do, trust me you do), then you have to have a separate backend source that will handle the posting, saving, and retrieving of the chats anyway, and that will need to be pulled from for the initial chat history. So the websockets part of the chat is actually only half of the equation, a different api and database is needed for long-term storage of the chats that were sent in a non-realtime way.

Overall, the reason I recommend going with a framework like amplify is because the complexity of getting it right in a bespoke way is pretty high. The initial "prototype" is simple with a websockets serverless backend, plus some UI, but there are hidden gotchas that come up invisibly later, like the need to ack the connection, and handle reconnections to the websocket server.

Even this minor task ("just create a realtime chat!") is actually much harder than it looks on the face of it.

Practical and pragmatic generic aria roles to use

Coco — Tue, 17 Jan 2023 04:58:56 +0000

It's sometimes difficult to pick an aria role, because they are defined vaguely. Because an aria role is often used to define the purpose of a piece of an application, it often doesn't pin down what specifically the role is, the description hems and haws around the purpose, it provides the vague shape of a thing but not the specifics. Roles are by nature vague. For example, if a role=button were exactly the same as <button> it would actually be redundant, there would be no point in the role, we would just use <button>! So the role descriptions are often more vague.

This results in some difficulty: How do we know what roles to use? For example, try to avoid using role=button. Why? Because you should use actual <button>s, and the actual buttons get for free the role=button.

So then what roles are actually, pragmatically useful when otherwise using html5?

Well, as app authors, what we are often doing is setting priority. We aren't just creating one <div> among thousands, it's a <div role=region>. It's not just a <span>, it's a <span role=comment>.

So what roles are NOT given to us for free by a html5 element like and ( a role=link element), and work well for when we need to make generic elements more specific?

role=region is a useful generic role to highlight an area
role=alert applies to really high priority, transient things that come up and may go away after, things that elicit a response
role=status for status updates, generally where the contents are going to update live
role=comment for where it is people writing comment responses
role=note for separated text

A huge majority of the remaining roles fall into two "try not to use" categories:

Don't use these because the role comes for free from the semantic html5 element: role=button, role=figure, role=code, etc
Abstract roles (they exist, but are not intended for use with authorship)

In Summary

I sorted through the aria roles trying to find 2023's most useful generic roles. For now, these are the most useful roles that I could find for authorship that aren't rendered redundant with other html5 builtins.

Using the software testing pyramid to capture high value with less effort

Coco — Wed, 23 Nov 2022 22:57:48 +0000

Let's talk about how to stabilize a complex application, back to front.

Recently I was working on a strategy to text a complex application, get some checks in place so that bugs in core processes would be caught before they could go live, even as I worked on the underlying systems (core creation and updating of information, generally).

To put it another way, I kept on breaking my own core application functionality in weird ways while trying to make improvements.

I can't tell you the number of times that I have had some core functionality (login, forgot password, the homepage, the first page a user sees) break and it took some time for me to catch it. I have broken http://ninjawars.net more times than I can count, over it's 19 years of being in existence. Usually in totally new ways.

So I wanted to get a check in place to try to prevent breakage, and get that check automated in the Continous Integration.

Well, it is time to get some stability on this beast. I built some unit tests via TDD while I was coding things, I created some basic integration tests run through python to check urls that are deployed in a local instance and check that they load. What I needed now was some End-2-End tests, with a real, true browser involved.

The Testing Pyramid

Before I get into what I did, I just want to mention that my approach was informed by the testing pyramid:

You have a lot of unit tests at the bottom, some integration tests in the middle, and fewer E2E tests at the top of the testing pyramid.

Some people I have spoken to get a little too caught up in the pyramid metaphor and say "Ok, so you have to write lots of unit tests first, right?" but nah, don't get too hung up on it being a pyramid like the pyramid of giza. Ironically the tiny tip of the pyramid (a few E2E tests) is more valuable than the wide base of the pyramid (lots of E2E tests).

DON'T GIVE IN, TEST FROM THE OUTSIDE IN!

Don't give in, test from the outside in! By this I mean don't do what's easy and write a few unit tests and stop at the layer of unit tests (there are always more unit tests you could write, so it's easy to get lost at the unit test stage). Instead consider ignoring integration tests until you know you need them, delay unit tests later, just skip right to writing E2E tests first! You can get a lot of stability value from working down from the top and up from the bottom, until you meet in the middle.

Cypress for E2E tests

So I started to set up cypress for E2E browser tests. Modern, powerful, and integrates well with continuous integration.

I ran through the core paths of the system, login, forgot password, player account profile, homepage and main page.

E2E tests are actually nice for this because knowing how to use the app allows us to create tests. We don't need as much to know how the infrastructure, the react components, the code itself is working, as long as we have a good sense of the behavior of the app itself, and how we expect it to behave. Also almost everyone is familiar with login, so testing a login page with E2E tests is highly familiar ground, a good place to start.

Test Flow

Homepage --> Login --> Main Page

This set of test flows for E2E tests cover a LOT of ground, if these pieces of your app are operational, at least people can get in and see your core functionality, and try to use it. So these tests flows are high value. Potentially much more valuable than testing 5-10 units of unit test functionality.

Forment Revolution

You don't have to do what they tell you. Write your E2E tests first, do unit tests later, if you can.

Just a brief final anecdote, I once worked at a company with thousands of unit tests (it had a full QA team dedicated to creating unit tests)... ...and the app still kept breaking because the app was much much more complex than the sum of it's units. If your current tests aren't working for you, don't do more of the same, get some E2E tests in place.

Thoughts on these ideas?

Moving an EC2 server Exim4 email to AWS Email systems

Coco — Tue, 01 Nov 2022 03:44:17 +0000

Background

I have an EC2 server running in production that connects to
Exim4 to send out emails, transactional and otherwise.

AWS's recommendation for simplest email seems to essentially be "Use AWS SES instead of pinpoint".

Pieces

Generally there are some email templates that have to be programmatically fed into the email system.

An internal API has to be created to accept requests for sending out new transactional emails. Implementation of this will be with an AWS lambda called first by local CLI, and later via an invocation event.

Initially, manual whitelisting of addresses in SES will be done via the console while the SES is still in sandbox mode.

Security checks need to be done to ensure email injection can not be done (there may be something that AWS SES will do to make email injection attackers less effective than Exim4, but that is always worth checking)

Create unsubscribe link system, requiring a url on the main domain that accepts url-based unsubscribes in one action, and adds the user to a blacklist

Eventually, getting the AWS SES out of sandbox mode (here are some example rationales, though your milage may vary, these are just drafts).

Getting out of SES sandbox mode

How do you plan to build or acquire your mailing list?
- Users must opt in to receiving transactional emails after they create an account and provide an email address. We do not cold-call/cold email users, they sign up and willingly provide their information.
How do you plan to handle bounces and complaints?
- Complaints: Complaints can just be forwarded through to the main contact email, and dealt with accordingly on a case-by-case basis.
- Bounces: Bounces are more complicated, and need an automated system to take out email addresses that bounce in real time. To handle bounces, an SNS notification topic will be created, which triggers an AWS lambda to add the email to a do-not-send list and refrain from sending further emails to the bouncing address.
How can recipients opt out of receiving email from you?
- We provide a simple on-step unsubscribe link in clear, blue text in the footer of every email. We also follow the structure that allows standard free email providers to use the one-click [Unsubscribe] button method.
How did you choose the sending rate or sending quota that you specified in this request?
- Currently there are XXX number of users in the system. Only X% of them opt in to receiving emails. We send a transactional email at the end of the day to those people who have opted in. As a result, we expect to send XXX * 0.0X emails a day, * 31 for monthly numbers. We also send out a monthly update email to those users who have opted in, thus adding another XXX * 0.0X emails per month. Give a buffer of 120%, results in YYY emails per month as an upper bound.

...More on this process later...

Useful references:

Hacking a quick closed-source react static site in aws s3 and cloudfront, with a lambda helper

Coco — Tue, 01 Nov 2022 03:05:48 +0000

We all know about deploying a static react build to an aws s3 bucket. We can build a react site, e.g. npm run build and get an output directory, which we can then serve as a static site, npx serve ./build for example.

What is the fasted robust site we can hack together easily ourselves, before we go to third-party services like vercel, netlify, and surge.sh (for instance if you are working for a company that does not want to use out-of-aws third-parties for infrastructure) ? A bucket+cloudfront site is a good option. I have been running nigh unbreakdable cloudfront sites that have been up for years without complaint and almost zero maintenance. One caveat is that keeping the caching settings and invalidations correct initially is tricky, cloudfront caches at the edge really really well, so we might want to automate invalidation of the / root for ourselves.

This is where we can have a little fun by using a lambda to automatically trigger on changes to the bucket assets, leaving us with a dead-simple, copy-and-paste - from local folder to bucket - deploy process.

So the first step step is to create a bucket & upload the files to it e.g. aws s3 sync ./build s3://your-bucket-name, and then we want a robust and wont-cost-much site DNS, which we can do relatively fast using cloudfront:

Creating a cloudfront distribution

Set the error pages of the cloudfront distribution to point error 404 at /index.html, error 403 at /index.html, so that the react app can route all urls that don't have static assets at them to the main index page. Set the path at / to have a caching policy of no-cache, must-revalidate, so that people hitting the site will always get a fresh / root result, but we can more aggressively cache other assets because they will get invalidated when we push up.

Creating a lambda watcher to invalidate on bucket changes: https://gist.github.com/supinf/e66fd36f9228a8701706
Trigger the lambda on bucket changes.
S3 is one of the standard triggers for a lambda function, so you can set S3 changes to trigger the lambda invocation.

s3 -> lambda -> cloudfront

In the first stage of development, you may want to wrap the cloudfront distribution behind a WAF (or WAF2, whatever) so you can control who has access to it while you are doing rapid development. If you don't care who sees or hits the site, skip the WAF, with cloudfront I find you'll rarely need it, your site will be so stable.

Deploy via CI

Because the infrastructure itself will handle the cache invalidation, your CI can simply be responsible for building the assets. In the beginning this could be as simple as having an index.html with hello world in it or you could skip right to react and npm run build and a github action that runs cp ./src/* ./build and then aws s3 sync ./build s3://your-bucket-name-here/.

(You will have to add an aws token secret to your github secrets to allow github action to upload to the bucket)

A diagram of the constellation of useful UI libraries

Coco — Thu, 11 Aug 2022 04:33:00 +0000

When refactoring a frontend, it can be useful to have a "toolbelt" of different libraries to move from a complex set of html/css to - for example - react or cleaner css principles. CSS is especially problematic to refactor because of how it cascades to other parts of an app.

Here are a few tools worth considering: