Forem: Lob

To Infinity and Beyond: Our Nomad Migration is complete!

Lob — Fri, 30 Dec 2022 21:34:46 +0000

Lob’s core API has been fully migrated to HashiCorp's Nomad, Lob’s Next Generation service platform. This is a major milestone for the Nomad Project, the Platform Team, and Lob Engineering. This migration is the culmination of a year of R&D, months of practice migrating other Lob services, and weeks of work on this particular service. It’s absolutely worth celebrating for the complexity and customer impact.

TLDR: Lob’s core API has migrated from Convox to Nomad. Our API is running better than it was before and with Nomad we have the tools to tackle platform issues that were previously not possible.

Why?

The way we run code for our API and dozens of supporting tasks has organically grown based on the needs and constraints of the business. Most apps at Lob ran on Convox, a straightforward service management tool we license, but a handful of workloads have needs that are not met by Convox! Historically, these run on AWS Lambda for security or scaling reasons, or Heroku for development ease, and some run on ECS for various customization needs.

The point is that this artisanal, organic, Non-GMO architecture was and continues to be a huge burden on Lob Engineering. The Platform team needs to know half a dozen tools very well and the rest of engineering must settle for a narrow feature set or spend weeks ramping up on a different technology. This drags on all new products, features, and bug fixes resulting in lower engineering velocity and a lot of hair-pulling.

It was clear back in 2021 that Lob needed to consolidate how we run code and none of our current tools were up to the task; it wasn’t a matter of if Lob would upgrade to something new, but when. The Platform team kicked off a research project to find Lob’s next service platform. Forever ago (back in 2019) we investigated migrating to Kubernetes, a popular but notoriously difficult-to-manage tool for this sort of thing, but that project fizzled out for many reasons, forcing us to consider something else. We chose Nomad which offers a comparable feature set to Kubernetes in a much more streamlined package. Nomad is developed by HashiCorp, a leader in the DevOps space, and is used by companies like Pandora, Cloudflare, Internet Archive, and Roblox.

Nomad is able to meet all of Lob’s business needs today and into the future, allowing us to consolidate where we run most of our code onto one platform. We are able to specialize in this tool, create custom workflows to meet Lob’s changing needs, and tune the underlying architecture to outperform our old solution(s). Where before we had to stretch ourselves thin, or live without some features, Nomad is a one-stop shop.

What does this get us today?

Our core API running on Nomad has performance parity with our old container orchestrator, but it’s even better in some ways. Here’s how:

Lob releases happen via GitHub Actions, giving us better visibility into when the last release was, what got deployed, if it failed, and who kicked it off.
Ad-hoc tasks like Database Migrations and Scripts are also managed in GitHub Actions, moving even more processes off of your laptop and onto the cloud.
Nomad offers a much richer UI, enabling Lobsters to gain better visibility into how their app is working or debug their app when it isn’t.
Nomad’s autoscaling is much more customizable, allowing us to scale our API more responsively to meet spikes in customer requests.
Tag-driven releases allow us to audit when code was deployed and promoted to customers.

What does this get us tomorrow?

The real reason to migrate to Nomad is for the laundry list of shiny new features that unlock major potential down the road…

Hosting containers on AWS Spot Instances should save us money on our AWS bill.
Middleware for handling authentication and other business needs at the edge.
Canary deployments, finding regressions before rolling them out to all customers.
Federated services, hosting our apps in AWS regions closer to our customers.
Autoscaling based on custom metrics like workload prediction.

The final piece of the puzzle

Most of our API’s migration to Nomad happened in August, but the key feature of autoscaling was not working as expected. This turned out to be a bug in Nomad which James Douglas tracked down. The issue was recently fixed and autoscaling works as expected, completing the migration!

This migration, as with all of Lob’s Nomad migrations, is intended to be zero downtime. This means we would slowly migrate traffic to Nomad and roll back if we noticed any regressions in order to minimize customer impact; in most cases, customers cannot tell that this type of change took place. Autoscaling was an optional feature—we could just run Lob at max—but that would be a waste of resources and wouldn't really bring us to parity with the old Convox deployment.

50+ services have now been migrated successfully!

Countless Kudos

This is the type of project that cannot happen in a vacuum. The Platform team is incredibly thankful for all of the Lobsters past and present who have helped us out. You have lent us your time, expertise, patience, and trust in a project that really needed it. While it’s easy to move on to the next big project, moving our core API to Nomad is an accomplishment worth taking a moment to celebrate.

Special thanks to Senior Platform Engineer Elijah Voigt.

I Know What You Shipped Last Summer

Lob — Tue, 20 Dec 2022 17:56:39 +0000

A hackathon is the perfect embodiment of several of Lob’s core values including Be bold, Move fast, take action, and Level up! It’s an opportunity to step outside the daily hustle, collaborate, innovate, and problem-solve. The only thing more fun than a hackathon is a themed hackathon—especially when the theme is Halloween!

We want to acknowledge the hard work our Product and Engineering teams put in by highlighting their projects; the following is a technical recap of Lob’s Halloween Hackathon 2023.

SQS-consumer gets a glow up

Many of our projects focused on improving the developer experience at Lob, but a few innovative Lobsters created a new open-source library that also benefits our community as well.

AWS SQS, or Simple Queue Service, is a piece of infrastructure offered by Amazon; it’s a powerful tool processing data in a controlled manner. “Sqs-consumer” is an existing open-source library for working with SQS.

Though it’s a very popular library—we use it in Lob API and other services—it is not being maintained (major issues and suggested features are not getting attention) and perhaps most importantly, it’s not been upgraded to the new version of AWS SDK (V3). This prohibits us from upgrading many of our workers, and Lob API as a whole, to V3 of the SDK.

Solution: A new repo for working with SQS @lob/sqs-consumer! It's an open fork of the previous package, but it makes a few key changes:

Upgrade underlying AWS SDK to V3
Upgraded all dependencies
Converted test suite from an outdated version of Mocha to Jest
Fixed challenges with FIFO queues

Even though we now offer a low-code solution to automate direct mail (via our Campaigns product), we will always be a developer-first company. This library is a big win for our engineers, and we look forward to feedback and contribution from the developer community.

Software Engineer, Sishaar Rao (with help from Staff Software Engineer, Landon Barnickle, and Senior Software Engineer, Hanqing Chen) tied for second place for this ingenious project.

o11y is the bee’s knees

Another hackathon project focused on increasing our understanding of the services at play at Lob. Staff Software Engineer Rich Sevoria and Senior Software Engineer Benny Kitchell teamed up to tackle observability.

Lob’s rendering engine transforms HTML templates into print-ready PDFs, at scale. Every so often, we’ll encounter a glitch, and debugging can be a very manual—and painful—process. Most of the logging is contained within an internal service so any one individual lacks full visibility into all the services that are connected to each other and where they are failing. Ideally, you could visualize what's taking place, how long it takes, and in what order.

Sevoria proposed a two-part solution.

First, utilizing Open Telemetry, or OTel, ”a vendor-neutral open-source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, logs.” Using traces and spans, we can get detail and visualization of all the events that took place across multiple services that aren't connected in any way outside of that they're all our services.
Then we can use Honeycomb for a deep-dive analysis of this information.

Unlike Datadog (which we use at Lob quite a bit), you don't need to know what axis you want to analyze in advance. (Anybody who spends enough time with Datadog knows that if you wanted to, for example, search aggregate information by a given data set, you better have known that already and have created a facet for it; if you haven't, you're not going to be able to get an accurate read off of it.) You can aggregate on any dimension at any time you want, which is fantastic for analysis.

Sevoria noted two other benefits of exploring Honeycomb for data analysis:

using industry-standard typically results in less technical debt (in form of a really tightly coupled adherence to a specific vendor's implementation), and
Honeycomb has a very transparent and easy-to-understand pricing model.

Simplifying local development

In addition to hops along the way to delivery, during creation, each mailpiece takes a trip through a network of distributed services. This means Lob services are difficult to stitch together when developing locally. We've historically called staging instances from our local environments to simulate the production behavior. Unfortunately, this process breaks down when the interaction is asynchronous or when multiple services need to be updated simultaneously.

This problem was introduced with ominous music and masterful editing. Remember the scene from “Saw” when Jigsaw says there is only one key? Masterful editing had Jigsaw following up with, “to find the key, you must integrate services in Docker.”

The team identified three reasons why it’s so difficult to integrate in Docker: differences in naming conventions, port bindings, and overlapping resources. Lob uses AWS so we communicate with each other through queues. This can get quite complex, (previously queue messages are problematic for example), but the team uncovered a way to simplify it.

They proposed standardizing starting conventions, staggering our ports, and sticking with one version (or one running container) for joint resources like one AWS stack and Redis. Before vs. After:

Each service can be started separately via a new startup script. This ensures shared localstack resources are started, adds the queues/buckets, etc., to the shared AWS localstack that is needed, and starts up only needed services in the Docker network: shared. Port binding conflicts can be avoided by referring to the new (very high-tech) shared Google spreadsheet.

This has the potential to standardize how Lob’s apps communicate with each other. This aligns with similar initiatives within Lob architecture and a continued push towards operating as a unit or as a whole ecosystem, not just individual teams.

Major kudos to Staff Software Engineers Krys Flores, Ken Pflum, and Adrian Fallerio, Senior Software Engineer David Nutting, and Software Engineer Nick Perri—their efforts resulted in a tie for second place.

GitHub productivity hack

Easily winning the best costume award was Senior Software Engineer, David Nutting, for his “Lob” Zombie getup.

Nutting kicked off our presentations with a GitHub productivity hack. He outlined the frustrating and time-consuming scenario where you are waiting on your code to be approved, then manually executing a merge, only to be served with the “This branch is out-of-date with the base branch” message (sometimes not just once, but twice).

As it turns out, automatically merging a pull request is a feature that's built right into GitHub. To turn it on, you just have to open up the repo settings and check two boxes. That's it!

Though not a feature of GitHub proper, there are also Github Actions that will essentially press that “Update branch” button for you. So when any other merge happens to main, a "push" event will trigger some code to tickle your branch and pull new changes. (If you are concerned about future conflicts, you can set up as many notifications as you want in GitHub.)

While better for some use cases than others, this certainly is a time-saver.

Changing of the guard

Karmbir (KC) Chima was on-call for a portion of the hackathon; this ended up being the inspiration for his project. He too was interested in saving time. At the end of each on-call shift, the primary engineer—or “goalie”—is responsible for a comprehensive handoff. This involves documenting all issues (resolved and ongoing) and other detail; essentially duplicating all the info captured in the tickets. But let’s be honest, at the end of two weeks, not everyone has the time/energy/memory to do this at a high level. Realizing that Notion can natively sync with Jira, the solution was obvious: Add an “on-call” label to the request form (a Slack Workflow that creates a Jira ticket); this automatically syncs into the Notion document, eliminating the need for recall and duplicate data entry.

Run, Dora, run

Staff Engineer Guy Argo also had efficiency on the brain. Lob’s address verification software, affectionately named “Dora,” is CASS-certified, which is a huge benefit to our customers, but does require quite a bit of work to maintain. For the latest round of certification, we have about 150,000 tests to run.

To get things running a bit quicker, he persisted all the data for the tests in the database but was still looking to shortcut the endless send/retrieve in Elasticsearch. So why not Redis? 20 lines of code and 1 cached index later, he saw a 20% improvement in runtime on the large test dataset—a perfect productivity hack! Correr, Dora, Correr!

Show me what you’re working with

The Partner Management & Tools engineering team has the unofficial mission to make life easier for Lob’s partners and our internal Partner Operations group; both have expressed the need for greater inventory visibility. Together the team built two new UIs that allow visibility of inventory by partner, and inventory change by day. Previously only available in Datadog; this information is now displayed in an easy-to-digest way; areas of concern can be quickly identified, as well as trends.

This was a huge win for both our partners and Lob, and for their efforts, the team took home the gold ribbon. Congratulations to Engineering Managers Adam Stodgill and Andrew Jorczak, Senior Software Engineers Martin Han, Tim Bright, and Zac Leids, and Software Engineers James Cho and Jessica Ho.

Filter what you’re working with

Up next, to make a case for “there's no small PRs, only small teams” was solo artist Staff Software Engineer, Landon Barnickle. Our Partner Ops team has also expressed a pain point around filtering. To make their lives easier, Landon added an admin route for managing filtered partners using a wrapper for our Redis cache, or as he put it: List, Add, Remove, WIN! He reminded our engineers of the importance of working closely with the end users; often little changes can make a big impact.

See the future, I can

Our Routing and Delivery engineering team (more commonly known by the awesome acronym RAD) also supports our partners. For example, we currently have a tool for our team to create dynamic rulesets around routing to our print partners. To use a very simple example, let’s say we wanted to route all mailpieces from a customer to a specific print partner. Using GitHub Actions, our team can create this rule quickly and easily—but they do so in production.

The downside of such agility is we don't have a good way of predicting the effect of any given change. Should the print partner in the above example be at capacity, or if there is a conflict with another ruleset (more common), we must make another adjustment.

Ken Pflum, Staff Engineer, developed a way to allow us to A/B test; in short, we can now work off a particular branch to test rule sets, then merge that branch into main. This predictive ability reduces risk and still allows us to adapt quickly.

A picture is worth a thousand words

Visibility was a common theme in this year’s hackathon. Lob sends millions upon millions of mailpieces each year which results in a lot of data. For those within the company, there's no easy way to digest that data. Looker dashboards have limitations and even those with SQL chops may run into difficulties writing queries.

Engineers on the RAD team aimed to provide a couple of visualizations depicting mail volume and our routing approach. Why visualizations? They are easier to digest and more descriptive than code and they help surface patterns and anomalies.

Sample data was used to showcase mail density over the last month across the US:

Using data from a sample postcard campaign, they also presented a simulation of what a routing profile could look like on the US map. It follows the campaign getting routed through Lob: The bigger bubbles represent Lob’s print partner facilities and the smaller bubbles represent the USPS facilities; the links showcase the USPS facilities where our partners sent our mail.

Props to Senior Software Engineer Sachin Muralidhara and Software Engineer Joseph Villanueva, with nods to Staff Software Engineer Landon Branickle and Engineering Manager Adam Stodgill. The team at large is excited to build on this work.

The coolest traffic cop

In another effort to standardize development and operations, Lob has just wrapped up our container orchestration migration from Convox to HashiCorp’s Nomad, led by Senior Platform Engineer Elijah Voigt. In this new ecosystem, one feature available to us is Consul Service Mesh (a feature of Consul, which is part of our Lob Nomad stack).

With Consul Service Mesh, we can have each of our services declare what their dependencies are, and enforce only traffic from those services. So you can tell your container orchestrator: this service should only accept traffic from this other service, and anything else should be blocked (and that service is encrypted). As we scale up, this offers us the same zero trust security much larger enterprises typically employ.

In addition to setting us up for a better security posture, Service Mesh also provides a greater understanding of how our services work together. Topologies allow you to quickly identify dependencies and interactions, and details are available for a deep dive.

A journey of a thousand miles starts with a single step

Senior Director, of Strategic Business Development, Tyler Dorenburg, and Senior Product Manager, Andrew Reagan are passionate about bringing new functionality to our customers. Working closely with our engineering team they continue to drive innovation.

Informed Delivery by USPS is a free daily digest email with digital scans of each mailpiece you will receive each day. Additionally, mailers can add a clickable advertisement underneath each mailpiece scan.

As of September 1st, USPS adjusted the requirements making it possible for Lob to support this functionality for our customers and their mail campaigns in the near future! The technical details of how we plan to implement this are both mindboggling and proprietary so we won’t share that here, but Staff Software Engineer, Landon Barnickle, and Software Engineer, Gage Guzman, used the hackathon as an opportunity to dive into some of the prework (like digesting a 92-page PDF with XML specs—sounds terrifying!).

Go big or go home

Arguably the most frightening moment of the hackathon demos was when Senior Infrastructure Engineer, James Douglas, showed the group our AWS bill. Ek! We are storing over 5 petabytes of data in the form of billions of objects. Our tables have grown so big that we're timing out on a lot of queries.

We developed a service called “Expunge” which deletes expired data on an ongoing basis but has a hard time working through our current backlog. It’s like using a lawn mower when a feller buncher is what’s needed. (You are not alone if you had to look up “petabyte” and/or “feller buncher.”)

True to the Lob value Be Bold, James went in search of the tsar bomba, that is, a one-time big-bang approach to clear all the old data.

He initially explored Amazon S3 Lifecycle which is a built-in wipe but came across a number of limitations.

For example, why not just delete everything that's X days old, right? Wrong. There are special circumstances where we want/need to keep data for longer.
What if we tagged them, then have S3 delete the rest? Wrong again; turns out you cannot exclude a tag in the filters.
Move them to a different storage class? Nope. Can't target only a storage class.
How about we evacuate the files, then copy them back in so it updates the date (to fall within the “keep” period)? Nope. Innocent bystanders would be caught in the slaughter, like templates, which are timeless.

Alas, the tsar bomba approach would be too indiscriminate. Since the goal is to target each S3 individually, James landed on the following instead:

He attacked the database first by carefully moving everything that qualifies for deletion into a new table and indexing it; this took about 19 hours. From there, you can use Expunge’s core functionality to attack this new table to delete resources. Testing proved this process would take mere minutes; minutes that will save us major moola. This is a huge win for Lob.

With S3, an ounce of prevention is worth a pound of cure" —Douglas.

Stripe Upgrade

Proactive upgrades are also worth their weight in gold. Software Engineers Sage Farrenholz and Patrick Thulen on our Billing team tackled an upgrade to Stripe, the system we use to automate and process payments. True to theme, Sage noted our legacy Stripe integration was pretty “scary”: we were using an older version, and we wanted to move from Charges API to the newer Payment Intents.

The work was fairly straightforward: They bumped Stripe (Node) Client package to 10.15.0, bumped Stripe API to the latest version 2022-10-0, and moved over to Payment Intents, but this involved multiple breaking changes and an awful lot of testing.

While not the sexiest of projects (although an accidental dropping of the “e” from “Stripe” on one of the presentation slides would suggest otherwise), this brings tremendous value to our Lob and our customers.

Wrap Up

After 48+ hours of non-stop hacking, our Lobsters looked a little more like ghouls. That said, each and every participant, and all those who got to witness the final presentations, had smiles on their faces when all was said and done. Hopefully, they were sincere…

“No matter the situation, always wear a smile.” –The Joker

Want more?

We love a good theme; read more about our spring hackathon, inspired by a favorite 80's movie: Hack to the Future.

h(API) holidays! Mail your holiday cards programmatically

Lob — Tue, 06 Dec 2022 17:43:04 +0000

It’s Holiday Card season! But my hand hurts just thinking about addressing all those envelopes, and I can’t remember the last time I bought stamps…let’s be real, ain’t nobody got time for that.

Fortunately, I happen to work for a company that sends direct mail programmatically. Many developers integrate with Lob using one of our SDKs in their preferred programming language, but we recently launched our low-code Campaigns and I thought I’d give it a whirl.

The following is a step-by-step tutorial so you too can create personalized full-color 6”x9” postcards (automatically printed and mailed) for under a dollar each.

Here is a sample of the postcard we will create.

Select and prep your photos

Choose two of your favorite photos: one landscape (for the front), one portrait (for the back). Use your photo editing tool of choice to format each correctly; I used Preview on my Mac. To ensure that they scale proportionally and print at the highest quality they should be 5x7 inches (2100 pixels x 1500 pixels in landscape or 1500 pixels x 2100 pixels in portrait) and at a resolution of 300 pixels per inch. (If you need more support on this part, and/or are snagging photos from your iPhone, see the Appendix.)

You will then need to host your images somewhere; I used a free account at imgbb.com.

Prep your data

You will then need to bring up the HTML for the front and back of the postcard. Click each link and view the HTML; we’ll use it in a later step).
Next, you will prep your address data. Make a copy of this spreadsheet and input your data:
Do not modify or add any column headers!
Add the full name in the name column
Add the address data (make sure that leading zeroes aren’t being dropped and that the state code is used)
Replace the “Lorem ipsum” in the message column with your own personalized message. Make sure that the message fits within the space by being shorter than the placeholder examples (about 65 characters) You can test your holiday card messaging HERE
Add the links to your front and back images to every row in the image_front and image_back columns (should look something like this: https://i.ibb.co/fpJ84Js/front.png)
Download and save your address file as a .csv.

Create your HTML templates

Now you are ready to create your postcard templates in the Lob dashboard.

First you will need to sign up for a free Lob account.
Once you have registered you will be taken to the dashboard. In the Developer tier there is no cost to test out the Print & Mail APIs, but if you want to send live requests—and have a postcard printed and mailed—it will run you $0.896 a pop.
Payment information can be entered here: Your name (top right) -> Settings -> Payment
Create & save HTML templates for the Front and Back of your postcard
On the left menu bar, Navigate to HTML Templates.
Make sure Live is highlighted!
Click on ‘Create’
Name this ‘Front Template’ and paste in the HTML code for the front. Click ‘Create’ to save.
On left menu, click HTML Templates to go back to main template screen. (You should see your new Front template there under Live templates.)
Once again, ensuring Live is highlighted, click ‘Create’ again. Name this ‘Back Template’ and paste in the HTML code for the back. Click 'Create' to save.
Click HTML templates to go back to main template screen; you should see your new Back template there under Live templates.

** Make sure you see both templates are listed when Live is highlighted before proceeding.

Create and send your postcards

On the left menu bar, navigate to Campaigns; click on “Create Campaign”

Step 1: Configure Campaign

Enter a Name and Description
Campaign Type: Marketing
Mail Type: Postcard; Size: 6x9
Postage references: Return address is not required; Select postage: Up to you, note the listed expected delivery times
Cancellation Window: This is set at 5 minutes; you will not have the option to edit on a Developer plan
Campaign-Level Metadata: Optional

Step 2: Add Audience

Upload your CSV file (created during prep stages), or drag and drop it into the dashboard
Map required address variable: These are mapped automatically.

Step 3: Choose Creative

Under ‘Postcard Front’, select the HTML Template you created from the dropdown menu. (Note a preview will not show your images or messages as they are not mapped yet)
Under ‘Postcard Back’, select the HTML Template you created from the dropdown menu. (Note a preview will not show your images or messages as they are not mapped yet)
Connect your creative to your audience:
Merge variable strictness: Ignore;
Connect/map your merge Variables from HTML to the appropriate column in the CSV: {img_front} to img_front, { img_back} to img_back, {message} to message

Step 4: Review Campaign

If something is not correct, click ‘Previous Step’ to edit. The free Developer tier does not allow you to edit cancellation windows. That said, once you place your order, after 5 minutes, the live request will be sent to Lob. Your postcards will be created and mailed, and you will be charged.

If all looks correct, click ‘Place Order’ and your holiday cards will be on their way! You can view your cards under Campaigns in the Lob dashboard.

Wrap up

Lob was started by two developers who needed to solve a problem, and direct mail APIs were the answer; the goal was to make sending mail just as easy as sending an email. This is a fun personal use case, but Lob is meant to automate direct mail—postcards, letters, checks, self-mailers, and more—at scale. Just like email, each mailpiece can be highly personalized, and for advanced tiers, Mail Analytics is available for each mailpiece to track each postcard along its journey to delivery (and if you included a QR code, you could even track open rate). Check out Lob.com if you want to learn more.

Appendix: How to prep a photo

Email/Download a photo from your iPhone onto your computer. The following steps are shown in the Preview app (Mac) but are a good guideline for any editing program.
Select your back photo (portrait orientation). Under Tools, select Adjust Size.

Change Resolution to 300 pixels/inch.
For this photo (portrait orientation): Change Width to 5 inches; and Height will automatically change. Alternatively, update the Height to 7 inches, and the Width will change. It’s highly likely you won’t end up with a perfect 5x7; that’s ok: Change the field that results in one of the measurements being slightly too big; you will crop in the next step).

Under Edit, Select All. Drag the dotted line to crop the image to the necessary 1500x2100 (as you move the line, the measurements will display; exaggerated below).

Under Tools, Assign Profile select Generic RGB. (Photos must be saved as RGB.)
Make sure your photo is named clearly as the BACK. Save!
Repeat these steps for **the FRONT photo, but in landscape orientation (7” wide x 5” high, 2100 pixels x 1500 pixels).

Tips for starting a new job as a Staff+ Software Engineer

Lob — Thu, 01 Dec 2022 17:18:56 +0000

My experience getting hired at Lob at the Staff level made me realize I may have some learnings to share. I've been at the Staff level previously, but that was at a company where I had been promoted up to those levels; I was already very familiar with my team, my company, our business, and our platform, so it was just a natural progression of responsibility for me. But getting my current job was the first time I'd started at a new company at such a level of seniority. And while at least for me, every new job is a major invitation for an Imposter Syndrome attack, this had the potential to be an even greater one. But I fought it, and I was really intentional about how I approached getting up to speed and be effective more quickly and smoothly than I have in previous new job experiences.

So you've landed the job, now what?

You worked hard to get through that interview cycle, you got the job offer, you accepted it, —you won! But then the insidious thoughts can start. "But what if they got it wrong and I'm actually not good enough? What if people there are smarter than me? What if they don't like me? What if their expectations of Staff are different than where I was before and I don't meet them?" The stories can go on and on if you let them.

This is where I stop the stories and I tell myself, "Hi, Imposter Syndrome. I see you. But I'm not going to let you control my life. Even though I'm feeling really nervous right now, here are some facts I can tell myself:”

I got the job. They liked me enough, they saw enough of something in me to offer me the job.
I have a proven track record for performing, getting positive feedback and reviews, and getting promoted. All of those people saw something in me and they can't all be wrong.
All I can ever do is my best. I can work hard, I can try my best, and if that's not good enough, then that's outside of my control.
Worrying about what might be before I have any information to act on isn't going to change or help anything.
There will be a number of people that are smarter than me or have more experience than I do in a number of things, and that's okay. If I'm the smartest person someplace, then who will I learn from? How will I grow?

No one is a know-it-all

Let’s address that last point. Starting at a new company, at a level of some seniority, it can feel like you should be expected to be the expert in all the things. Well, there's this concept that I constantly remind myself of called being T-shaped; inspired by this article in Forbes: Why T-shaped teams are the future of work.

While generalists know a little about a lot of subjects, and I-shaped employees are experts in a single area, a T-shaped person is a subject-matter expert in at least one area and knowledgeable or skilled in several others.

If we lined up several t-shaped engineers in an organization we would have plenty of overlap in general knowledge but bring a different mix of experience and specialization to the table. So, I don’t have to be the expert in everything; we can share the load.

One of the most powerful blog posts I've read, and it really helped me personally with this concept was written by a well-known engineer I respect, Dan Abramov, titled The Things I Don't Know. He very openly and publicly shared with thousands of followers that while he may be an expert in a number of things, there are plenty of things that he doesn't know very well or at all. What was really empowering about that list is there were some things that he didn't know that I did. So if Dan Abramov can be brilliant at some things and struggle with others, I can too.

Behaviors to Avoid

You’re really going to want to show everyone as soon as possible that you know what you’re doing and deserve the title you’ve been given. But, here are some things to watch out for so you don’t end up starting off on the wrong foot and acting like a jerk.

1. Don’t assert things just to show off what you know

It's fine and absolutely appropriate for you to be sharing knowledge and helping to educate others, but just make sure the context is right. **If you find yourself constantly feeling the urge to talk about the things you've done before or point out something you know that doesn't actually add value to the conversation, try to hold back. **This can make you either sound arrogant or desperate if you do it too much. You will have plenty of opportunities to demonstrate your skills and knowledge.

2. Don’t try to change too much too soon—take some time to observe and learn first

Maybe something isn't exactly the way you would've done it or have done it in the past, but that doesn't necessarily mean it's wrong. Make note of these things that seem to go against what you would've done in the past, but resist the urge to jump in immediately and call it wrong. Part of your role is to improve things, and teach others how to do things in a better way, but until you've had time to really get your surroundings, you may not have enough context yet.

Take some time to get the full picture. Gather data, ask questions, but withhold judgment until you've had a little more time to get to know people, processes, and history. Ask questions humbly and from a place of curiosity.

Along that note, pick your battles. When you have that instinct to assert that something should be done differently, continue to ask yourself, “How important is this right now?” Make a note, and you can revisit it later with more context.

3. Resist perfectionism

During this time of high insecurity, you're going to want to project nothing but perfection. But perfectionism is a myth and thinking that you have to be perfect is toxic, not just for you but for the others around you. Part of being a leader is mentoring others, and a portion of that is done by what behavior we are modeling to them. If we show others that we never make mistakes, there's nothing we don't know, we work however many hours it takes to get the job done, we're always available, we're never stressed or tired—I could go on and on—we send that message to others that in order for them to be successful, they have to be that way, too.

It encourages them to set unhealthy, unfair, and unrealistic expectations of themselves. So as much as it goes against that urge to prove yourself to everyone right now, you are doing a greater service to others if you can model some vulnerability. More junior folks need to see that everyone makes mistakes, no matter how much experience you have.

What_ is_ important is demonstrating how you handle yourself when you make a mistake. Calmly own your mistake openly and early. "Here's what happened and here's my game plan for what I'm going to do about it." Take responsibility without being embarrassed or overly apologetic. Just be transparent about it and then get it fixed.

We should foster an environment of continual learning. There were many times when I'd have to say, I don't know how to do that yet, but I'll get back to you. Then I'd use that as an opportunity to go figure that thing out. Don't act like you're just so smart that nothing is hard for you, but rather show others it doesn't have to be scary to do something you don't know how to do.

As software engineers, we need to be comfortable with being uncomfortable and able to figure things out. I see a lot more junior folks trying to stay in the areas they're already familiar with and avoid branching out. They're hanging out in the shallow end of the pool. Show them by example that swimming into the deep end doesn’t have to be intimidating.

Behaviors to Adopt

Now let’s address what you should do to get up to speed quickly and start providing value ASAP.

1. Start taking notes early

Your role as a Staff + Software Engineer is to be a sharpened tool. You bring with you a lot of experience and skills and you can be deployed to accomplish tasks for your team, your manager, or even the engineering organization level. You may be given the responsibility and freedom to work on special projects or tasks that cross team boundaries. You may be given a level of autonomy to decide or at least propose what you think you should be working on.

That said, you can only do so many things at a time. So you have to be very thoughtful about how you prioritize your time and what you choose to spend that time on (more on that in a moment). But as you're learning your way around, start taking notes. You should be keeping your eyes out for opportunities to improve, optimize, and make systems, processes, and people better, but early on, just make notes on all of those things you notice for reference later.

2. Prioritize what you're spending that time on

As you're finding yourself getting caught up on the learning curve, and you're starting to feel more comfortable with your surroundings, that's a good time to start reviewing that list of items you've been building. Is everything on it still something worth spending time on? Can/should any of those things be delegated to others? Whatever's left will need prioritizing.

In the book _Staff Engineer _by Will Larson (which I highly recommend) he references a quadrant of impact versus effort.

“Work on What Matters” is worth a read to understand where you might go astray, but, in short, the goal is to aim for the top half of the quadrant at the “high impact” tasks. Keep checking in with yourself, both before you prioritize your work, and after you’ve started a task, to make sure it’s the right thing at that time for you to be doing.

3. Start meeting people

If you're an introvert or you have some social anxiety this could be really difficult. But I've learned how to push through that anxiety when it's important. Inside, I may be wanting to curl up in a ball and hide, but outside I try to project comfort and confidence; I think that's a crucial skill for being successful as a software engineer.

I can't take credit for this next idea, but now that I've done it, I think it's a must-do when starting a new job. At Lob, every new hire is given a list of people by their manager whom in their first two or three weeks, they have to set up one-on-one meetings with. These are people that are going to be important for the new hire to get to know in their role, or who can provide context or history about the company and business. These can be in-person or occur remotely. (If remote, I would really encourage you to turn your camera on. I think that it is so important to see people's faces and to really feel like they're in that room with you to build that familiarity and rapport).

This was an invaluable exercise for me; it provided me with context on the decisions and directions that had been made before me, and helped me quickly develop relationships and build my network of peers and go-to people for the various things I would need in my job.

Try to distill from others that you talk to what are the biggest problems, pain points, and challenges that they think the organization is facing today? These are good things to take note of so that as you settle into your role, you could try to find ways to improve on these things.

4. Learn in public

While you're getting up to speed on so many things, it's a perfect time to start a “learn in public” culture at your company if it doesn't already have one. Learning in public is where you share what you've learned as you're learning it.

We don't have to hide while spending all night trying to cram on some framework we don't know just so we can log on tomorrow and act like we've been an expert all along. We can be open about where we're at today, and at the same time, we can share the learnings so that others can benefit from what we've discovered. Ask questions in public channels and keep knowledge sharing out of DMs. Show others that they don't have to be embarrassed to ask questions.

In turn, when you learn something new, share it publicly and share often. At Lob, we have a Today-I-Learned channel in Slack, where anyone can share anything in there and the whole engineering organization can see it.

Another great way to share learnings is to demonstrate them. If you've learned how to do something or if someone asks you to show them how to do something, instead of just showing them privately, consider recording a video. Now you can share that video and more people can benefit from that demonstration.

With all of this, try to keep the barrier low. Don't spend a ton of time putting together a perfect dissertation or a superbly polished video. Just hit the record button and start talking. The more you do this, the more others will catch on and not feel intimidated to do it themselves too.

5. Know your learning styles; use them to get to know the landscape more quickly

If you aren’t familiar with learning styles and what yours are, a simple Google search can educate you as well as lead you to a number of quizzes to pinpoint what works best for you for learning and retaining knowledge. Once you know what styles work well for you and which really don't, you should lean on that awareness to really focus your plan for how to get up to speed.

That being said, you should probably do at least a little of each approach, because we shouldn't limit ourselves to just our preferred methods. We need to be flexible and perhaps with practice, styles that haven't been working well for us can be improved.

Visual or Spatial learners do best when looking at visual or spatial representations of information. This could include things such as charts, diagrams, images, and anything that shows ideas in an illustrated way. You may want to lean on using UML diagrams to help you learn more about the platform, interactions between applications, and even within applications. Build your own integration and activity diagrams, or even just flow charts as you trace through things.
Auditory learners do best when hearing information presented to them. Reach out to those people you've been meeting in that network or peer group you've been building and ask them to walk you through some things. Ask people to pair program with you, either on your coding tasks or theirs. (This can be a gift to more junior folks because it gives them an opportunity to teach and explain something that they know to someone else!)
Verbal learners do better when reading and can retain that information better when they write or speak about it. So read the docs! As you learn more, start writing and contributing to those docs. Read code, especially PRs, even if you're not the requested reviewer. This can help you see firsthand where things are being done and how. This can be a great way for verbal learners to quickly pick up languages, frameworks, and libraries they're not familiar with yet by seeing exactly how those technologies are being used.
The kinesthetic learning style is learning by doing, and I've lumped this together with the logical learning style, which involves the use of logical reasoning when processing data and solving problems. In software engineering, I think these two often go together. It may seem like the chicken or the egg situation. If you're trying to learn and get up to speed, but the best way you learn is by doing and solving problems, how can you do things if you still don't know enough? Sometimes you just have to jump into the deep end and be over your head for a little bit. So find projects or tickets to work on that force you to learn your way around.

Last but not least

Try to have fun! Hopefully through all of this, you can keep that Imposter Syndrome monster at bay and really believe that you are exactly where you're supposed to be—and deserve to be, and you can just enjoy the process. Enjoy meeting your new teammates and forging new relationships. Get to know your manager and let them be an ally for you. Build that peer group so you have other people you can confide in and share with if you're feeling stressed, frustrated, or overwhelmed. And celebrate and be proud of your accomplishments—you worked hard to get here and you should feel good about it!

Content adapted from Lob Staff Engineer, Erin Doyle’s, presentation at REFACTR.TECH: How to Hit the Ground Running Starting as a Staff Software Engineer at a New Company.

The Definitive Guide to Node.js Backend Development

Lob — Tue, 15 Nov 2022 16:41:29 +0000

I was poking around our engineering blog when I came across this post from 2021, highlighting the release of this book. Though no longer employed by Lob, I hope you'll agree it's still a fantastic resource from Thomas Hunter II worth sharing!

Node.js has grown at such an incredible rate that the reference materials have struggled to keep up at times. Now Node developers finally have a complete source for learning to build robust applications, thanks to former Lob Staff Engineer, Thomas Hunter II.

Hunter’s new book, Distributed Systems with Node.js: Building Enterprise-Ready Backend Services is a hands-on guide to help intermediate and advanced developers build their skills, and learn to harness the incredible capabilities of Node.js in a production environment.

The book is an indispensable tool for intermediate and advanced developers looking to build the backend skills they need to run Node.js in an enterprise production environment. Developers will learn everything they need to know to build, deploy and scale robust applications, integrate and communicate with other services, monitor application health and above all, ensure reliability in the apps they build.

While there are plenty of resources for frontend Node developers, the book has filled a much-needed gap in backend instruction, earning praise from developers, execs and entrepreneurs.

Mixmax CTO Brad Vogel has called the book, “The missing manual for scaling Node applications,” and made it required reading for engineers at the company.

It’s also an important milestone in the rapid growth of Node.js from an innovative environment for startups into a crucial, enterprise-grade tool.

The Growth of Node.js

From its creation by Ryan Dahl in 2009, Node.js has driven innovation, with features and paradigms absent from industry mainstays like PHP. Node quickly solved the C10k problem — the challenge of optimizing a server to concurrently handle 10,000 connections — a significant challenge for developers at the time.

Innovative characteristics like its event loop design pattern and asynchronous operations quickly began to shake things up, from the startup world to some of the world’s biggest digital enterprises. LinkedIn launched an overhauled mobile app that works with an API written on Node,js in 2011, just two years after the initial release. In the years since, it has been harnessed for the ecommerce backends of eBay and AliExpress, and helped power many of the world's busiest websites like Netflix, Groupon and PayPal.

While originally designed with the needs of servers in mind, Node.js has also found a home on the desktop, powering Microsoft Visual Studio Code, NVIDIA drivers, and many other important applications.

As an early adopter, Hunter grew alongside the platform, garnering knowledge and experience that the next generation of Node.js developers will find invaluable.

The Agility of Node.js

While many tech leaders use Node to power their servers, Hunter points out that its most enthusiastic adopters have been small, agile companies. Many large enterprises already have substantial resources invested in more traditional programming paradigms and are committed to them, as Hunter has experienced first-hand.

Before he came to Lob, Hunter worked for a startup that built a security tool for Node.js-based applications. However, when it was acquired by a major company that was committed to Java, he felt less than enthusiastic about the slow, plodding approach to development typical of large, traditional enterprises. He left for Lob, a company that has built with Node from the beginning, and upholds an agile development philosophy that matched his own.

Why does he prefer Node? One reason is the speed and flexibility of the language.

“It’s so quick to prototype or get something running,” says Hunter. “Turnaround is so fast. You can innovate in ways competitors can’t.”

That speed has enabled Lob developers to maintain an ambitious release schedule, often deploying changes ten times per day. That kind of schedule isn’t generally possible with Java apps, which can take hours just to compile, often limiting companies to quarterly deployments.

Node’s asynchronous operation is also a great fit for Lob’s services in particular, and an engaging challenge for Hunter. The software can proactively send notifications at each stage of a process, such as creating, printing and delivering a postcard, without waiting for a client to request a status update from Lob. This keeps customers continuously appraised of progress and in control of the process, while minimizing resource usage.

Hunter is also a big fan of Node’s open-source identity, enabling anyone to contribute instead of giving one company control of the platform. That has enabled Node to benefit from its huge user community. Node.js users have built the world’s largest package library, with prebuilt packages available for anything from obscure databases to new protocols.

A Book Six Years in the Making

Hunter was initially inspired to take up programming as a teenage game developer, first starting with languages like Perl and PHP. When he later discovered Node.js he was excited by its applications for building multiplayer games where players could interact with other players in near real time. However, it took him longer to learn more niche Node applications, such as standing up databases and synchronizing processes. While the tools were there, the educational resources hadn’t been fleshed out, making it difficult for new developers to learn backend skills.

Hunter points out that this experience is common for application developers, who often get thrown into situations with use cases that they aren’t familiar with, like load balancing or communicating with another server. So as he worked for companies as a Node.js developer over the past six years, Hunter gained knowledge applicable to the book, eventually building a resource to help other developers thrive in both enterprise and startup roles.

By teaching Node.js developers advanced skills, Hunter also hopes to encourage developers and CTOs to make their applications more robust and scalable.

“One of my hopes is that a CTO or early employee at a startup might realize they should dedicate resources to system observability early on in the process.”

Resources

The book is available directly from the publisher, O’Reilly, as well as from Amazon and other book vendors.

You can learn more about Hunter, books he’s published, tech talks, and more on his blog.

Stop Wasting Connections, Use HTTP Keep-Alive

Lob — Fri, 04 Nov 2022 16:28:08 +0000

With the proliferation of third-party APIs and microservice architectures, modern web servers can make as many outgoing HTTP requests as the number of incoming HTTP requests they serve. A typical web application can interact with third-party APIs to handle payment processing, send email, track analytics, dispatch text messages, verify mailing addresses, or even deliver physical mail. A server can also rely on internal APIs to fetch account information, start asynchronous processes, or perform complex searches. Programs that initiate a high volume of outgoing HTTP requests must minimize the overhead of each in order to remain performant and optimize resource utilization.

One of the best ways to minimize HTTP overhead is to reuse connections with HTTP Keep-Alive. This feature is commonly enabled by default for many HTTP clients. These clients will maintain a pool of connections—each connection initializes once and handles multiple requests until the connection is closed. Reusing a connection avoids the overhead of making a DNS lookup, establishing a connection, and performing an SSL handshake. However, not all HTTP clients, including the default client of Node.js, enable HTTP Keep-Alive.

One of Lob's backend services is heavily dependent on internal and external APIs to verify addresses, dispatch webhooks, start AWS Lambda executions, and more. This Node.js server has a handful of endpoints that make several outgoing HTTP requests per incoming request.

Enabling connection reuse for these outgoing requests led to a 50% increase in maximum inbound request throughput, significantly reduced CPU usage, and lowered response latencies. It also eliminated sporadic DNS lookup errors.

Performance Benefits of HTTP Keep-Alive

Running a benchmark validates the performance benefits of HTTP Keep-alive. The following chart displays the total time taken to make 1000 GET requests for both non-reused and reused connections with a varying number of requests made concurrently. It shows that for all levels of tested concurrency, reusing connections reduces the total run time by a factor of roughly 3.

Another observed benefit of reusing HTTP connections is reduced CPU utilization. On Mac OS X, this reduction manifests in the Node process itself and in a process named mDNSResponder, an operating system service responsible for resolving DNS. Running top -stats pid,command,cpu | grep -E "(mDNSResponder|node)\s" during both benchmarks shows the contrast in CPU usage.

Without Connection Reuse

With Connection Reuse

Inspecting the flamegraph of the benchmark script without connection reuse reveals the reason for increased CPU utilization in Node. A large percentage of CPU time is spent on establishing connections and performing SSL handshakes. For example, the flame fragment below shows that 14% of measured CPU ticks occurred while creating a socket.

It should be noted that initiating connections also incurs overhead for HTTP servers. Therefore, reusing connections also reduces overhead for servers handling these requests.

Flamegraphs of each benchmark are available to explore: flamegraph without connection reuse, flamegraph with connection reuse.

The benchmarking scripts are documented in node-keep-alive-benchmark.

Reduced DNS Errors

Reusing connections also eliminated a set of DNS errors that occurred sporadically within our service. When connections are not reused, a new connection is initialized for each outgoing request. In Node, this initialization includes a DNS lookup to determine the IP of the domain to send the request to. A high volume of DNS lookups can lead to sporadic errors of the form Error: getaddrinfo ENOTFOUND.

Based on several issues in the Node repository (nodejs/node-v0.x-archive#7729, nodejs/node-v0.x-archive#5488, nodejs/node#5436) this error can occur when a DNS server fail to respond, perhaps due to it rate-limiting requests. Reducing DNS lookups can reduce or eliminate these errors.

Tips when Reusing Connections

Check Your Timeouts

In some cases, reusing connections can lead to hard-to-debug issues. Problems can arise when a client assumes that a connection is alive and well, only to discover that, upon sending a request, the server has terminated the connection. In Node, this problem surfaces as an Error: socket hang up.

To mitigate this, check the idle socket timeouts of both the client and the server. This value represents how long a connection will be kept alive when no data is sent or received. Make sure that the idle socket timeout of the client is shorter than that of the server. This should ensure that the client closes a connection before the server, preventing the client from sending a request down an unknowingly dead connection.

Don't Use Node's Default HTTP Agents

For Node services, the agentkeepalive library provides HTTP and HTTPS agents that enable connection reuse by default. These agents also have other sensible defaults that the standard libraries agents do not.

Wrap Up

Connection reuse should provide significant performance improvements to services written in any language that are making numerous outgoing HTTP requests. Some HTTP clients enable this behavior by default, but not all. Some widely used languages and libraries do not enable HTTP Keep-Alive by default, such as Node, so be sure to check the documentation and source code.

More on HTTP Keep-Alive from Mozilla

Predicting deliverability with confidence

Lob — Wed, 12 Oct 2022 17:48:10 +0000

TLDR: Win trivia night with more than you may ever need to know about USPS deliverability data…but when that data is key to driving decisions, you’ll be glad you went on the following ride with us.

Our customers send millions of mailpieces through our direct mail automation software every single day, and to do so requires a great deal of trust. But what happens when we lose confidence in the data driving our product? (Spoiler alert: We level up.)

Houston Lob, we have a problem

Lob has a large indirect marketing customer that sends direct mail campaigns via our Print & Mail APIs. They own their own address data but verify that data against our Address Verification API; month-over-month they have about a 2.2% return to sender (RTS) across all campaigns. But a few months ago, they reported individual campaigns returning as much as 97% of mail submitted back to sender. Eek.

They sent us pictures of piles of postcards with the dreaded “nixie” label (Return to Sender) slapped on each. Mail can be RTS for a variety of reasons, some common examples are insufficient address, unable to forward, or vacant, but we saw the following:

As the verbiage would indicate, this means there was no mailbox or USPS-approved container to receive mail.

This customer facilitates direct mail campaigns for their customers by allowing them to geographically select an area to send mail to. Cases in which a user may be attempting to send to a rural area, a specific complex that rejects marketing mail, or perhaps entire neighborhoods without mail receptacles may be partially responsible for the high volume of RTS, but this answer is insufficient as it leaves room for doubt in the quality of our AV product. The confidence in our solution becomes dubious if there are numerous failed campaigns and irritated end-users, regardless of reasoning. A boost in efficacy in our product became critical at this juncture, and one solution arose from re-evaluating the finer data points of our Address Verification API.

Digging into the data

The thing is, the data we had led us to believe these were clean addresses. Our API response would have looked something like this example in our API documentation. Notable fields include ”deliverability”, “valid_address”, "deliverability_analysis", and “lob_confidence_score.”

Lob’s Address Verification product is CASS certified, and we include USPS deliverability prediction in the API response. Upon investigation, these postcards were returned to sender despite USPS confidence they would be “deliverable.”

The “valid_address” field indicates whether an address was found in a more comprehensive address dataset that includes sources from the USPS, open source mapping data, and our proprietary mail delivery data. These postcards were being returned despite a value of “true.”

Speaking of our proprietary data, in 2020, our team developed the Lob Confidence Score. Because we process addresses and ingest USPS tracking events from the mailpieces we send, we have an enormous amount of data we can marry together to calculate a Confidence Score that predicts the likelihood of delivery. But these postcards were returned to sender despite Lob having 100% confidence they would be deliverable.

Digging (deeper) into the data

To get to the bottom of this we had to dive into the USPS deliverability analysis. USPS includes details in their metadata including Delivery Point Validation (DPV) information for each address. This includes “dpv_footnotes,” or 2-character codes that provide more specific information or context. (For example, there is a footnote code to indicate the address input matches a military address or a code to indicate a valid address that is missing a secondary number.) DPV is intended to confirm USPS addresses but also identify potential addressing issues that may hinder delivery. These codes are one of many variables factored into USPS decision-making when determining deliverability.

Though not shown in our sample address, one of the DPV footnotes present was ”R7”, which is short for Carrier Route R777, also known as Phantom Route. If you get asked on trivia night, this is code for an “address confirmed but assigned to phantom route R777 or R779 and USPS delivery is not provided” (source). In practice, this means the mailpiece is not eligible for “street delivery” but is still technically deliverable at the doorstep. Addresses with the R7 footnote are still deemed deliverable by the USPS since they are able to receive mail at the door. That is, these addresses are technically deliverable but practically undeliverable.

Historically, our AV team followed suit. That is, when this data was present, we also reported this address as deliverable.

But when our customer became buried in returned postcards, we determined Lob needed to change our response when we see the R7 DPV footnote. If any part of the USPS response says “undeliverable,” then we should err on the side of caution and note it as undeliverable. The customer still has all the data available in the response and can choose to proceed or not.

Problem solved, right?

Not entirely. We still had returned to sender mail where R7 was not present!?

Digging (deepest) into the data

We opened the investigation to our contacts at the USPS to determine why these addresses were indicated as deliverable by the USPS, had been delivered to previously (that is, had a 100% Lob confidence score), and lacked the R7 code—but were still being returned.

The USPS informed us they were returning these to the sender based on a different identifier on the mailpiece:

“The Service Type Identifier (STID) came back as a Full-Service First Class Mail Piece using Informed Visibility with No Address Corrections. In this case, we return the pieces if they are undeliverable based on the address on the mail piece. That could be due to the customer moving, the address could be missing a suite or apartment number, etc.“

Sure, these reasons could be valid, but since all data prior to the send predicted a successful delivery, were they?

Our team did not let up.

We sent 10 specific addresses to the USPS for a deeper dive and we eventually got details from individual mail carriers. For our samples in Burlington, WI, “neither address has a mailbox.” For zips in 98947, they reported “no street delivery for any address in the city of Tieton, WA.” This makes sense; in any geolocation, there may be a whole suite of houses that don't have mailboxes. Ideally, the USPS could update each of these addresses to reflect their R7 status, but understandably they do not have the resources to tackle such a problem. And as we have learned, even with the correct DPV code, the USPS would still return a prediction of “deliverable” if queried.

That leads us to the unfortunate truth, which is that USPS data on deliverability is prone to imperfection.

Light at the end of the tunnel

But wait. We're not just surfacing USPS data alone, are we?

Our Address Verification team is back in the ring, and look who has a chair? Let’s not forget about the Lob Confidence Score!

Now when we evaluate the same set of “problem” addresses, the accuracy of our Confidence Score has improved significantly. For example, when we re-analyzed the addresses for the 97% campaign, the Confidence Score shifted to more closely reflect reality. The majority of these addresses now have a Confidence Score of 0. The lower the score, the lower the confidence in successful delivery.

Confidence Scores have and will continue to improve as more data is collected on deliveries. That is, the more mail we send, the more tracking data we get, and the more valuable our Confidence Score is for our customers.

Oh, and NBD, but this customer is now seeing a 0.8% RTS rate across sends, cutting down their original amount by over half! Needless to say, both the customer and our team are thrilled with this outcome.

Wrap up

And so, this is how we gained even more confidence in the Lob Confidence score. The fact is address data is prone to faults. It's manual data subject to clerical error and imperfections and many elements are outside of our control. It would be easy to simply roll over and accept this. But here at Lob, we own the outcome; to stand behind having the most comprehensive deliverability data available in the market, our commitment to product improvement will never cease.

Content adapted from a presentation by Solution Engineer Michael Morgan.

Tracking changes in your Elixir code with the Audit library

Lob — Fri, 09 Sep 2022 14:23:28 +0000

Part of my work with Lob as a Staff Engineer is to implement various USPS standards pertaining to address autocorrection (aka CASS Cycle ‘N’ and ‘O’). These standards have a lot of subtle details, so the Elixir code that implements them can be intimidating. It is difficult to get this complex code to do what you want, so I started to investigate techniques that would help me track how my changes impacted the results. Before we dive into my solution, let’s first give a quick recap on how functional data structures differ from their imperative counterparts.

A recap on Functional Data Structures

One of the counterintuitive aspects of writing functional code is the constructive nature of data structures. You cannot mutate an existing data structure — if you want to make a change, you must create new elements containing the change that you desire without altering the existing structure. Essentially you make a new version of the data structure that embodies your change that will share common parts of the previous version that stayed the same.

Let’s take the concrete example of a binary tree. The tree, T1, has six nodes: A, B, C, D, E, and F:

Destructive vs constructive binary tree insertion

The traditional imperative way to add G to this tree would be to overwrite the right field of the parent node containing F to point to a newly created node containing G as illustrated by the left side of the diagram. But in a functional language like Elixir, we don’t have the ability to overwrite fields. Instead, we must create a new spine for the tree containing the nodes leading to the newly created G node while sharing the unchanged nodes from the original tree, T1.

At first, this seems terribly inefficient — after all the imperative counterpart just needed to perform one (destructive) update. But it’s not all bad news in the functional world; although in the tree example we create log₂(n) extra nodes, our old version and new version can happily co-exist. In fact, by merely keeping a pointer to old versions of the tree, we can trivially implement an infinite undo facility (space permitting). In the imperative world, this is a bit trickier — we would need to record the changes via a list of deltas and rerun those deltas if we wanted to access an earlier state. In the functional world, all these versions can simultaneously coexist for a small investment in space. Neat! In the imperative world? Not so much…

This also illustrates why it’s easier to reason about functional code versus imperative code — in the functional code, everything stays constant. In the imperative code, a pointer to a structure is no guarantee that structure will remain intact. This is also the reason that imperative concurrent programming is fraught with difficulty whereas functional programming lends itself quite naturally to working in parallel. No need for critical sections, semaphores, monitors, etc. Those constructs are all about protecting some piece of code that side effects from being executed simultaneously in different threads/processes. No side effects — no problem!

Implementing data structures functionally is actually a fascinating area of study — I highly recommend Chris Osaki’s Ph.D. thesis on the topic: Purely Functional Data Structures.

The problem

Now that we’ve refreshed our understanding of functional techniques for implementing data structures, we can move forward with an explanation of the basic idea. The core of our program doing address correction manipulates a very large %Address{} struct. The program works in phases and in each phase a different set of fields are populated with new metadata either deduced from machine learning sub-systems, or extracted from various USPS- supplied metadata. The final Address struct will have the answer we seek (we hope) and a whole lot of additional metadata that might help us understand how it came to that conclusion. But with a data structure as large as Address with over a hundred fields, IO.inspect at some strategic locations isn’t going to cut it — the amount of information printed to the screen is just too overwhelming. We need something a little bit more precise.

Making a difference

A key component of our library is the difference engine. This takes two objects and computes a list of the differences between them which makes it easier for us to visualize the evolution of our data structures over time. First, let’s decide how to represent a difference. A delta has three possibilities:

update, replace one item with another
delete, remove an item
add, add an item

With our representation settled, we need to figure out our implementation strategy. We simultaneously recurse over the two objects we wish to compare until we find a point where they diverge and we return the difference. So our main function will look like this:

This delegates to a three-argument version that takes the current path as the first argument. We flatten the result, reverse the paths (because they’re call stacks) and then order the results by length of path.

So now we tackle the major cases for delta: struct, map, list, tuple, and everything else:

Let’s tackle these one at a time. For structs, if they are the same kind of struct, convert them into a map and carry on; otherwise, they’re already different.

For maps, we want to compute the overlap of keys: a_only, only in a; b_only, only in b; and common, appears in both. With that information, it’s simply a matter of computing the difference of the values of the common keys:

For lists, I take a simplistic approach of zipping them together to compare the elements pairwise. (I think computing the Levenshtein edits would be overkill).

For tuples, if they are the same size convert them to lists and let delta_list process them, otherwise, they’re different.

This just leaves the base case and we’re done.

Ok, I suppose we should define empty?

And that wraps up our description of our difference engine.

Elixir’s macro magic

A lot of folks don’t realize that Elixir is actually quite a small language; most of the features that you know and love are implemented via the powerful macro languages. Don’t believe me? Check out all the language features implemented inside the Kernel module: @variables, def, defdelegate, defexception, defguard, defimpl, defmacro, |> etc., etc. This also explains why the syntax feels is a little clunky in places — it’s likely implemented as a macro!

The basic idea

Our basic idea is to define a macro, audit, which embeds a paper trail inside an audittrail field. It needs to be a macro because we want to capture the file and line number from where the macro was called. Basically, this field will contain a triple of: the last version of the data structure, the filename, and the line number.

Let’s get started by defining some types:

So our audit consists of a snapshot of the struct, a filename, and a line number. It’s considered good macro design practice to isolate as much of the functionality as possible in a function. So here’s our audit_fun:

So now our macro is extremely simple:

To put it to use, we want to be able to generate a changelist given a struct that has our magic audit_trail field:

Now we just need some helper functions to turn that changelist into a human-readable string:

There are two nuances in this code. Firstly, as we’ll be looking up our sources files a lot, it behooves us to cache them. So we implement a simple Agent to accomplish that:

Secondly, by their very nature source files are subject to change. So for this information to be useful at a later point in time, we should use git SHAs to specify the version of the file that we’re looking at. To accomplish this, we have a module to do all the GitHub URL wrangling:

The final result

With all these components in place, we’re finally in a position to demonstrate our library in action.

This is a somewhat trivial example. The ideal use case is for much larger struct where it becomes hard to see the wood from the trees when you IO.inspect results. The use of the difference engine shines in these scenarios.

For extremely large %Address structs in our Address Verification code, this library has saved my bacon on several occasions. I hope you find it equally useful.

Conclusion

We’ve described a simple library to track changes to data structures. If you’d like to kick the tires and try it out, you can find it in hex.pm under audit.

‍

DEI+Tech event in ATL: REFACTR.TECH

Lob — Thu, 01 Sep 2022 14:39:15 +0000

Everything you need to know about this event is summarized in the logo: A super-fun, colorful mashup of Diversity, Inclusion, and Tech. REFACTR.TECH is all about growing and showcasing powerful voices of marginalized people and allies in tech. The event will focus on technology while creating a safe space for thoughtful and nuanced conversations around diversity, inclusion, and intersectionality in tech. This three-day event is a combination of workshops, keynote, and track sessions by an incredible lineup of speakers to expand knowledge, hone technical skills, and make important connections.

The event is held in Atlanta, GA, from September 14-16. Staff Engineer Erin Doyle will be presenting on behalf of Lob; we caught up with her to learn more.

Why do you want to speak at this event?

I've been to the CONNECT.TECH conference a number of times in the past and it's an amazing conference. When the same organizers started up REFACTR.TECH I was immediately excited and wanted to experience a conference that brings tech together with a focus on diversity and inclusion. It's really an honor for me years later to be able to speak at a conference that literally hands the mic over to marginalized folks to give them a space to be heard. As a female in a technical leadership position, I feel like it's important for me to speak at events like this and show other women that they can achieve success in this industry.”

What are you going to talk about?

Joining a new company as a Staff+ Software Engineer—versus being promoted at a company you've been at for a while—can be an invitation for an Imposter Syndrome attack. I'll cover tips for battling that Imposter Syndrome and ways that you can be the most effective in your role to hit the ground running. Key takeaways will include:

How to fight imposter syndrome that spikes when starting a new job
How to get ramped up on the new technical landscape quickly
Methods for building rapport more quickly
How to be useful and effective quickly

Which talk are you most excited about and why?

Of all the presentations at Refactr, I'm most looking forward to attending Jameson Hampton’s "Reframing Shame & Embracing Mistakes.” For too many years I personally struggled with a self-imposed pressure to outwardly portray perfection in everything I did. I didn't realize that I, in turn, was modeling and sending that message to more junior folks that they also had to be perfect—or else they were less-than. Since learning how incredibly toxic that way of thinking and acting was, I've been working to try to model and encourage others to see that perfection is a myth, we're all human, we all make mistakes and it's totally ok. So I'm really excited to hear more on that topic and other ways I can battle my Imposter Syndrome-induced perfectionism and help others do the same.

It’s not too late to get tickets to this unique event!

Swipe Right on Redshift

Lob — Thu, 25 Aug 2022 16:30:00 +0000

As a “top pick” in the direct mail automation space, Lob prints and sends millions of mailpieces each year. As such, it's been Lob's commitment from the get-go to minimize our footprint and build a more sustainable approach to direct mail. As a part of this initiative, in addition to ensuring we use responsibly sourced raw materials, we plant two trees for every one that we use in the production of our mailpieces, through our partnership with Eden Reforestation Projects. (For more on our partnership check out this blog post/video: “The Power of Trees: Partnering for Positive Environmental and Community Impact.”)

As awesome as that is, this is not a sustainability story, it’s about the technology we use to accomplish this and why we shifted (excuse the pun) tools.

During a hackathon in 2017, our Engineering team developed automation that runs every month. This involves a Postgres query that pulls how much mail (letters, checks, postcards, etc.) Lob sent the previous month, a calculation the equivalent in paper, which is then translated into a number of trees. (Then we offset our carbon footprint by sponsoring Eden Reforestation to plant double that number of trees.)

Snippet of query:

The result is a summary communication sent via Slack to the community and stakeholders.

Unfortunately, as of late, the automation was pretty flaky: often the query would time out requiring an engineer to run it again in off-peak hours when the Postgres database was under less load; this is not a scalable solution and was yielding less and less success.

Since the query looked fine (accurate), an initial idea was to break it out into separate queries and then combine the results. But this and some other brainstorms just seemed like a hack; sure they would fix the current problem but eventually, those would get big enough to timeout too. So, Platform Engineer Elijah Voigt turned to the rubber ducky channel.

Enter Jack Meussdorfer from the Data team to the rescue.

His keen sense of smell indicated this was a too much/big data problem. He suggested instead of running the query against PostgreSQL, we run it instead against Amazon Redshift, which the Data team already uses for similar data pipelines.

In short, Postgres is a transactional db, whereas Redshift is specifically designed for large-scale data storage and analysis. Because Redshift stores data in a different way—it uses column stores instead of row stores—there is less overhead on queries spanning a large selection of data. (This also means differences in how constraints and indexes are implemented.) Redshift also has a higher capability of processing large amounts of data because it runs massively parallel processing (MPP). If you want a deep dive check out this article Redshift vs. Postgres: Detailed Comparison of Performance and Functionality.)

Is that really the right tool for the task?

Since the necessary data already exists in Redshift, Jack suggested we:

Update the query to be compatible with Redshift (alter the query syntax and add the proper schema/tables) and then
Point the Reforestation query at Redshift.

Boom!** Lifting and shifting the data source from Postgres to Redshift solved the problem immediately: the query went from timing out after 1 hour to literally taking 40 seconds.**

The takeaway? Rubber ducky rules. And, it’s important to evaluate whether you are using the right tool for the job at hand. Postgres was the right tool for the job in 2017, but our data changed—which means our problem changed. While we are big fans of Postgres, our automation needed to scale to our new needs, so for this important query, it’s Redshift FTW.

Use Klaviyo to Automate Direct Mail

Lob — Tue, 23 Aug 2022 16:57:30 +0000

Klaviyo is a popular marketing automation software tool that empowers its customers to build campaign automation by
syncing their tech stack with their ecommerce store to scale their communications and business. While it has traditionally served as a great resource for automated email and SMS, direct mail has historically been missing from the equation.

But marketers now have a solution: integrating Lob’s direct mail APIs. Klaviyo can connect to Lob to automate direct mail at scale in the same manner that they are automating their digital channels today: Klaviyo’s webhook functionality within their Flows interface provides an easy connection point to trigger mailpieces.

Read on for a walkthrough of exactly how to connect Klaviyo with Lob.

Getting started

We’ll assume you have a Klaviyo account and you have already registered with Lob (to get your API Keys) to follow along.

In order to send mail from Klaviyo, Lob requires a mailing address to be stored for each profile. Klaviyo does not require mailing addresses by default, so before attempting to send mail, ensure that your recipient profiles contain properties for:

Address
City
State
Zip Code

If you are importing new profiles to Klaviyo, simply ensure that you have CSV columns for each of those properties. An example CSV to import Profiles is linked here.

Creating a Flow

In the Klaviyo dashboard’s left-hand panel, click on Flows.

Click to Create Flow, then select a predefined goal, or click Create From Scratch and give your flow a name.

Choose a Trigger Setup option in the left panel. For this example, we will be selecting List, and choosing a list from the resulting drop-down to trigger the flow.

Create Webhook

Once you have configured your trigger, an Actions menu will appear on the left panel. Select Webhook, and drag it into your flow beneath the trigger.

Select the Webhook in your Flow, and the configuration panel will appear on the left side. You can configure it as follows:

Destination URL: This is where you will enter the Lob API URL for the form factor you would like to trigger. In this example we’ll choose postcards: https://api.lob.com/v1/postcards. (Note that Lob has separate endpoints for postcards, letters, checks, and self-mailers.)

Headers: This is where you will enter the headers Lob requires in order to correctly process and authenticate the incoming requests. You will be required to enter at least two headers: Content-Type and Authorization. Click Add Header.

For Content-Type Header, enter the following:

Key: Content-Type
Value: application/json

For Authorization, you will need your API Keys.

Adding API Key

For the Authorization header, you will need to have your Base64-encoded Lob API key ready to enter, so Lob can associate the request with your account. Retrieve these credentials from your Lob dashboard by clicking on the Settings menu on the sidebar, then clicking on the API Keys tab.

IMPORTANT: ENSURE YOU ARE USING YOUR TEST API KEY.

You can use a Base64 Encoder tool to encode your (test) API Key.

For example, if your API key was test_0dc8d51e0acffcb1880e0f19c79b2f5b0cc, you would enter test_0dc8d51e0acffcb1880e0f19c79b2f5b0cc: into a Base64 Encoder tool, the result of which would be something like dGVzdF8wZGM4ZDUxZTBhY2ZmY2IxODgwZTBmMTljNzliMmY1YjBjYzo=

Note that we added a “:” to the end of the API key before encoding it, please ensure you do the same.

Once you’ve encoded your API key, add the Authorization Header and use the encoded key as the value. For example:

Key: Authorization
Value: Basic dGVzdF8wZGM4ZDUxZTBhY2ZmY2IxODgwZTBmMTljNzliMmY1YjBjYzo=

JSON body

This is where you will pass Lob the information that informs the mailpiece. The field names (name, address_line1, etc.) represent the information that Lob’s API will require. The values on the right represent what you will pass to Lob to populate the given field.

The values can be mapped to attributes from Klaviyo profiles by using Klaviyo’s Profile Variables. You can find the Profile Variables for each attribute by taking the following steps:

First, click on the linked word profile in ‘View profile properties’, directly beneath the JSON Body input field.

This will pull up an example Profile. Clicking on an attribute will automatically copy the corresponding Profile variable, which you can use to populate the appropriate value in the JSON Body.

An example of a populated JSON Body could look something like the following. (You can find more information on each of these fields in Lob’s API documentation.)

Update Creative

Note that if you are copying the above example, you will want to replace the values being passed for ‘front’ and ‘back’ with your own Lob template IDs or linked creative. For example:

"front": “https://s3.us-west-2.amazonaws.com/public.lob.com/lobcom/template_gallery/gtmpl_524eb6ff6c6998_Increase_Customer_Loyalty_Postcard_4x6_front.html"

"back": "https://s3.us-west-2.amazonaws.com/public.lob.com/lobcom/template_gallery/gtmpl_524eb6ff6c6998_Increase_Customer_Loyalty_Postcard_4x6_back.html"

(For more about designing/formatting creative for Lob, go here.)

Testing

You can test that your webhook is configured correctly by clicking on Preview Webhook.

The resulting panel will show an example of the data being passed to Lob for a specific Profile. You can look at the JSON to verify that the Klaviyo Profile Variables are being correctly replaced. For example, instead of person|lookup:'$address1'|default:'' you will see an address, such as 185 Berry St.

If this all looks correct, double-check that you used your Test API Key (rather than your live one), then click on Send Test Request. If it was configured correctly, you will see the following message:

You can then log into your Lob account to verify that your postcard was generated. (The below is the result of the example creative.)

Go time!

You will need to add payment information to your Lob account should you actually wish to print and send mailpieces. When you are ready, you can replace your Test API key with your Live API key; if you have your LIVE API key in the template it will create mail. If you are using other test resources like address IDs or template IDs, those will also need to be transferred to your live environment.

Make sure when you are testing that your cancellation windows are set to an ample time (2 hours is recommended) so you can cancel mail if you make a mistake.

Have fun exploring this integration; if you have questions or need support, don’t hesitate to contact us.

How Docker memory caching works & finding a hidden hapi vulnerability

Lob — Wed, 10 Aug 2022 21:20:00 +0000

It was a dark and stormy night in May when our internal metric readings for memory usage spiked in our Address Verification service for no obvious reason. Even weirder than that, our largest API and main service had suddenly started crashing and restarting every several hours—a dark night indeed. The following is a summary of our painful investigation over several weeks, involving multiple team members and endless :confused_dog: and :computer_rage_bang_head: emojis. But (spoiler alert!), after the storm comes a rainbow.

TL;DR: We learned how Docker memory caching works and found a 7-year-old file management bug which led to a secret hidden hapi vulnerability; all of which helped stabilize our containers and product offering.

What you need to know about Docker

Traditionally for those who have worked with Docker and Datadog, the main way to measure memory usage in a container is to look for Docker.mem.in_use. Now for reasons unbeknownst to me, Docker.mem.in_use used to only measure Rss memory usage rather than the combination of both Docker.mem.rss and Docker.mem.cache:

Docker.mem.cache: The amount of memory that is being used to cache data from disk (e.g., memory contents that can be associated precisely with a block on a block device).
Docker.mem.rss: The amount of non-cache memory that belongs to the container's processes.; used for stacks, heaps, etc.

RSS memory usage is essentially what most would think about when they see a memory leak; a piece of the heap not being garbage collected, or an object that continues to grow infinitely. And that’s why a large chunk of our investigation looked towards traditional sources of memory leaks. But after several weeks with little progress, I started to piece together why our container memory readings had spiked and why the memory leak in our main service had started at roughly the same time.

During my investigation I discovered that on May 9th, Datadog made a change in how memory data is read: mem_in_use is now a combination of both rss and cache usage.

This was when a lightbulb went off: None of our Docker containers had any mounts of volumes attached, so whenever we wrote to disk, we would be writing to memory instead. Logically it makes quite a bit of sense, if you don’t have disk space the only other place you can write to is memory(ram).

This actually even goes a step further, there might be instances where you may have a mount or volume but you still notice docker.mem.cache going up. Well certain linux distributions will freely cache disk read/writes to memory if there is ample space, and will automatically start freeing it up once you start running low. For a deeper breakdown, here’s a great article: Linux Ate my RAM!

So:

import { promises as fs } from 'fs' 

await fs.writeFile(...)

…wasn’t writing to disk but straight to memory. Specifically for our Address Verification Service, we load nearly 4 gigabytes of files into memory at runtime in order to operate.

Now this helped us identify and resolve the problem in our Address Verification service: we were improperly using cached Docker builds. Once we pushed a fix for that, our memory readings went back to normal. For those that are confused why that might be the case, the details need to be kept in house, but in short, if our assets had been properly cached as a docker layer, the 4 GB of data in that case would be counted as part of the image size and would be part of Docker’s storage driver. That way it would no longer be using memory to store those files and instead would properly be a part of the disk space.

What you need to know about hapi file management

With one service fixed, I just had to fix the memory problem in our core API. To understand how I fixed it, I think it’s relevant to provide some background into hapi.js. We use hapi as our main framework for our api and are currently running v20. As a company that deals heavily with turning assets into printable mail pieces we needed a framework that makes it easy to work with parsing payloads.

The main options of dealing with payloads that Hapi provides are:

Data
Streams
Files

The first option, Data, is fairly straightforward: if you pass a JSON body into a Post request it’ll parse it into JSON data. Streams will do the same thing, but if you get a multipart upload (i.e., a file is uploaded), it’ll read that file into a stream without any additional tooling needed. The third is similar, but it’ll write a file to a specified directory rather than give you a stream:

‘File’: the incoming payload is written to a temporary file in the directory specified by the uploads settings. If the payload is 'multipart/form-data' and parse is true, field values are presented as text while files are saved to disk. Note that it is the sole responsibility of the application to clean up the files generated by the framework. This can be done by keeping track of which files are used (e.g. using the request.app object), and listening to the server 'response' event to perform cleanup.

Now there’s two things that developers should be aware of:

If you’re running a Docker container with no storage method, hapi will write your file to memory. This goes back to what I mentioned earlier. It’s quite intuitive but you cannot go with an assumption that disk storage actually exists with how popular services like s3 are now and how volumes are built in by default.
The second is, that it is the sole responsibility of the application to clean up these files. Unfortunately we missed this line in the docs and hadn’t actually been cleaning up those files. Now this in of itself isn’t the biggest deal breaker. There were several endpoints that were failing to delete these files after we had performed a major refactor, so I quickly cleaned those up. Immediately after those changes, our containers were showing a significant decrease in memory leaking but it was still leaking. Our containers were now crashing every 3 days rather than every 3 hours. This led me to have a final discovery…

The Hapi.js request lifecycle doesn’t make a lot of sense

While the hapi docs do let us know that we should be deleting assets, they fail to raise a pretty important concern in that it’s super important to listen to a server response event rather than manually handling file deletion within your normal code paths. The huge issue is that payload parsing occurs first in the hapi lifecycle before almost anything else—stuff like authentication and request validation happen after a payload is parsed—creating somewhat of a vulnerability if you’re not careful.

Say a malicious actor, let’s call him Bob, had a personal vendetta against Lob. They could continually try to hit our endpoints with ill-formatted requests, along with a massive 200 mb asset attached. That request payload would be parsed and the file would be written to memory. Now we’re smart enough to create some validators that will check any files attached to make sure they’re less than a certain size, let’s say 20 mb in this case. That same request would certainly fail payload validation and then we would send a 422 response. But that 200 mb of data is now sitting in memory, and with a couple more requests it would start causing our containers to run out of memory. So rather than making sure these files were deleted our normal code path and controllers, we had to make sure they were being deleted regardless of whether they made it that far in the pipeline.

All it took was a little chunk of code to clean up response payloads right before a response.

Say we’re handling a request like this:

/v1/upload

{

“Upload_name”: “Testing”,

“file_we_want_to_upload”: “/tmp/file.html”

}

We need a response handler that’ll delete that file right before sending a response regardless of whether it was successful or not.


module.exports = {

 name: 'cleanUpAssets',

 register: async function (server) {

   server.events.on('response', async function (request) {

     rimraf(request.payload.file_we_want_to_upload) }

};

The end of the rainbow

And with these changes our memory was no longer spiking! The best part? Even after we shrunk our container memory limit by 1/4th, we were still seeing stable memory readings Celebration ensued! Now the emojis were of the :partyporg: and :dancingpickle: variety.

This reduction in cpu and memory is likely to result in AWS savings from right sizing containers; there may even be future savings when we complete our migration to Nomad.

Engineers may occasionally get called stubborn, but in the Case of the Mysterious Memory Leak, tenacity paid off. In leveling up our Docker and hapi knowledge, we uncovered a legacy bug and a potential vulnerability, and the changes we made resulted in a more stable, secure, and cost-effective operation.