Forem: Joshua Wood

Aggressively part-time

Joshua Wood — Fri, 31 Mar 2023 02:43:02 +0000

I recently used the phrase "aggressively part-time" to describe the culture and work ethic at Honeybadger. We typically work on Honeybadger for 20-30 hours a week. To get things done, we have to be focused and efficient.

This approach allows us to maximize the time we spend on life outside of work.

I've been thinking about applying this concept to the rest of my life. Work is just one area where I desire progress. I want my relationships, interests, and hobbies to improve too.

In family life, I want to become a better partner to my wife and father to my children and to enjoy meaningful experiences together.

I want to expand my knowledge and improve my thinking to grow intellectually and solve problems.

I don't want to have hobbies—I want to be good at them. I want to develop skill and experience success in each area.

These areas encompass my life goals.

Each area demands focus to improve. But, like work, I can’t pursue everything at once—I need to prioritize.

Aggressively part-time leaves room for your other endeavors—whatever they may be.

What are your part-time pursuits? Leave a comment below!

openpgp proof

Joshua Wood — Fri, 02 Dec 2022 20:47:35 +0000

openpgp4fpr:4E5DC0D7B51CF6AB3664FCD83C185ED56E760F73

My notes from Aaron Patterson's RailsConf 2020 keynote

Joshua Wood — Thu, 07 May 2020 02:04:17 +0000

Aaron's talk was very Aaron: 20 minutes of jokes, 40 minutes of brain bending Rails performance show-and-tell. His goal: to teach you how to profile your Rails app. If you haven't watched his keynote yet, here's the gist:

https://www.joshuawood.net/notes/2020-railsconf-aaron-patterson-keynote

A few takeaways:

Not all code is equal
Similar behaviors should have similar performance
Poor performance is a bug

Miss Aaron's talk? In THIS economy?

👌

My notes from DHH's RailsConf keynote interview

Joshua Wood — Wed, 06 May 2020 21:31:34 +0000

For those of you who haven't had the chance to watch yet:

https://www.joshuawood.net/notes/2020-railsconf-dhh-keynote

One of the parts I liked best was this comment:

Ruby is great because you can know a little JavaScript and then jump to Ruby, understand it, and own the full stack.

What do y'all think about the state of Ruby and Rails in 2020?

Deploy a new Rails app to AWS Lambda using Lamby

Joshua Wood — Thu, 30 Apr 2020 16:42:44 +0000

These are my notes from deploying a boilerplate Rails app to AWS Lambda (yeah, you read that right) using Lamby. Basically this is me going through the Quick Start guide. (Here is my production endpoint--not sure how long this will be up.)

We made a community error dashboard for DEV

Joshua Wood — Tue, 17 Mar 2020 15:48:14 +0000

Software development is more fun with friends; that's why we've built tons of collaboration features into Honeybadger over the years, making it easier for teams to fix errors.

Earlier this year the team at DEV emailed us with a feature request: could we make it easier to involve the broader DEV open source community in the error-fixing process?

Today, we're excited to announce a new feature that does exactly that: Public Dashboards.

[Read on to learn more about public dashboards and how to enable one for your own Honeybadger project. If you want to help DEV squash bugs in Honeybadger, they wrote about how to get involved.]

A public dashboard is a secret URL that you can share with outside users (such as contractors and open-source contributors). When you use the share button (see below for details) from the error details page, that error appears on the public dashboard for your project—making it easy for everyone to discover new errors to fix.

Backstory on the share button

Most DevOps tools do not adapt well to community projects, where the team is continually changing. It's is a problem we've thought a lot about in the past: open-source benefits from anyone in the world being able to jump in and fix an error, but sometimes the needed production data is private.

To begin to address the problem, we added the ability to "share" an error with an outside user, scrubbed of all PII (Personally Identifiable Information, such as user IDs and session data). This single feature made it easy for core teams to share vital debugging information (such as error messages and stack traces) with community members without a ton of copy/pasting:

The Community Dashboard

DEV wanted us to take this a step further and create a public dashboard where the community could see existing production errors and contribute a fix. We loved the idea so much that we fast-tracked it and are launching it today.

Of course, not everyone wants a public dashboard for their Honeybadger project, so it's disabled by default. To enable it, head over to Settings -> Advanced -> Enable Public Dashboard, and flip the bit:

Feel free to send the unique dashboard link to just a few team members, or share it with the world.

Conclusion

We are stoked to have the opportunity to work with the DEV team on this, and can't wait to see what other open source communities do with it.

If you're involved in an open-source community that runs production applications, Honeybadger is for you. It's also free for non-commercial open-source projects. Sign up today!

I never deploy on Fridays, but when I do...

Joshua Wood — Fri, 21 Feb 2020 22:21:08 +0000

I know you're not supposed to. I may have been the one to tell you not to. And yet, sometimes I can't help myself... 😬

Anybody else deploying today? If so, why? What are you shipping that's so important?

I'll tell if you tell. :)

F*ck-That Money

Joshua Wood — Tue, 11 Feb 2020 15:22:08 +0000

I've met some hugely successful people. Some of them are happy. Some of them... aren't. I've noticed some things about the ones who aren't:

They're obsessed with their careers
They spend a lot of time at work
They aren't fulfilled by the work they do
They aren't content with what they have

In many circles, these people are way more successful than most of us. They probably have better networks. They might have fancier job titles. They certainly make more money.

The Silicon Valley meme goes, "start a company and sell it for so much money that you can say 'f*ck you' to anyone in the world." Or maybe an alternate career path is to climb the corporate ladder at Google until you've banked that privilege.

So much toxic bullshit stems from that line of thinking.

I'd rather have f*ck-that money—e.g., a moderate amount of money that empowers you to live a meaningful life. F*ck-that money isn't about being a pretentious asshole; it's about keeping your priorities straight. For example:

Miss your kid's soccer practice to catch up on emails? F*ck that!
Power through lunch? F*ck that!
Skip the gym? F*ck that!
Work over the weekend? F*ck that!
Save that side project for when you have more free time? F*ck that!
You get the idea...

At Honeybadger, this mindset has led us to turn down acquisition offers and investors—to stay small and grow naturally. We're not in this business to be rich; we're in it to be free.

Regret the best years of our lives?

Fuck that.

Btw, we're currently hiring a developer to join us.

Cold Showers

Joshua Wood — Sat, 08 Feb 2020 05:19:51 +0000

Next time you rinse off, do this:

Standing right under the showerhead, gradually increase the hot water. When you can’t handle it anymore, turn the temperature all the way to cold (and brace yourself). Wait to stop hyperventilating before turning the water off.

Do this every day. Not only is it refreshing, but it’s also a daily reminder that for many things, getting past the initial resistance is the hardest part. 🚿

Breadcrumbs for JavaScript in Honeybadger

Joshua Wood — Fri, 07 Feb 2020 01:00:20 +0000

One of the things that make fixing JavaScript errors so difficult is that everything happens on the client-side. When an obscure error happens in a callback, you often lack the context to reproduce it. If the error is critical, you may even resort to deploying debug code to get more information about the events leading up to it.

We added a feature to help, and it's called Breadcrumbs.

A Breadcrumb is a client-side event that happened before an error. Breadcrumbs are collected in realtime as users interact with your client-side application. When an error happens, the breadcrumbs leading up to it are included, filling in the missing context.

Since launching breadcrumbs for Ruby and Elixir last year, our customers have been fixing errors faster and with more confidence. Today we're excited to announce that breadcrumbs are available for JavaScript!

How Can You Use It?

Breadcrumbs are available as of honeybadger-js version 2.1, but disabled by default. To capture breadcrumbs, you must explicitly enable them. We plan to enable breadcrumbs by default in version 3.0.

To enable breadcrumbs in your project:

Update your honeybadger-js package to the latest 2.x version (2.1.1 at the time of this post)
Enable breadcrumbs in your honeybadger-js configuration:

  Honeybadger.configure({
    // ...
    breadcrumbsEnabled: true
  });

Automatic Breadcrumbs

Honeybadger captures the following breadcrumbs automatically by instrumenting browser features:

Clicks
Console logs
Errors
History/location changes
Network requests (XHR and fetch)

Sending Custom Breadcrumbs

No one knows your app better than you. In addition to the default events, you can report custom breadcrumbs to Honeybadger:

Honeybadger.addBreadcrumb('Loading User', {
  metadata: { user_name: userName }
});

When an error is subsequently reported, you should see it in the Breadcrumb stack:

You can also customize the category of custom events that are displayed. For
more information, check out the guide in the Honeybadger
docs.

Let us know how it goes!

We hope that Breadcrumbs will be a helpful addition to your JavaScript toolbox. Try it out, and give us a shout if there is anything you would like to see added.

What if I called FLUSHALL on your Redis instance? 😱

Joshua Wood — Mon, 05 Aug 2019 19:11:53 +0000

At Honeybadger, we use Redis a lot. It's our Swiss Army Knife; it's a cache, a single source of truth, it stores background jobs, and more. Basically, Redis is one of those services that should never fail.

I was pondering the DevOps apocalypse recently, as one does (could Redis be one of the four horsemen?), which led me to jump into our #ops channel to ask Ben a simple question:

josh [13:58]

what are the risks if someone executed flushall on our redis instances?

Ben [13:58]

you just gave me a micro heart attack

josh [13:59]

rofl

Sorry :)

I didn’t do it, for the record

Ben [14:00]

all kinds of badness would happen... the job queues would be flushed, there would be a potential for duplicate notifications, timeline charts would start hitting ES instead of being cached, and more :)

it would make for a very bad day :)

Don't worry, Ben recovered after a few hours, and is mostly back to his old self again. I should have prefaced my question; I didn't mean to suggest that I'd actually flushed our Redis cluster. Still, it kind of proved my point. Maybe trolling your SREs and measuring their sweat is a good way to plan for catastrophes...

SO. I'd identified a potential issue. It would be really bad if Redis got flushed. What are the risks of that happening?

Our Redis clusters are deployed with primary and secondary instances across multiple availability zones in AWS, with automatic failover to the secondary in case of primary failure. That's a pretty rock-solid Redis deployment; we can lose entire instances without losing data or even impacting other services.

Unfortunately, preventing human error is much harder, and for some reason, Redis makes it dead-simple to delete all your data with a single command entered in the wrong console. Our architecture did not guard against that. While our team is aware of this, there's a pretty good chance that a future developer could make this mistake.

In fact, my friend Molly Struve, Sr. SRE at Kenna Security, remembers a situation where something similar happened:

We made a change to some code that caused the old cache values in Redis to break with the new code. So we would request the old value and it was not what the code was expecting. Rather than rollback the code, one of our engineers thought it would be fine to run Rails.cache.data.flushdb and just start with a fresh cache.

Like Honeybadger, Kenna uses Redis for several things—one is the cache-backend for their Ruby on Rails application. The command Molly mentioned, Rails.cache.data.flushdb, is the Ruby on Rails equivalent of opening up a Redis console and calling FLUSHDB (which deletes all data in the current database).

Unfortunately, Redis was also being used to cache report data from Elasticsearch (something that we also do at Honeybadger, incidentally), and that's where things went wrong. When the Redis database was flushed, the cache had to be rebuilt from scratch, which overwhelmed their Elasticsearch cluster:

We have a "Dashboard" page where clients can load part of ALL their reports (think hundreds), and when clients started to hit that without the cache, Elasticsearch lit up like a Christmas Tree. CPU maxed out on all nodes across the board. In the end it was a mad scramble to open multiple consoles to re-cache the reports.

After Kenna's systems were restored, Molly worked with the development team to identify steps they could take to prevent the same thing from happening in the future. They came up with a creative safeguard for new developers who might not realize that clearing the Rails cache is a destructive action: they made all production application consoles read-only by default.

Luckily, Kenna's incident was not catastrophic—they were able to recover from it after just a little downtime. It would have been much worse had the unlucky developer accidentally called FLUSHALL—which flushes all Redis databases—instead of FLUSHDB. It would have been an easy mistake under pressure, especially when exception reports are already rolling in (did I mention they also use Honeybadger?)

Let me ask you a quick question: what would happen if someone called `FLUSHALL` on your Redis console?

If the answer is "all hell would break loose", then you might consider taking preventative action. Here's what we went with; of course, this is us (I'm more than a little paranoid)—your mileage may vary.

First, access to Redis through clients (i.e., in the Rails console) should disallow use of the FLUSHALL and FLUSHDB commands entirely. Developers never need to run these commands in production; doing so would cause serious problems, so why have them at all?

If you're a Ruby/Rails user, feel free to steal this gist. If you want something a bit more comprehensive, see Molly's gist. If you use a different programming language, hopefully there is a way to disable these commands, but even if there isn't, don't worry, I got you.

The Redis config that everyone should use

Mike Perham, the creator of Sidekiq (by far the most popular Ruby background job system, with an awesome business model), knows a thing or two about Redis. Sidekiq is built on top of Redis to provide an incredibly reliable and efficient job system.

I asked Mike what best-practices he recommends to his customers, many of which have mission-critical Redis deployments (think Netflix and Oracle). He told me that users who are concerned about the safety of their Redis data should disable destructive commands entirely via Redis's configuration file.

This approach has the added benefit that the commands are disabled everywhere, including in redis-cli consoles. The following config should be added to redis.conf:

rename-command FLUSHALL ""
rename-command FLUSHDB ""
rename-command CONFIG ""
rename-command SWAPDB ""

Renaming the above commands to empty strings means that they will no longer exist as Redis commands. If you still want to be able to call them in rare (and intentional) circumstances, you can rename them to something secret:

rename-command FLUSHALL SUDO_FLUSHALL_222ed15a
rename-command FLUSHDB SUDO_FLUSHDB_2a3bdd5e

For instance, you could put those commands in a doomsday ops playbook which only your operations team has access to. Treat them like your company's nuclear codes.

Of course, there's always a caveat 🤦

We use Amazon's ElastiCache service to host our Redis clusters, and after some research, I learned that ElastiCache does not provide direct access to redis.conf, and it doesn't provide a Redis configuration parameter for rename-command. So unfortunately, while our application consoles are safe, we still must handle redis-cli with care.

In the end, we added a note about this to our internal Redis playbook, and will revisit the ElastiCache documentation occasionally to see if Amazon gives us access to rename-command.

Failures are inevitable

If it were possible to prevent 100% of failures before they occur, our jobs would be much easier. We wouldn't need on-call rotations or postmortems, and we could all code full-time. Unfortunately, we live in the real world, where chaos rules, and entropy ensures that our systems are constantly deteriorating.

There's risk inherent in everything we do. To ship stable applications, we should take actions which minimize risk. In doing so, we reduce (but not eliminate) the potential for failures.

Being able to evaluate the risks associated with your actions dramatically increases your value as a developer.

Molly's story and others convinced me that the same thing could easily happen to us; the (perceived) risk was high. The solution—disabling potentially destructive commands, or making them extremely difficult to execute—was relatively easy.

There's a name for the combination of high value and minimum effort: low-hanging fruit. In the context of software, it's the idea that if you can gain a lot by making a small change, it's probably worth doing. This felt like it was in the sweet spot of eliminating a big risk for a small amount of effort. Kind of like the first time I installed Honeybadger... ;)

This story was adapted from an email I recently sent to our community newsletter, Leveling Up. If you liked it, feel free to subscribe.

Ship it like Honeybadger

Joshua Wood — Tue, 05 Feb 2019 18:40:56 +0000

A long time ago, my friends and I were in trouble. An app we used almost every day to debug software for our clients had basically stopped working, and it was giving us all headaches. There wasn't really a viable alternative on the market at the time, so we decided we'd build one. Three months later, our app—Honeybadger—was launched.

Over the next 5 years, we'd build Honeybadger into something we were really proud of, used by thousands of developers to debug exceptions and monitor their web apps for errors. We also built a company that we really wanted to work at, turning down a tempting acquisition offer at one point.

We did all of this on our own, with no funding and just 3 developers.

In case you don't believe me, here's a pic from a recent company meetup (we're all remote). If you're wondering why we're wearing blue, it was for Autism Awareness Month:

👋 Sup, I'm the fuzzy one on the left.

When I tell other devs how small our company is, I usually get the same reaction: "No way, I thought you were like 20 people" (one time someone said 100 🤔), "I would love to [do my own awesome thing] someday".

My response is always the same: if we did it, so can you.

Building Honeybadger was a lot of work—and we got a bit lucky with our timing, tbh—but we're not special. In fact, we're pretty average (I didn't even finish college 😬).

I can think of a few things we did right with Honeybadger, but there's one very basic thing we did that I blame most of our success on: we shipped a scrappy version 1.

It wasn't perfect, and it didn't do all the things our customers love us for today. That didn't matter, though, because it was enough to make us happy, and as it turns out, there were other devs like us out there. Had we wasted time trying to make it perfect, it would have never taken off, and I wouldn't be here to tell you that—whatever it may be—you can ship it too.

Shipping doesn't mean you have to start your own app (although you certainly can). You may want to write blog posts, record screen casts, contribute to open source, or just kick ass at your job/career. You can do all of these things—the fundamentals are the same.

In hindsight, it sounds too simple... "oh yeah, just ship it."

I know it's harder than that. There are all the times you don't start, because you don't know how, or don't believe you can. There's the paralyzing fear that drops in every time you think about finally executing on your plans. There's the cynical voice that whispers "this will never work" whenever you're nearing the finish line.

If you add up all of these excuses, you get what Seth Godin calls "the resistance." The 10,000 foot view is that most of what you know is holding you back, keeping you from creating your best work. If you want to ship, you need to overcome the resistance.

You can beat the resistance through small, iterative changes—it's time to level up.

No matter where you are in your career, there's always a next level. Building software is easier when you make small changes over time (nobody wants a big rewrite). Habits work the same way. The more things you ship, the easier shipping becomes. Why not start today?

Pick the smallest thing you can do right now, and then do it. Send a pull request. Share something you know with someone else. Investigate a new tool or technology. Read Linchpin, by Seth Godin (the book I mentioned earlier—it's good).

Get in the habit of shipping small things; when the big one arrives, you'll ship that too. 🚀

If you found this story inspirational, reply and let me know a) what you're shipping today, and b) what you'd like to ship in the future. There are no wrong answers. 😄