Forem: John Bintz

The ouroboros manifest

John Bintz — Thu, 29 Oct 2020 16:36:05 +0000

“Oh come on...not again!”

Norbert’s mug starts overflowing with coffee. He rushes over to the sink to grab the Coffee Towel. Jenell watches him over her own cup of coffee, sipping gently.

“Ugh, this stupid machine! When are they gonna get this fixed?” After wiping down the machine and mug, Norbert carefully takes a sip, wincing.

“Let it cool down a bit, Norb,” Jenell suggests.

Norbert takes another sip, wincing again. “No, it’s just...they need to clean the lines. It’s nasty.” He gulps down a bit more, then reaches for the French vanilla creamer sitting nearby.

The office is quiet, except for a lone mechanical keyboard clicking away and the whirr of the HVAC, which building management allowed them to run for a bit longer than normal.

“Norbert, if it’s that bad, toss it and start over.” Jenell chugs the rest of her coffee without complaint. “Or, lower your coffee standards. Our servers need caffeine-fueled humans to ensure they’re configured correctly, and us devops folks can’t wait around for Cocoa Connection to open to get quad-shot skinny mochas.”

Norbert puts the creamer away. “Look, if I’m gonna get wired, I’m gonna do it right. Your little YAML configs can handle crappy coffee, Jenell. My code enjoys the finer things.” He grins.

This type of response used to bother Jenell a lot, but now she smiles and shakes her head. “Bad coffee or not, this feature needs to ship so we can all go home.”

Norbert and Jenell walk up to the clicking. Surrounded by cans of diet soda and packs of cheese crackers, headphones on and glowing green, Domencia is completely focused. She’s illuminated by the blue glow of her laptop screen and the rainbow backlight on her keyboard. Peeking at her screen, Jenell can see that Domencia has almost cycled through one of her “focused work” playlists

Norbert looks quizzically over Domencia’s shoulder, thinking. Jenell leans back against an opposing desk. “So, you learning a lot from Domencia?”

Norbert watches the code fly by. Code and tests. Norbert leans back next to Jenell. “She’s not even being sloppy about it. Well, not this time, at least. I think I learned more from reviewing one pull request from her than I did my entire first semester in college.”

Domencia stops typing suddenly, takes off her headphones, and stretches her arms up. “Well, that stupid feature request is done. Sorry, Norb, all you get tonight is a code review.” Domencia swivels and looks at Norbert. “Here’s another lesson—if a customer promises another customer that a third customer can do a thing with the project you’re working on, and they give you a week to do it, just, like, delete the whole thing. It’s idiots all the way down from that point.”

Norbert chuckles. “Yeah, that product meeting was rough. You could see ’em sweating when they asked if it could be done.”

Domencia stands, swiveling her neck around. It cracks a few times. “Well, the review is up. I found something else I think we can tackle tonight while our esteemed devops liaison is gracing us with her presence.”

Jenell bows. “Here to serve, O Great Developers.”

Domencia sits back down and motions Norbert over. “Here, look,” she says, pointing to an open file on the screen.

“Whoa, the version of mochurl we’re using is ancient,” Norbert exclaims. The beeping of a snow plow and the flashing yellow light through the windows punctuate his statement. “I’m surprised this hunk of junk could even function.”

“Sounds like the snow’s starting,” Jenell says, looking into her mug. “Let’s get going soon, before we get stuck here.”

Domencia points at a few other lines. “We should be able to upgrade to the latest version and maybe drag along some other deps, too.” She taps at her keyboard and a browser appears. “Look, the mochurl API seems to be the same. Worst case scenario we tweak a few tests and we’re good.”

Jenell taps her mug. “Well, I’m all about reducing risk. I don’t want to sleep under my desk, though, so if it’s too big, delay it until Monday.”

Domencia salutes. “YES, SIR, I will execute to the best of my abilities.” Jenell rolls her eyes, shakes her head, then heads over to the coffee maker for her next cup.

Norbert pulls up a chair. “It’s been really interesting working on this legacy project. So many issues I never had to deal with at the last place.”

“Only working on greenfield projects makes you soft, Norb,” Domencia responds. “Real coders are forged in the hells of nine year old legacy apps, like this thing.” She types a little more, then purposely hits the Enter key on her keyboard just a little harder. “Looks like a few binaries need to be rebuilt. Time to relax a bit.”

Jenell returns with a fresh, steaming cup. “Yeah, you see all sorts of weirdness with legacy apps. I once worked on a project that had a very...peculiar...deploy process.” She sips. “I was nursing a Halloween candy hangover in 2015 when I got a page at 2 a.m. A developer decided to get his late night code on and did a deploy, and, like, everything exploded. Turns out we were using some tool in the chain that had no concept of Daylight Saving Time, and…”

Domencia pfffts. “DST is a jerk. I’d move to Arizona if it means never dealing with it—” She motions to the window, now full of white flakes. “—or with that.”

Domencia’s laptop beeps. She gives it a quizzical look. “Hrmm…”

“What is it, Domencia?” Norbert leans in closer.

“I can upgrade mochurl, but there’s a transitive dependency that I can’t upgrade due to conflicting requirements…” Domencia pokes at her keyboard. “There, we were using it, too, and now I’ve changed out requirements to match the dep’s. Again!” She slams the Enter key once more.

Norbert and Domencia watch the terminal output closely. After a few moments, the laptop beeps. “...again?” Norbert asks.

“Ugh, it’s etheda this time.” Domencia types some more. “OK, we’ll upgrade that to the latest version, too…” Another Enter key, another beep.

“WHAT?!?” Domencia cries out. “Are you KIDDING ME?” Jenell walks over, curious. “What’s up?”

“etheda requires...a version of mochurl that’s lower than the latest. But I guess the new mochurl requires a version of globench that is higher...but…it’s like a big, nasty cycle of dependency requirement mess.”

Domencia cracks her knuckles. “I got this. You know how git bisect works, right, Norb?”

“Kinda...I read a blog post, but I haven’t tried it yet.”

Jenell pulls back. “Domencia, the snow’s getting bad, let’s save this for Monday.”

“Nope, I can do this. Go indent some YAML or something and let me work.” Domencia puts her headphones on, starts a new album playing, and begins typing. Her headphones start glowing green.

Norbert looks at Domencia’s laptop, then outside at the increasing snowfall. “A John Carpenter anthology album seems like a bad choice right now…”

Domencia types furiously, changing version requirements in larger, then smaller, increments, backing out of changes, modifying tests, modifying code. Norbert and Jenell stand back, watching. The glow of Domencia’s rainbow keyboard starts to take on a red tint. The lights on Domencia’s headphones start to flicker, then turn from gamer green to solid red.

“I didn’t know they could do that…” Jenell remarks. The beeping of the plow starts moving past the window again—

And disappears. The yellow flashing light stops, too. The coffee machine starts brewing, unattended. “What the —” Norbert exclains. He starts to walk over and Jenell grabs his shoulder.

“Norbert…” She’s staring at Domencia.

Norbert looks.

Domencia, chair, keyboard, and laptop are all hovering an inch off of the ground. Her screen has taken on the same red tinge as her keyboard and headphones. Domencia types furiously, muttering, “...bump this point release, reduce and narrow down minor version…”

Domencia’s chair starts...bending. Bending in a way it shouldn’t be. So is Domencia.

“Norbert,” Jenell asks quietly, “does it look like Domencia’s laptop is...eating her?”

Domencia, chair, and laptop float away from the desk, and the bending becomes more prominent. The top of Domencia’s head is almost touching the top of her laptop, and her feet bend unnaturally toward the bottom of the computer.

Norbert gasps. “From both ends! She’s trying to fix this problem, and it’s just eating itself, and her!”

Domencia continues typing mindlessly, turning into a developer croissant, oblivious to what’s happening. “...fork this, change requirements, fork adjacent project…”

Just as Domencia’s forehead and toes are about to touch the laptop, Norbert rushes over, shoving Domencia’s hands aside. He quickly types and, as confidently as Domencia, slams the Enter key.

Immediately, all the lighting returns to normal. The plow, seemingly unstuck in time, continues its rounds. Domencia and her equipment fall to the floor.

“...dude...I was in the zone,” Domencia exclaims, breathless. She looks at the laptop, askew on the ground, and gets irate. “You deleted the whole directory?! But all the changes I made! I almost—”

She looks around. The HVAC system cuts off, and everything becomes silent. She stares up at Norbert and Jenell.

“Look, I was doing this for you, Jenell. I just wanted to make your deploys of ouroboros a lot easier.” She stands up. “But it’s snowy and I’m...really tired for some reason.” She closes up her laptop, unplugs her keyboard, and grabs her backpack from under the desk.

As Domencia slides the keyboard into her bag, Norbert notices, out of the corner of his eye, the backlights briefly flicker red.

Every developer gets dragged to dependency hell at some point in their career. It’s inevitable, especially since we’ve found that up to 20% of commonly used dependencies are unmaintained. Preventing this painful journey is one issue we hope a managed open source subscription like Tidelift can help solve.

Dependency Hell is inevitable...and that’s OK (and you’re OK, too!)

John Bintz — Mon, 13 Apr 2020 14:35:18 +0000

At a recent meetup, a few of us developers got into a discussion about "dependency hell"—the condition when one or more pieces of software need two or more conflicting dependencies (usually transitive ones) installed in the same environment.

Almost every developer has run into some version of this: library A depends on version 1 of a piece of software, but library B depends on version 2. The latest video card driver works for one game, but fails on a different game, requiring a downgrade (until the vendor fixes the issue), the list goes on.

I started recanting a story about a time I ran into a dependency hell situation, and I randomly blurted out, "Everyone's going to get into dependency hell; don't be ashamed, it happens to everyone!"

We’re all going to (dependency) hell

You pinned a dependency for a reason and never got around to updating it. You forked a project to add some functionality and the upstream package moved merrily along without your changes. A native extension needs one version of a system library, and a second app on the system needs another. The road to dependency hell is paved with good intentions, those intentions being "get this thing working so we can achieve our organizational goals."

It's going to happen to you, it's going to rear its ugly head at the worst time, and it's going to be painful to escape.

Dependency hell is not technology specific either. I've run into it in the Ruby/Rails ecosystem, in the Clojure ecosystem, and in the NodeJS ecosystem. I know folks who have run into it in C++ and Python, too. You name the language, operating system, framework...it's going to happen.

The way out

I wish there was one simple answer, but the paths out of dependency hell lead all over the place:

Manually experimenting with (or git bisecting) the versions of dependencies in your manifest file until things get happy (how are those integration tests working for you? Time to strengthen them up a bit, maybe?)
Forking and fixing open source dependencies on your own repos, hopefully being able to push those changes back upstream into the main releases.
If, for some reason, the maintainer can't/won't accept the patch, you've got two choices—stick with this dependency, forever git rebaseing to keep it up to date with your changes; or saying YOLO and never touching this forked dependency again.
Finding or writing a replacement dependency, one that requires as little code change as possible (I ask again, how's that integration test suite?)

The technical aspects of fixing dependency hell are frustrating, difficult, and not fun. There's another aspect that you'll probably experience, only because your brain is trying its best to protect you from being kicked off the proverbial island.

Shame vs. guilt

Shame is a feeling associated with a negative evaluation of yourself. While there are several “flavors” of shame, the one we’re talking about here is the one that arises when you break a social norm. You want to keep in good graces with the community around you (your coworkers, boss, and company), so you’re compelled to believe yourself faulty in order to change your behavior and fix what went wrong. It's different from guilt, a feeling you get when you break a personal code of ethics. Shame is all about what you believe others think of you. For example:

You feel guilty when you declared that you'll keep the integration test suite running to catch dependency hell issues and you find that you've let it lapse (see, I kept asking you about it for a reason). Your body primes you to hide your mistake so you don't get caught, or you must muster the strength to fess up.
You feel shame when, upon finding this out, your team confronts you about the failures of the integration suite. You don't want to have to face looking like a fool in front of your coworkers, so your body primes you to either deflect and point the finger at others, or to own up to the fact you disappointed your colleagues and that you'll work to make it right.

While a little bit of shame in these situations is normal and unavoidable, it's when we are left swimming in it for long periods that it becomes a bad thing. Dwelling on what others may think of you and latching on to “what if” scenarios prevents you from working well with others, and with yourself, to get the issue resolved.

Finally, you're probably going to have some physical effects with shame—blushing, turning away from others, a quiet voice. It's just your brain and body doing what it decided was an appropriate response to this situation tens of thousands of years ago. It's perfectly natural.

Dependency hell is guaranteed to happen

We aim big at Tidelift. We want to make your dependencies as safe to combine in whatever way you can possibly want to combine them. We want to work with maintainers to make their projects as safe as possible for subscribers to combine together to make their amazing apps.

We at Tidelift, however, are not perfect (for shame!). Nor are the maintainers who become lifters. Nor are the maintainers who work on non-lifted dependencies. Nor is the universe.

There will eventually be a combination of factors that cause the myriad of dependencies in your application to get into gridlock. You and your team will be stuck in dependency hell, quite possibly through no fault of your own. And it's gonna suck, and it's gonna take time to unravel. At that point, acceptance of the situation and overcoming your shame are the best approach to get a project, a team, and yourself, through the gridlock in one piece.

Acceptance for managers

If you are a manager, the worst thing you can do when dependency hell rears its head is start pointing fingers. While a post-mortem to determine what happened and how to hopefully prevent it in the future is both useful and essential, and while a little bit of pressure to get a problem fixed isn't a bad thing (a 100% stress-free job can really prevent growth), nothing demotivates an already demotivated team better than publicly shaming them (or an individual) for a problem that they might have not even been able to predict.

A compassionate-yet-stern approach where the whole team (including yourself) is responsible for fixing it will accomplish much more, and can help turn an awful experience into something that, in a weird way, could be looked back upon fondly months or years from now.

Acceptance for teams

The right approach to facing a dependency hell challenge is to take an attitude akin to “welcome to the firefight,” where your combined goal is to fix this problem as efficiently and comprehensively as you can so you can get back to the business of delivering features. Everyone will take their piece of the puzzle, whether it be improving test suites, trying different version combinations, or rewriting code to work around dependency issues, and you all will bring it together into a stable fix.

Much like the advice for managers, the last thing you should do is angrily point the finger at a teammate to shame and blame. Yes, someone might have introduced an issue for the reasons listed above, but unless they were cackling like a mustached villain who tied your project's Git repo to some train tracks, the choice they made was probably the best they could make with the information they had at the time. Software development is always about tradeoffs, and you can never know the full effect of a choice made now months or a few years down the road. Be kind and understanding and you’ll get this situation fixed up a lot sooner.

Acceptance for you!

Finally, take care of yourself. An encounter with dependency hell is not the time to drive yourself into the ground. It is not the time for slamming tons of caffeine to keep yourself going beyond your limits. Eat well, take a break to exercise (even a walk around the building or block is useful), and rest when you can, because if you're tired, you'll make mistakes and start losing your temper, which will not get the problem solved any sooner.

I’ve experienced dependency hell many times in my career, and have learned, sometimes the hard way, that the best way forward is the one where accepting the situation for what it is—as inevitable—and working together as a team to resolve it and prevent it as best as possible in the future, is the one where everyone makes it through happy and healthy.