Forem: David Morgenthaler

Reflections: Culture as a Competitive Wedge in DevOps, with Google & Autodesk

David Morgenthaler — Tue, 01 Mar 2022 19:02:07 +0000

Armory’s Stephen Atwell sat down recently with Brandon Leach of Autodesk and Dave Stanke of Google for a roundtable interview. During their far-reaching discussion, they explore various ways in which company and team culture can make or break modern DevOps teams. I listened to the conversation, and a handful of takeaways struck me as elegant in their simple, yet genuine power to contribute to a culture of success.

Seek to learn, not blame: seemingly small details in team culture can actually ripple out to have massive impacts on technical teams;
Automate, then let go: identify and eliminate slow, outmoded manual processes, both in the development and deployment phases in order to build and take advantage of new systems that automate and accelerate workflows;
Cross-team collaboration: Generative culture can often be found in mutually inclusive teams who are invested in each others’ success, and who unlock value by breaking down barriers and tackling common challenges via mutually developed systems and processes;
Be a booster, not a laggard: it might seem obvious, but you can easily improve team morale (and your reputation) by showing with your code and yourself early, consistently; and
Keep a beginner’s mind: those who think they know less about a topic—but throw themselves at learning and especially doing something about it—often become perceived by their peers as experts… potentially even more so than those who call themselves experts.

I hope you find this talk to be as impactful I did, and welcome your questions or comments. See you at the next one!

The post Reflections: Culture as a Competitive Wedge in DevOps, with Google & Autodesk appeared first on Armory.

Cultural aspects of creating a highly evolved DevOps organization

David Morgenthaler — Tue, 02 Nov 2021 17:00:03 +0000

Join Armory for a roundtable discussion with DevOps leaders to examine the impact that company and team culture have on DevOps adoption rates.

Following in the footsteps of Puppet’s 2021 State of DevOps report, Armory’s Thomas McGonagle sat down for a discussion with

• James Bland Global Tech Lead — DevOps, AWS,

• Margaret Francis, COO & President, Armory, and

• Nigel Kersten, Field CTO, Puppet

to examine the myriad factors that affect a company’s success in adopting DevOps. This talk was born from some of the key findings of Puppet’s report. In light of Puppet’s finding that “cultural blockers are keeping mid-[DevOps-]evolution firms stuck in the middle,” we wanted to get out a magnifying glass to better understand the cultural differences that make certain companies great and keep the rest from reaching peak DevOps adoption.

We hope you enjoy this topic, which covers a wide range of topics, including team organization principles, securing executive buy-in, diversity & inclusion, and various tips from each of the speakers’ companies. Enjoy and please engage with us to share your thoughts!

WATCH NOW

The post Cultural aspects of creating a highly evolved DevOps organization appeared first on Armory.

Continuous Delivery is no longer optional

David Morgenthaler — Wed, 08 Sep 2021 02:07:05 +0000

The landscape has changed

Going into the pandemic, the global market was already moving rapidly towards digital transformation, due to universally rising demand for elegant and effective digital solutions. Peaking in 2017, the last six years have seen the most mergers and acquisitions in history; 2021 is set to be another banner year. The last 18 months have created an unprecedented tidal wave of demand for digital services. As McKinsey’s Global Managing Director said, “We have seen 10 years of digital innovation in roughly three months.”

If the pandemic has taught us anything, digital transformation is now mandatory for any business that wants to survive. Given the hockey-sticking demand for digital services and falling tolerance for “the pandemic” as an excuse for poor service, the need to create engaging, stable, and secure, and scalable experiences at speed has never been higher. So how can companies incorporate excellence into their software development life cycle?

What is continuous delivery & why does it matter?

Prior to the cloud, companies had to deliver code into their data centers; in a way, each company had to reinvent the wheel to deliver software. Continuous delivery (CD) allows organizations to codify and automate otherwise manual practices. According to Jez Humble, CD fosters low risk releases, faster time to market, higher quality, lower costs, better products, and happier teams. Given the clear benefits, how widespread is adoption?

Despite increasing automation, adoption of CD is slow

As of 2021, roughly 70% of companies have committed to ramping up automation. However, the adoption rate of CD principles and technologies remains relatively slow. Out of the 19k developers surveyed for the 2021 State of Continuous Delivery report:

nearly ⅔ reported that code takes ≥ a week to go from commit to production,
69.1% pushed to production ≥ 1x/week, and
over ½ take ≥ a day to fix an unplanned outage.

Why do so many companies take so long to push new features, breakfixes, and ship code? Says Frederic Rivain, CTO of Dashlane, “[CD] requires deep engineering practices around code, QA, and operations,” not to mention, the right tools; in short, it’s difficult and expensive.

The costs of not adopting CD are untenable

Autodesk Software Architect Brandon Leach said, “There [are] simply too many manual steps and decisions that need... to happen to launch a service into a new region.” Summing up the challenge elegantly, Sr. Director of Engineering, Ben Cochran, said, “Making our products and services available in new regions takes months. It needs to take minutes."

Manual processes slow deployment speed and reliability and often translate to unwieldy scheduled releases. Edith Harbaugh, CEO of LaunchDarkly called this way of releasing code “the push and pray release,” so named for the apparently divine luck involved in the outcome. “If you move slowly,” she continued, “it's... more likely you’ll break something.”

The costs of breakage are not strictly limited to reputational harm. A 2015 study by AppDynamics uncovered that “A critical application failure costs a staggering $500,000 to $1M per hour [and] downtime costs the Fortune 1000 between $1.25 billion and $2.5 billion every year.” Yet six years later, it still takes nearly 20% of companies over a month to restore services after an unplanned outage.

It’s not too late to act

Autodesk Senior Software Architect Shrirang Moghe had this advice: “align with the best early on… don’t waste time… Amazon and Armory, they’re best in class.” The cost of not adopting CD is rising, and the delta in outcomes between those who do and don’t adopt it will diverge at pace.

In short, for companies that wish to thrive in the future, adopting continuous delivery is no longer optional.

Heroku is to IaaS as Apples are to Apple Seeds

David Morgenthaler — Mon, 19 Oct 2020 20:17:14 +0000

A Hard Look at the Total Cost of Ownership of Heroku vs. IaaS Platforms

Executive Summary: This began as an email to a customer who wrote in asking Heroku to lower its prices to counter an offer of free credits from AWS. Over time, this email has morphed from a retention email into an detailed examination of the ways in which Heroku allows companies to laser-focus on innovation and business value — the number one thing that makes and retains customers — instead of on keeping the lights on. Heroku’s best-in-class Developer Experience — and the freedom that comes from abstracting away a massive amount of operational complexity — is exactly why we do not compete directly on price with IaaS providers. Please read on through this open letter, and feel free to reach out with any questions or (constructive) criticism.

Introduction

Hi Developer/Engineer/CTO,

I’ve put together what I hope is a helpful — if perhaps overly thorough — rubric by which you can compare Heroku against IaaS. It seems there can be a perception that Heroku is expensive when compared with IaaS. However, my thoughts below have been collected from > two years of witnessing businesses grapple with this deceptively challenging question: “Which IaaS/PaaS should I trust to host and scale my core service?” My answer to that question here primarily references AWS, however, the comparison is also still quite apt to Azure, GCP, and others. For the sake of laying out my case, I’ve structured my case into four sections, which are as follows:

I. Comparing Heroku to AWS

II. How do you address additional DevOps burden/complexity if you decide to leave Heroku?

III. How have things gone for customers who’ve left Heroku for AWS?

IV. So, what to DO?

Before I continue, I should admit that I obviously have a bias towards the platform that is my employer; however, I hope you’ll trust that I’ve strived to base my opinion here on unbiased 3rd party sources to help show why a move to an ostensibly “cheaper” IaaS provider is almost never as simple as it seems. For that reason, TL;DR: Heroku unit prices for customers paying on our monthly credit card billing model are fixed; however, I outline some ways you can take advantage of 3rd party services while still benefitting from Heroku’s market-leading DevEx in 3rd and 4th sections below. Finally, while this piece will read most cohesively from top to bottom, if you think of yourself as clearly within the target demographic and are short on time, feel free to skip to section IV.

I. Comparing Heroku to AWS

As you may know, Heroku is built as a layer on top of AWS, so there has never been a time in Heroku’s history when we have been—or would want to be—in direct competition with them. Due to the substantial value that Heroku adds for developers and businesses — by abstracting away the true complexity of a direct deploy to AWS, comparing us (and our prices) to AWS is less like comparing apples to oranges and more like comparing apples to apple seeds. To put it mildly, the work that it would take simply to get an app or database—let alone an entire CI/CD pipeline—on any of the major IaaS providers to reach parity with what Heroku provides out of the box is massive, and is perhaps best illustrated by the maroon part of the graph in this image.

Image credit: Guru99: https://www.guru99.com/heroku-vs-aws.html

Comparing Heroku to AWS in terms of raw, surface-level costs completely ignores the value that we have built into our battle-hardened, reliable, durable, and secure platform. Over the last several years, we’ve been very public about the work we do to improve our platform’s compliance with International Standards and to address critical security gaps. Here are some posts from the Heroku blog covering ways in which we’ve handled security and compliance issues that you will need to address yourself on IaaS:

Why Frequent Maintenances Are Essential for Secure Heroku Data Services, from July 2019
Announcing ISO 27001, 27017, 27018 Certification and SOC2 Type I Attestation, from August 2018
Heroku Behind the Curtain: Patching the glibc Security Hole, from March 2016 (or check out our response to Meltdown/Spectre in 2018 and Heartbleed from 2014)

As you can hopefully see from the above, when you decide to run directly on IaaS, dealing with vulnerabilities and bringing your apps into compliance with internationally accepted standards falls to you. As a scary reminder of how frequently businesses forget this fact, Palo Alto Networks’ Unit42 recently found over 34 MILLION (!) preventable vulnerabilities in apps deployed across AWS, GCP, and Azure. Their key takeaway is “the threats are not the result of cloud providers themselves but the applications customers deploy on cloud infrastructure.” Read SiliconAngle’s summary here.

Moreover, the simplicity of continuous integration and delivery (CI/CD) on Heroku is hard to overstate. I work with many developers who routinely tell me that Heroku pipelines — and especially Review Apps—are one of their favorite parts of our platform because they allow them to easily share their PRs with non-technical stakeholders to solicit feedback prior to merging into master. When they’re ready to merge, click a button to promote to staging or prod, and on to the next. While a variety of IaaS providers have been gaining ground, none of them can boast the same “Apple of App Dev” DX (<—real quote from a customer) that we offer. For more about why that is and why that’s important, check out Heroku Master Technical Architect Greg Nokes’ terrific overview video from earlier this month here.

Greg Nokes, Distinguished Technical Architect at Salesforce Heroku, is often asked how he describes Heroku. Here’s his take. What is Heroku? What is the difference between Heroku and IaaS? What is Heroku’s value?

For further unbiased reading, check out these 3rd party assessments of AWS vs. Heroku.

II. How do you address additional DevOps burden/complexity if you decide to leave Heroku?

As a wide variety of blog posts have illustrated over the years, Heroku’s main value add is that we abstract away the complexity of managing your infrastructure yourself. Put another way, while some larger Heroku customers use us even with full-time DevOps on staff, we significantly reduce or delay the need to hire DevOps, particularly for small companies. The flip side to that, of course, is if you decide to go straight to IaaS, operational complexity increases significantly, and usually, you’ll need to hire at full-time DevOps engineer.

Let’s say you do hire a DevOps Engineer. The average DevOps Engineer Salary in the US is $131k per year according to Neuvoo. Glassdoor says it’s $99k. Either way — even if you are spending that much on Heroku each year, do you anticipate that a single DevOps engineer could do everything for your org that I’ve described above, with a nearly flawless track record?

Perhaps you have a family friend in Turkey — as one of my earliest customers did — who is pro DevOps and works for pennies on the dollar. Even then, quickly Googling “Average DevOps Salary [[insert your geography here]]” will not address a crucial factor: the cost to actually recruit, train, and retain good DevOps talent. While I don’t have specific $ values for that, other folks have already written extensively about the costs associated. Take a look at these two excellent blog posts discussing the costs to find and keep top-tier DevOps engineers:

“Well, we have a pretty lean startup. We’re just going to do DevOps ourselves!” I’ve heard many developers say this over the years, and of course, it’s an option. Yes, technically, you can skip hiring DevOps altogether and assign the added workload to your existing team. However, even after assuming that someone on your team has the requisite knowledge to effectively take on pro-level DevOps, many companies have small technical teams, and as such, each team member’s time is at a premium. So with that in mind, being lean and agile is exactly why you need to ask yourself the following question:

“What is the opportunity cost of taking time away from our existing day-to-day activities to focus on operational tasks that don’t contribute towards our core differentiators to our customers?”

Put another way, will your customers benefit more from an operational change that requires you to spend more of your time “wearing the pager”, so to speak? Or will they benefit more—and hopefully, stick around to keep paying you longer—if you instead take that time to iterate on and improve your core product or service?

III. How have things gone for customers who’ve left Heroku for AWS?

Candidly, this is far from the first time I’ve had to make this case. In August 2018, I had a company reach out, asking me to discount Heroku based on an offer of $40k in credits from AWS. In response, I wrote to him almost exactly what I’ve written here. Experience being the best teacher, he decided to take the credits and migrate to AWS anyway. In February 2019 (about 6 months later), I received this response from him:

Just an update on this. We attempted the migration to AWS and… we failed. It turns out that Elastic Beanstalk isn’t right at the same level as Heroku in terms of PaaS features. Congrats on you guys [sic], as it shows that you’ve built a really good product over the years.

So I just wanted to let you know that we’ll be with you guys for the time being. It turns out you were right.

-Anonymous Heroku Customer, Feb 2019

This customer is now back and is happily running on Heroku today. You can also read about one of our other customers, Connect Space, who went on the record and wrote a full blog post about exactly this topic a year or so ago. Check out their story here.

Here’s another example from Bryan Woods, CTO of Rhino:

AWS gave us a ton of free credits to use, like tons of money. And I thought as a cost saving measure, maybe it would make a lot of sense to move our databases out to AWS just to save like 100 bucks a month on hosting. That ended up being a fool’s errand because once we got established, we lost those startup credits and then we had to do a project to move those databases back into Heroku…

So [now, on Heroku] we have all these tools, we’ve built a data warehouse, we can now roll back to any arbitrary point in time. We have these automated backups, we have these great metrics dashboards… All this stuff that we would have otherwise had to invest time and resources, if not building our own, then tying all these third party services together, it would just be silly. We’re not an infrastructure business.

You can also listen to Bryan tell this story on episode 67 of our podcast Code[ish]: “Launching a Startup in a Regulated Industry"

This only covers three customers, so obviously I’m not presenting a scientific study. However, the issues that these businesses wrangled on IaaS are extremely common complaints amongst Heroku customers past and present. Most IaaS platforms are (on the surface, at least) endlessly customizable and as unfathomably vast in their power and capabilities as they are affordable in their prices. In reality, however, while you could certainly optimize your unit cost on apple pie if you grew your own apple trees from seeds, for most people, it’s just a whole lot simpler to go buy some apples.

The ASCII tree is representative of Heroku’s famous Zen-like simplicity. We make the most complex and arcane aspects of application development, deployment, and management easily approachable.

On a related note, my team doesn’t only deal with customers jumping from Heroku -> IaaS; we’ve also seen plenty come from IaaS -> Heroku because they’ve spent years dealing with the ever-increasing operational complexity of managing a fully IaaS-based architecture and simply don’t want to deal with it anymore. For a deeper dive into why, check out John Vester’s 4 part series:

Ultimately, the marginal utility of spending a ton of money and time doing SRE rarely becomes a competitive advantage in 2020, or at least not until you are routinely signing uptime SLAs with your customers. And well, let’s just say, Azure doesn’t guarantee better than 99.9% uptime and they just won the $10B JEDI contract… meanwhile, Heroku is showing 99.9985% (US) and 99.9999% (EU) uptime over the last 60 days.

https://status.heroku.com; Screenshot Thursday, October 8th, 10:26 AM UTC

So unless you can get in a time machine to 2002 and beat Benjamin Treynor Sloss to the punch by a year, chances are high that taking on this particular type of complexity in 2020 will neither create a core differentiator for your business, nor will it fill your garage with McLarens like it did his.

IV. So, what to DO?

Keep Going

A lot of cost bloat on Heroku can be solved with one word: Optimize! Relatively few customers seem to take advantage of seemingly basic cost optimization tools like auto-scaling. For more information, check out these add-ons.

FlightFormation for worker autoscaling and scheduling
Adept Scale for web dyno autoscaling
AutoIdle staging, dev, and review apps after 30 minutes of inactivity

For further questions about case-by-case cost optimization strategies, feel free free to reach out to our team here or check out the reams of info on StackOverflow.

Go Multi-Cloud

For what it’s worth, there are absolutely still ways that you can benefit from using cutting edge technologies from IaaS providers while simultaneously keeping your mission-critical app pipeline on Heroku. And if you have credits, use them! Instead of thinking that this has to be some sort of Heroku OR AWS zero-sum game, do what Heroku customers have been doing for years now and leverage hybrid- or multi-cloud architectures. AWS offers a wide variety of services for which Heroku does not provide a corollary: Redshift, Cassandra, Spark, S3, Cloudfront, etc. And if you don’t have credits kicking around, check out the wide array of data store and data utility add-ons in Heroku’s Elements Add-on Marketplace—which allow you to provision S3 buckets and a wide array of other 3rd party services—directly from the dashboard or CLI of your Heroku account. For further research, check out the following resources:

“Better Together: Hybrid Cloud Architectures Powered By Heroku," our presentation from Dreamforce 2018 on how you can integrate with 3rd party platforms or even with on-prem.
“Pretzel Tech Handles Extreme Peaks in Demand with a Multi-Platform Architecture Centered on Heroku": This Customer Story provides a walkthrough of Pretzel Tech’s hybrid architecture, which leverages Heroku, Netlify, and AWS Lambda to enable extreme scaling in seconds.
You can also listen to Nate Beck, Pretzel Tech’s Founder and CTO, discuss their use case on episode 63 of our podcast, Code[ish]: “Streaming Music to the Livestreamers”.

Go Enterprise

Finally and critically, many Heroku customers find that they’re able to unlock more attractive unit pricing via a Heroku Enterprise annual contract. Enterprise isn’t a panacea by any means. However, it is the primary vehicle that growing businesses on Heroku use to access crucially important features and benefits, such as:

Private Spaces — your own network-isolated version of Heroku with the same DX, for dramatically enhanced security, local presence through broader region availability, and VPC peering and VPN connections for hybrid architectures;
Shield Private Spaces — for the same simplicity as standard Private Spaces, but for companies who need HIPAA or PCI compliance;
Heroku Connect — for point-and-click, bidirectional synchronization of Heroku Postgres and one or more Salesforce orgs (and a strong step towards a “single source of truth”);
24/7 support with a 1-hr guaranteed urgent ticket response time
Customer Solutions Architecture Program — for Enterprise-grade professional service consultations on applications, datastores, and wide-ranging technical assistance across your service architecture
Fine-grain access controls, 2FA, SSO, and Audit Trails — for security and enforcement of Principle of Least Privilege; and last but certainly not least
Volume-based pricing for unit cost optimization on a case-by-case basis.

While Heroku has gradually been making Enterprise-grade services available to credit card customers (on the monthly model), Enterprise is often the best way for companies of all sizes to access the innovations that our product and engineering teams have put their blood, sweat, and tears into building and hardening over the last half-decade or so. To be abundantly clear, this is not a paean to the virtues of Heroku Enterprise. This section is instead a call to action to any CTOs, Lead Developers, Engineers, Product Managers, or other technical stakeholders, especially in the SMB to Mid-Market space.

Go email us!

If you feel as though you’re bumping up against some perceived limitations in your Heroku credit card account, you owe it to yourself and to your team to take a good, hard look at Heroku Enterprise (read: contact us!) before deciding that you need to lift and shift to another platform or provider.
— Me, Just Now

In Sum

If you made it this far, thank you for reading! I hope this perspective has (a) helped clarify the value Heroku provides and (b) provided some ideas as to how you might proceed going forward.

I welcome any and all questions thoughtful feedback.

Best,
David