Forem: Ash Isaac

DevOps is for Humans

Ash Isaac — Tue, 09 Mar 2021 01:51:29 +0000

Here's an question I'd like to present for your consideration:

Should DevOps adoption influence your decision to work at an IT organization?

DevOps has overtaken the software world. Recent surveys show some 80% or more of IT executives already have or expect to adopt DevOps within the next 5 years. The primary driver behind adoption is the awareness of it's positive impact on businesses. Advantages such as reduced time to market and increased quality are often cited.

But I'd like to come at this topic from a slightly different perspective by making this statement:

DevOps is for humans.

What I mean by this that beyond the business value that DevOps' provides, DevOps provides direct positive effects to our experience as humans working with technology. My opinion is that, if evaluating the health of a current or potential workplace, you would be wise in considering DevOps adoption as a positive trait to look for in an IT organization.

Allow me to present 3 specific areas in which DevOps has direct, positive effects on our experiences as human operators of software systems:

Reduced stress and anxiety
Offloading mind-numbing, repetitive work
Increasing collaboration and trust

Reduced Stress & Anxiety

One of the universal, shared experiences of folks in IT Development & Operations is the pain of pushing out software to production. The stress around software deployments, the pressure to make the right decisions on the fly, to fix issues in the midst of outages that are costing your company thousands of dollars per hour in lost revenue, is massive.

This stress is often exacerbated by lack of sleep over deploys that can last days. Once the deploy is complete and the seemingly inevitable issues are exposed, the feelings of frustration and dread as blame is assigned and decisions are cross-examined is truly horrible. Such repeated experiences are a major contributor to engineer burnout.

DevOps addresses this IT pain point. DevOps makes software delivery a routine, practiced, controlled event, with robust tests and automation to minimize human involvement -- in contrast to the traditional "big bang" deploys where the expectation of "all hands on deck!" is the norm.

Of course, DevOps does not eliminate this stress instantly or completely -- but it does provide a path forward that if embraced, can provide substantial relief in this area. The phrase "if it hurts, do it more often" is a DevOps mantra capturing this idea.

DevOps reduces stress by making software delivery a routine, practiced, controlled event.

Ask any physician if reducing stress is worth it, and you're likely to get a earful. No two ways about it, DevOps reduces work stress!

Offloading Mind-numbing, Repetitive Work

Another central idea behind DevOps is the drive eliminate repetitive, error-prone manual work within the software value stream. Unnecessary human intervention in a software delivery stream is a DevOps anti-pattern. "Automate all the things" is how this concept is captured in the DevOps world.

So why does DevOps favor automation? DevOps drives towards codified automation because human intervention is often slow, error-prone, harder to audit, tends to conceal domain knowledge and offers less guarantee of parity across various environments.

Good software pipelines on the other hand, are fast, self-auditing, expose domain knowledge and provide parity across every environment, all of which automation can provide.

But a valuable side benefit of this practice of automation is that this mind-numbing, repetitive work is offloaded from humans to machines to execute. Isn't following 200+ steps in an operational checklist to configure a software system a terrible waste of human ingenuity, when a machine can execute 10x faster, and flawlessly every time?

Manual QA testing is another area where this is particularly applicable. Should humans be clicking through thousands of test cases for hours on end, week after week? DevOps answers with a hard "No!"

Laborious, mind-numbing work is ideal only for machines. QA personnel should be applying their creativity towards building and managing complex & powerful testing frameworks or doing exploratory testing that computers cannot emulate.

The practice of automation offloads mind-numbing, repetitive work from humans to machines to execute, freeing them to apply their ingenuity elsewhere.

DevOps maximizes human ingenuity and offloads mindless labor to the machines that exist to serve us.

Increasing Collaboration and Trust

DevOps only succeeds in a culture of collaboration across the entire software value stream. It is impossible to embrace DevOps without fundamentally changing the traditional team dynamics of operating within silos of "Development", "Operations", "SQA" etc.

DevOps views organizational silos as a deterrent to the collaborative mindset essential to rapid, robust software delivery pipelines.

But what exactly are silos? If you've been a part of Dev team where you are given a narrow set of capabilities and workitems, and once you're done with coding and unit testing you are encouraged to "throw it over the wall" to the Ops Team or the QA team and forget about it, you've been in a siloed organization.

This is not the DevOps way.

Rather, DevOps promotes adopting "system thinking" where the whole team, no matter their function, cares about the entire software delivery pipeline. It pushes to eliminate slow, manual hand-offs in favor of continuous integration and deployment. Information and power is never hoarded, but democratized and teams are empowered to do more.

DevOps promotes the adoption of "system thinking" where every team member, no matter their function, cares about the entire software delivery pipeline.

While DevOps does not mandate a particular team configuration (such as the cross-functional product teams model), it is practically impossible to be successful without members across the various IT functions banding together to ensure that software moves fast from development through testing and into production.

But this has interesting side-effects: it effectively demands increased information sharing, understanding, empathy and trust towards all members of your team.

So the question becomes "Where would you prefer to work?" -- in a traditional siloed IT organization or in a highly cooperative, collaborative, trust-oriented & empowering environment where information is freely shared across IT functions?

DevOps is for Humans

No one wants to work with unhealthy levels of stress or be assigned mind-numbing, frustrating work, or have to navigate a dysfunctional, low-trust work environment. Is anything worth trading on your physical and mental health?

Interestingly, here is where the magic of DevOps comes into play:
DevOps marries both positive business outcomes (such as profitability & market share) which are hugely important with positive human outcomes like lower stress, creative work and empowered collaboration.

I don't know about you, but I am convinced enough that I always look for DevOps adoption in potential workplaces -- because DevOps is FOR humans, not just for business success.

Going Serverless with AWS Lambda? Here's what you should know

Ash Isaac — Sun, 23 Jun 2019 16:11:14 +0000

Serverless has gone mainstream now: it's the natural next-step in the evolution of cloud-computing.
The value proposition is that you can:

1) Focus exclusively on your application code
2) Pay only for actual compute time and resources consumed.

All infrastructure is completely abstracted away. Well :) in theory, perhaps... in practice things are a bit different.

Here is a very high-level overview of AWS Lambda, Amazon Web Services' implementation of serverless computing.

Runtime Context:

AWS Lambda's current implementation runs your code within a Docker container under the hood. Your code has access to the Docker scratch volume for a filesystem.
Lambda supports specific versions of runtimes. Custom runtimes can be built or shimmed.

Execution:

Lambda functions can have a "cold start" time when executed after an interval since the container needs to be booted up. It's possible to keep the function "warm" via a ping or health check.
You can design your function to reuse expensive resources (db connections etc) between immediate invocations.
Lambdas have built-in retries on failures. Retry is triggered based on exit code or callback error.
Lambdas can have a DLQ (Dead-letter Queues) to collect failed events for investigation or additional processing.
Each Lambdas can run for a max of 15mins, after which they will be killed. The available time remaining to execute can be inspected within Lambda.
The amount of compute and memory allocated to the Lambda determines how fast it will run. Pricing is based on 100ms increments and reserved CPU/Memory (vs. actually used). The first million Lambda invocations per account are free every month.
AWS Lambda has concurrency limits - max 1000 concurrent lambdas executing at one time. You can reserve concurrency for critical Lambda functions if necessary.
AWS Step Functions can be used to compose complex workflows using Lambda functions.

Triggers:

AWS Lambdas are typically invoked in response to (the ever expanding) list of event sources.
Lambdas can be "manually" invoked as well from other code, cli tools or via the AWS Console.
Invocation can be synchronous (Request/Response) or Asynchronous (Fire and Forget)
Lambdas permissions are handled via IAM Roles. Event sources may need to be given IAM permissions to invoke Lambda functions.
Lambda integrates well with the AWS API Gateway service. External APIs can trigger Lambda functions and Lambdas can perform Authentication/Authorization or serve as a proxy for APIs as well.
Lambdas can be triggered via internal Application Load Balancers, via target groups and on schedule via CloudWatch Scheduled events.

VPCs:

Lambdas can execute either outside a VPC (i.e only uses other AWS Services i.e S3, DynamoDB) or within a VPC (can access VPC-specific resources - RDS, EC2 instances etc).
When executing within a VPC, every function will provision an ENI (elastic network interface) within the VPC in order to access resources, say in a private subnet. One ENI (& IP address) per subnet.
To make AWS service calls, AWS Lambdas running inside a VPC will need internet access, even inside a private subnet i.e via NAT-ing (NAT Gateway or Instance). You can get around this for some services by using VPC endpoints.

Other:

AWS Lambda offers encryption via KMS (Key Management Service) for environment variables during execution.
AWS X-Ray (tracing) support is available for Lambda functions. Helpful to see where latency is being introduced.
Lambda logs are automatically sent to CloudWatch LogStreams. The request ID and time billed and memory used are logged per invocation.
There are many tools to facilitate deployment to AWS Lambda. Serverless.com is big. AWS has it's own tool called SAM.

I am a huge fan of serverless computing and find AWS Lambda to be an extremely powerful tool, as long as you understand and use it within it's limitations.

Questions? Comments? Concerns? Feel free to ping me.

DevOps in 3 Sentences

Ash Isaac — Fri, 30 Nov 2018 02:26:02 +0000

DevOps is a hot topic right now, and amidst the marketing hype and buzzword-frenzy it can be hard to get to the essence of the concepts it introduces.

What really is at the heart of DevOps? Perhaps my 3 sentence summary of the "DevOps mindset" can help...

The DevOps Mindset:

Make your software-building process into an automated pipeline and optimize it for speed of delivery.

The software pipeline should ensure security, quality and stability by automating the building and testing of infrastructure and applications and progressively prove their fitness by providing feedback (via tests), and when ready, deploy it to end-users.

Work to continuously improve the flow (speed) and feedback of the software pipeline

Okay, so I cheated on that 2nd sentence... it's more of a paragraph :)

The key elements of the pipeline are:

It automates the building of both infrastructure and applications.
It applies a series of automated tests to both infrastructure and applications (unit, integration, performance etc) to prove they meet their functional and non-functional requirements.
When proven to be fit to release, it automates the deploy to end-users.

The Business Value of DevOps:

DevOps primarily applies to businesses that build or assemble their own software, versus just using pre-built software.
DevOps offers the greatest value to businesses where speed of delivery provides a significant competitive advantage.

Additional Insights:

To increase speed without compromising quality, limit the size of the changes pushed through the pipeline at any one time.
Errors should cause the pipeline to fail dramatically, forcefully calling attention to the change that caused the breakage.
To achieve speed, eliminate slow, repetitive and error-prone human effort. Limit human intervention to decision-making, analysis and other creative effort (ex. exploratory testing).
Departmental silos (Product/Development/QA/Operations) are antithetical to software pipelines. Taking a software build through the pipeline successfully requires cross-functional expertise and collaboration. Thus DevOps requires changes in org. culture and team structure.
Build infrastructure and applications via a version-controlled codebase for traceability and repeatability.
Build security into the product by integrating it into the pipeline.
Software architecture matters: DevOps is most effective when you can build, test and deploy discrete components of your product.
To improve the pipeline, make it's health and metrics visible to stakeholders.

DevOps & Happier Teams

Obviously this is a very high-level overview of DevOps concepts and principles -- applying them and building a solid software pipeline is a challenging but worthwhile goal.

Did you know that peer-reviewed research shows that DevOps reduces the pain/stress involved in releasing software, thus reducing engineer burnout? Adopting DevOps can improve the quality of life for your key engineering teams.

Did I miss or misstate something? Feel free to let me know your thoughts & comments.

An engineering perspective on Tech debt

Ash Isaac — Sun, 29 Jul 2018 13:07:48 +0000

One of the inescapable realities of enterprise software is the necessity of facing down "tech debt". But what exactly is it? Is it always something to avoid? The central issue here, of course, it how you define this term.

My definition of tech debt is this: it's the ongoing technical impact of business decisions made in order to secure a market advantage. Tech debt functions just like the availability of credit to a growing business, providing it with a much-needed boost during it's rapid-growth stage. Sometimes you've got to take on a calculated "shortcut" to reduce time-to-market and gain a strong market share. It's a business decision, a calculated gamble, so to speak.

So true tech debt, the "good" kind, is a compromise knowingly made on the engineering side of product-building in order to gain business advantage.

What you'll find is that the term "tech debt" is also used when addressing long-standing issues arising from bad programming practices or brittle & shallow design. But those are just a fallout of poor hiring practices, inexperience and/or unprofessional engineering leadership. I believe this to be an incorrect usage of the term that just muddies the waters.

Here's the kicker: Can the engineering burden you are facing down be explicitly tied to decisions made in order to gain a business advantage? If so, then you are now "paying back" tech debt by setting things right. If not, it's not tech debt at all -- it's just cleaning up a mess you (or someone else) made.

And here's the final word from Ward Cunningham's explanation:

"... the ability to pay back debt, and make the debt metaphor work for your advantage depends upon your writing code that is clean enough to be able to refactor as you come to understand your problem."

So, the next time you use the term "tech debt", think it through: Is this the "good" kind of debt? Or is it something else entirely?

Software Architecture "Virtues"

Ash Isaac — Mon, 23 Jul 2018 00:14:22 +0000

Software architecture is all about appropriate compromises. For small projects, having a highly involved software architecture is probably an overkill. For large enterprise applications, it becomes a necessity.

Finding a suitable software architecture for a project requires both technical and domain knowledge: you need to have a solid grasp on the problem your client needs you to solve AND understand the technical trade-offs involved.

Here is a shortlist of architectural virtues to think through when evaluating various architectural patterns and frameworks:

1. System Comprehensibility

Does this architecture enable developers and business owners explain, understand and collaborate well? Does it hide away less-important implementation details and help accurately frame the problem domain?

2. Modularity (Separation of Concerns)

Does this architecture allow for the problem domain to be broken out into smaller, more easily managed parts? Does it promote the healthy isolation of components? Does this allow component reuse and recombination? Does it reduce unnecessary coupling and make dependencies explicit?

3. Testability (Mockable Dependencies)

Does the software architecture facilitate testing at the unit, integration and system level? Does it allow mocking and stubbing of services and infrastructure? Does it allow testing failure modes?

4. Hexagonality (Infrastructure Agnosticism / Domain Isolation)

In this architecture, is possible to isolate the domain (or business) logic from implementation details? Can you segregate and swap out the transport or persistence layer without needing to make major internal changes to the system?

5. Aspect-orientation (Cross-cutting concerns)

Does the architecture allow for uniform, centralized handling of cross-cutting concerns? Can logging, caching, authorization, serialization etc be handled independently from the core logic?

6. Plugin-orientation

Does the architecture allow decoupled and optional processing? Does it allow for integration with external systems as and when needed? Does it offer hooks into it's internal operations without compromising security?

7. Scalability

Does the architecture framework allow for room for growth at least for the foreseeable future? Does it facilitate the measurement and monitoring of system performance?

There are, of course, many more issues to think about... but this is a pretty good set of parameters to think through while making architectural choices.

The value of technical leadership

Ash Isaac — Sat, 14 Jul 2018 18:21:55 +0000

I've been in technical leadership roles for over a decade and I have to say, in large measure it is quite unglamorous. What I mean, of course, is that you don't get to re-design or architect large-scale systems every week.

Technical leadership typically involves a daily influence over small decisions -- those tiny course corrections that keep things moving in the right direction.

Perhaps you suggest a more flexible architectural model for a subsystem or you work to prevent data-coupling between services. Maybe you write a bit of glue code to stitch together parts of your software pipeline or you weigh in on how using a specific Java library can make code cleaner, easier to understand and maintain.

Of course, your hope is that these small, consistent adjustments, when aggregated over months and years, will promote software excellence on a broad scale. Occasionally though, you do run into situations that are a bit more daunting.

Here's one scenario to think about:

A team has gotten stuck -- they are consistently producing poor results.

When you take a more in-depth look, it's obvious these are due to the adoption of bad software practices. The problem is that these poor results have, over time, caused a lot of pain to all stakeholders involved (Dev, Ops, QA, Product etc and most importantly, your customers). This causes everyone to fear and therefore resist any sort of change.

So what does technical leadership offer to this situation?

I believe that courageous, persuasive technical leadership is the only way to get a team unstuck.

Courage is needed because where fear and suspicion dominate, speaking out could be dangerous to your career. And you will need all your persuasive skills to navigate these types of situations and to start, in a reasoned and methodical fashion, to guide the adoption of good software engineering practices.

The good news is that the adoption of good software practices has it's own positive feedback loop.

Adopting best practices should result in improved results, and you can start to track and measure these small wins. This in turn will produce gains for stakeholders (or at the very least, reduce pain points) which produces an openness to hearing and adopting more of your suggestions. And so a positive feedback loop starts to kick in.

So, when you face those daunting and seemingly intractable situations -- stay steady, speak up and get that positive momentum going!

Docker Swarm: An overview

Ash Isaac — Wed, 06 Dec 2017 14:06:07 +0000