Forem: Dan Lebrero

Book notes: Design It!: From Programmer to Software Architect

Dan Lebrero — Tue, 28 Mar 2023 21:54:33 +0000

These are my notes on Design It!: From Programmer to Software Architect by Michael Keeling.

"From Programmer to Software Architect" is a spot on subtitle.

Key Insights

SW design is a constant struggle to find the right balance between the things you want and the reality you must accept.
Every team has at least one architect. The best teams have several.
For every SW system you build, briefly describe the system and what you learned during your time developing it.
Design Thinking Principles (From Design Thinking: Understand – Improve – Apply):
1. All design is social in nature: for and with people.
2. Preserve ambiguity:
  - Minimalist architecture:
    - Less is More with Minimalist Architecture.
3. All design is redesign.
4. Make ideas tangible to facilitate communication.
Emphasize:
1. Treat solutions as experiments.
2. Focus on reducing risks.
3. Work to simplify problems.
4. Iterate quickly to learn quickly.
5. Think about the problem and solution at the same time.
Use risk to decide what to focus on.
Architecturally Significant Requirements:
1. Constraints:
  - Well-chosen constraints simplify the problem.
  - Early design decisions can become constraints in the future.
2. Influential functional requirements:
  - Identifying influential functional requirements is equal part of art and science.
3. Quality Attributes.
4. Others:
  - Tech trends.
  - Architecture skills, knowledge.
  - Team skills, knowledge.
  - Team organization.
Our confidence in a decision increases after seeing multiple alternatives.
Decision matrix: Do not use numbers as they give a false sense of confidence and precision.
Complexity is an inevitable by-product of every successful software system.
Discussion lacking in disagreement may seem like positive progress, but it is more likely to be the opposite.
Sometimes a simple table is all that is needed.
Precision gets in the way of communication: accurate, not precise.
A good-looking document tells readers the content is trustworthy and was created by a professional.
Sometimes your pile of rejected decisions can be more telling than a lengthy explanation.
Keep design authority when risks of failure are high.
The discussion is often more important than the option picked.
Empathy map.
The Agile Inception Deck.

All activities

Understand

Activity	Time	Participants	Notes
Choose one thing		All stakeholders	Clear prioritization and disagreement.
Empathy map	10-30m	Solo or 3-5. Devs + Arch	For the team to develop empathy.
Goal-Question-Metric workshop	15-90m	1-5 people.	Prioritized metrics to gather.
Interview stakeholders	30-60m per interview	1-2-1 or small stakeholder group
List assumptions	15-30m	Whole team in 2-5 groups	Take assumptions out of the shadows.
Mini-Quality attributes workshop	90-180m	Arch + 3-5 stakeholders
Point-of-View mad lib	30-45m	1-3 people. Any stakeholder	Summarize business goals and other needs.
Response measure straw man	Combine with other	Architect prepares	Get conversation moving.
Stakeholder map	30-45m	1-25. Whole team + stakeholders	Relations and interactions.

Explore

Activity	Time	Participants	Notes
Personify the architecture			It is ok to feel silly.
Architecture flip book	30-45m	1-3 people	Record all steps on design iteration.
Component responsibility collaborator cards	30-90m	Solo or 3-5	Propose architectural elements.
Concept map	30-60m	1-3 technical stakeholders	Uncover missing domain concepts.
Divide and conquer		Groups of 2-4	To cover a lot of ground in parallel.
Event Storming	90m	2-12. Expert + dev team	Identify domain events.
Group Posters	20-30m	Groups 2-5 people	To summarize outcomes.
Round-Robin design	15-45m	3-12 tech stakeholders	Combine ideas + build consensus.
Whiteboard Jam	Up to the group	3-5 tech stakeholders

Make

Activity	Notes
Architecture Decision Records
Architecture haiku	One-page uber-terse summary
Context diagram	Agreement with system scope
Greatest hits reading list
Inception deck	10 important questions at the start of a new project
Modular decomposition diagram	Tree diagram that shows relationships at different granularities
Paths not taken
Prototype to learn or decide
Sequential diagram
System metaphor

Evaluate

Activity	Time	Participants	Notes
Architecture Briefing	45-60m preparation. 30m present. 30m questions	Arch + wide audience
Code review	Ongoing	2 to whole team
Decision Matrix	Varies	Arch + Stakeholders
Observer Behaviour			Observability in production
Question-Comment-Concern	30-90m	Whole team
Risk storming	60-90m	3-7 devs
Sanity Check	5-10m	Whole team	Pop quiz
Scenario walk-through	20-30m per quality attribute	3-7 people	Use early
Sketch and Compare	20-30m	3-5 people

Part I - Introducing Software Architecture
- Chapter 1 - Become a Software Architect
- Chapter 2 - Design Thinking Fundamentals
Part II - Architecture Design Fundamentals
- Chapter 3 - Devise a Design Strategy
- Chapter 4 - Empathize with Stakeholders
- Chapter 5 - Dig for Architecturally Significant Requirements (ASR)
- Chapter 6 - Choose an Architecture (Before It Chooses You)
- Chapter 7 - Create a Foundation with Patterns
- Chapter 8 - Manage Complexity with Meaningful Models
- Chapter 9 - Host an Architecture Design Studio
- Chapter 10 - Visualize Design Decisions
- Chapter 11 - Describe the Architecture
- Chapter 12 - Give the Architecture a Report Card
- Chapter 13 - Empower the Architects of Your Team
Part III - The Architect's Toolbox
- Chapter 14 - Activities to Understand the Problem
- Chapter 15 - Activities to Explore Potential Solutions
- Chapter 16 - Activities to Make the Design Tangible
- Chapter 17 - Activities to Evaluate Design Options

Part I - Introducing Software Architecture

Chapter 1 - Become a Software Architect

What architects do?
1. Define the problem from an engineering perspective.
2. Partition the system and assign responsibilities.
3. Keep an eye on the bigger picture:
  - System as a whole.
  - SW design is a constant struggle to find the right balance between the things you want and the reality you must accept.
4. Decide trade-offs among quality attributes.
5. Manage technical debt.
6. Grow the team's architecture skills:
  - Architecture design is a social activity.
A system's software architecture is the set of significant design decisions about how the software is organized to promote desired quality attributes and other properties.
Software structure's elements and relations:
1. Module: structure that exists at design time (class, DB table).
2. Component and connector (C&C): exits at runtime (object, process).
3. Allocations: how modules and C&C elements correspond to each other.
  - Aka: mapping structures.
Quality attribute is any externally visible characteristic by which stakeholders judge a SW system's goodness.
Every team has at least one architect. The best teams have several.
For every SW system you build, briefly describe the system and what you learned during your time developing it.

Chapter 2 - Design Thinking Fundamentals

Design thinking: problem-solving approach that puts humans at the center.
Principles (From Design Thinking: Understand – Improve – Apply):
1. All design is social in nature:
  - For and with people.
2. Preserve ambiguity:
  - Minimalist architecture:
    - Less is More with Minimalist Architecture.
    - Only focus on how high-priority quality attributes are achieved.
3. All design is redesign.
4. Make ideas tangible to facilitate communication.
To design an architecture:
1. Pick a mindset.
2. Pick a practice within that mindset.
3. Apply practice to learn something new.
4. Goto (1).
Design mindsets:
1. Understand:
  - Requires empathy.
2. Explore.
3. Make.
4. Evaluate.

Part II - Architecture Design Fundamentals

Chapter 3 - Devise a Design Strategy

Emphasize:
1. Treat solutions as experiments.
2. Focus on reducing risks.
3. Work to simplify problems.
4. Iterate quickly to learn quickly.
5. Think about the problem and solution at the same time.
How much design up front:
- Barry Boehm, Architecting: how much and when. Chapter 10 of Making Software.
- Total project time == dev time + architecture + rework.
- Architecture reduces rework time.
- Bigger systems benefit the most from more up-front architecture work.
- COSYSMO and COCOMO II.
Use risk to decide what to focus on:
- [conditions] might [consequences]
- Deal with risk:
  1. Reduce probability.
  2. Reduce impact.
  3. Push out the time frame of the risk.
  4. Remove condition.
  5. Accept it and do nothing.
- Which mindset:
  - Understand: risk is about problem.
  - Explore: risk is about solution.
    - Have you seen enough solution alternatives?
  - Make: risk is about communication.
    - Do stakeholders fully understand design and architecture?
  - Evaluate: risk involves a design decision or the design's overall fit.
    - Do we need to make a design decision?
Create a design plan.

Chapter 4 - Empathize with Stakeholders

Create a stakeholder map with stakeholders, stakeholders groups and their relationship.
Record business goal statements:
- Subject, outcome, context.
- Outcome: measurable.
- Context: insightful and not obvious.
Most systems only have three to five business goals.

Chapter 5 - Dig for Architecturally Significant Requirements (ASR)

4 Categories:
1. Constraints:
  - Unchangeable. Non-negotiable.
  - Well-chosen constraints simplify the problem.
  - Technical: programming lang, platforms, components, technology.
  - Business: team structure, schedule, budget, legal.
  - Early design decisions can become constraints in the future.
  - Keep a XLS: constraint, origin, type, context.
2. Influential functional requirements (FR):
  - Identifying influential functional requirements is equal part of art and science.
  - Sketch notional architecture.
  - Identify classes of requirements:
    - Group FR with the same architecture elements (like persistence).
    - FR that seem difficult to implement.
    - High-value, high-priority FR.
  - Try each class of requirements against the notional architecture:
    - If it is not obvious how to implement, the it might be an influential FR.
3. Others:
  - Tech trends.
  - Architecture skills, knowledge.
  - Team skills, knowledge.
  - Team organization.
4. Quality Attributes:
  - Use quality attribute scenarios:
    - Parts:
      - Stimulus: event that triggers the scenario.
      - Source: who.
      - Artifact: part of the system.
      - Response: externally visible action by the artifact as result of the stimulus.
      - Response measure (Success criteria, Specific and measurable).
      - Environment context: "normal", peak load, failure condition.
      - Keep in XLS:

Quality Attribute	Scenario	Priority
Availability	When DB does not respond, respond with stale data within 3 secs	high
...	...	...

Build an architect significant requirements doc:
- Purpose and scope.
- Intended audience.
- Business context.
- Quality attribute requirements:
  - Top scenarios.
- Influential FR:
  - Top users or user persona.
  - Use cases or user stories.
- Appendix A: Glossary.
- Appendix B: Quality Attributes Taxonomy.

Chapter 6 - Choose an Architecture (Before It Chooses You)

Our confidence in a decision increases after seeing multiple alternatives.
You need to explore:
1. Architecture elements and their responsibilities.
2. How elements interact with each other. Their interfaces.
3. Understand the domain.
4. Tech and framework that promote quality attributes.
5. How app will be deployed.
6. Look at past designs.
Decision matrix:
- I personally like text + color (red, white, green).
- Do not use numbers as they give a false sense of confidence and precision.

	Option A	Option B
Property 1	promotes	+
Property 2	inhibits	-
Property 3	neutral	o

Create an element responsibility catalog:
- Assign to each element the influential functional requirement.

Chapter 7 - Create a Foundation with Patterns

See Fundamentals of Software Architecture.

Chapter 8 - Manage Complexity with Meaningful Models

Complexity is an inevitable by-product of every successful software system.
Abstractions help us focus on specific details at the expense of others.
Good architecture models:
- Establish the vocabulary of design.
- Direct attention to interesting details.
- Allows to reason about quality attributes.
- Captures the architect's intention.
Design a meta-model: concepts and rules to use those concepts.
Building models into the code:
- It is possible to shrink the model-code gap, but it cannot be closed completely.
- Use architecture vocabulary.
- Organize code following architecture elements.
- Enforce relations, so that it becomes impossible to violate the architecture.
- Add comments.

Chapter 9 - Host an Architecture Design Studio

Design studio encourages group collaboration and has strict time constraints to help the team see a broad range of ideas in a short time frame.
Creates buy-in of everybody involved.
Outcomes:
- Things to make.
- Things to research more.
- New open questions.
Typically, few hours to a day.
Steps:
1. Prepare:
  - Business and workshop goals.
  - Quality attributes.
  - ASRs.
  - Choose participants:
    - 3-10.
    - Diverse background.
  - Discussion lacking in disagreement may seem like positive progress, but it is more likely to be the opposite.
2. Kick-off:
  - Share context.
3. Create:
  - Choose design activity:
    - Tell-show-tell to explain the activity.
    - Time boxed. Minimum 5 mins.
4. Share:
  - 3-5 mins.
  - Only main points.
  - No questions or comments.
5. Critique:
  - Specific.
  - Focus on facts.
  - Point also good things.
6. Iterate:
  - Go to 3.
  - Tweak the group dynamics.
7. Close:
  - Reflect.
  - Decide specific actions.
Use a parking lot to record interesting ideas for other time, and keep the workshop moving.

Chapter 10 - Visualize Design Decisions

Sometimes a simple table is all that is needed.
Sometimes precision gets in the way of communication:
- Accurate, not precise.
Tips:
- Always include a legend.
- Add a descriptive title.
- Highlight patterns.
- Be consistent.
- Provide descriptive prose, and/or text annotations.

Chapter 11 - Describe the Architecture

Which architecture description approach should I use?
Tribal:
- Oral tradition, informal sketching, story telling.
Communal:
- When you find yourself telling the same story to more than a few people.
- Architecture Haiku, ADR, architecturally evident coding style.
Formal:
- High-risk systems or architecture decisions.
- Traditional SW architecture decision (SAD).
- Templates:
  - SEI views and beyond.
  - ISO/IEC/IEEE 42010.
Always keep the audience in mind:
- Use empathy map.
A good-looking document tells readers the content is trustworthy and was created by a professional.
Establish viewpoints:
- Arch from the point of view of a related set of stakeholder concerns.
- Example standard viewpoints:
  - SEI views and beyond.
  - Simon Brown's C4 model.
  - Phillipe Krutchen's 4+1 view model.
- Usually organized around quality attributes or stakeholder needs.
Explain rationale:
- Sometimes your pile of rejected decisions can be more telling than a lengthy explanation.

Chapter 12 - Give the Architecture a Report Card

Benefits of evaluation:
- Educate team.
- Create buy-in.
- Reduce delivery risks.
- Improve architecture.
Design Rubric:
- 2 parts:
  1. Criteria:
    - ASR as a guide.
    - Important, essential, distinct, observable and measurable, precise and unambiguous.
  2. Rating:
    - Select scale:
      - 1-2: all or nothing. Few reviewers.
      - 1-3: minimum acceptable threshold. Multiple reviewers.
      - 1-4: Detailed feedback desired.
    - Discuss any "1" score, even if the average is acceptable.
Organize an architecture evaluation workshop:
- Steps: prepare, prime reviewers, assess, analyze, follow-up.
- It is important to understand why the design is fit (or not).
- Great designs can always be improved.
- Gold standard:
  - ATAM.
  - See Software Architecture in Practice.
  - Multi-day or multi-week.
Evaluate early, often, continuously:
- Risks.
- Unknowns.
- Problems.
- Gaps.
- Arch erosion.
- Contextual drift.
Start with low ceremony methods.

Chapter 13 - Empower the Architects of Your Team

Modern SW dev teams need a different kind of leader than the traditional top-down architect.
- Coach, mentor, technical guru.
Teams who embrace the idea of SW architect as a way of thinking instead of as a role, produce better SW:
- More eyes.
- Team buy-in. Ownership.
Create opportunities for safe practice:
- Pair design.
- Create scaffolding:
  1. Build templates for delegated work.
  2. Provide feedback during peer reviews.
  3. Create checklists.
- Introduce architectural guide rails:
  - Guide rails (constraints) decreases the chances to mess up the architecture.
- Host information sessions.
Keep design authority when risks of failure are high.

Part III - The Architect's Toolbox

Chapter 14 - Activities to Understand the Problem

Choose One Thing:
- Discuss priorities by presenting an extreme choice.
- If you only get one thing, what will it be?
- Clear prioritization.
- Clear disagreement.
- The discussion is often more important than the option picked.
- Use early.
Empathy Map:
- Brainstorm and record responsibilities, thoughts and feelings for the team to develop empathy.
- Updated version.
Goal-Question-Metric Workshop:
- Identify metrics and response measures to connect data with business goals.
- Question to answer to know if we have met the goal.
- Result: prioritized data and metrics to gather.
Interview Stakeholders:
- Unstructured interviews should still have a checklist.
List assumptions:
- Take assumptions out of the shadows.
- Write down everything mentioned, even obvious ones.
- Pause to discuss surprising assumptions.
Mini-Quality Attribute workshop:
- Prepare:
  1. Quality attribute taxonomy (max 7).
  2. Quality attribute scenario templates.
  3. Quality attribute web.
  4. As homework, refine top raw quality scenarios. Present in a follow-up meeting.
Point-of-View Mad Lib:
- Summarize business goals and other needs.
- Template:
- Consensus is not required.
- Outcome focused.
Response Measure Straw Man:
- Give stakeholders something to beat up until they arrive at their own answers.
- Response either:
  - Honest: if you are confident.
  - Outrageous: to find boundaries.
Stakeholder Map:
- To visualize relationships, hierarchies and interactions between people involved or impacted by the system.
- Once completed ask who are the most important stakeholders.

Chapter 15 - Activities to Explore Potential Solutions

Personify the architecture:
- Give it human qualities so that you can explore interactions among elements.
- How it "reacts" or "feel", make it memorable through story telling.
- It is ok to feel a little silly.
Architecture flip book:
- Record every step of the design journey so others can follow along afterwards.
- Sketch + notes about incremental changes.
- Teach others how to think about design and modeling.
- On a slide deck, keep copying the last slide and editing it.
Component responsibility collaborator cards:
- Propose architectural elements, their responsibilities and how they come together to form a view of the architecture.
Concept map:
- Visualize how concepts in the domain relate to one another.
- Helps to uncover missing, hidden, or implied domain concepts.
Divide and conquer:
- Tight feedback loops work better.
Event Storming:
- Collaborative brainstorming technique used to identify domain events.
- To better engage subject-matter experts.
- Concrete and specific examples, not abstract.
Group Posters:
- Create a poster that conveys their design ideas for the architecture outcomes.
- To summarize outcomes.
Round-Robin design:
- Quickly explore a range of ideas, and then combine to start building consensus.
- Steps:
  1. Sketch 5m.
  2. Rotate.
  3. Critique 3m.
  4. Rotate.
  5. Improve 5m.
  6. Review all sketches.
Whiteboard Jam:
- Collaboratively draw diagrams on a whiteboard.
- Pictures won't make sense to someone who wasn't there.

Chapter 16 - Activities to Make the Design Tangible

Architecture Decision Records:
- Capture architecture design decisions using a lightweight text-based template.
Architecture haiku:
- Create bit-sized architecture summaries that stakeholders will actually use.
- Think through and articulate the essential parts.
Context diagram:
- Help stakeholders understand where the software system fits in the world.
- Agreement with system scope.
Greatest hits reading list:
- Help navigate the morass of design artifacts, so they can find relevant information.
- Links list with tile, overview and caveats.
Inception deck:
- Answer 10 important questions at the start of a new project to avoid common failures and align stakeholders.
- The Agile Inception Deck
- Customize questions to your context.
- Effort to create should be proportional to size and cost of project.
Modular decomposition diagram:
- Simple tree diagram that shows how abstractions at different granularities are related.
- Root is the whole system.
- Child are more and more specific.
Paths not taken:
- List options discarded with a brief explanation of why.
Prototype to learn or decide:
- Always prototype as quickly and cheaply as possible.
Sequential diagram.
System metaphor.

Chapter 17 - Activities to Evaluate Design Options

Evaluation is a continuous activity.
Architecture Briefing:
- Brief presentation on some part of the architecture, to get meaningful feedback.
Code review:
- From architecture point of view.
Decision Matrix:
- Visual comparison of alternatives.
- No more than 7 factors.
- No more than 5 design options.
- Take notes when filling the matrix.
Observe Behaviour:
- Add instrumentation to see runtime behaviours.
Question-Comment-Concern:
- Collaborative, visual activity to shine light on knowledge gaps, articulate issues, and establish known facts about the architecture.
- Team puts sticky notes (comments, questions, concerns) on views of the architecture.
Risk storming:
- Collaborative, visual technique for identifying risks in the architecture.
Sanity check:
- To expose issues in team communication or understanding.
- Pop quiz-like.
Scenario walk-through:
- Describe step-by-step how the architecture addresses a specific quality attribute scenario.
- Use early.
- Avoid problem-solving during the session.
- Architect + note taker + session moderator + few reviewers. 3-7 total.
Sketch and Compare:
- Create two or more alternatives of the same design, so it is easier to see the pros/cons.
- Always summarize the findings.

Are you asking too much from your team/tech lead?

Dan Lebrero — Tue, 24 Jan 2023 23:05:43 +0000

Somebody recently asked:

How do you learn to be a tech lead?

Knowing her background, I replied with another question:

Do you mean tech lead or team lead?

The difference was not obvious to me until I worked as a tech lead in a team that had a team lead.

Team vs Technical lead

On that particular job, as a tech lead, my responsibilities were:

Choose the right tool for the job: be aware of existing tools/libraries/frameworks, their trade-offs and what was appropriate for the company's context.
Code contributions.
Code quality (aka bugs).
Production issues/outages.
Design and architecture, including NFRs.
Developer productivity: from testing, CI/CD, release, monitoring, …
Technical mentoring.
Awareness on other parts of the system: functionality available, technologies used, trade-off, high level workings, lessons learned.
Applying the company's architecture constraints, principles and practices, and raising concerns when we wanted to deviate from those.
Technology proof of concept.
Technical assessment of candidates.
Technical debt prioritisation.

My fellow team lead would take care of:

People's career progression.
Performance appraisals.
1-2-1.
Reporting up and down.
Recruitment, team composition.
Onboarding.
Internal and inter-team conflicts.
Project management.
Inter-team coordination and dependencies.
And a myriad of other things that I was not aware of.

What we will both do:

Backlog grooming: me for feasibility and rough sizing. Team lead to know if we had the capacity and for stakeholder management.
Development process improvements: me more to suggest, team lead more as Scrum Master and lean/agile coach.

Basically I will become a software architect while the team lead would become an engineering manager.

Choices

Continuing with the conversation:

Me: "What path are you asking about? You cannot do both."

Tech lead: "Both is actually what I have been asked to do."

Me: "How is the experience of trying to do both?"

Tech lead: "Is it a lot. I spend most of the time doing management, admin and planning."

Me: "So you are team leading, not tech leading. But if you are not doing the tech leading (because you have no time), then you should raise it with your manager."

Tech lead: "Haven't you ever act as both the team and tech lead?"

Me: "In my very early days (~2002-2004), in a very small company in a team of ~6 people. Of the list above, I know I did not had to (or did not know I had to):

Production issues/outages: never got anything successful in production, hence no users, hence no production issues.
Design and architecture: system that we built were simple.
Developer productivity: mostly unknown things for me back then.
Awareness on other parts of the system: there were no other parts, not other teams.
Applying the company's architecture constraints: No such a thing.
Technology proof of concept: no time for this.
Technical debt prioritisation:
People's career progression: we were trying to survive, not planning long or mid term, also I was probably not aware.
Performance appraisals: see above.
1-2-1: see above.
Reporting up and down: company was small enough to not need this.
Recruitment, team composition: recruitment yes, but the team composition was pretty static.
Internal and inter-team conflicts: Internal yes, but no other teams.
Inter-team coordination and dependencies: no other teams."

That is a long list of things I did not do.

Are you asking too much from your team/tech lead?

If so, what is she neglecting? Do you agree with her choices?

But life had other plans ...

Dan Lebrero — Wed, 22 Jun 2022 18:28:48 +0000

In summer 2018 my mother-in-law was diagnosed with cancer.

Further testing identified a BRCA2 gene mutation as the cause. Having the BRCA2 mutation increases your chances of developing breast cancer to 45-70% and ovarian cancer to 11-17%.

The mutation is hereditary, with a 50% chance to pass it to your children.

In May 2020, my wife’s BRCA genetic test was positive.

Prevention

Doctors recommended preventive surgery to remove breasts, ovaries, and Fallopian tubes which reduces the cancer risk by 90-95%.

On top of the heavy physical and psychological toll, and the inherent risk of any operation, the surgery has two additional consequences: early menopause and no more children.

A sibling for our lovely four-year-old daughter was in our plans, but with three heartbreaking miscarriages on our backs, and both of us over 40, we were ready to move past the childbearing stage of our lives.

So given what my mother-in-law had gone through, we deemed the operation and its consequences to be the lesser evil, and my wife joined the public health service's waiting list.

Operation and recovery

The operation was scheduled for the last quarter of 2021, but we were given the chance of moving it forward to May.

We expected the recovery to take a couple of months, so having the surgery in May would mean that we will have enough time to recover strength before September's school season.

But it wasn't like that.

My wife was in constant pain, and the painkillers would make her sick and throw up, so she had no other option but to bear it day and night.

But by the end of summer, something unsettling happened: her belly slowly started to swell.

The swelling was too localized to attribute it to fattening, and it was unlikely to be an infection as she had no fever.

These left us with the possibility that either something had gone wrong with the surgery, or …

Doctor's visit

We scheduled a visit with a doctor as soon as possible.

The doctor (a gynecologist) explained about peritoneal cancer, recommended an urgent tumor marker blood test, and, as the swelling was so bad, made an ultrasound scan to see if anything useful could be found.

And indeed she did.

My wife was pregnant.

Wait, what?!?!? Hasn't she just gone through surgery to remove her ovaries? How was this even possible?

Given that my wife didn't recall talking with any angel lately, the simple explanation was that she was already pregnant when she went through the surgery.

She was five(!) months pregnant.

At the time of the surgery, the embryo was young enough to not be detected but old enough to be out of the Fallopian tubes and not depend on the ovaries' hormones anymore.

Older than one week, but younger than three. A two-week window.

Puzzles and odds

The pregnancy made the pieces fall in place: no menopause symptoms, the anemia, throwing up every morning (not due to the painkillers!), and obviously the belly:

When people ask why we didn't ever contemplate the possibility of a pregnancy, first we shout "no ovaries!", and then we explain that for 15 years we have been unable to conceive naturally.

Our daughter and all the unsuccessful pregnancies have been thanks to the more advanced types of fertility treatments.

Never ever naturally. In 15 years.

It feels like a prank.

After 15 years of trying with no luck, after giving up for good, my wife gets pregnant with her very last egg; the embryo narrowly avoids death and being detected, and then it survives to such a nasty surgery and recovery procedure, while previous miscarriages had a much easier time.

But the most incredible thing: when have you seen the public health service being earlier than planned?

Life.

A healthy Lucas Lebrero was born on February 4th, and my mother-in-law helps with him every day.

CTO last day: reflections, mistakes, and some learnings

Dan Lebrero — Tue, 31 May 2022 18:26:49 +0000

In April 2021, I left my job at Akvo as CTO, after about two years in the position.

Today, ~~six~~ ~~seven~~ ~~eight~~ ~~nine~~ ten months later, it seems like about time to stop procrastinating and write something about it.

Leaving

I realised that I needed to leave when Sunday evenings became the worst part of the week when I felt exactly the same as in my old school days: anxious and unhappy.

But it was not only Sunday evenings. I have never minded some work problem popping in my head out of working hours -- I love solving problems -- but now it will just suck all the joy of whatever I was doing.

And of course, the sleeping troubles that came with all of this.

True happiness occurs only when you find the problems you enjoy having and enjoy solving.
Mark Manson, The Subtle Art of Not Giving a F*ck.

Root Cause Analysis

No need to go through the 5 whys to see what the cause was: the compounded effect of the company’s financial situation, the impact of the decisions I was involved with, and the feeling of being completely unprepared.

As an example, my translation of the new business strategy: stop product development.

Who I was to push for halting the development of what has been the company’s flagship product since its inception, twelve years ago? Would a more experienced CTO see otherwise? Would a more experienced CTO change the business strategy? Changed how product development was financed? How contracts were negotiated? Find external investment?

And the impact on the development team …

This was the last decision that I helped make.

Mistakes and lessons learned

Mandatory “lessons learned” section:

Bye-bye coding.
Complaints are good.
Clarity is hard work.
Business first, tech second.
Conflicts: avoid being a broken telephone.
Difficult conversations (with data).
It is always a system problem.
Self-blaming is pointless.
It is all about teams.
Don't make changes, run experiments.
Feedback loops are loooong.
Ring-fencing is a necessary evil.
Lonely.
I don’t dislike people management!

Bye-bye coding

As much as it hurts, there will be always something more important than coding.

Complaints are good

Not all complains and not all the time, but some level of complaining means that people trust you enough and that they care enough about the job.

Complains are an opportunity to either:

Make an improvement.
Provide clarity.

Best outcome is delegating the improvement in the complainer after giving some additional clarity.

Clarity is hard work

Not only communication is messy and you will need to repeat yourself ten times, finding the clarity yourself in the first place requires a lot of work.

Business first, tech second

I understood that my main responsibility as CTO was to improve how we delivered software. Major mistake.

I should have dug deeper from the very beginning on how the business works and how software supported and enabled it.

Conflicts: avoid being a broken telephone

Foo complains about Bar because of Buzz. You talk with Bar about Buzz but Bar says Quux, which you have no clue about. So you go back to Foo to clarify Bar's Buzz's Quux and Foo replies Flob.

What a waste of time, energy, and a source of confusion.

The underlying issue is that Foo does not want to have a difficult conversation with Bar, which, as much I can empathise with, it needs to happen.

So either, depending on circumstances:

Ask Foo to have a difficult conversation with Bar.
Sit with Foo and Bar to have a difficult conversation.

Difficult conversations (with data)

Two ingredients that made difficult conversations less difficult to have:

Following the respectful confrontation script, which has a wonderful mix of hard-core objective real data and fluffy feelings.
Internalizing that all problems are system problems.

It is always a system problem

Everybody is doing their best, under the current circumstances.
Gerald M. Weinberg, Becoming a Technical Leader

With this in mind, it is amazing how much more productive conversations are once you move from blaming people to understanding, inspecting, and changing the system.

The steps:

State that we are trying to understand how the system leads to the outcome.
Repeat point 1.
Agree that the outcome was undesirable.
Collect the timeline/facts about the system behavior, to understand "the circumstances".
Agree on what was the desirable outcome.
Agree on what system change is needed to avoid (1) and promote (2).

Note that "circumstances" can be a lack of knowledge or competency by the person doing the work. This is something that can be fixed.

Change the system and people will follow.

Self-blaming is pointless

I always thought that acknowledging that "It is my fault!" was a brave thing to do, but if you really believe that something is your fault, then you are ready to believe that others can be at fault.

Blaming and self-blaming are counterproductive. It is always a system problem.

It is all about teams

What teams, composition, responsibilities, the internal and external dynamics, ... this is going to be your main leverage point for system change.

Don't make changes, run experiments

Change is scary and comes with a lot of resistance, but running an experiment sends a different completely different signal: it is a temporal thing aimed to learn.

Experiments are expected to "fail", which brings the psychological safety required for honest feedback and increases the willingness to participate.

Feedback is high quality as it comes from empirical evidence, not theoretical "this will never work here".

And the most important reason:

Change culture by first changing what people do, not how people think.
John Shook, How to Change a Culture: Lessons from NUMMI

Feedback loops are loooong

You will miss those 72 hours test suites.

Ring-fencing is a necessary evil

I made several times the mistake of asking people for important-but-not-urgent work and expecting them to find the time for it. It rarely worked.

As much as it pains me, the workaround is to explicitly ring-fence people's time, so that if other conflicting priority arises, they can rely on your authority to push back, or escalate the conflict to you.

Lonely

I have never talked so much and been in so many meetings, but nobody in your company will be in your same position, so I actually felt quite lonely at times.

I ended up reaching out to random CTOs on LinkedIn, which was surprisingly useful.

I don’t dislike people management!

You probably do not care but it was a big surprise for me.

Even if the 1-2-1 days were really really draining, I actually enjoyed the time spent with other human resources.

What is next?

The experience was like my first beer, first blog post or first conference talk.

Trying once is not enough to make up my mind.

But before starting a new job, the original plan was to take the rest of 2021 off, get a respite from work and COVID, and take care of my family while my wife went through some nasty surgery.

But life had other plans ...

Book notes: Staff Engineering: Leadership beyond the management track

Dan Lebrero — Fri, 27 May 2022 13:28:16 +0000

These are my notes on Staff Engineering: Leadership beyond the management track by Will Larson.

Key Insights

Staff-plus roles archetypes:
1. Tech lead.
2. Architect.
3. Solver.
4. Right hand.
What do Staff engineers actually do?
1. Setting technical direction.
2. Mentorship and sponsorship.
3. Providing engineering perspective.
4. Exploration.
5. Being glue.
It is normal to end some days feeling like you haven't accomplished anything.
Your work maintains an aloof indifference to your sacrifice rather than rewarding it.
Look for low effort-high impact work.
Durably useful engineering strategy and vision are the output of iterative, bottom-up organizational learning.
Synthesize 5 design docs into a strategy, extrapolate 5 strategies into a vision.
Design docs (RFC):
- When:
  - Project capabilities will be used numerous times in the future.
  - Big impact on users.
  - Work takes longer than a month.
- Start from the problem: The clearer the problem statement, the more obvious the solution.
- Gather and review together, write alone.
Specific statements create alignments; generic statements create the illusion of alignment.
Best strategies feel too obvious.
Vision:
- Reconcile strategies, extrapolate how their trade-offs will play out over the next two to three years.
- Assume everything is going to go well, but don't assume infinite resources.
- Don't expect load of excitement:
  1. Vision is for people writing strategies and these are few.
  2. Great visions are so obvious that it bores.
At a well-run and successful company, most of your previous technical decisions won't meet your current quality threshold.
Architect's decision degrade the further they get from doing real work on real code in real process.
Your career depends more on being easy to involve than being technically correct.

Overview

Staff-plus roles archetypes:
1. Tech lead:
  - The most common type.
  - Lead one team or a cluster of teams.
  - Scope complex tasks, coordinate teams to solve it and unblocks them.
  - Keep cross-team relationships.
  - Roadmap shuffling.
2. Architect:
  - Responsible for the success of a specific technical domain.
  - Intimate business knowledge and technical constraints.
  - Either deep in codebase or no coding at all.
3. Solver:
  - Moved from one high execution risk problem to the next.
  - Problem identified and prioritized by the org.
  - Require little org-level chiropractices.
  - Individual vs team.
4. Right hand:
  - Least common.
  - Senior org leader without direct managerial responsibilities.
  - Borrow authority from senior leader.
  - Dive into a fire, edit the approach, delegate execution to the most appropriate team, go to the next fire.
  - Problem are never purely technical.
What kind of work energizes you?
What do Staff engineers actually do?
1. Setting technical direction.
2. Mentorship and sponsorship:
  - What does sponsorship look like?.
3. Providing engineering perspective.
4. Exploration.
5. Being glue.
Time frames are longer than dev work.
It is normal to end some days feeling like you haven't accomplished anything.
Advantages:
1. Allow you to bypass informal gauges of seniority:
  - Do not need to recurrently prove yourself, which will free time and energy.
  - More respected from the outset.
2. Being in the room:
  - To provide input for important decisions.
3. Compensation.
4. Access to interesting work:
  - But not always.

Chapter 2 - Operating at Staff

Work on what matters:
- Your work maintains an aloof indifference to your sacrifice rather than rewarding it.
- Look for low effort-high impact work.
- Avoid snacking: low effort-low impact work that is "just" psychologically rewarding.
- Stop preening: low impact-high visibility work.
- Stop chasing ghost: low-impact high-effort:
  - Ghosts from previous org.
  - When joining a new company, first understand context before making changes.
What work?
1. Existential issues: company going through an existential risk.
2. Work where there is room and attention:
  - Teaching a company to value (pay attention) something it does not care about is the hardest sort of work you can do, and it often fails.
3. Foster growth of your team.
4. Edit: small, quick changes that shift a project's outcome with little effort.
5. Finish things:
  - Push project to finishing line, by helping rescope or remove roadblocks.
6. Things that only you can do.
Writing engineering strategy:
- Durably useful engineering strategy and vision are the output of iterative, bottom-up organizational learning.
- If you have rehashed the same discussion three times, it is time to write a strategy.
- When the future is too hazy to identify investments worth making, it is time to write another vision.
- Technical Decision-Making and Alignment in a Remote Culture.
- Synthesize 5 design docs into a strategy, extrapolate 5 strategies into a vision.
Design docs (RFC):
- When:
  - Project capabilities will be used numerous times in the future.
  - Big impact on users.
  - Work takes longer than a month.
- Start from the problem: The clearer the problem statement, the more obvious the solution.
- Keep template simple.
- Gather and review together, write alone.
- Prefer good to perfect.
Strategy docs:
- Look for controversial decision that came up in multiple designs.
- Guide trade-offs and explain rationale.
- A Framework for Responsible Innovation.
- How Big Technical Changes Happen at Slack.
- In Good Strategy, Bad Strategy terms:
  - Strategy doc == diagnosis + guiding principles.
  - Design doc == coherent action.
- Advise:
  1. Get started.
  2. Write the specifics:
    - Write until you start to generalize.
    - Specific statements create alignments; generic statements create the illusion of alignment.
  3. Be opinionated.
  4. Show your work.
Best strategies feel too obvious.
Strategies worth writing:
1. When should we write a design doc?
2. Which DBs do we use for which cases?
3. How should we stage migration from monolith to microservices?
Vision:
- Reconcile strategies, extrapolate how their trade-offs will play out over the next two to three years.
- Advice:
  - Grounded on serving your business and your users.
  - Be ambitious, but not audacious.
  - Assume everything is going to go well, but don't assume infinite resources.
  - Stay concrete and specific.
  - One or two pages long.
- Don't expect load of excitement:
  1. Vision is for people writing strategies and these are few.
  2. Great visions are so obvious that it bores.
- Measure successful vision by comparing design docs before/after it.
Managing technical quality:
- At a well-run and successful company, most of your previous technical decisions won't meet your current quality threshold.
- Approaches, from cheaper to most complex:
  1. Fix hot spots:
    - Unless your company is creating problems faster than you are able to fix hot spots.
  2. Best practices:
    - Limit to 1 new practice at a time.
    - Steps:
      1. Document the approach.
      2. Experiment with few engaged teams.
      3. Sand down rough edges.
      4. Improve doc.
      5. Go to (2).
    - See Accelerate recommendations.
  3. Leverage points:
    - When you want to roll out more than one practice at a time.
    - Leverage points: prevent gross quality failures and reduce the cost of future quality investments.
    - 3 most typical:
      1. Interfaces between systems: encapsulate implementation.
      2. Stateful systems: Failure modes, consistency behaviour, performance.
      3. Data models: Prevent invalid states, evolve over time.
  4. Technical vectors:
    - Architect's decision degrade the further they get from doing real work on real code in real process.
    - Technical vector: where technical decisions point to, hopefully towards the vision.
    - Tools:
      1. Direct feedback.
      2. Refine engineering strategy.
      3. Enforce in workflows and tooling.
      4. Train during onboarding.
      5. Use Conway's law.
      6. Curate technology change with architecture reviews or similar.
  5. Measure technical quality:
    - Some examples in page 61.
    - Be detailed in your definition of quality.
  6. Technical quality team:
    - aka Dev Productivity, DevTools, Product Infra.
    - 1 for each 15 devs.
    - For success:
      - Trust metrics over intuition.
      - Keep intuition fresh: team embedding.
      - Treat it as a product.
      - Do fewer things, but better: Tool quality will impact the whole org.
      - Don't hoard impact: Decentralized approach.
  7. Quality program:
    - Hire a Technical Program Manager.
    - Org wide change.
    - Approach on page 66.
Stay aligned with authority:
- Don't surprise and don't get surprised by your manger.
- Feed them context:
  - Bring problems to inform them, make it clear that is not to solve them.
To lead, you have to follow:
- Leader:
  1. Has a vision on how things should be.
  2. Sees how things are.
  3. Takes action to narrow (1) and (2).
  4. This approach only creates early success.
- Most effective leaders spend more time following than they do leading:
  1. Follow if you disagree just in a minor way.
  2. Follow other trustworthy leaders.
  3. Make your feedback comments explicitly optional (non-blocking).
Learn to never be wrong:
- At the same time:
  1. Have a deep perspective on technology and architecture.
  2. Remain skeptical of yourself.
- Meetings with multiple failed reframings almost always end with scheduling another meeting.
  - Ask three good questions before sharing your perspective:
    - Good questions are asked with the desire to learn, and they are specific.
- Your career depends more on being easy to involve than being technically correct.
Create space for others:
- A good discussion is one that it turns out you didn't need to attend.
  - When you make a key contribution, think what needs to happen for someone else to make it next time.
- In meetings:
  - Ask questions.
  - Pull people not participating.
  - Be the one taking notes (less time to speak).
- Sponsor other to do "your" work.
Build a network of peers:
- Regulars with previous managers/colleges.
- Blogging/public speaking.
- Internal networks.
- Ambient networks: follow interesting people in Twitter.
Present to executives:
- Always for planning, status reports or resolve misalignment.
- Goal is always to extract as much perspective from the executive as possible.
  - Never to change their mind.
- Barbara Minto, The Pyramid Principle:
  - The clearest sequence to present your ideas is always to give the summarizing idea before you give the individual ideas being summarized.
- Recommended doc structure:
  1. Situation: relevant context.
  2. Complication: why situation is problematic.
  3. Question: what is the core question to address.
  4. Answer: what is the best answer.
- Mistakes to avoid:
  1. Never fight feedback.
  2. Don't evade responsibility or problems.
  3. Don't present a question without an answer.
  4. Avoid academic-style presentations.
  5. Don't fixate on your preferred outcome.
- Share early drafts with executives attending the meeting asking for feedback.

Chapter 3 - Getting the title where you are

Opportunities are not evenly distributed across the company.
Promotion packets:
- Write them before, so it becomes a map to accomplishing your goal.
- Template:
  1. What staff project?
    - Your role, project impact, complexity.
    - Links to design documents.
  2. High-leverage ways you have improved the org.
  3. People mentored and through what.
  4. Glue work.
  5. Which teams/leaders are familiar with and advocate for your work.
- Process:
  1. Answer why you are doing it.
  2. Take it easy, it takes time.
  3. Bring your manger into the fold:
    - Make her aware of your goal.
    - Ask for feedback, guidance.
  4. Write your promotion packet.
  5. WAit two days and edit.
  6. Edit with trusted peers.
  7. Edit with manager:
    - Ask for gaps nad how to address them.
  8. Review periodically with manager.
Find a sponsor.
Staff project is usually not required, but worth pursuing for personal growth and ease the promotion.
Must get into the meeting where important decision are made. You need:
1. Bring something useful that is different:
  - Expertise, detailed knowledge, similar past experience.
2. Sponsor.
3. Make your sponsor aware that you want to be there.
Folks evaluate leaders on how aligned their teams are with their approach.
Being visible.

Chapter 4 - Deciding to Switch Companies

Salary Negotiation.

Book notes: Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith

Dan Lebrero — Wed, 04 May 2022 16:17:57 +0000

These are my notes on Monolith to Microservices by Sam Newman.

Practical and useful book on the subject.

Key Insights

Microservices are independently deployable services modeled around a business domain.
Don't focus on size but:
1. How many microservices can you handle.
2. How to define boundaries to get the most out of the microservices, without everything becoming a horrible coupled mess.
Modular monolith still have the challenge of a monolith DB.
DDD:
- Aggregate: self-contained unit that have a life cycle around them.
- Bounded context (BC) + Aggregate == unit of cohesion.
- The aggregate is a self-contained state machine that focuses on a single domain concept in our system, with the BC representing a collection of associated aggregates, with an explicit interface to the wider world.
Microservices are not the goal.
- What are you going to achieve that you cannot with your current architecture?
- Have you considered alternatives to using microservices?
- How will you know if the transition is working?
Reuse it not an outcome.
When not to use microservices:
1. Unclear domain.
2. Startups.
3. Customer installed and managed software.
Dr John Kotter's eight-step process for implementing org change (from Leading Change):
1. Establish a sense of urgency.
2. Creating the Guiding Coalition.
3. Developing a Vision and Strategy.
4. Communicating the change vision.
5. Empowering employees for broad-based action.
6. Generating short-team wins.
7. Consolidating gains and producing more change.
8. Anchoring new approaches in the culture.
Small changes, little steps.
- The biggest the bet and bigger the accompanying fanfare, the harder is to pull out when it is going wrong.
Pattern: Change Data Capture (CDC):
- Use when there is no other option.
Pattern: Database-as-a-Service interface.
When you are relying on network analysis to determine who is using your database, you are in trouble.
Split the DB first or the code?
- Code first.
Static reference data:
- Duplicate, dedicated schema, static library.
- Data service: only when creating microservices is cheap.
BPM tools: issue is that they are non-dev tools that end up being used by devs.
The more coupling, the earlier the pains will manifest:
- 2-10 services:
  - Breaking changes.
  - Reporting.
- 10-50:
  - Ownership at scale.
  - Developer experience.
  - Running too many things.
- 50+:
  - Loads of teams.
  - Global vs local optimization.
  - Orphaned services.
- In all:
  - Robustness and resilience.
  - Monitoring and troubleshooting.
  - End-to-end testing.
    - Solutions:
      - No cross team tests.
      - Consumer-driven contracts.
      - Use automated release remediation and progressive delivery in addition to end-to-end tests.
Without strong code ownership (one and only one team can change service, other teams can do pull requests to propose changes) a microservices' architecture will grow into a distributed monolith.

Chapter 1 - Just Enough Microservices

Microservices are independently deployable services modeled around a business domain.
- Type of SOA.
Start with the technology that you know.
Don't focus on size but:
1. How many microservices can you handle.
2. How to define boundaries to get the most out of the microservices, without everything becoming a horrible coupled mess.
Modular monolith still have the challenge of a monolith DB.
Other monoliths:
- Distributed monolith.
- Third-party black-box systems, both on premise and SaaS.
Coupling types:
- Types.
- Implementation coupling.
- Temporal coupling (in the sense of sync calls).
- Deployment coupling (release trains).
- Domain coupling.
DDD:
- Aggregate: self-contained unit that have a life cycle around them.
- Bounded context (BC):
  - Hide implementation and internal details.
- BC + Aggregate == unit of cohesion.
- The Aggregate is a self-contained state machine that focuses on a single domain concept in our system, with the BC representing a collection of associated aggregates, again with an explicit interface to the wider world.
Start by targeting services that encompass entire BC.
- You can split them further latter, hiding this decision.

Chapter 2 - Planning a Migration

Microservices are not the goal.
Key questions:
- What are you going to achieve that you cannot with your current architecture?
- Have you considered alternatives to using microservices?
- How will you know if the transition is working?
Reuse it not an outcome.
Limit the scope of expected outcomes.
Possible whys:
- Improve team autonomy, alternatives:
  - Modular monoliths.
  - Assign responsibilities based on functional grounds.
  - Self-servicing.
- Reduce time to market, alternatives:
  - Value stream mapping to identify bottlenecks.
- Scale cost-effectively for load, alternatives:
  - Vertical or horizontal scaling.
- Improve robustness:
  - Being able to react to expected variations.
  - Alternatives:
    - Running multiple copies of your monolith.
    - Use more reliable SW/HW.
    - Automate manual processes.
- Scale number of developers, alternatives:
  - Modular monolith (but less).
- Embrace new technology, alternatives:
  - None.
When not to use microservices:
1. Unclear domain:
  - Getting services wrong can be expensive due to large number of cross-service changes and overly coupled components.
2. Startups.
3. Customer installed and managed software:
  - Due to increased operational complexity.
Dr John Kotter's eight-step process for implementing org change (from Leading Change):
1. Establish a sense of urgency.
2. Creating the Guiding Coalition:
  - Try to bring somebody from "the business".
3. Developing a Vision and Strategy:
  - Vision: realistic yet aspirational.
  - Commitment to vision is important, but overly commitment to strategy can be dangerous
4. Communicating the change vision:
  - Face to face + broadcast.
5. Empowering employees for broad-based action:
  - Bandwidth change: increase capacity or reduce load.
6. Generating short-team wins.
7. Consolidating gains and producing more change:
  - Keep pushing on.
8. Anchoring new approaches in the culture:
  - Communicate successes and failures.
Small changes, little steps.
Whiteboard is where the cost of change and the cost of mistake is the lowest.
Where to start?
- Identify BC and their relationships:
  - Just enough DDD to get started.
  - Maybe use Event Storming
  - Relationship show how easy/difficult should be to extract that BC (warn: code may disagree).
- Plot BC in:
Is the transition working?
- Regular checkpoints. Agenda:
  1. Restate what you are trying to achieve. Does it still make sense?
  2. Review quantitative metrics.
  3. Ask for qualitative feedback: Happier in BVSSH.
  4. Decide if any change is needed.
Avoid the skunk cost fallacy:
- The biggest the bet and bigger the accompanying fanfare, the harder is to pull out when it is going wrong.

Chapter 3 - Splitting the Monolith

You have more options if you can change the monolith.
- Best if you can copy (not move) existing code.
- Consider refactoring into modular monolith first.
Pattern: Stranger Fig Application:
- Steps:
  1. Identify what to migrate.
  2. Copy to microservice.
  3. Reroute calls to new microservice.
- Safe to rollback routing.
- There must be a clear way to redirect the calls to the new service.
- In case of HTTP, a reverse HTTP proxy in front of the monolith.
  - Consider Ngnix + Lua if need something custom.
  - If custom logic is very complex (like changing protocol from SOAP to gRPC), consider:
    1. Avoid putting the logic in the proxy, as it is a shared service between teams.
    2. Implement it in the microservice.
    3. Use service mesh.
- In message systems, the proxy consumes the monolith queue and does a content-based routing to two queues: new monolith queue and microservices one.
- While migrating, avoid any functionality change.
Pattern: UI Composition:
- UI to call new microservice.
- Widget or page level.
- Mobile apps are monoliths, unless you can make changes without resubmitting them.
- Micro-frontend.
Pattern: Branch by Abstraction:
- Steps:
  1. Create abstraction for the functionality to be replaced.
  2. Change clients to use new abstraction.
  3. Create new implementation that uses the new microservice.
  4. Switch over the new implementation.
  5. Cleanup.
- Verify branch by abstraction:
  - If call to new implementation fails, call the old implementation.
Pattern: Parallel Run:
- In strangler and branch by abstraction, call both implementation and compare results.
- Can check also performance.
- N-version programming:
  - Implement functionality in several different ways, and do a parallel run, choosing the "correct" (quorum) one.
  - For fault tolerance and to avoid bugs.
- Verification techniques:
  - Use spy in microservices to record what will do, but without doing it.
    - When duplicated side-effects are not ok.
  - Github scientist.
- Not trivial, use just when there is a high risk.
Pattern: Decorating Collaborator:
- When cannot or don't want to change monolith.
- Proxy works as a decorator that calls new microservice in addition to old monolith.
Pattern: Change Data Capture (CDC):
- React to changes happened in the data store.
- Typical implementations:
  1. DB triggers:
    - Use very sparingly.
  2. Transaction log poll.
  3. Batch delta copier:
    - Process that on a regular schedule scans the DB to find what data has changed since last run.
- Use when there is no other option.

Chapter 4 - Decomposing the database

Shared DB:
- It is ok with:
  - Read-only static reference data that is stable and has a clear owner.
  - Pattern: Database-as-a-Service interface.
- Pattern: Database view:
  - Less coupling than shared DB.
  - When you are relying on network analysis to determine who is using your database, you are in trouble.
  - Useful only for read-only.
- Pattern: Database wrapping service:
  - Move database dependencies to service dependencies.
  - When is too hard pulling the schema apart.
  - More flexible than DB view.
  - Can take writes.
  - Stepping stone, buys you time.
- Pattern: Database-as-a-Service interface:
  - Create a specific and dedicated DB to be accessed externally.
  - Mapping engine options:
    1. CDC.
    2. Batch process.
    3. Build from an event log.
Transferring Ownership:
- Pattern: Aggregate exposing monolith:
  - Monolith expose a proper aggregate API.
  - When a newly extracted microservice still needs data owned by the monolith.
  - Maybe a future microservice.
- Pattern: Change data ownership:
  - When monolith still depends on newly extracted microservice data.
  - Ideally monolith should call new service API:
    - Copying the data back to the monolith DB as an alternative.
Data synchronization:
- Pattern: Synchronize data in application:
  - Steps:
    1. Bulk synchronize data:
      - If monolith was kept online, implement CDC to copy data since snapshot created.
    2. Synchronize on write, read from old schema.
    3. Synchronize on write, read from new schema.
    4. Decommission old schema.
- Pattern: Tracer write:
  - Same as synchronize data in app, but one table at a time (instead of the whole bundled context) using the new microservice API.
Split the DB first or the code?
- DB first:
  - Easy, little short-term benefit.
  - When concerned about performance or data consistency.
  - Pattern: repository per BC:
    - First step to understand dependencies.
  - Pattern: database per BC:
    - Bet for future split.
    - Recommend for greenfield.
- Code first:
  - Most common.
  - Short-term improvements.
  - Pattern: Monolith as data access layer:
    - Same pattern as "aggregate exposing monolith".
  - Pattern: Multischema storage:
    - New microservice uses new schema for new functionality/data, old schema for existing data.
- DB and code at the same time:
  - Avoid.
Schema separation examples:
- Pattern: Split table:
  - When table is owned by 2 or more BC.
  - Lose referential integrity.
  - Will need to chose the owner of the data.
- Pattern: Move foreign-key relationship to code:
  - Increase latency: from 1 join query to 1 select + n service calls.
  - Data consistency, deletion options:
    1. Check with all services before deleting:
      - More coupling, reverse dependency.
      - Don't use.
    2. Handle 404/410 gracefully in dependant service.
    3. Don't allow deletion:
      - Soft delete/tombstone record.
Static reference data:
- Duplicate:
  - As it changes infrequently maybe ok.
  - For large volume of data.
  - Background process to update it.
- Dedicated schema:
  - No duplication, always up to date.
  - Allows for cross-schema joins.
- Static library:
  - For small datasets.
- Data service:
  - When creating a new microservice is cheap.
  - Maybe can emit change events.
Avoid distributed transactions.
Sagas:
- Model transactions as business processes.
- Specially for long lived transactions.
- Rollback with compensating transactions.
- Two implementations:
  1. Orchestrated:
    - Central coordinator. Command and control.
    - Pro: easier to understand.
    - Cons: coupling.
    - Warn: Anemic microservices: coordinator having too much logic that should be in microservices.
    - Different services can play the coordinator role for different flows.
    - BPM tools: issue is that they are non-dev tools that end up being used by devs.
    - Camuda and Zeebe are targeted to microservices developers.
  2. Choreographed:
    - Distributed responsibility. Trust but verify.
    - Usually event based.
    - Pro: decoupled.
    - Cons: harder to understand whole flow.
    - Use correlation Id to build/know the state of the saga:
      - Service to read all events to show view.

Chapter 5 - Growing Pains

More detailed in "Building microservices" book.
Based on anecdotal experience.
The more coupling, the earlier the pains will manifest:
- 2-10 services:
  - Breaking changes.
  - Reporting.
- 10-50:
  - Ownership at scale.
  - Developer experience.
  - Running too many things.
- 50+:
  - Loads of teams.
  - Global vs local optimization.
  - Orphaned services.
- In all:
  - Robustness and resilience.
  - Monitoring and troubleshooting.
  - End-to-end testing.
Ownership at scale:
- Without strong code ownership (one and only one team can change service, other teams can do pull requests to propose changes) a microservices' architecture will grow into a distributed monolith.
Breaking changes:
- Avoiding accidental breaking changes:
  1. Explicit schema to avoid structural breakages.
    - Protolock to prohibit incompatible changes.
  2. To avoid semantic breakages: testing.
    - Make it hard to change a service contract:
      - Make it obvious, no magic, or generate schemas from code.
- Non-accidental:
  1. Do not break, but accrete.
  2. Give consumers time to migrate:
    1. Run two versions of the service.
    2. Support old and new endpoints in the service.
- Be more relaxed if changes are within a team.
Reporting:
- Build a reporting specific schema.
Monitoring and Troubleshooting:
- Log aggregation:
  - First thing to do when implementing microservices.
- Tracing:
  - API GW or service mesh to generate the correlation ID.
- Test in production:
  - Synthetic transactions.
- Towards observability:
  - Distributed Systems Observability.
Local developer experience:
- How many services to run locally?
- Solutions:
  - Stubs.
  - Point to instance running elsewhere.
  - One remote env per developer:
    - Slow to deploy to test changes.
    - Cost.
  - Mix local/remote:
    - Telepresence for Kubernetes.
    - Azure's cloud functions.
Running too many things:
- Deployment, configuration and management of instances becomes more difficult.
- Solution:
  - Kubernetes.
  - Function-as-a-Service (preferred).
End-to-end testing:
- Even slower and even more brittle.
- Solutions:
  - No cross team tests.
  - Consumer-driven contracts:
    - Pact.
  - Use automated release remediation and progressive delivery in addition to end-to-end tests.
Global vs local optimization:
- Solving the same problem twice.
- Divergent tech stack.
- Solutions:
  - Cross-cutting group to raise awareness/make tech decisions.
Robustness and resilience:
- See Release it!.
Orphaned Services:
- In-house service registries that combines service discovery + code repository data.

The self-inflicted denial-of-service (DDoS) attack

Dan Lebrero — Wed, 27 Apr 2022 11:43:35 +0000

I was happily complaining reviewing some pull request, when the following piece of code make me shudder:

var unsent-messages = new List();

function log(message) {
    unsent-message =+ message
}

every (5 secs) {
    send-all(unsent-messages)
    unsent-message = new List();
}

Then it made me smile. I had seen a similar piece of code cause a major outage on a billion dollar company.

Outage 1

As all good outages, it all started with some network misconfiguration.

In this case, the network issue made all the company's systems unreachable from the outside, causing a few minutes outage.

Change was reverted, service was restored, blame was assigned, bureaucracy was added and we all went back to work.

At least for some minutes.

Outage 2

A side effect of fixing the first outage was that in few seconds, like a thundering herd, thousands of our client's browsers were trying to reconnect to back.

All services hold their ground except for one: the browser statistics service.

The Browser Statistics Service (stats service)

The stats service was born to collect feature usage data for one team, for one particular subset of the web application.

The service log into disk the usage data sent by the browser, Splunk read the log files, and the developers build in Splunk their dashboards and alerts.

It was all self-service and very low bureaucracy friction, which meant that all front-end teams ended up adopting it.

Obviously, no stats is complete without knowing how times something fails (aka. exceptions) and in what way (aka. stacktrace).

And here is where trouble started.

A big pile of errors

During the first outage, as none of the backend services were reachable, the client's browsers have been collecting the stats of all failed attempts.

When the first outage was fixed, for the stats service was like:

The service could not write fast enough to disk, so requests started to pile up in the stats service.

The piled up requests increased the memory pressure, causing the stats service to GC like crazy, adding yet more latency.

When the Apaches noticed that an stats service instance was not timely responding, they did the opposite of what you would like: they resent the requests to another instance, basically doubling the pressure.

And, as all these request started to fail or timeout, the client's browser diligently recorded the failure and added it to the pile of errors to report.

DDoS attack

To add salt to the injury, the front-end stats library had been designed to not lose data, hence a failed request to the stats service would not reset the stats.

In fact, the failed stats requests would be recorded and added to the next stats service call, which made the stats message size bigger and bigger, adding more and more pressure to the already struggling stats service and everything on its path.

The amount of traffic caused the Apache layer to start choking, making requests to other downstream services slower and slower.

This raised the number of errors in the client's browsers, which increased the stats request's size even further, which caused the Apaches to choke even more, which added extra latency, causing more errors, bigger requests, more latency, more errors, ...

End result: thousands of browsers all over the globe sending every few seconds megabyte long POST requests to the stats service, telling the stats service that the stats service - and everything else - was failing.

A complete outage thanks to a self-inflicted distributed denial-of-service attack.

Steady State pattern

Multiple things went wrong (and multiple things we fixed afterwards).

Focusing on the initial piece of code, applying the Steady State pattern could have avoided the issue.

Steady State just points out the obvious: nothing is infinite. Not memory, not CPU, not bandwidth, not disk, not time, not money.

So you need to set some limits: the max size in the unsent messages array in this case.

Adding those limits is scary, mostly because the fear of missing out.

But without those limits, your system's stability is at risk, specially when things start to fail.

And remember that ...

Book notes: Turn the Ship Around!

Dan Lebrero — Wed, 20 Apr 2022 12:57:43 +0000

These are my notes on Turn the Ship Around! by L. David Marquet.

Classic.

Key Insights

Divested control only works with a competent workforce that understands the org's purpose.
Disengaged, dissatisfied employees break the spirits of their colleagues.
Thinking that we know something is a barrier to continued learning.
It is contradictory to have an empowerment program whereby you are empowered by your boss.
I was at my best when give specific goals and broad latitude in how to accomplish them.
Are you optimizing the org for your tenure, or forever?
It doesn't matter how smart a plan is if the team cannot execute it.
When performance of a unit goes down after an official leaves, it is taken as a sign that he was a good leader, not that he was ineffective in training his people properly.
Avoiding errors == no new initiatives, no innovation.
Don't move information to authority, move authority to the information.
Worries to delegate control usually fall either in lack of technical competence or lack of clarity on what the org is trying to accomplish. Both can be resolved.
Short, early conversations make efficient work.
Demand of perfect products the first time around results in significant waste and frustration.
Trust == I believe you don't lie, but you can still be wrong.
Use "I intend to ... " to turn passive followers into active leaders:
- It has to include all the rationale of the decision which caused subordinates to think at the next higher level.
Adherence to the process frequently becomes the objective, as opposed to achieving the objective that the process was put in place to achieve.
When I heard what my officers where thinking, it made it much easier for me to keep my mouth shout and let them execute their plans.
Lack of certainty is strength and certainty is arrogance.
Inspectors == disseminate ideas in the org, learn from others.
Balancing the courage to hold people accountable for their actions with the compassion for their honest efforts.
Control without competence is chaos.
How to design a training program.
A briefing is a passive activity for everyone except the briefer.
Provide your people with the objective and let them figure out the method.
One hour session with supervisors:
- Rule: only talk about long-term issues, and primarily people issues.
Steps to move to leader-leader.

Mechanisms

Compiled list of mechanism:

Control mechanisms:
- Find the genetic code for control and rewrite it.
- Act your way to new thinking.
- Short, early conversations make efficient work.
- Use "I intent to ..." to turn passive followers into active leaders.
- Resist urge to provide solutions.
- Eliminate top-down monitoring systems.
- Think out aloud (both superiors and subordinates):
  - Also a clarity practice.
- Embrace the inspectors.
Competence mechanisms:
- Take deliberate action.
- We learn (everywhere, all the time).
- Don't brief, certify.
- Continually and constantly repeat the message.
- Specify goals, not methods.
Clarity mechanisms:
- Achieve excellence, don't just avoid errors.
- Build trust and take care of your people.
- Use your legacy for inspiration.
- Use guiding principles for decision criteria.
- Use immediate recognition to reinforce desired behaviours.
- Begin with the end in mind.
- Encourage a questioning attitude over blind obedience.

Introduction
Starting Over
- Chapter 1 - Pain
- Chapter 2 - Business as Usual
- Chapter 4 - Frustration
- Chapter 5 - Call to Action
- Chapter 7 - I Relieve You!
Part 2 - Control
- Chapter 8 - Change, in a word
- Chapter 9 - Welcome Aboard Santa Fe!
- Chapter 10 - Underway on Nuclear Power
- Chapter 11 - I Intent To ...
- Chapter 12 - Up Scope!
- Chapter 13 - Who's Responsible?
- Chapter 14 - A New Ship
- Chapter 15 - We Have a Problem
Part 3 - Competence
- Chapter 16 - Mistakes Just Happen!
- Chapter 17 - We learn
- Chapter 18 - Under Way for San Diego
- Chapter 19 - All Present and Accounted for
- Chapter 20 - Final Preparations
Part 4 - Clarity
- Chapter 21 - Under Way for Deployment
- Chapter 22 - A Remembrance of War
- Chapter 23 - Leadership at Every Level
- Chapter 24 - A Dangerous Passage
- Chapter 25 - Looking Ahead
- Chapter 26 - Combat Effectiveness
- Chapter 27 - Homecoming
- Chapter 28 - A New Method of Resupplying
Chapter 29 - Ripples

Introduction

Disengaged, dissatisfied employees break the spirits of their colleagues.
Leader-follower:
- People who are treated as followers have the expectations of followers and act as followers.
  - Those who take orders run at half speed.
- Performance of the organization is closely linked to the ability of the leader.
- Leader leaving is a major event.
Leader-leader:
- Evolutionary steps.
- Divested control only works with a competent workforce that understands the org's purpose:
  - Control, competence, clarity.
- As control is divested, both tech competence and organizational clarity need to be strengthened.

Starting Over

Thinking that we know something is a barrier to continued learning.

Chapter 1 - Pain

It is contradictory to have an empowerment program whereby you are empowered by your boss.
I was at my best when give specific goals and broad latitude in how to accomplish them.
- Irritated when given a task list.

Chapter 2 - Business as Usual

Are you optimizing the org for your tenure, or forever?
It doesn't matter how smart a plan is if the team cannot execute it.
When performance of a unit goes down after an official leaves, it is taken as a sign that he was a good leader, not that he was ineffective in training his people properly.

Chapter 4 - Frustration

Being curious (make sure you know) vs being questioning (make sure subordinate knows).
Bad: Technical knowledge as the basis of leadership.

Chapter 5 - Call to Action

Flashlight anecdote: leading by example.
Little things like lack of punctuality are indicative of much bigger problems.
Avoiding mistakes became the primary driver for all actions:
- Focus exclusively on satisfying minimum requirements.
Leave request required 7 approvals == huge delay.
- It was the system, not the people, that failed.

Chapter 7 - I Relieve You!

Self-reinforcing downward spiral:

Avoiding errors == no new initiatives, no innovation.
Shift from avoiding errors to achieve excellence.
Connecting our day-to-day activities to something larger was a strong motivator for the crew.
Clarity: Start with Why by Simon Sinek.

Part 2 - Control

Don't move information to authority, move authority to the information.

Chapter 8 - Change, in a word

Start at the mid-level managers:
- Starting top-down contrary to a bottom-up leadership philosophy.
- Starting at the bottom: too inexperience people.
When delegating decision-making authority, I was worried ... that the interests of the command would not be protected. That didn't happen.
Find the genetic code for control and rewrite it:
- Senior leadership exercise:
  1. Identify in the org's policy documents where decision-making authority is specified.
  2. Identify decisions that are candidates for being pushed to the next lower level in the org.
  3. For easy decisions, draft change. For large decisions, maybe disaggregate.
  4. Ask each participant to complete the following sentence in a 5x8 card:
    - "When I think about delegating this decision, I worry that ..."
  5. Post cards on the wall, go on a long break, and let the group mill around them.
  6. Sort and rank worries and begin to attack them.
- Worries usually fall either in lack of technical competence or lack of clarity on what the org is trying to accomplish. Both can be resolved.

Chapter 9 - Welcome Aboard Santa Fe!

The fear and cost of being different.
How to embed a cultural change in your org:
1. Identify what cultural change is required.
2. In a 5x8 card, people complete the sentence:
  - "I'd know we achieved [this cultural change] if I saw employees ..."
  - Expectation is that the answers will be specific and measurable.
3. Allow 5 mins, tape card to wall, go to break, have everyone mill around reading cards.
4. Depending on discussions and quality of answers, give everyone a second shot.
5. Short and prioritize answers.
6. Discuss how to code the behaviours into company's practices.
7. Write new practices in appropriate company procedures.

Chapter 10 - Underway on Nuclear Power

Adding additional verification/review steps to a process == extra work without making anything better.
Short, early conversations make efficient work.
- Early feedback of unfinished work.
- Not to order but to provide clarity.
- Hierarchy is supposed to protect Commanding Officer time (== highly valuable).
- Demand of perfect products the first time around results in significant waste and frustration.
- Early feedback raised the question:
  - "Don't you trust me?"
  - Trust == I believe you don't lie, but you can still be wrong.

Chapter 11 - I Intent To ...

Use "I intend to ... " to turn passive followers into active leaders.
- Disempowered phrases that passive followers use:
  - Request permission to ...
  - I would like to ...
  - What should I do about ...
  - Do you think we should ...
  - Could we ...
- Empowered phrases that active doers use:
  - I intent to ...
  - I plan on ...
  - I will ...
  - We will ...
  - "I intend to" has to include all the rationale of the decision which caused subordinates to think at the next higher level:
    - Basically training them to be their boss.
Book recommendations:
- 7 Habits of highly effective people.
- The 8th Habit: From Effectiveness to Greatness.

Chapter 12 - Up Scope!

Mechanism: resist the urge to provide solutions:
- If there are a lot of short notice decisions, you are in reactive mode:
  - No time to training the team.
  - No time to think.
- To get the team thinking:
  - Urgent decisions: make it, then the team "red-team" the decision.
  - Soon decisions: ask for team input, make the decision.
  - Decision can be delayed:
    - Force team to provide inputs.
    - Don't force team to come to consensus.
    - Cherish dissension: if everybody thinks like you, you don't need them.

Chapter 13 - Who's Responsible?

Mechanism: Eliminate top-down monitoring systems:
- Avoid system whereby senior personnel are determining what junior personnel should be doing:
  - Gives ownership and responsibility.
- You want to keep data collection and measuring processes that simply report conditions without judgment.
- Adherence to the process frequently becomes the objective, as opposed to achieving the objective that the process was put in place to achieve.

Chapter 14 - A New Ship

Mechanism: Think out loud:
- When I heard what my officers where thinking, it made it much easier for me to keep my mouth shout and let them execute their plans.
- As the captain, impart important context and experience to subordinates.
- We aren't comfortable talking about gut feelings or anything with probabilities.
- Lack of certainty is strength and certainty is arrogance.

Chapter 15 - We Have a Problem

Mechanism: Embrace the Inspectors:
- No being defensive, but taking ownership.
- Inspectors == disseminate ideas in the org, learn from others.

Part 3 - Competence

Chapter 16 - Mistakes Just Happen!

Balancing the courage to hold people accountable for their actions with the compassion for their honest efforts.
7 Steps to Learning from Our Mistakes.
Mechanism: Take deliberate action.
- Usual ways to avoid mistakes:
  1. More training:
    - Implies lack of knowledge, which can be detected with a test.
      - What question would detect it?
  2. Add more supervision: unlikely to help.
- Mistakes due to people in autopilot.
- We want to engage the brain before action:
  1. Prior to action, vocalize and gesture towards what you are about to do, even when alone.
  2. Pause.
  3. Execute action.
  4. Even when in emergency.

Chapter 17 - We learn

Control without competence is chaos.
Mechanism: We learn (everywhere, all the time):
- Training program that employees will want to go to:
  - Purpose: increases technical competence, so that
  - Increases delegation of decision making, so that
  - Increases engagement, motivation and initiative.
- To design, in the next leadership meeting:
  1. Hand out 4x6 cards.
  2. Complete:
    - "Our company would be more effective if [level] management could make decision about [subject].".
    - You fill [level] but the group fills the [subject].
  3. Cards to the wall, break, mill.
  4. Select a couple of subjects.
  5. Ask: what, technically, do the people at this level of management need to know in order to make that decision?
  6. Cards to wall, break, mill.
  7. Collect list of topic for training.

Chapter 18 - Under Way for San Diego

A briefing is a passive activity for everyone except the briefer:
- It is not a decision point.
- No one listens to those briefings.
Mechanism: Don't Brief, Certify.
- In a certification, the person in charge asks them questions:
  - It is a decision point: no certification means no go.
  - Requires more management: both what to do and who will do it.
  - Active: Shift the onus of preparation to participants.
    - Everybody responsible to know their job.

Chapter 19 - All Present and Accounted for

Mechanism: continually and consistently repeat the message:
- Day after day, meeting after meeting, event after event.
- People think they understand, but if they haven't seen it before, it is unlikely that they do.

Chapter 20 - Final Preparations

Mechanism: specify goals, not methods:
- Provide your people with the objective and let them figure out the method.
- Also a mechanism for Clarity.
- Warn: compliance with procedure supplanting achieving the objective.

Part 4 - Clarity

Chapter 21 - Under Way for Deployment

Mechanism: built trust and take care of your people:
- Take care != protecting them from the consequences of their own actions.
- Take care == give them every tool and advantage to achieve their aims in life, beyond the job.

Chapter 22 - A Remembrance of War

Mechanism: use your legacy for inspiration:
- Use past examples to bring clarity to org's purpose.

Chapter 23 - Leadership at Every Level

My job as the command is to tap into the existing energy of the command, discover the strengths, and remove barriers to further progress.
Guiding principle: faced with a decision between two courses of action, provide criteria to chose.
Mechanism: use guiding principles for decision criteria:
- Real ones aligned to org's goals.
- Use them for awards or evaluations.

Chapter 24 - A Dangerous Passage

Mechanism: use immediate recognition to reinforce desired behaviours:
- Don't delay it for official/admin rubber-stamp.
- Awards should not pit a teammate against other, but the team against the external world.
  - Example: avoid "Top 10%".
  - Would hinder collaboration otherwise.
Gabe Zichermann's blog on gamification.

Chapter 25 - Looking Ahead

Mechanism: begin with the end in mind:
- One hour session with supervisors:
  - Rule: only talk about long-term issues, and primarily people issues.
- Long term: write down end-of-tour awards/goals (1-3 years in future).
- Make goals specific and measurable:
  - "How would you know if xxx was improved?"
Not mentor-mentee sessions, but mentor-mentor:
- Both pars learn as much.

Chapter 26 - Combat Effectiveness

Mechanism: encourage a questioning attitude over blind obedience.

Chapter 27 - Homecoming

Steps to move to leader-leader:
1. Identify where excellence is created:
  - Generally interfaces with the customers and the physical world.
2. Figure out what decision the people responsible for the interfaces need to make to achieve excellence.
3. Understand what is needed for those employees to make those decisions:
  - Technical knowledge.
  - Understanding org's goals.
  - Authority.
  - Responsibility for the consequences.

Chapter 28 - A New Method of Resupplying

Mechanism: don't empower, emancipate:
- Empowerment is necessary but not enough:
  - It is still a top-down structure as leader empowers followers.
- You know you have an emancipated team when you no longer need to empower them:
  - You are not their source of power.

Chapter 29 - Ripples

Additional benefits:
1. Excellence after leader leaving.
2. Development of new leaders.
https://intentbasedleadership.com.

My personal path to Senior developer ... and beyond!

Dan Lebrero — Tue, 12 Apr 2022 18:35:00 +0000

I have always struggled to answer the question "How do I become a senior developer?".

It is one of these things that happened so long ago, that I barely remember when or how it happened. Age takes its toll, and I was not clever enough to be blogging back then ( nudge, nudge, wink, wink).

But, as I was today moving some book notes to the blog, it dawned on me what my path was to become a senior developer.

And it happens to be the same path that moved me to the higher echelons of the IT hierarchy.

It all starts with books

For me, it all starts with reading a book (or books):

Books give me enough (false) confidence to become a big mouth, to have too many opinions, and to make too many suggestions.

These suggestions translate to a boss getting the (wrong) impression that I know what I am talking about, hence being appointed to solve a hairy problem. Or maybe the boss is trying to teach me a lesson.

And once I am given the chance... OMG! The little gaps in your fake knowledge become huge chasms that you need to jump, circumvent, or fill up.

It is all blood, sweat, and tears.

But the toil nurtures the precious experience that cements your knowledge, your most valuable asset (unless your parents are rich, in which case, it is your parents).

Only books?

No. I am sure that other people will have more efficient ways, but, distilled, the general formula is:

Obtain some proto-knowledge.
Find the courage to speak up, to have an opinion. Remember that there is a thin line between courage and stupidity.
Be in an environment that will give you an opportunity. This requires both courage and forward-thinking.
Struggle. Be ready to be wrong. Toil.
Reward! Real world experience == actual knowledge.

Repeat enough times and your knowledge will become wisdom.

And don't hurry, this will require a lot of time.

Cut the chase, what books do I have to read to become a senior software developer?

Here are some lists.

For the record, I disagree with all of them.

Book notes: Sooner, safer, happier: Antipatterns and patterns for business agility.

Dan Lebrero — Tue, 29 Mar 2022 22:31:48 +0000

These are my notes on Sooner, safer, happier: patterns and anti-patterns for business agility by Jonathan Smart with Zsolt Berend, Myles Ogilvie and Simon Rohrer.

The book is like The DevOps Handbook, but less technical and more org-wide.

Key Insights

Better Value Sooner Safer Happier (BVSSH):
- Better == quality.
- Value is unique. OKRs.
- Sooner != faster:
  - Flow efficiency.
  - Lead time.
- Safer: governance, risk, compliance.
- Happier: none of the previous "at any cost".
"It is not my problem. I did my bit. The hole is on the other side of the boat".
SW binary is agile-created (complex) and the path to production is lean (complicated).
- Milestones make no sense on emergent (complex) domains.
Not "the business" but "our business".
Imposing Agile is not agile.
Edgar Schein: learning only happens when survival anxiety is greater than learning anxiety (fear of being a noob again).
"Why" for change must appeal all primary motivators (autonomy, mastery, purpose).
- You cannot over-communicate the why.
Change is a social activity.
- Invite over Inflict.
Orgs are complex adaptive systems:
- Any experiment that takes place in them are not experiments because you cannot undo them.
You are never done improving.
“We don’t have time. We need to go big.” is a fallacy and self-fulfilling prophecy.
Cognitive overload: willpower is a limited resource, and when it runs out you fall back on habits.
As more teams adopt the new ways of working, the impediments will be more systemic and harder to resolve.
Pattern: Descale Before you Scale.
Joseph Campbell: if the path ahead is clear, you are probably on someone else’s.
- Impediments are not in the path; impediments are the path.
Improvement over time is more important than the absolute value.
Silence is unhealthy.
Instead of reporting lines, supporting lines.
People need to be recognized for the lowest cost and quickest time to failure.
Water-scrum-fall:
- Urgency paradox: ideas sit 12-18 months on big upfront prioritization, and when they hit the dev team they suddenly become urgent.
Value streams are not oriented around customer personas.
Funding is a constraint on which to maximize value.
Applications and puppies have something in common: they are for life, not just for Christmas.
Install Jira and Jenkins and you're agile.
Don't manage dependencies, break team. > Ignorance is the single greatest impediment to throughput. Dan North at Introducing Deliberate Discovery
Following the plan == not learning. > Humans need to belong to groups, but the tribal instinct is also an instinct to exclude Amy Lynn Chua at Political Tribes
When a measure becomes a target, it ceases to be a good measure.
Tacit knowledge can only be shared by socialization (pairing, face-to-face) and a bias for action.
Learning is more effective when exercises are in the learner context.
There is no such thing as best practice. Your context is unique.

List of Patterns and Anti-Patterns

Pattern	Anti-pattern
Focus on Outcomes	Doing an Agile Transformation
Start with Why; Empower the How	Using Old Ways of Thinking to New Ways of Working
Achieve Big Through Small	The Bigger the Capital “T” Transformation, the Bigger the Change Curve
Descale Before you Scale	Scaling Agile before Descaling Work
Scale Agility, not Agile, Vertically then Sideways	Grass Roots Hits a Grass Ceiling
Not one size fits all	One Size Fits All
Invite over Inflict	Inflict over Invite
Leaders Go First	Do as I say, not as I do
Psychological Safety	Psychological unsafe
Emergent Mindset with Servant Leadership	Deterministic Mindset
Optimize for Fast End-to-End Flow	Local Optimization
Outcome Hypothesis	Milestone-driven Predicted solutions
Intelligent Flow	Headless Chicken
Stop Starting, Start Finishing	Start Starting
Safety within Safety	Lack of Safety within Safety
Organize Safety by Value Stream	Role-Based Safety Silos
Intelligent Control	Fixed Mindset to Risk
Go Slower to Go Faster	Going Faster Leads to Going Slower
Continuous Technical Excellence	Agile Hollow Shell
Architect and Organize for Flow	Misalignment of Teams and Architecture
Smart People and Smart Teams with Robot Friends	Tools Over People
Optimize for Learning	Information and Learning Silos
Nested Learning with Built-in Feedback Loops	Output over Outcomes
Communicate, Communicate, Communicate	The Bubble Effect
Be Comfortable with Uncertainty	Applying a Deterministic Approach to an Emergent Domain
Measure for Learning	Weaponized Metrics

A Sense Of Urgency
How We Got Here
Chapter 1 - Focus On Outcomes: Better Value Sooner Safer Happier
Chapter 2 - Achieve Big Through Small
Chapter 3 - Optimization over One Way; Invite over Inflict
Chapter 4 - Leadership Will Make It or Break It
Chapter 5 - Build the Right Thing; Intelligent Flow
Chapter 6 - Building the Thing Right; Intelligent Control
Chapter 7 - Continuous Attention to Technical Excellence
Chapter 8 - Create a Learning Ecosystem
Chapter 9 - The Best Time to Plant a Tree is Twenty Years Ago; the Second Best Time is Now

A Sense Of Urgency

There is no such thing as best practice. Your context is unique.
Better Value Sooner Safer Happier (BVSSH):
- Better == quality.
- Value is unique.
- Sooner != faster.
- Safer: governance, risk, compliance.
- Happier: none of the previous "at any cost".
Practices = principles + context.

How We Got Here

Maximize outcomes with minimal output.
Doing Agile vs being agile.
"It is not my problem. I did my bit. The hole is on the other side of the boat".
SW binary is agile-created (complex) and the path to production is lean (complicated).

Chapter 1 - Focus On Outcomes: Better Value Sooner Safer Happier

They apply to the whole org:
- Not "the business" but "our business".
Value == OKRs.
Sooner:
- Flow efficiency:
  - work time / end to end lead time.
  - Typically, 10% or lower.
- Lead time:
  - Measure the 85% percentile and its change over time.
  - Throughput:
    - Items per period.
    - Should increase when reducing lead time.
Values balance each other.
Cheaper == anti-pattern:
- Increases hidden costs via a reduction in flow efficiency.
Anti-pattern: Doing an Agile Transformation
- Reasons to try an Agile Transformation:
  - “We are being disrupted. If we don’t change we won’t survive”.
  - Issues:
    1. Optimizes for the agile tool, not the outcomes.
    2. Cargo cult. “Doing agile”.
Anti-pattern: Using Old Ways of Thinking to New Ways of Working.
1. Imposing Agile is not agile:
  - Top-down, command-and-control.
  - Big upfront planning.
  - Deadlines.
2. Emotional reactions:
  - Fear and resistance.
  - Loss aversion.
  - Agentic state:
    - Blindly obey orders, passing the responsibility for the consequences of the person giving the orders.
3. Remove intrinsic motivators:
  - Autonomy: no choice on Transformation.
  - Mastery: new thing, hence becoming a beginner.
  - Purpose: If the reason for AT is reducing cost or increasing benefits.
Pattern: Focus on Outcomes.
Pattern: Start with Why; Empower the How:
- Simon Sinek: people don’t buy what you do, they buy why you do it.
- Edgar Schein: learning only happens when survival anxiety is greater than learning anxiety (fear of being a noob again).
- “Why” for change must appeal all primary motivators.
- You cannot over-communicate the why.
- Orgs are complex adaptive systems:
  - Any experiment that takes place in them are not experiments because you cannot undo them.
- Start with:
  1. Identify early adopters by running a voluntary Community of Practice.
  2. Create a central Ways of Working Center of Enablement.
You are never done improving.

Chapter 2 - Achieve Big Through Small

Anti-pattern: The Bigger the Capital “T” Transformation, the Bigger the Change Curve.
- Change curve:
- “We don’t have time. We need to go big.” is a fallacy and self-fulfilling prophecy.
- Cognitive overload: willpower is a limited resource, and when it runs out you fall back on habits.
Anti-pattern: Scaling Agile before Descaling Work.
- Parkinson’s law.
- Also Parkinson: bureaucracies expands at a rate of 5-7% each year irrespective of the variation in the amount of work.
  - Policies, standards and controls expand with the number of control staff employed.
- Scale of the org itself can be an impediment for BVSSH.
Anti-pattern: Grass Roots Hits a Grass Ceiling.
Pattern: Achieve Big Through Small:
- People have a limited capacity to unlearn and relearn.
- The only way to determine the optimal course of action is by safe-to-learn doing.
- Rule of one:
  - 1 experiment.
  - 1 customer or team.
  - 1 location.
  - In production.
As more teams adopt the new ways of working, the impediments will be more systemic and harder to resolve.
- Don’t continue until these are alleviated.
Adoption as an S-curve:
- One per business unit and key systemic enabler.
- One small WoW CoE per business unit, reporting to the business unit, not the central CoE.
Pattern: Descale Before you Scale:
- Descale work and system of work.
- An org is a network of interdependent services:
  - Descaling should include breaking dependencies.
Pattern: Scale Agility, not Agile, Vertically then Sideways.
- New middle management role:
  - Ensure that the mission and desired outcomes are clear, then listen, coach and remove impediments.
  - Coaching Kata.

Chapter 3 - Optimization over One Way; Invite over Inflict

Anti-pattern: One Size Fits All
- Joseph Campbell: if the path ahead is clear, you are probably on someone else’s.
- Your scaling and cultural context is unique.
  - And culture will vary between teams, business units, geography,...
Anti-pattern: Inflict over Invite.
Pattern: Not one size fits all:
- Unique VOICE:
  - Values and principles:
    - Behavioral guardrails that apply across context.
  - Outcomes and purpose:
    - Improvement over time is more important than the absolute value.
    - Targets should be avoided as they invite cargo cult and gaming the system.
  - Intent-based leadership:
    - High autonomy with high alignment.
  - Coaching and support:
    - Coach should be omnists: recognize and respect all bodies of knowledge without fundamentalism.
  - Experimentation:
    - Complex system: probe, sense, respond.
    - Edgar Schein: you cannot understand a system until you try and change it, and when you do try to change it, only then will the underlying mechanisms maintaining the status quo emerge.
“No”, “Not yet” or “What do we stop in order to do this?”
Impediments are not in the path; impediments are the path.
Visualize (flow metrics), stabilize (limit WIP), optimize.
Evolution or revolution (by invitation), depending on your context.
- Only Kanban and Disciplined Agile support evolution.
- Revolution if org is facing an existential crisis, there is support for a bigger dip in the change curve, there is sufficient personal psychological safety, people are volunteering and there is prior experience.
Pattern: Invite over Inflict:
- Improving BVSSH must be communicated as high priority from the board, and that taking action will lead to recognition.
- Exemplary Community:
  - Voluntary.
  - Agree to seek to become exemplary.
  - Benefits:
    - Access to external speakers and experts.
    - First to access new tools.
  - Keep the innovators on the bleeding edge.
  - Members have x20 improvement on most measures.

Chapter 4 - Leadership Will Make It or Break It

Leader: guiding on a journey, following is optional.
Commander: control, obey is mandatory.
Anti-pattern: Do as I say, not as I do:
- Leaders go on the same journey:
  - Become a beginner, trying something new, failing.
Anti-pattern: Psychological unsafe:
- Boeing as a bad example.
- Cost and schedule should not be the primary focus.
- Leaders’ actions and ability to provide or withhold rewards communicate their preferences, which then become the preoccupation of the workforce.
- Silence is unhealthy.
Anti-pattern: Deterministic Mindset:
- All non-trivial change and product development is emergent.
  - The notion that we can predict what is going to happen is […] wasteful, demoralizing and dangerous.
- Can be caused by leadership being uncomfortable exhibiting vulnerability: not knowing.
Instead of reporting lines, supporting lines.
Pattern: Leaders Go First:
- Transformational Leadership (vs transactional), from James Burns’ Leadership:
  1. Role model.
  2. Vision.
  3. Intellectual stimulation.
  4. Coach: care for individuals.
Pattern: Psychological Safety:
- People need to be recognized for the lowest cost and quickest time to failure.
- Steps to build psychological safety:
  1. Set the stage:
    - Not personal incompetence but of the system of work.
    - Failure is learning.
  2. Invite participation.
  3. Respond productively.
Pattern: Emergent Mindset with Servant Leadership.

Chapter 5 - Build the Right Thing; Intelligent Flow

Anti pattern: Local Optimization:
- Water-scrum-fall:
  - Urgency paradox: ideas sit 12-18 months on big upfront prioritization, and when they hit the dev team they suddenly become urgent.
Anti-pattern: Milestone-driven Predicted solutions:
- Milestones with RAG status are culturally toxic:
  - Red is shame and reprisal.
- Milestones make no sense on emergent domains.
Anti-pattern: Headless Chicken.
- Feature factories disconnected from customer needs and company strategy.
- Teams incentivized by output.
Anti-pattern: Start Starting:
- Reducing WIP increases the chances of identifying causality.
Pattern: Optimize for Fast End-to-End Flow:
1. Identify value-streams:
  - Value streams are not oriented around customer personas.
2. Map products to value streams. Identify:
  - Duplication.
  - Monolith that straddles multiple value streams.
3. Create small long-lived teams:
  - T-shaped, multidisciplinary.
  - Roles:
    1. Value Outcome Lead (VOL):
      - The What (value).
      - Outward focus.
    2. Team Outcome Lead (TOL):
      - The How for the system of work and people.
      - Inward focus.
      - Alleviate impediments, build continuous improvement as a daily habit, coach.
    3. Architecture Outcome Lead (AOL):
      - The How of the tech implementation within the org’s broader tech architecture and engineering principles and standards.
      - Solutions architect == hands on.
      - Enterprise architect provide support and governance across value streams.
    4. These roles at every level.
    5. Top level:
      - CEO = VOL.
      - CIO = TOL.
      - CTO = AOL.
4. Funding the Flow of Value:
  - Fund by value streams, not by project.
  - Funding is a constraint on which to maximize value.
Pattern: Outcome Hypothesis:
1. Emergence: Toyota Kata.
2. Nested outcomes: OKRs.
  - Outcome do not cascade, they align.
  - Levels:
    - Strategic objective (multi year).
    - Portfolio objective (<3 years).
    - Portfolio epic (<12 months).
    - Business outcome (<3 months).
    - Experiment (<1 month).
3. Business Outcome: > Due to <this insight> > We believe that <this bet> > Will result in <this outcome> > We will know that we are on the right track when: > Measure 1,2,3,...: quantified and measurable leading or lagging indicators.
4. Rolling roadmaps and fixed dates:
  - Planning is done continuously at the multiple nested cadences.
5. From Project Management Office to Value Enablement Teams:
  - Coaches and supports measuring business outcomes.
  - Assist with prioritization.
  - Coaches the limiting of WIP.
  - Ensures alignment of nested outcomes.
  - Helps collecting lagging and leading data.
  - Consolidates data from value streams.
From “is it done yet?” to “what value we observed, what we have learned, and what steps shall we take next to get closer to the desired outcome?”
Pattern: Intelligent Flow
- Horizontal nested value streams with the vertical nested outcome hierarchy == high alignment.
Pattern: Stop Starting, Start Finishing
- Reduce WIP.
- Pull, don't push.

Chapter 6 - Building the Thing Right; Intelligent Control

Anti-pattern: Lack of Safety within Safety:
- Safety (governance, risk, compliance) people scared of raising concerns due to blame culture.
- Lack of psychological safety in the Safety teams.
Anti-pattern: Role-Based Safety Silos:
- Each safety specialization (Infosec, Data Privacy, Fraud, ...) being its own silo.
  - It causes:
    - Duplication.
    - Information bubbles.
    - Infight between silos and silos vs delivery teams.
Anti-pattern: Fixed Mindset to Risk:
- Risk management (current/future threads) vs compliance (past issues).
Pattern: Safety within Safety.
Pattern: Organize Safety by Value Stream:
- Cross-functional, long-lived safety teams aligned to value streams, with a Centre of Excellence for support, plus a risk catalog.
Pattern: Intelligent Control:
1. Product teams engage with their safety teams on a continuous basis.
2. Safety teams reassess on the quarterly business outcome the frequency of engagement, depending on those business outcomes.
3. Risk stories validated through automated test, or by Value Outcome Lead, or by Safety team.

Chapter 7 - Continuous Attention to Technical Excellence

Applications and puppies have something in common: they are for life, not just for Christmas.
Constant Improvement (Kaizen) and occasional step changes (Kaikaku): Punctuated gradualism.
Anti-pattern: Going Faster Leads to Going Slower:
- Features, and only features.
- No time to follow good dev practices because:
  1. Lack of partnership with "Business".
  2. No time to apply lessons learned.
  3. System Entropy.
Anti-pattern: Agile Hollow Shell:
- Agile without technical excellence is Agile work management.
Anti-pattern: Misalignment of Teams and Architecture.
- Team topologies.
Anti-pattern: Tools Over People:
- Install Jira and Jenkins and you're agile.
- Should be: People -> Process -> Tooling.
- No technical career path for hands-on software developers.
- Not having testers: too much automation.
Pattern: Go Slower to Go Faster:
- Visualize flow of features, defects, risks, debts.
- Prioritize improvement of daily work over daily work.
- Coach for technical excellence/practices is much rarer than "Agile process coach".
Pattern: Continuous Technical Excellence.
Pattern: Architect and Organize for Flow.
- Cognitive Load.
- Don't manage dependencies, break team.
Pattern: Smart People and Smart Teams with Robot Friends:
- Technical career, good example: UK Government digital services career framework.

Chapter 8 - Create a Learning Ecosystem

Anti-patterns: Information and Learning Silos: > Up to 50% of info is lost on every hand-off. Poppendieck at Implementing Lean SW Dev.

Ignorance is the single greatest impediment to throughput.
Dan North at Introducing Deliberate Discovery

Anti-pattern: Output over Outcomes:
- Learning whether we have delivered something valuable or not comes from the customer.
Anti-pattern: The Bubble Effect:

Humans need to belong to groups, but the tribal instinct is also an instinct to exclude
Amy Lynn Chua at Political Tribes
- Silo mentality: not want to share with other tribes.
- Poor learning retention: when people leave, their learnings are lost.
- Duplication.
Anti-pattern: Applying a Deterministic Approach to an Emergent Domain:
- Following the plan == not learning.
Anti-pattern: Weaponized Metrics:
- When a measure becomes a target, it ceases to be a good measure.
Pattern: Optimize for Learning:
- Unlearn.
- Tacit knowledge can only be shared by socialization (pairing, face-to-face) and a bias for action.
- In Clear and Complicated domains: explicit knowledge.
- In Complex and Chaotic: tacit knowledge.
- Identify and break silos with Value Stream Mapping.
Pattern: Nested Learning with Built-in Feedback Loops:
- Learning is more effective when exercises are in the learner context.
- Levels:
  1. Individual:
    - Sharon Bowman four Cs, from Training from the Back of the Room:
      - Connections: Learners connect to the topic exploring prior experiences and knowledge.
      - Concepts and concrete practices: Self-learn.
      - Conclusions: Learners do self-reflection on the learning experience and teach back to the class.
  2. Team:
    - Stand-ups, weekly reviews, conferences, unconferences, retros, show and tell, pair and mob programming.
    - Toyota Improvement Kata.
    - Dojos.
  3. Organization:
    - Internal meet-ups, conferences, unconferences, webinars, brown-bag sessions, show-and-tell.
Pattern: Communicate, Communicate, Communicate:
- Internal BVSSH Awards:
  - Team has to proved factual data and quotes from customers and colleagues.
  - Teams will connect with winners for help and advise
- Communities of Practice:
  - Voluntary, open to all, Darwinian.
  - Meet once every two weeks.
- ASREDS loop: Align, sense, respond, experiment, distill, share.
Pattern: Be Comfortable with Uncertainty.
Pattern: Measure for Learning.

Chapter 9 - The Best Time to Plant a Tree is Twenty Years Ago; the Second Best Time is Now

How to continue from your current starting point and org memory to get better at fast learning in order to optimize for outcomes across the org?:
1. Start with Why (why of change, not why agile).
2. Focus on outcomes: BVSSH.
3. Leaders go first.
4. Create a Ways of Working Center of Enablement.
5. Start small and create an s-curve of change.
6. Invite over Inflict: Start with the natural champions.
7. Involve everyone from top to bottom.
8. Bias to action: communicate, communicate, communicate:
  - Change is a social activity.
9. Become a Relearning org:
  - You are never done.
  - It takes years.

Implementing DORA key software delivery metrics

Dan Lebrero — Thu, 17 Mar 2022 07:27:23 +0000

To ensure that your software process improvement process is having the desired results, you need to measure the before and after of any change.

Following the research in DORA, summarized in the Accelerate book, the key metrics to use are:

For performance: delivery lead time and deployment frequency.
For stability: time to restore service and change fail rate.

If you are not familiar with them, you can find their definition here.

This post is a deep dive on how we implemented them at Akvo both for regular applications and blue/green ones.

Regular applications

Let's start explaining what is our development process:

Any commit to trunk/main is deployed to the Test environment.
Sometimes, after some manual QA, the Test environment is promoted to Production.
- Promotion == take whatever is running right now in Test and deploy it to Production.
- The promotion script looks at the git SHA in Test and tags it with promote-$datetime which will trigger a CI build that will do the production deploy.
On promotion, no new binaries/containers are created, hence no build or test is required, only the deployment is needed.

1. Deployment frequency

For regular applications, calculating the deployment frequency is almost straightforward: look at the git tags with the pattern promote-$datetime, or look at the CI builds for such tags.

The caveat is that the CI deploy may fail, and if it fails, it may fail before or after the deployment step, so it starts to get tricky to decide if the deployment happened or not.

Ideally you want to look at the system itself to find out what has been deployed, something like Google App Engine's gcloud app versions list or Kubernetes' kubectl rollout history.

At Akvo, we chose the simple git tag count, as the most likely case for the CI build to fail was that the application failed to start, which we will still want to count as a deployment.

Here is the code to get all the tags using GitHub's GraphQL api, here for getting all the TravisCI builds and here all the Semaphore's ones.

2. Delivery lead time

We need to calculate the amount of time it takes a commit to get into production, so we just need to know on which deployment a commit was deployed in, when a commit was committed, and when a deployment was deployed.

Commit deployment

"Just" a matter at looking at the closest promote- tag that it is in the future:

The only thing to be careful with is that the commits must be sorted in topological order, not chronological order:

The Git command git log --graph --pretty=oneline --abbrev-commit draws the commits in topological order.

Commit timestamp

Easy peasy lemon squeezy: Git contains the commit's authored timestamp ... except:

When your team has the bad habit of squash merging their pull requests, then we just have the merge commit timestamp.
We should ignore the merge commits, as those commits have no code review time, so it is not "fair" to count them as they will skew the results towards a better (lower) lead time.

GitHub keeps a link between a commit and the PR in the Commit's associatedPullRequests field (for any kind of PR), and from the PR object we can find the original commits.

We could replace the merge commit with all the commits done in the PR, plus we should also ignore any merge commits from regular PR, but instead we decided that for all kinds of branches we will just take into account the authored date of the first commit and ignore all other commits.

This means that if you have been working for 10 days in a branch committing once a day, then your median delivery lead time will be 10 days, instead of 5 days if we were counting all the commits. This is more in accordance with the practices of small batches and trunk-based development encouraged in the DORA report.

Continuing the previous example:

So we keep the merge commit for the topological order, but we will use the first commit's authored date when calculating the delivery lead time.

Deployment timestamp

The actual easy peasy lemon squeezy: we have the timestamp in the tag (careful to not use the commit timestamp the tag points to), in the name of the tag itself and in the CI build.

We opted for the CI build finish time as one of our application's deployment pipeline took a significant amount of time to run, and the speed of the pipeline impacts the MTTR, so including the CI pipeline time is important.

3. Change fail rate

First instinct to implement the change fail rate is to have a way, maybe a UI, for developers to mark a deployment as failed.

But adding extra bureaucracy and another place to click around when trying to restore service, can only end up with people forgetting and managers complaining about developers not being "disciplined" enough to follow the process.

Instead, we assumed that a failed deployment will be followed by a deployment to fix whatever the issue was, so the deployment script will ask:

Does this deployment contain a hotfix for a previous deployment? [Y/n]

This reduces the friction to record a failure, as the developer knows why the new deployment is happening and is already running the deployment script, plus is just a Yes/No question.

The answer be stored as part of the promote- annotated tag, which we will be able to read from Git.

What is a failure?

A failure is any deployment that result in degraded service (for example, lead to service impairment or service outage) and subsequently require remediation (for example, require a hotfix, rollback, fix forward, patch).

As this definition didn't seem clear enough for the team, we defined a failure as an issue that both:

Some user has noticed or suffered it.
You think (or say) "Oh, shit" when you learn about it.

Much more clear :).

4. Mean time to restore (MTTR) service

In theory, we were recording any incident in Atlassian's StatusPage, from we can easily extract the timing information needed.

In practice, we opted for complaining about developers not being "disciplined" enough to follow the process of filling up StatusPage.

A possible automated way to calculate the MTTR is to look at the deployment history and use the time from a fix deployment to the previous failed deployment. This will still miss some outages, but it will require no additional bureaucracy.

Blue/Green applications

For applications with blue/green deployments, the release process requires one additional flip step:

Flips in Test happen automatically after a successful automated smoke test run.
Promotion happens from Live Test to Dark Production.
Flips in Production are triggered manually after some QA the dark production cluster.

Deployment frequency

For blue/green applications, you have to decide what a deployment is. From the Accelerate's site:

By "deployment" we mean a software deployment to production or to an app store. A release (the changes that get deployed) will typically consist of multiple version control commits.

Side note: I personally prefer this deployment and release definitions.

But should deploying to the production dark cluster count towards the deployment frequency?

We argued that no: if a change is not receiving production traffic, it cannot (or it is very unlikely to) cause a failure, hence it would not count towards the change fail rate, hence it should not count towards the deployment frequency.

So for blue/green applications, we want to look at the flips between dark/live.

Similar to the promotion script, the flip script just tags a commit with flip-$datetime which will trigger a CI build that will do the flip.

Delivery lead time

Same as regular applications, except that you want to compare commit timestamps against the flips.

Change fail rate

As the information will be more fresh in the developer's mind at the time of promotion, we decided to keep the failure question in the promote script.

Then, to know if a flip should be counted towards the failure rate, we will aggregate all the promotions for the flip, and mark the flip as failure if any of the promotions was marked as failure.

Mean time to restore (MTTR) service

Same as regular applications :(.

There are still some scenarios to think about: feature flags, dark launches and canary releases.

But for us, this was a good enough starting point.

You can find all the (Clojure) code for the service that collected these metrics at GitHub.

Book notes: Grokking Simplicity

Dan Lebrero — Wed, 09 Mar 2022 08:12:49 +0000

These are my notes on Grokking Simplicity by Eric Normand.

Excellent introduction to functional programming, or at least how I understand it.

Key Insights

World is divided in actions, calculations, and data.
Actions: anything that depend on when they are called or how many times they are called.
Data is useful mostly because of what it can't do: it cannot be run.
- Reads to mutable data are actions.
- Reads to immutable data are calculations.
Groups actions by layers of meaning.
Principle: design is about pulling things apart.
Code within a function should have the same level of detail.
- Being able to ignore the same details is one clue that the code is at similar level of abstraction.
- Don't move unclear code to another layer, make it straightforward.
Design discussions run the risk of getting too abstract.
We often write a lot of code just in case something might change in the future.
There is constant tension between design and the need for new features. Let comfort guide you when to stop.
A function cannot call other functions in its same layer.
Simplify collection processing:
- Transform code into data so that you can name and inspect.
- Prefer small simple step to big complex ones.
Use abstraction barrier on deeply nested data to reduce cognitive load.

Chapter 1: Welcome to Grokking Simplicity
Chapter 2: Functional Thinking in Action
Part 1: Actions, calculations, and data.
- Chapter 3: Distinguishing actions, calculations, and data
- Chapter 5: Improving the design of actions
- Chapter 6: Staying immutable in a mutable language
- Chapter 8: Stratified design: Part 1
- Chapter 9: Stratified design: Part 2
Part 2: First-class abstractions
- Chapter 10: First-class functions: Part 1
- Chapter 13: Chaining functional tools
- Functional tools for nested data
- Isolating timelines
- Sharing resources between parallel execution
- Reactive and onion architecture

Chapter 1: Welcome to Grokking Simplicity

Book explains the two main functional programming (FP) skills:
1. Distinguish actions, calculations, and data.
  - Actions: anything that depend on when they are called or how many times they are called.
2. Using first-class abstractions.

Chapter 2: Functional Thinking in Action

Stratified design:
1. Business layer: Least stable.
2. Domain layer.
3. Tech stack layer: Most stable.

Part 1: Actions, calculations, and data.

Chapter 3: Distinguishing actions, calculations, and data

Data:
- Facts about events.
- Is useful mostly because of what it can't do: it cannot be run.
- Must be interpreted to be useful.

Chapter 5: Improving the design of actions

Principle: minimize implicit inputs and outputs.
- Code is more reusable.
Choose better level of abstraction: closer function/parameters names to business language (ubiquitous language from DDD).
Groups actions by layers of meaning.
Principle: design is about pulling things apart.

Chapter 6: Staying immutable in a mutable language

Reads to mutable data are actions.
Reads to immutable data are calculations.

Chapter 8: Stratified design: Part 1

Software design: using one's aesthetic sense to guide programming choices to improve the ease of coding, testing, and maintaining software.
Stratified design:
- Pattern 1: straightforward implementation.
  - Code within a function should have the same level of detail. This is the Composed Method in Implementation Patterns.
  - Being able to ignore the same details is one clue that the code is at similar level of abstraction.
  - If a layer is using for its implementation several other layers, then it is not straightforward implementation.
  - Don't move unclear code to another layer, make it straightforward.
Design discussions run the risk of getting too abstract.

Chapter 9: Stratified design: Part 2

Stratified design:
- Pattern 2: Abstraction barrier.
  - A layer that let us ignore the same thing when working above that layer.
  - Most important benefit: they let you think more easily about the problem you are trying to solve.
Warn: we often write a lot of code just in case something might change in the future.
- Pattern 3: Minimal interface.
  - It ask us to consider where code for new features belong.
  - A function cannot call other functions in its same layer (really??).
- Pattern 4: Comfortable layers.
  - No codebase is ideal.
  - There is constant tension between design and the need for new features. Let comfort guide you when to stop.
"Ilities":
- Maintainability: code at the top layers is easier to test.
- Testability: code at the bottom layers is more important to test.
  - Because it changes less frequently and all other code depends on it.
    - Do I strongly disagree???
- Reusability: code at the bottom layers is more reusable.

Part 2: First-class abstractions

Chapter 10: First-class functions: Part 1

Code smell: implicit argument in function name:
- Similar function implementation.
- Name of function indicates the difference in implementation.
- Refactoring: express implicit argument:
  - Argument becomes first-class and part of the API.
- Treat data as data (data orientation):
  - Bespoke interfaces allow one interpretation of data while prohibiting others == less reusable.
  - Entities at the bottom layers should be reusable/generic.
- Refactoring: replace body with callback.

Chapter 13: Chaining functional tools

Simplify collection processing:
1. Make data:
  - Transform code into data so that you can name and inspect.
2. Operate on the whole array:
  - Try to process uniformly.
3. Many small steps:
  - Prefer small simple step to big complex ones.
4. Replace conditionals with filter().

Functional tools for nested data

Use abstraction barrier on deeply nested data to reduce cognitive load.

Isolating timelines

Timeline diagram:
- Sequence of actions.
- Show sequential or parallel execution.
- From Manning site:

Sharing resources between parallel execution

Queue to guarantee order.

Reactive and onion architecture

Reactive architecture:
- Instead of "Do X then do Y", it says "Do Y when X happens".
- Atom watchers: when atom value changes, do Y.
- ReactiveManifesto.org
Onion architecture:
- Layers can only call/know about inner layers.
Do domain rules need actions?
- No. You can always implement it as calculations.
- In truth, it depends on:
  1. The terms used to place the rule in a layer.
    - Domain language in DDD sense.
  2. Readability and awkwardness:
    - Some programming languages makes a non-functional implementation many times clearer.
    - Take technical debt.
    - System performance.

Forem: Dan Lebrero

Book notes: Design It!: From Programmer to Software Architect

Key Insights

All activities

Understand

Explore

Make

Evaluate

TOC

Part I - Introducing Software Architecture

Chapter 1 - Become a Software Architect

Chapter 2 - Design Thinking Fundamentals

Part II - Architecture Design Fundamentals

Chapter 3 - Devise a Design Strategy

Chapter 4 - Empathize with Stakeholders

Chapter 5 - Dig for Architecturally Significant Requirements (ASR)

Chapter 6 - Choose an Architecture (Before It Chooses You)

Chapter 7 - Create a Foundation with Patterns

Chapter 8 - Manage Complexity with Meaningful Models

Chapter 9 - Host an Architecture Design Studio

Chapter 10 - Visualize Design Decisions

Chapter 11 - Describe the Architecture

Chapter 12 - Give the Architecture a Report Card

Chapter 13 - Empower the Architects of Your Team

Part III - The Architect's Toolbox

Chapter 14 - Activities to Understand the Problem

Chapter 15 - Activities to Explore Potential Solutions

Chapter 16 - Activities to Make the Design Tangible

Chapter 17 - Activities to Evaluate Design Options

Are you asking too much from your team/tech lead?

Team vs Technical lead

Choices

But life had other plans ...

Prevention

Operation and recovery

Doctor's visit

Puzzles and odds

CTO last day: reflections, mistakes, and some learnings

Leaving

Root Cause Analysis

Mistakes and lessons learned

Bye-bye coding

Complaints are good

Clarity is hard work

Business first, tech second

Conflicts: avoid being a broken telephone

Difficult conversations (with data)

It is always a system problem

Self-blaming is pointless

It is all about teams

Don't make changes, run experiments

Feedback loops are loooong

Ring-fencing is a necessary evil

Lonely

I don’t dislike people management!

What is next?

Book notes: Staff Engineering: Leadership beyond the management track

Key Insights

TOC

Overview

Chapter 2 - Operating at Staff

Chapter 3 - Getting the title where you are

Chapter 4 - Deciding to Switch Companies

Book notes: Monolith to Microservices: Evolutionary Patterns to Transform Your Monolith

Key Insights

TOC

Chapter 1 - Just Enough Microservices

Chapter 2 - Planning a Migration

Chapter 3 - Splitting the Monolith

Chapter 4 - Decomposing the database

Chapter 5 - Growing Pains

The self-inflicted denial-of-service (DDoS) attack

Outage 1

Outage 2

The Browser Statistics Service (stats service)

A big pile of errors

DDoS attack

Steady State pattern

Book notes: Turn the Ship Around!

Key Insights