Forem: Plandek

Ship features faster - lessons from elite sports for engineering teams

Plandek — Fri, 17 Nov 2023 10:42:38 +0000

A lot is written in business about lessons to be learned from great sporting achievements. Elite sporting stars, coaches and sports scientists are regular visitors to business strategy days and Board rooms.

Less is written about the lessons that might be learned for Agile delivery teams faced with the relentless challenge of delivering endless roadmap features for often thankless stakeholders.

So here are seven lessons borrowed from elite sports that every hard-working product delivery team can benefit from to help them ship the right features faster.

The core theme of each lesson comes back to the effective use of the intelligent insight available from the new generation of engineering intelligence tools.

Roadmap delivery and the need to ship features faster – common problems

The common problems that these sporting lessons are designed to solve in the context of Agile software delivery are:

Inability to deliver features predictably (on time)
Difficulty in balancing speed and quality
Done doesn’t actually mean ‘done’
Team fatigue over time.

So, if your teams have suffered from any of these common challenges, read on…

The seven lessons from elite sport to ship software features faster and accelerate your roadmap delivery

Here, we discuss the seven key lessons from elite sports that we see our best-performing and most innovative customers apply day in and day out. These are:

Plan to succeed
Stay focused
Speed with precision
Conserve energy – small steps perfectly
Look ahead – find and mitigate hidden risk
Keep the fans cheering
Champion teams!

1. Plan to succeed

“It’s not the will to win that matters. It’s the will to prepare to win that matters”. Paul ‘Bear’ Bryant

This is a universal tenet of elite sport and is clearly directly applicable to Agile software delivery.

The nature of Agile makes planning tricky to get right. Forecasting when a feature will be delivered is often compromised by the fact that much of the feature may not be scoped/estimated upfront.

This ambiguity of Agile means that the planning that is done ahead of delivering a product feature must, at the very least, do two things well:

Estimate accurately – poor estimation is the root of many delayed feature releases.
Analyse and manage backlog effectively – poorly scoped and managed backlog is often a source of inefficiency and poor forecasting.

The new generation of engineering intelligence tools like Plandek enables product managers to do both of these things better.

For example, Sprint Velocity and Sprint Volatility analytics (see below) enable analysis of teams’ estimation skills when planning the delivery of a new epic or feature.

Plandek example graphics – Sprint Velocity and Volatility analytics

It provides a clear understanding of teams’ past Sprint Completion and Carried Over Work, which is the best quantitative measure of teams’ ability to plan and execute sprints accurately (and therefore a proxy estimate of the probability of them delivering ‘to plan’ in the future).

As per (2), Delivery Managers can analyse the backlog at the outset in order to understand the proportion of new functionality that has been scoped and the proportion that remains un-scoped, which will have a huge bearing on delivering timing estimation.

Then, as teams get underway, they can keep a continuous eye on the key Backlog Health metrics, which are critical to increasing velocity. These include:

Story Points Ready for Dev: (the number one Backlog Health metric which should be considered relative to the velocity of teams to ensure that there are always 1-2 sprints of backlog ‘cushion’).
Orphaned Stories (ensuring that all stories are linked to a ‘parent’ (e.g. an epic), rather than unassigned or ‘orphaned’, thereby adding to potential planning inaccuracy).

As the old Chinese proverb says…

“The best time to plant a tree is 20 years ago. The second-best time is now.”

2. Stay focused

Great athletes have a unique ability to avoid distraction and remain focused on a single objective for extended periods of time.

Our research shows that one of the key reasons for problems in delivering new roadmap features is a lack of focus at two levels:

At the macro level, there is a lack of team resources available to actually build new features – versus resources diverted away to either planned technical debt or unplanned work such as bug fixing.
At the micro-level, there is a tendency for teams to get diverted into working on tickets outside planned sprints and epics.

The example graphic below shows how the new generation of engineering intelligence tools like Plandek enables product managers to improve at both these levels.

At the macro level, it shows very clearly how much resource is actually available to deliver new features and how this varies over the duration of the delivery period. It is very common for less than 50% of total resources to be available to build new features, taking into account time lost to addressing technical debt of unplanned work (e.g. bug fixing).

Plandek example graphics – showing the proportion of work done by new features/priorities, tech debt and unplanned work

Similarly, at the micro-level, elite engineering teams will use engineering intelligence platforms to analyse the make-up of the tickets constantly worked on within sprints to reduce work outside the sprint, carried-over work and unplanned tickets added after the sprint starts.

This ability of high-performing engineering teams to remain focused at both macro and micro levels is shared with great athletes.

Carl Lewis, one of the greatest US track athletes of all time, sums this up…

"My thoughts before a big race are usually pretty simple. I tell myself: Get out of the blocks, run your race, stay relaxed. If you run your race, you'll win... channel your energy. Focus." Carl Lewis (Track and Field multiple Olympic Champion)

3. Speed with precision

A key challenge for delivery teams is how to increase velocity without negatively impacting quality.

This is clearly something that elite sportspeople in all disciplines become very good at doing.

"It's a round ball and a round bat, and you got to hit it square." - Pete Rose (Baseball Great)

Elite athletes constantly track their progress in the ability to manage the speed of execution and flawless execution. So, too, do high-performing delivery teams.

Again, an engineering intelligence platform can really help in this regard, enabling Delivery Managers to keep a very close eye on the relationship between team output and quality.

At the aggregate product level, it’s important to track the overall relationship between throughput (expressed in tickets or story points completed) and quality (expressed, for example, in escaped defects).

Plandek example graphic – showing the relationship between throughput and quality at the product level

This balance of speed and precision develops rapidly in engineering teams that relentlessly review the relationship between velocity and quality.

And as the golf great Gary Player famously said…

“The harder I practice, the luckier I get.” Gary Player

As this virtuous circle of speed and quality develops, teams start to see the time it takes to deliver an increment of software reduce. Again, elite engineering teams will closely track this over time through metrics such as Lead Time to Value and Cycle Time.

The teams in the example below are relatively high-performing, having reduced their Lead Time to under 20 days (the time taken to ship an increment of software - from entering backlog until delivered to the live environment). Similarly, the Cycle Time is less than five days (one working week). This represents the time it takes from dev-start until deployed to live.

Plandek example graphic – Lead Time and Cycle Time

Conserve energy – small steps perfectly

“Excellence is not a singular act but a habit. You are what you do repeatedly.” – Shaquille O’Neal

Our research shows that high-performing engineering teams that ship features faster and consistently deliver roadmap goals have a great ability to conserve energy over long periods because they execute the small things very well.

This is most notable in two key areas:

When running scrum Agile, they execute sprints extremely accurately as this is the basic building block of predictable and fast-paced delivery.
They constantly look to remove ‘dead time’ from the delivery process – so that they limit the amount of time tickets remain in an ‘inactive’ status (e.g. ‘awaiting QA’)

Engineering intelligence tools such as Plandek enable elite engineering teams to address (1) by developing a far better understanding of their sprint execution than is possible with tools such as Jira and ADO.

The example Sprint Velocity graphic provided by Plandek shows how high-performing teams keep a very close eye on sprint accuracy as a key building block to longer-term roadmap delivery.

The analytics show Sprint Volatility, Sprint Completion, Carried over Work (often a big issue) and Story Point Change (also a common problem.).

Elite teams look for Sprint Completion of 80-85% as the sweet spot of pace and precision.

Plandek example graphic – Sprint Velocity analytics to track and drive sprint accuracy

Similarly, as per (2), the constant review of metrics like Flow Efficiency enables teams to take a more holistic view of the efficiency of the SDLC over time.

It enables them to understand how long tickets remain in ‘inactive’ statuses (such as ‘Waiting for QA’) in order to remove ‘dead time’ from the delivery process and hence reduce friction and delay.

Plandek example graphic – Flow Efficiency analytics to identify and remove friction from the delivery process

Unsurprisingly, great artists and elite sportspeople understand the importance of doing the small things very well.

“Great things are done by a series of small things brought together. If you cannot do great things, do small things in a great way” Vincent Van Gogh.

Much like elite athletes, the best agile teams always review, learn from what went wrong, and adapt; metrics also play a key role in this process.

As Michael Jordan famously said...

“I've missed more than 9000 shots in my career. I've lost almost 300 games. 26 times, I've been trusted to take the game-winning shot and missed. I've failed over and over and over again in my life. And that is why I succeed”. Michael Jordon

And Lolo Jones, the dual world champion hurdler and bobsledder, noted…

“A failure isn't a failure if it prepares you for success tomorrow” - Lolo Jones.

5. Look ahead - find & mitigate hidden risk

“The will to win is important, but the will to prepare is vital”, Joe Paterno, the most victorious coach in NCAA FBS history.

A universal challenge for delivery teams is managing risk during the delivery journey, particularly as a go-live milestone approaches.

Understanding and mitigating risk requires a holistic view of the end-to-end delivery process – from design all the way through to deployment to live software. This is where the next generation of engineering intelligence tools are invaluable.

Plandek’s intelligent algorithms work at sprint and epic levels. They examine all tickets sitting at every stage in the SDLC to identify hidden risks. As such, they work as a virtual Scrum Master with an ability to ‘see’ risk that their real-life counterparts are very likely to miss.

The graphic below shows how elite engineering teams use the Plandek Virtual Scrum Master (known as LiveView) to quantify risk and identify ‘action items’ that need addressing in order to reduce risk and improve timing accuracy.

Plandek example graphic – LiveView for epics Virtual Scrum Master to identify and mitigate hidden epic delivery risk

6. Keep the fans cheering

“The fans are the ones who make the game. Without them, there would be no game. They’re the ones we do it for.” Michael Jordan

Like sports stars, high-performing engineering teams excel at generating box-office (i.e., keeping their customers happy and asking for more!)

In Agile software delivery, this means better understanding the relationship between shipping features faster and value delivered.

It is well known that Deployment Frequency correlates positively with product NPS scores – i.e. the faster you deploy new features to live, the happier your customers (and the more value you generate).

Plandek allows teams to analyse the relationship between DevOps metrics and multiple other data sources (e.g. financial, HR and customer data).

Hence, elite engineering teams will tend to use Plandek to track the relationship between Deployment Frequency and product NPS score, as shown in the graphic below.

Plandek example graphic – Deployment Frequency versus product NPS score

7. Champion teams!

"The secret of winning football games is working more as a team, less as individuals. I play not my 11 best, but my best 11." Knute Rockne, College Football Hall of Fame Coach

But perhaps the most important sporting lesson of all is the importance of teamwork. The key challenge in software delivery will always be maintaining a motivated and relaxed team, as Scrum Agile is relentless in requiring teams to run sprint after sprint in the pursuit of roadmap glory!

Our research shows that delivery leaders who keep a close eye on individual team members’ well-being and satisfaction achieve the highest levels of performance. This can be done formally through ‘pulse’ surveys and informally.

In addition, the balance of expertise within the team is crucial. For example, the Plandek Pull Request Collaboration analytic shown below is helpful in understanding the balance of capabilities within teams and the reliance within teams on ‘star’ engineers.

Plandek example graphic – Code Knowledge (Pull Request Collaboration) analytic to understand and manage the balance of capabilities within teams

So there we have it!

There are big commonalities between high performance in software engineering and elite sports.

And the key to unlocking greatness in both disciplines is intelligent insight applied effectively over time. In software engineering, this is provided by the next generation of intelligent analytics tools that give engineering teams a level of intelligent insight that was previously unattainable from underlying DevOps toolsets.

As Steve Nash concludes…

“Data analytics has become the lifeblood of professional sports, and it’s changing the way we view and compete in the games we love.” Steve Nash, Two-time NBA most valuable player

About the author – Charlie Ponsonby
Charlie started his career at Andersen Consulting (Accenture). He was a senior executive at Sky until 2007 before leaving to found and run Simplifydigital 2007. Simplifydigital, a tech-enabled e-commerce business, was three times in the UK Sunday Times Tech Track 100. It was acquired by Dixons Carphone plc in 2016.

Charlie is CEO of Plandek, which he co-founded with Dan Lee in 2018. Charlie is responsible for Plandek’s overall strategy and growth and works with clients and domain experts globally to help engineering teams apply intelligent analytics to deliver software better.

About Plandek
Plandek is an intelligent analytics and performance improvement platform to help software delivery teams deliver valuable software faster and more predictably.

Plandek enables technology teams to track and drive their improvement and share understandable KPIs with stakeholders interested in accelerating value creation/ improving delivery efficiency.

Plandek works by mining data from delivery teams’ toolsets (such as issue tracking, code repos and CI/CD tools) to provide actionable and intelligent insight across the end-to-end software delivery process.

Plandek is recognised as a top global vendor in the DevOps Value Stream Management space by Gartner and Forrester and is used by private and public organisations globally to optimise their technology delivery and accelerate R&D ROI.

For more information, please visit www.plandek.com

Lost in translation: communicating delivery metrics to the wider business

Plandek — Fri, 10 Nov 2023 10:24:00 +0000

Recent debates, spurred by the McKinsey report “Yes, you can measure software developer productivity”, have centred on the effective use of metrics within engineering to report to the C-Suite level. A broader question surfaces of how to ensure what is measured is valid and useful in achieving its intended purpose.

As GWI have become more diligent about metrics, their approach to measurement and reporting has evolved to help tackle this question. Join George Nikolaropoulos, GWI’s SVP of Engineering, to discover how they adapted their metrics to suit their engineering teams, VPs, and C-Suite to stimulate the conversations driving continuous improvement at multiple levels of their organisation.

Additional supplementary reading:

Speaker: George Nikolaropoulos, SVP of Engineering, GWI
Currently the Senior Vice President of Engineering at GWI, George is an accomplished architect and team leader who inspires innovation and drives success and growth. His experience spans banking, insurance, maritime, and marketing research industries, but his current focus is growing and leading engineering teams, product development, tech strategy, and artificial intelligence – with a passion for creating products that make customers’ jobs easier and more enjoyable.

Calculate R&D Tax Credit with Plandek

Plandek — Tue, 19 Sep 2023 10:45:22 +0000

Increase the accuracy and speed of your R&D Tax Credit calculation with Plandek dev team analytics

What are R&D Tax Credits?
Research & Development (R&D) tax credits are a (UK) government incentive designed to reward your company for investing in innovation. They are a valuable source of cash for businesses that invest in accelerating their R&D, hiring new staff and ultimately growing their business.

Whatever size or sector your company operates in, if you attempt to “resolve scientific or technological uncertainties” through “creating new products, processes or services” or “changing or modifying an existing product, process or service”, then it is likely you will be eligible for R&D tax credits.

What costs qualify for R&D Tax Credits?
The following costs would qualify for R&D tax credits in the UK:

Staff, including salaries, employer’s NIC, pension contributions and reimbursed expenses.
Subcontractors and freelancers. Materials and consumables, including heat, light and power, are used up or transformed by the R&D process.
Some types of software.
Payments to the subjects of clinical trials.

How to calculate R&D Tax Credits using Plandek?
Plandek is an analytics and performance improvement tool for software delivery technology R&D) teams.

Its principal use is to help technology teams understand better and analyse the software delivery process (from design through to the delivery of live software) to improve efficiency and accelerate the delivery of value.

As such, Plandek can also be used to help calculate your R&D Tax credit by providing detailed data on the first two cost elements in the section above:

related salary costs for staff;
and cost of subcontractors/freelancers.

Software development teams spend their time working on ‘tickets’ (increments of work tracked in Workflow Management tools like Jira or the Azure DevOps equivalent.)

Plandek offers instant visibility of all tickets and the proportion of completed tickets related to R&D.

This means a user can quickly and easily see the amount of time and, therefore, cost spent on qualifying R&D spend for any reclaim.

Indeed, here at Plandek, we use our own software to easily calculate the proportion of work done on R&D by our teams and then apply that to our salary costs.

Identifying qualifying teams
In the illustrative example below, we show how Plandek is used to identify the teams and individuals that have undertaken work that qualifies for an R&D Tax Credit.

Plandek shows the relevant data split out by Teams.

As you can see from the example graph above, the “Enablement” team (which consists of known individuals with known fully loaded salary costs) has completed 314 tickets that relate to R&D out of a total of 468 tickets during the tax year (the period for which the R&D Tax Credit is being applied for).

Other teams’ data can be easily split out and investigated further in the same way using Plandek.

The teams selected for the claim should be any team undertaking any R&D activity.

It is essential to include any R&D activity, no matter what size. Every little helps!

The number of tickets in each section is determined by the nature of work completed using data from Jira that is seamlessly imported into Plandek.

Calculation of the R&D Tax Credit Benefit
To fully calculate the benefit, salary information is needed. Salary information for each team member for the teams you are analysing should be included to maximise your claim.

Salary information should be sourced from your Finance Department.

Once salary data has been collected, the simple calculations below are done:

The number of tickets for R&D forms the basis of the calculation. Of the total tickets completed by “Enablement” (468), 314 relate to R&D, equating to 67% of the completed tickets.

This percentage is then applied to the total staff costs for the fiscal year for that team, and the process is repeated as necessary for your teams to give the total qualifying expenditure, which forms the basis for the claim.

As per UK HMRC, the qualifying expenditure should be uplifted to 230%, with the tax credit of 14.5% being payable on that amount.

These figures are correct for an SME as of December 2022 and are subject to change.

Conclusion
Using Plandek takes out the guesswork when calculating qualifying expenditures for R&D tax credits. There is no need to estimate percentages of time taken on specific projects or implement expensive and troublesome time-tracking processes.

Plandek is easy and quick to implement, offering end-to-end delivery and engineering metrics, analytics, and intelligent alerts to align and accelerate value delivery for your software teams.

R&D tax credit calculation is just another added benefit.

If you’re interested in learning more about how Plandek can help you with your R&D tax credit calculations, feel free to get in touch with us today.

Skyscanner's journey to a "continuous improvement" team culture

Plandek — Wed, 06 Sep 2023 10:41:48 +0000

How Skyscanner embedded a team metrics culture for continuous improvement

Team metrics are in vogue, and there is no shortage of literature espousing the value of various combinations of metrics that will lead your team(s) to delivery stardom.

In this webinar, Skyscanner shared their journey to build a data-driven Agile organisation with Plandek, from acknowledging that data was critical to realising their competitive advantage, to changing their engineering management culture to one that valued and utilised these metrics to drive greater improvement at scale. They shared what originally took them down this path including the signs that change was needed and the impact it has had.

In this webinar, we covered:

The values Skyscanner created to assess the various tool providers to find one that they felt would enable their team and more importantly, reinforce the cultural change that was the fundamental goal of this exercise.
How their understanding of what they needed evolved throughout the exploration process and how they ran trials with teams to understand this better.
Their interpretation of the most important metrics, the sources they were inspired by and how they expected to need to augment them.
The challenges they discovered with adoption, those they predicted and how they altered their plans to adapt to them.
Some of the improvements that teams are starting to see.
Some of the greater company initiatives that this change has empowered.

Skyscanner is still very much on its journey and at the moment they are guiding 80 squads (~600 engineers) through this change. They are seeing good levels of adoption and insights being acted on, having created cross-organisation communities of practice to support this change and bed it in for the long term. They will talk about the principles we have used to drive this adoption.

Visit our website: https://plandek.com/resource/client-webinar-series-skyscanner/

Speaker
Ramsay Ashby is a Senior Software Delivery Operations manager, with over a decade of experience working both in and with agile teams. He’s also worked in environments with more traditional methodologies, as well as being part of two agile transformations. He began his career in engineering, initially as a test engineer where he focussed heavily on developing automated tests and automated test frameworks, before moving into development engineering where he was a team lead for a globally distributed team for several years. Today, Ramsay is responsible for presenting and maintaining the suite of internal agile training courses that Skyscanner offers to its employees. Ramsay works closely with a broad spectrum of employees across the business, ranging from recent university graduates up to director level, helping them and their teams to perform at their best.

Plandek - Introducing the enhanced LiveView for Sprints and Epics

Plandek — Tue, 05 Sep 2023 12:56:40 +0000

Since the launch of LiveView in April 2022, the Plandek team has been extremely busy collecting feedback and designing the next generation of AI-driven software delivery tools. We are thrilled to unveil the enhanced LiveView for Sprints and Epics, an innovative tool for teams that leverages machine learning to spot risks that traditionally disrupt delivery flow and allow them to react in real time.

Effective product delivery is paramount in today's rapidly evolving software landscape. Our data scientists, engineers, and Agile delivery experts have intelligently designed LiveView to revolutionise how teams keep track of and manage their Sprints and Epics. It allows product delivery teams to track the delivery of their work across the entire toolchain.

How LiveView can change the way teams deliver

1. Understand the health of your Sprint/Epic through our new Risk Score
The new Risk Score reflects the current health of a Sprint/Epic based on real-time insights throughout the delivery toolchain. LiveView evaluates various behaviours that have a material impact on a team’s ability to meet their delivery goals successfully and provides early indicators when they get off track so they can quickly react.

The risk score is calculated by analysing your data to identify insights throughout the toolchain that indicate potential risks. The impact of each insight is given an impact weighting, and the combined overall weighting is used to determine the risk score (0-100). Critically, this algorithm is constantly updating based on how a team works to keep the insights fresh and meaningful for teams.

2. Get a better understanding of where risks/issues might exist through Plandek Insights
Sprints and Epics tend to go wrong for many of the same reasons, and it’s often because teams distance themselves from executing on the basics. LiveView helps you identify potential problems as they arise by analysing your data to surface actionable insights, giving you the opportunity to respond and keep your progress on track.

The insights you can monitor in LiveView include identifying where teams are working outside of their Sprint/Epic (called Distractions), where tickets and pull requests are stuck, when work goes into development, but no commits are being made, and where engineering complexity indicators suggest that a team or individual may be struggling or the work is more complex than originally expected.

3. Understand who's overloaded so you can redistribute work if needed
Teams can often find themselves overly dependent on certain individuals, which can quickly disrupt their overall flow. With LiveView, you can easily spot where there are potential “work in progress” (WIP) issues or too much work piling up for certain team members so it can be redistributed quickly before bottlenecks form.

4. Track the progress of your tickets through the tools you've connected
LiveView not only enables you to track a ticket’s progress, but you will also be able to see all the critical events related to the delivery of the tickets, including pull request events, builds failing, deployment events, and so on. It’s a unique view on the delivery of value that consolidates all the relevant information teams need throughout the toolchain to work effectively.

5. Ultimately, increase the chances of achieving your delivery goals more successfully
High-performing teams have two things in common: they perform the fundamentals well and demonstrate high levels of awareness to quickly identify and remove blockers (achieving a high “flow state”). LiveView provides teams with a consolidated view throughout the toolchain to help increase their awareness of the fundamentals of good delivery, where issues may exist, so they can quickly react and improve their delivery effectiveness.

“We are very excited about the launch of the new LiveView. While historical trends are central to any good continuous improvement culture, it is critical to have the ability to spot issues bubbling up in real time with enough time to resolve them before they impact milestones. LiveView not only helps our clients get better at managing the basics with Sprints and Epics, but it shows them where the fires are that they need to put out quickly.” - Will Lytle, COO, Plandek

For more information about the enhanced LiveView, please visit https://plandek.com/liveview/ to learn more. You can also sign up for a free account to try out the new feature or book a demo with our team.

How to measure the impact of GitHub Copilot on engineering productivity

Plandek — Fri, 11 Aug 2023 12:58:38 +0000

Most software delivery teams are considering adopting AI in some form to help engineers accelerate their value delivery and increase delivery effectiveness.

GitHub Copilot is one of the first examples of AI-powered engineering assistance. It is self-styled as “your AI pair programmer’ and can autocomplete lines of code.

Early adopters report ‘productivity improvements’ of up to 20% using GitHub Copilot. Still, it is not cheap (at c$10 per user per month), so how can you build a business case for GitHub Copilot?

As such, what methodology (and metrics) should you use to accurately assess the impact on productivity and value delivery of a tool like GitHub Copilot?

GitHub Copilot: first-generation AI assistance for software engineers

Copilot is clearly the start of a long and accelerating journey as AI is applied to many areas of the SDLC. It uses the OpenAI Codex to suggest code and entire functions in real time from your editor.

Other AI tools arrive almost daily and can help software engineers in a myriad of time-saving and efficiency-enhancing ways. Here are just a few examples:

Grit.io will help manage technical debt
Mintlify provides automated documentation for developers
Code AI helps translate (some) languages, debug, navigate code and act as a pair programmer
Tools like AdrenalineAI use AI to improve understanding of your code base

So, at a time when cost control is the order of the day, you may be considering how you can accurately quantify the impact of tools like these and justify the added expense.

*A methodology for measuring the impact of a tool like GitHub Copilot *

To robustly measure the impact of GitHub Copilot (and similar AI engineering-enhancement tools), the methodology must be:

Quantitative - based on hard, measurable data
Holistic – considering all benefits and potential impacts across the end-to-end SDLC (software delivery lifecycle)
Balanced – inclusive of subjective survey data alongside software delivery data

This requires a metrics scorecard that fully captures the benefits and potential costs of GitHub Copilot.

The metrics reflect the SPACE framework for measuring developer productivity, emphasising the key areas GitHub Copilot will likely impact.

These metrics can be tracked over time for a representative group of GitHub users to see the ‘before and after’ effect. We suggest that a representative sample would include engineers of different seniority and activity – and the time period for analysis would be at least three sprint cycles (e.g. 6 weeks+).

*Key metrics to quantify the impact of GitHub Copilot *

An end-to-end software delivery analytics platform like Plandek provides a single pane of glass to measure the real impact of a tool like GitHub Copilot.

It surfaces a range of engineering and software delivery metrics to capture the impact of GitHub Copilot on five key variables that determine ‘productivity’:

Velocity and throughput – measures of team ‘output’
Time to value – time taken to deliver an increment of software
Quality
Dependability – a key benefit if teams more reliably deliver against their plans.
Developer satisfaction - impact on speeding up repetitive/less interesting tasks.

These metrics can be tracked over time for a GitHub Copilot control group versus non-users.

Velocity and throughput metrics

Throughput is a core measure of ‘output’ over time for Scrum and Kanban teams - and can be calculated in tickets, story points, pull requests, builds or value points. A tool like Plandek will easily calculate Throughput per engineer for users and non-users of GitHub CoPilot. This can be expressed as a percentage increase.

Sprint Velocity considers the rate of work achieved within a sprint and how it varies over time. It can be calculated in tickets or story points. Advanced analytics tools like Plandek will also show you the amount of work carried over by Sprint to see an even better underlying measure of delivery.

This would be a key metric when considering the impact of GitHub Copilot.

Time to Value

Cycle Time is a core agile software delivery metric which tracks an organisation’s ability to deliver software early and often. It calculates the time taken to deliver an increment of software from dev start to deployment. The shorter the Cycle Time, the shorter the feedback loops, hence the quicker the organisation is going to receive new features and respond to customer needs. This is a vital KPI when assessing technology delivery efficiency.

Code Cycle Time typically accounts for 20-30% of overall Cycle Time. It calculates the average time taken from a pull request (PRs) opening until it is merged/closed. The bulk of this time is usually spent during the approval process.

In theory, GitHub CoPilot enables quicker, easier development. Therefore, developers should have greater availability to review each other’s PRs. If code quality is improved, then the outcome of the reviews should result in fewer changes requested and an approval time.

Quality

Escaped Defects is a simple but effective measure of overall software delivery quality. It can be tracked in numerous ways, but most involve tracking defects by criticality/priority.

Any analysis of delivery efficiency pre/post the implementation of GitHub Copilot should include consideration of Escaped Defect rates as it would be a poor trade-off to increase velocity and ‘productivity’ at the expense of quality.

Build Failure Rate identifies the percentage of builds which fail and the overall risk this poses to a team working productively. Notable changes to the failure rate after implementing GitHub Copilot indicate that code quality may be impacted.

Dependability

Sprint Target Completion tracks the percentage of the sprint goals achieved each cycle. ‘Scrum Teams’ and ‘Sprints’ are the basic building blocks of Scrum Agile software delivery. If Scrum Teams consistently deliver their Sprint goals, Agile software delivery becomes relatively dependable, enabling the prediction of delivery outcomes across multiple teams and longer time periods.

Scrum team predictability is, therefore, a critical success criterion in Agile software delivery. If GitHub Copilot can improve the likelihood of a team delivering their tickets faster and with fewer bugs, then this is a major contributor to the overall improvement in effectiveness.

Developer Satisfaction

eNPS tracks employee satisfaction and loyalty within teams and organisations. Anecdotal reports suggest that developers find that GitHub Copilot makes the more tedious aspects of coding less taxing and positively impacts wellbeing. An employee NPS makes this straightforward to validate and quantify.

Although an important factor of productivity measurement, it shouldn’t be viewed in isolation from the other metrics when quantifying overall developer productivity.

The above are some examples of relevant metrics to consider when analysing the impact of GitHub Copilot on delivery productivity. The key is to take a balanced set of metrics that holistically considers software delivery a complex process.

Combining the balanced scorecard of metrics to create a business case for GitHub Copilot

Typically, we would combine data from the ‘balanced scorecard’ of metrics discussed above using simple weightings to create an overall Productivity Impact Assessment (PIA) of GitHub Copilot. See the below table:

GitHub CoPilot Productivity Impact Assessment - example template

The weighted average productivity improvement calculated in the PIA can then be applied to the estimated cost of the delivery capability (headcount x fully loaded staff costs). This provides a productivity improvement monetary calculation based on resource costs. It excludes the potentially (larger) benefits of delivering more value to customers earlier, which is not a benefit that is easily or necessarily calculated.

Productivity improvements from using GitHub Copilot – the empirical data

There is a distinct lack of independent data in this regard.

GitHub’s own survey of 2,000 developers showed that 88% of developers claimed ‘to be more productive’ when using the tool, while a task test undertaken by 95 developers saw the group that used GitHub Copilot was 55% faster and had a 7% higher rate of completing the task (see below).

GitHub’s own Survey data – the impact of GitHub Copilot on users (2022)

Our own analyses show improvements using a PIA (as shown above) of circa 5%. However, this is bound to improve further as AI technology improves so rapidly.

About Plandek

Plandek is an intelligent analytics platform to help software delivery teams deliver valuable software faster and more predictably. 

Plandek enables technology teams to track and drive their improvement and share understandable KPIs with stakeholders interested in accelerating value creation/improving delivery efficiency. As such, Plandek is a key global vendor in the fastest-growing area of DevOps, known as Value Stream Management.

Plandek is recognised as a top global vendor by Gartner and Forrester and is used by private and public organisations globally to optimise their technology delivery.

For more information, please visit www.plandek.com