Introduction
We’ve working in many environments where the organisation has picked up a standard set of Agile metrics in a hope that this will inform them that they’re making the correct level of progress. But, what constitutes progress?
There are a huge number of metrics that can be measured, and a lot of industry literature associated with these metrics. In most cases this has happened in a functional vacuum. In other words, we were only measuring to the boundary of our control, and not the end-to-end business outcomes. This meant we were measuring what we thought represented success, but it was only part of the picture and didn’t tell us if we are working on the right things.
This article looks at the indicators that represent successful outcomes for the wider business, why they are important and how you can start measuring these indicators.
A lot of the information used in this post is covered in more detail in The Manual - Measurement.
The Problem
We measure and track so that we can make informed decisions within the delivery process. Having the right detail at our fingertips means we can make timely decisions to avoid unnecessary waste and effort. However, there are many dimensions of activity and behaviour that are important indicators to our success.
So the problem we’re really trying to solve is the ability to react early and quickly based on how we’re progressing against our objectives, but also any changes in the business predictions against our targets. Changing priority and pivoting are expensive and disruptive activities, thus having reliable data to justify these decisions is crucial.
We also want to drive the continuous improvement feedback loops at all levels. This helps the delivery organisation focus on waste reduction, quality, re-use, etc. Anything that supports the principles and continuous improvement culture.
Finally, we are often blocked by our constraints and dependencies. These constraints and dependencies can have a dramatic impact on our delivery predictability. So, how do we control these and drive positive decisions to improve our overall predictability.
Which Indicators?
Which indicators should you chose to measure?
Let’s enumerate the dimensions of progress and success.
- Business - Are we meeting our business objectives? Depending on the type of product and sector, this can mean different things, but in reality we’re supporting the business and its goals. Without high performance in this dimension, the other dimensions are quite meaningless.
- Delivery - The efficiency and predictability of taking ideas/requirements into the beginning of the delivery pipeline to getting them into the hands of the customer.
- Quality - There are many levels of quality. Scrutinizing our quality at all stages of the delivery pipeline and operational drives continuous improvement and insights to our ways of working
- Organisational - A delivery organisation has many roles and responsibilities. Co-ordination and efficiency of the teams contributing to the overall delivery has a significant impact on the overall result
- Operational - How the product performs both commercially and technically in the hands of our customers
The key point is that no single indicator can give you the complete picture. Each indicator will drive insight and will allow you to make adjustments, but they cannot (and should not) be used in isolation.
What we have discovered is that you need to pick a set of indicators that fit your intent and culture. When we first started, we tried to measure too much and then got swamped in information overload with very little time to analyse the data. This devalued the information in the eyes of the teams, thus the whole exercised backfired.
This leads to the other finding in this area. Measurement is an iterative process. You try it, testing and then adapt, just like any other agile process.
For is post we’ll describe the indicators we landed on and why.
Business Value
Lead Time
Deployment Frequency
Change-Fail Percentage
Meant Time to Restore
Availability
Measurement
Capturing data manually dramatically decreases the value of the insights. Automate everything!
Summary
References
- Demystifying Leading Indicators in Product Development and Innovation
- Measuring Business Value in DevOps and Agile Projects
- Learning from the Accelerate “Four Key Metrics”
