Agile KPIs: How to measure agile success

The promise of agile development maps well to today’s disruptive, fast-moving business environments. But measuring the success of your agile efforts can be tricky, especially when it comes to ascertaining alignment with business objectives. How do you gauge success or failure? Are you even measuring the correct things?

Any usable metric needs to answer three simple questions, says Dave West, product owner and CEO at Scrum.org: Can I learn something about how I work or the value I deliver from this metric? Can I measure this frequently enough that I can use it to monitor the cause and effect of the changes I make? Does it connect to my purpose or mission?

But before organizations can attempt to answer these questions, they must first assess whether they’re functioning in an agile way, says Jeff Dalton, CEO of Broadsword Solutions. Here, going back to the basics of agile is essential. And for Dalton, every agile metric that matters can be traced back to trust. Most organizations that claim agility are not high-trust, which leads to a breakdown in being able to align metrics with business priorities, he says.

“The entire goal of agile is to gain trust and be trusted,” Dalton says. Before drilling down to more granular metrics, he says, you first have to address five questions: “‘Do we have an agile culture? Are we credible in the eyes of our own teams, managers and customers? Are we doing the best we can to meet our commitments? And are we producing quality products and services? And are we showing the outside world our best?’” Dalton says. The answers to those questions, he says, tell a manager everything they need to know.

“If there’s trust, then those five things are happening — there’s cultural alignment; there’s high-quality production; there’s speed; there’s collaboration,” he says. “Then, on that foundation, you can start to drill down into answering and measuring the results of those questions.”

Here’s how to assess how your organization measures up to those five fundamental agile questions — and how to measure success once it does.

Fundamental agile questions

Assessing your organization’s agile practices begins by answering the following five questions.

1. Do we have an agile culture?

There are three ways to gauge the agility of your culture — or lack thereof, Dalton says. The first is to use an industry standard performance model to measure your organization’s agile maturity. Dalton advocates using Agile Performance Holarchy (APH), which organizes agile best practices into five interconnected “performance circles”: Leading, Crafting, Teaming, Affirming, and Envisioning. Each of these areas is given a maturity rating — adopting, transforming, or mastering — depending on which level the organization has achieved.

The second way to gauge your agile culture is through values traceability, a process for determining how many of your processes can be traced directly to agile values, Dalton says.

“For example, do you use sprint demonstrations to share with customers? Then, yes, that ties directly to agile values. So, you’re measuring how many of the ‘ceremonies’ and processes tie to those agile values; the ultimate goal, of course, is to have all of them in place,” he says.

Third is your team score, which measures how engaged the various stakeholders are in those agile processes and ceremonies. Do you have full engagement in sprint demos? Does everyone attend daily standups?  This metric is fuzzier than most, as it mainly measures subjective feelings of team members. But it can tell you whether team members are engaged, burning out, or overloaded, enabling you to  reassess how tasks are allocated, how estimates are done and how the workload is shared.

2. Are we credible?

There are a few metrics to measure credibility, Dalton says. Estimated versus actual throughput is the first; at the release level, are teams producing what was promised? The second is revised versus actual releases; the third is sprint count per release (actual versus predicted).

“These metrics allow us to say ‘Yes’ or ‘No,’ because you can easily tell if you’re credibly able to deliver on what you promised,” he says.

3. Are we meeting our commitments?

Velocity is the most common measurement used to gauge this metric, Dalton says; most agile teams use velocity to determine whether they’re meeting or missing their sprint forecasts and thus their overall commitment to delivering features, products and projects.

Velocity is the amount of work a software team can accomplish within a set amount of time. Different organizations measure it in different ways, either by using person hours, number of tasks per sprint, story points — whatever unit of measurement works best for the team in question.

4. Are we producing quality products and services?

This is arguably the easiest metric to determine quantitatively, Dalton says, as it’s measured by:

• Defects per story
• Code quality, which itself is measured in three subcomponents: unit tests, complexity, and adherence to industry coding standards
• Defects per sprint
• Number of failed code merges/integrations
• Number of escaped defects — aka defects or bugs that made it all the way to the customer

5. Are we showing the outside world our best?

This final question can only be answered in the affirmative if the previous four are accomplished to the best of an organization’s ability, Dalton says, but it’s incredibly important and shouldn’t be left out.

Establishing valuable metrics

Once an organization can answer those five questions affirmatively, it can begin measuring the value of its agile efforts in earnest.

Typically, in Scrum teams, two sets of metrics are used, says West. The first pertains to quality, as measured by defect counts and number of production tickets issued. The second is work throughput, measured in velocity. But while these metrics offer insights, they don’t always illuminate the value an organization’s agile efforts deliver, nor do they always connect to an organization’s purpose or mission.

“These aren’t bad metrics, but we are starting to see Scrum teams focus more and more on value, learnings and outcomes on the product or business level in response to the fundamental question of velocity over value,” West says. “For example, it does not matter how fast your delivery or how high-quality the software you’re delivering is if it does not add any business value; either because it did not have any value to the user or stakeholders and it was never used, or, when it was used, it did not do what anyone wanted,” West says.

The Evidence Based Management (EBM) framework

To address this gap, Scrum.org focuses on an Evidence Based Management (EBM) framework, which provides four Key Value Areas (KVAs) that help teams focus on value:

1. Current value
2. Ability to innovate
3. Unrealized value
4. Time to market

“By providing metrics for these four areas, teams and organizations can effectively see the impact of the work they are doing, and can explore the relationship between the four,” West says. “For example, if we focus on time to market, how does that increase or reduce innovation and current value? Another important perspective on these metrics is not their absolute value, but how that value changes over time: What is the trend?”

The EBM framework can also help IT leaders ensure they’re asking the right questions of their delivery organizations, says West.

“For example, if time-to-market is a problem, and the competition is delivering more frequently, then by focusing on that metric, teams could look at improving flow, reducing waste and putting automation in place. If the rate of innovation is a problem, teams could focus on reducing technical debt, or improving quality. The metrics themselves provide the executive with the ability to ask the right questions and drive the right changes,” he says.

And, West adds, value-based metrics double as business metrics that can help build a better relationship with business stakeholders.

“From an executive point of view, the biggest challenge to moving to value-based metrics such as EBM is how it changes the relationship between IT and the business,” West says. “By introducing metrics, like value, that can be understood and owned by both IT and the business, a better relationship is enabled.”

That shift allows for in-depth discussions about where and how to deliver value, and changes the perception of IT from a “factory” that churns out business products to that of a valuable partner, he says.

But what about things like cost, efficiency and predictability? These will be reflected in value-based metrics, West says. “For example, cost will — or can — affect current value; efficiency will impact time to market. It is important that we focus on external, rather than internal, metrics,” he says.

Applying agile to the metrics themselves

EBM metrics are the ultimate product metrics that drive enterprise agility, West says. But that does not mean that each backlog item would not require additional, more granular metrics or measurement.

“It is crucial, for empirical processes to work, that work is linked to direct measures. This allows for inspection and adaptation based on real, direct evidence — and then these granular metrics help drive learning at the backlog level,” he says.

Finally, every organization will take the KVAs of EBM and pick metrics that are right for their unique situation, determined by what is right for the product and organizational context, West says. For example, a website’s current value might be measured by its Net Promoter Score (NPS), while a software product’s value might be measured by revenue or profit. Getting to this point, though, can be difficult for many organizations that lack access to this specific information. In that case, West says, IT and business should collaborate and determine good proxies for value.

“For example, if measuring NPS is not possible, then perhaps an organization could look at number of users and how long they stay on a web page as a proxy,” he says. “The objective is not to find the perfect metric, or to use the exact metric as everyone else, but to inspect and adapt based on experience. Picking the right metrics is a complex problem, and thus by its very nature needs to be worked on in an agile manner.”