A year into an agile transformation, the sponsor asks a fair question: is it working? The usual answers are not encouraging. “We’ve trained 400 people” describes activity. “Velocity is up 30%” describes a number that teams can raise by estimating more generously. Neither tells you whether customers get better software sooner.

If you’ve paid for a transformation, you deserve better evidence. This is how we think about it.

Metrics to be wary of

Two kinds of numbers show up in nearly every progress report, and both mislead.

Velocity comparisons across teams are the first. Velocity is a team’s private planning tool, measured in units the team invented. One team’s 40 points and another’s 80 mean nothing side by side. Rank teams by velocity and you’ll get inflated estimates, not better delivery.

Training and certification counts are the second. The number of teams “trained,” or people certified in Scrum or as a Project Management Institute Agile Certified Practitioner (PMI-ACP), shows that money was spent. It says little about how people work on Monday morning. Certification can be a useful start, but it isn’t capability.

Signals worth watching

Better measures look at outcomes the business can feel:

• Lead time: how long an idea takes to go from request to running in production. Shorter, steadier lead times mean fewer handoffs and less waiting.
• Delivery predictability: how often teams deliver what they said they would, within a reasonable range. Predictable is more useful than fast and erratic.
• Quality: defects that escape to production, and the effort spent on rework.
• Customer feedback: whether users find the product better, gathered from the people who use it.
• Employee feedback: whether people on the teams feel they can do good work. Frustrated teams often tell you about a problem months before the schedule does.
• Business outcomes: revenue, cost, or customer retention tied to the product, not to the project plan.

Consider a utility that rolls out agile across eight teams. Before the change, a small regulatory update took about 14 weeks from request to release, with long waits for approvals. If that drops steadily to six weeks while escaped defects hold flat, the transformation is doing something, whatever the velocity charts say.

Trends, not targets

One more rule: watch the direction of a measure, not its value against a target. As soon as a number becomes a goal, people optimize the number. Set a target for lead time and you’ll find tickets split or redefined to make the clock look better. A trend that moves in the right direction over several quarters is harder to fake and easier to trust.

Start with three or four measures, take a baseline before the next training wave, and review the trends with the teams themselves, not just with executives. If you can’t say what you’d expect to see if the transformation were working, it’s worth agreeing on that now. We’re glad to help you shape that conversation.