As an engineering manager, I have used many metrics and systems over the years to measure team performance.
I experimented with both simple Kanban systems and more structured AGILE Sprints. Both had their respective share of advantages and disadvantages.
Although a Kanban system was easy to manage with less logistical overhead, it led to less accountability. By not committing to a set amount of work in a given time (like you do in Sprints), Kanban resulted in more “infinite open loops” and a lack of retrospectives.
On the other hand, a Sprint system created a platform that encouraged greater accountability, however, that came at the risk of increased meeting load (Sprint Planning, Sprint Retro, Backlog Refinement, etc).
Having run both systems for extended periods, I concluded that it doesn’t matter how you measure your team’s productivity, but what you measure.
Let me explain.
Creating products through code is complex.
More than writing code, it involves alignment between people and teams. And, whenever you move away from the world of 1s and 0s to the world of people, things can take much longer than expected.
Whether it is deciding on designs, choosing experiment metrics, reviewing technical specifications, or getting approvals on security or data requirements — whenever discussions are required, it will impact project timelines.
That’s why, choosing "objective" metrics does a poor job at measuring productivity. Let’s look at a few of these metrics:
Lines of code shipped
Number of PRs (Pull Requests) shipped
Number of tests written
Total Story points completed in a Sprint
Number of tickets closed in a Sprint
When you pick one of these metrics to optimize, you are telling your team what you care about the most.
And when you do that, you are incentivizing your engineer to make these metrics look good.
Let’s look at a few concrete examples.
Lines of code shipped: your engineers can make the code unnecessarily verbose.
Number of PRs shipped: your engineers can intentionally break PRs down into atomic units.
Number of tests written: your engineers can write redundant tests.
Total story points completed in a Sprint: your engineers can inflate effort estimates.
Number of tickets closed in a Sprint: your engineers can break a single task into multiple tickets.
It’s simple really.
When you pick bad metrics that are easy to game, you will incentivize people to focus on the wrong thing.
As you incentivize people to optimize the wrong thing, the metric you think is a measure of team productivity, will end up measuring something different.
That brings the question — why do you want to measure your team’s productivity? What’s actually important here?
The most important deliverable for any engineering team is simple: business value.
That’s why you want to measure your team’s productivity. You want to make sure your team is providing value to the company, healthily and sustainably.
So, that’s how you want to incentivize your engineers. You want to empower them to care about delivering business value.
It shouldn’t matter how many lines of code they write, how many PRs they ship, or how many tests they write. They are not even good proxies, let alone the sole measure of productivity.
To sum up: you need to create a system that incentivizes your engineers to create more business value.
Now, the question is: how do you do that?
It’s no surprise that measuring value is difficult. That’s why people look for proxy metrics in software engineering.
The value of an engineering project needs to be determined at a very early stage.
Each project must be connected to some business metric — $$$ revenue, # of clicks, $$ saved, # of views, etc.
Whatever the business cares about, there should be a direct way to tie engineering projects to it. Once that’s done, you have an overall business value associated with a project.
The next step is to break that down into smaller chunks (milestones).
Each milestone should have a logical connection to the bigger picture. The milestones should be broken down in such a way, that at the end of every Sprint (say 2 weeks), something should be delivered to show progress.
The deliverable doesn’t have to be a fully fleshed-out feature. It can be something as simple as:
A written document that’s out for review
Approval of a specification
API contract
API skeleton with test data
Creating an empty database
Aligning on a database schema
A non-functioning UI element
If each milestone can become a “deliverable”, it should be easy to understand how these chunks fit into the larger puzzle to deliver business value.
Each milestone should be led by one engineer. They should be empowered to set their timelines, as long as they can commit to showing progress at the end of every Sprint.
It doesn’t matter how they do it — # of PRs, # of lines of code, etc — what matters is at the end of the Sprint, they have something to show progress. More importantly, given they were empowered to tell you what they could realistically demo, they are measuring their progress. Instead of being told when to deliver, they have the stopwatch in their hands.
So now, you are incentivizing engineers to deliver business value at some set frequency. You are not incentivizing them to optimize logical metrics, such as, the number of PRs closed or story points completed.
They know that their success will be the delivery of the milestone which would directly translate to business value delivered in the larger project.
Even though the incentives are aligned with this system, there are a few glaring issues.
When engineers are incentivized to deliver value, regardless of how they get there, they can cut corners when deadlines are looming.
In practice that can mean tradeoffs such as:
Write fewer tests
Shorter time horizon — choosing short-term over long-term gains
Poor individual task tracking
I think that’s just a side effect of this approach.
There’s no perfect approach.
However, in my experience, this is the closest way to align the incentives of engineers and the business.
There’s another huge benefit to incentivizing continuous business value delivery at the end of every Sprint.
Red flags or mistakes can be caught much earlier in the development lifecycle.
If there are constant missed demos, that’s your sign to dig below the surface.
Okay, folks, that’s all for today.
Thank you for your time. I hope you found it valuable.
If you want to stay connected, here are a few ways you can do so: follow me on Medium or subscribe to my website.