Product Managers, do you know how much your bugs cost?

by Dean Peters

TL;DR — Not knowing the cost of bugs is about as wise as learning to juggle using chainsaws. Sure, you learn from your mistakes, but at what cost?

Counting the Cost

Yesterday I wrote why it’s important to know how much your teams cost per sprint. Today I’d like to build on that conversation by offering thoughts on how one might calculate the cost of a bug that escapes into production following a process awesomely diagrammed by Emanuil Slavov below.

Ok, here goes nuthin:

  1. We’re going to assume this is a small bug that got post our internal testing that will take two days to turn around. We double this amount for medium-sized and quadruple it for biggie bugs; bug = 1, time = (2*8).
  2. A customer opens a support call, an engagement that will total up a quarter of their available bandwidth over the life of the bug fix in both the creation and resolution calls for the escaped bug; cs = .25;
  3. Depending on the severity of the bug, an account executive or product manager gets involved, again a quarter for the combined front end and then back-end efforts for any customer-to-bug interactions; mgr = .25
  4. Let’s say it takes a single developer 100% of their available time to take on our bug. We add .5 developers for a medium-sized bug and another .5 for a big bad bug; dev = 1.
  5. Then there is some QA, whether it’s a full-time tester or perhaps a combined effort of the said role and the product owner, consuming half of their availability over a two-day fix cycle; qa = .5
  6. If we average about $50 an hour for all involved, or some similar variation of those involved, then wage = $50, which brings our cost to fix an escape to $1,600.
((($50 per hour * (1 dev + .5 qa + .25 cs+ .25 mgr) sans tools & goodies)*8 hours)*2 days) 
= $1,600 to fix a small bug (* 1), 
= $4k for medium (* 2.5), 
= $9.6k for big (* 6.25).
(wage * (dev + qa + cs + mgr)) * (hrs * days)) = cost * size
($50 * ( 1 + .5 + .25 + .25)) * (2*8) = $1,600 * size

If you merely want to use 2.5 as a multiplier to scale up to each bug severity level, $4,000 for a medium sized bug and $10,000 for a big bug; size = (1sml, 2.5med, 6.25lg). Yeah, I like that, let’s go with this.

So using our math from yesterday, where we calculated the cost of a team of six at $24,000 per two-week sprint, we can see just how expensive an escape can be relative to feature delivery. Moreover, this amount doesn’t include other costs your organization incurs such as:

  • lost opportunities
  • erosion of customer goodwill
  • reliable usage analytics
  • team care and feeding
  • delays in delivering new functionality
  • leave comments if you think of other costs …

Bottom line, if a team member tells you they need extra time for testing, automation, technical debt remediation, or simply ‘doing it right the first time,’ listen to them and politely push back on those pressuring you for something new for their next sales demo … perhaps even mentioning the cost enumerated above.

But What about Barry Boehm?

Hey Dean, I can see how you got to $1,600, but the following popular graphic puts the cost of a post-release bug at $16,000! What gives?

Yes, the above chart by Capers Jones from his 1996 classic ‘Applied Software Measurement: Assuring Productivity and Quality’ is accurate, however, I also believe it was influenced in part by the writings of Dr. Barry Boehm in his 1981 book titled ‘Software Engineering Economics.’

In other words, even when adjusted for inflation, the $26,000 per post-production bug cost is offered within a waterfall context.

My calculations are based on an Agile setting, where you’re hopefully delivering functionality in much smaller increments over a much shorter timebox with faster feedback loops and better tooling to deploy fixes.

Below is a visualization depicting these deltas from an article titled ‘The Agile Difference for SCM’ that also explains in greater detail the cost differences.

Yet another argument for not going back to the dark days of the Waterfall SDLC, dontchathink?

Moving Forward

Regardless of how you slice-n-dice your bug costs, it’s always expensive, wasteful, and painful. As a product manager you can help your organization mitigate these costs in three ways:

  1. Don’t deprioritize items such as test automation nor technical debt remediation.
  2. Provide your teams with product backlog items that are small and testable.
  3. Begin to bug upper management to invest in an experimentation framework, preferably one fueled by feature toggles (more on this topic in later posts).

Meanwhile, I’m hoping some Ph.D. candidate in computer science, or perhaps a professor of economics rips my math to shreds and provides us all with a more durable and better-vetted formula for calculating the costs of bugs in both Agile and Lean development contexts. Here are some articles to get that research ball rolling:

And even if you’re not an academic type, feel free to tear me a new one, just so long as you provide us all some better numbers and calculations.

What about the cost of dev per sprint?

Glad you asked, here ya go:

YGWYPF