A guide on how to measure DevOps ROI


DevOps is commonly introduced into organizations for some well-understood, and usually highly justified, reasons. DevOps speeds the overall pace of the software development and release cycle, improves the quality of deliverables, and helps to resolve problems more quickly. But all of these intangibles should, and do, flow down to the bottom line. Which means that any well-designed program should include a healthy perspective on ROI.

So why isn”t DevOps ROI routinely measured?

Measuring the unmeasurable?

One school of thought holds that DevOps ROI isn’t measured because it really can’t be.

Michael Cote is the poster child for this idea at the moment. He recently wrote that the question of “what is the ROI for DevOps?” was “simultaneously absurd and important.” In a recent blog post for FierceDevOps, he says, quite accurately, that DevOps is not so much an investment as it is an operational process, and that the number of variables in an organization is too great to make this calculation possible. “DevOps is an unmeasurable process with respect to ROI,” he writes. “It has value, to be sure, but is nearly impossible to measure independently and precisely.”

Cote goes on to offer suggestions for CIOs who are tasked with making a DevOps ROI calculation. Typically, these involve fudging numbers so executives will be comfortable with the adoption of the strategy.

DevOps ROI can’t be measured, but the ROI of the projects you initiate and the products you develop and release after DevOps is instituted can be measured, Cote argues.

Is Cote right? Is DevOps too intangible to be measured with the degree of accuracy that leads to a proper ROI calculation? The other side of the argument is, of course, that it can and should be measured, and that there are a variety of mathematical constructs that can lead you to a valid ROI.

Consider a few of these approaches.

Time is money

One of the biggest benefits, if not the biggest benefit, of DevOps is the promise of speed. DevOps enables more and faster code deployments, which means a decreased time to market and more opportunities to capture revenue from customers.

The relatively simple math works like this.

Presume you experience the bulk of your sales when a product is first released, and these sales gradually decline until your next product (or update of the same product) is released. Over time, your revenue develops a familiar pattern, first spiking, then declining, then spiking again, creating a sawtooth graph.

What happens if you speed up the pace of development and accelerate your release schedule by a factor of two? You get the same sawtooth graph, but with twice as many spikes. In this example, your overall revenue is doubled, likely making your ROI from DevOps quite high even after accounting for any costs incurred in the course of implementing a DevOps framework. DevOps.com explains this (including an easy-to-understand graphical explanation).

This is an oversimplified example, and it raises more than a few questions. Not every product release is equally profitable. In fact, not every product release is profitable at all. DevOps releases typically function as more incremental, smaller updates, and many may not be focused on generating revenue at all but rather responding to customer feedback, fixing bugs, or doing other maintenance-related updates. It would be foolish to make an assumption that every software release has a significant financial impact.

At the same time, these are all numbers that can indeed be quantified. Overall revenue is a fundamental business metric. Sales can be broken down for each product and on a daily (or even hourly) basis. With some careful but relatively simple analysis, you can generate a model that describes this pattern. If you’ve implemented a DevOps program or even piloted one, you can use those numbers to generate a contrasting picture of revenue plotted against the original data.

Financially measuring quality improvements

Another way to measure DevOps ROI is a corollary to the velocity-of-revenue calculation outlined in the previous section. This approach considers the increased quality of work performed under a DevOps framework and calculates ROI from a cost-avoidance standpoint, rather than a revenue one.

Toon Verbeek, developer evangelist for Wercker, used Keen IO analytics to graph the pass/fail rate of Wercker’s development team. Through a system like this, Verbeek is able to see explicitly whether a DevOps framework improves build quality as promised. He has found that it has, with actual metrics to prove it.

He believes his findings can be converted into a monetary value suitable for an ROI calculation. “You can break it down with the hourly rate of development time,” says Verbeek. “Assuming an average salary of $120,000, developer labor roughly correlates to $60 per hour. So let’s say each failed build takes an average of three hours to fix, and you’ve reduced build fails by a factor of two, week-over-week. You’re looking at a cost ‘savings’ of $360 week-over-week.”

This is a limited example, but it does showcase a legitimate, real-world ROI calculation that can be extrapolated on a broader scale and tracked over time.

Enhanced productivity as an ROI driver

A third way to look at DevOps ROI considers the team’s overall productivity, specifically a calculation of idle time and how it is avoided.

“IT leaders need to connect the strategic objectives of the business to the work their teams do in order to create a DevOps transformation,” says Pete Cheslock, senior director of operations and support at Threat Stack. “If a goal-oriented culture is being achieved, it will lead to more efficient collaboration. The easiest way to measure this is by looking at the number of deployments per day per developer.

Additionally, IT leaders who are able to manage more infrastructure with the same number of team members can point to that improvement as a measurement of success.”

Of course, productivity is more difficult to quantify from a financial standpoint than some of the metrics we’ve discussed above. Says Cheslock, “Attempting to put a monetary value on deployments per day does not give an organization an accurate gauge of success. It’s not the value of deployments per day; it’s the value of efficiency in your deployment process, which could potentially result in more deployments per day and a reduction in the cost of deployment.”

Efficiency and frequency

That doesn’t mean that financial metrics surrounding IT productivity are impossible to generate. You just have to go about creating them in a slightly different way.

Nicole Forsgren, director of organizational performance and analytics at Chef, has done extensive research into DevOps deployments and has found that high-performing DevOps organizations are twice as likely as other companies to exceed profitability, market share, and other financial goals.

One of the key measurements she has identified involves metrics that quantify what she calls overall IT performan
ce, which is in turn composed of three key factors: deployment frequency, lead time to deploy, and mean time to restore. These are all designed to show throughput and stability in the development process. “This is an important measurement,” says Forsgren. “My research has shown that throughput and stability move in tandem.”

Forsgren’s research classifies organizations into three groups based on these IT performance metrics. Specifically:

  • High IT performers. Have high deployment frequency, fast lead times, and fast mean time to restore. These companies are the best at significant statistical levels on all of these fronts.
  • Medium IT performers. Have medium deployment frequency, medium lead times, and medium time to restore.
  • Low IT performers. These perform the worst, statistically, on all measures.

Now, when you look at the financial performance of these three groups, Forsgren says that you find that the high IT performers group is strongly aligned with high overall organizational performance, which means higher overall profitability, greater market share, and overall better financial results. Similarly, the low IT performers group does comparatively poorly on the same financial measures. “This shows the metrics are tightly related and suggests that IT performance is predictive of organizational performance,” she concludes.

What this fundamentally means is that some of the softer IT-focused metrics that measure…


Read More:A guide on how to measure DevOps ROI