How to survive the great DevOps culture shift


As IT organizations seek to bring developers and operations staff into harmony through a DevOps approach to collaboration, automated technology and process improvements play a crucial role in continuous, iterative development and deployment. But above all else, a sound DevOps culture fundamentally solves one of IT’s biggest people problems—bridging the divide between dev and ops teams in order to get them to stop working at cross-purposes with one another.

In fact, Gartner estimates that when infrastructure and operations teams try to drive a DevOps initiative without nurturing a cultural shift as well, they’ll fail 90 percent of the time. In order to truly get DevOps to work, both sides of the house must be willing to cooperate in the behavioral modifications necessary to effect cultural change.

“With respect to culture, DevOps seeks to change the dynamics in which operations and development teams interact,” explains Laurie Wurster, research director at Gartner, in a piece by Consultancy UK. “Key to this change are the issues of trust, honesty, and responsibility. In essence, the goal is to enable each organization to see the perspective of the other and to modify behavior accordingly, while motivating autonomy.”

Of course, all this is easier said than done. Dev and ops represent the biggest mismatch in perspectives and temperament since Oscar and Felix, the original Odd Couple. Developers are innovators, paid to be agents of change. Operations staff pray at the altar of stability, tasked with maintaining uptime and continuity.

Up until now, it might seem that IT leadership has set up this schism by design. At most organizations today, software engineers and operations staff are pressured by a different set of objectives handed down by executives. They rarely work together enough to understand the value each group brings to the ultimate objective of improving customer interactions. Both sides operate in fear of missing their objectives, and too frequently they engage in finger-pointing when experiments go south. If organizations want to tap into the efficiency and speed of innovation afforded by DevOps, IT leadership must find a way to bridge the gap between these two technology tribes to get them to remember they’re playing for the same team and encourage them to find new ways to achieve mutual objectives.

Gain developer buy-in

The first step in bridge-building is getting developers to buy into the DevOps ethos, which is sometimes a tall order, considering that a lot of the initial impetus behind DevOps can be attributed to the pain felt by operations staff when developers foisted unstable code on their fine-tuned environments. This is a pain that grew in regularity as dev embraced agile philosophies.

“One of the issues of agile is it was all about speed and we lost the reliability and quality because we took all of these shortcuts,” says Mike Kavis, vice president and principal architect for Cloud Technology Partners, an IT consultancy. “And I really believe the DevOps movement came from those system administrators tired of getting paged at four in the morning.”

In a collaborative culture, developers both empathize with their operations brethren and also understand why ops problems are development problems, too. But how do you get there? DevOps success stories show that there are some key ingredients to lasting cultural change.

Incentive shifts

In most organizations, the reason that dev and ops work on such divergent planes is that they’re paid to. “[IT organizations have] created a situation in which dev and ops are incented to do different things, and this is at the heart of where the conflict between dev and ops arises,” explains Kurt Bittner, analyst for Forrester Research.

When developers are only measured by the speed at which they roll out new features, and operators are only judged by uptime, getting the two groups to see eye-to-eye is next to impossible.

“One of the challenges that I see is that people say ‘Hey, we are going to move to DevOps,’ but they don’t change people’s incentives to behave differently,” explains Kavis. “I come from the app side and I remember at this one company, we had our ‘top 10’ list and the infrastructure had their top 10 list, and they didn’t match. So if I needed a server and it wasn’t on their top 10 list, then I was SOL.”

Reduce feature guilt

As both Kavis and Bittner explain, the more leadership drives both groups against a common set of incentives, the easier it will be to start shifting culture. “Essentially what we need to do is measure everybody the same way,” Bittner says. “And part of that means measuring devs on stability, and part of that means measuring ops on innovation. But the best thing is to reward everybody on improving the customer experience.”

This is exactly how teams at Orbitz started to transform their culture in order to drive a DevOps practice that ultimately helped the travel reservation company greatly improve the customer experience.

Ori Rawlings, senior software engineer for Orbitz Worldwide, spoke at Velocity last year: “One of the things we did early on was established a shared goal across dev and ops teams: to double the hotel search capacity across our whole site.” Setting up that shared goal helped drive developers to work incrementally on obscure bottlenecks that had been plaguing the site for a long time and ultimately achieve big wins in performance and capacity.

Changing that incentive structure also jump-started a shift away from a developer mentality that can really kill DevOps collaboration—it’s called feature guilt. According to Greg Burton, software engineer for Orbitz Worldwide, one of the reasons it can be tough to get developers to help operations solve long-standing performance problems is that they consider it “unproductive” time. “We’ve got to view operations-driven development as productive work. That is a cultural thing that has to change,” he says. “The belief goes all the way from management, down through your peers, and all the way to the business that features work is the productive work and operations work is the thing you have to do real quick on the side get it done—out of the way—and get back to the feature work. But that will kill you.”

As Orbitz went on its quest to improve the customer experience, one of the things it worked on was getting developers to the point where they understood that tending to the health of a web app was just as important as developing new features for that app.

Making failure less expensive

One of the major drivers of the DevOps continuous delivery practice is the experimentation that is possible once an organization is running on all cylinders. And innovation thrives on experimentation. Unfortunately, one of the big cultural hang-ups that impedes DevOps progress is an environment that heavily penalizes failure.

“The cost of failure has to be as small as possible. Right now in the enterprise the fear of failure is endemic,” explains Randy Bias, vice president of technology for EMC and director at the OpenStack Foundation. “It’s practically a culture of fear. It’ll take a long time to change that culture of fear but one of the primary ways to do it is to start driving down the cost of failure.”

This can be a bit of a catch-22 for organizations with monolithic application infrastructures, but the more organizations can make it possible to move toward bite-sized development chunks, the less expensive these iterations become, says Kavis. “Get to small deliverables,” he recommends. He also believes that blameless post-mortems will not only help give developers confidence that they won’t be raked over the coals every time they try and fail, but it will also present a huge learning opportunity.

“Even if your experiment failed you probably learned something from it,” he says. “A lot of technology we have today is from failed experimentation.”

In fact, one of the great DevOps success stories credits blameless post-mortems as playing a huge part in instilling a collaborative culture. E-commerce giant Etsy was able to grow rapidly based on this experimentation-friendly practice.

“One of the things I allowed people to do is make mistakes more freely,” Chad Dickerson, CEO of Etsy told Business Insider. “The best way to learn to ride a bike is to ride the bike and fall down. We have a ground rule that the purpose of a post-mortem is to find out…


Read More:How to survive the great DevOps culture shift