DevOps

DevOps, the combination of Development and Operations: How to create agile, reliable, & secure technology organizations.

More than ever, how technology work is managed and performed predicts the success of the entire organization. DevOps describes an approach to create agile, reliable and secure technological organizations. The core of the concept is the cooperation of different functions (See graphic: DevOps as the intersection of Development (Software Engineering), Operations and Quality Assurance (QA)) in small teams which leads to overall organizational success. Safe systems of work are created that are able to quickly and independently develop, test and deploy code and therefore value to customers safely, securely, and reliably. DevOps can have a major impact on the way technology organizations operate. It can be understood as the application of principles taken from physical manufacturing to the IT value stream. DevOps relies for example on Lean manufacturing concepts, on the Theory of Constraints, and the Toyota Kata movement. However, it may also be viewed as the continuation of agile software development. (Page 4)

DevOps as the intersection of Development (Software Engineering), Operations and Quality Assurance (QA)

It is therefore directly relevant for the project management knowledge base as a principle for Agile Project Management and Lean Project Management. It allows for a project to achieve general established goals described by PMI standards, for example to meet business objectives. IT work is hereby important for nearly all business objectives. It should be mentioned that DevOps can in addition be applicable for every organization, no matter what the core business is. Nowadays, every organization, company or NGO, needs a stable and reliable technology infrastructure to be successful. Therefore, DevOps has a relevance for all industries and the DevOps principles can be considered vital for effective and efficient project management. It can be viewed as a strategic competency to allow organizations to: (Page 52)

• Tie technology project results to business goals,
• Compete more effectively in their markets,
• Sustain the organization, and
• Respond to the impact of business environment changes on projects by appropriately adjusting project management plans

Even though the principles can be applicable to every organization this article will focus on IT organizations and on transforming IT project and portfolio management by using the DevOps principles. In addition, DevOps is not the "one answer suits all" solution for IT organizations. It requires changes in the company culture and the transition must be carefully considered since it may be costly. These shortcoming will be further discussed after an introduction to the theory and the application of DevOps.

A main concept of DevOps is to create shared goals between software development and IT operations. In addition, continuous integration practices are used to make deployment part of daily work. This was first described at the 2009 Velocity conference by John Allspaw and Paul Hammond. This leads to the theoretical foundation behind DevOps, the three ways, which lay the basis for further study and application of the principles. After an organization has made the decision to start their DevOps journey, there are three main areas, correlating to the three ways, which need to be accelerated.

DevOps has been subject to a novel published in January 2013 and written by Gene Kim, Kevin Behr, and George Spafford. “The Phoenix Project” describes the typical issues faced by an IT organization and how DevOps helps to solve them. This homage to “The Goal”, the 1984 novel by Eliyahu M. Goldratt, was accompanied by “The DevOps Handbook” in 2016, which describes how organizations can replicate the success of DevOps.

The three ways

To understand the big idea behind DevOps, several important movements in management and technology must be explored. Most importantly, value streams, the application of Lean principles to the technology value stream as well as the three ways must be understood: (Page 3)

• The principles of Flow (accelerate the delivery of work from Development to Operations and to the customer)
• The principles of Feedback (enabling the creation of safer systems of work)
• The principles of Continual Learning and Experimentation (promote trust and a scientific approach to improvement)

What is a value stream? A value stream is defined as "the sequence of activities an organization undertakes to deliver upon a customer request" or "the sequence required to design, produce, and deliver a good or service to a customer, including the dual flows of information and material". (Page 16) In technology work, the value stream is defined "as the process required to convert a business hypothesis into a technology enabled service". (Page 8) The picture below (The DevOps Toolchain) shows how the value stream can look like in DevOps.

What is Lean? In lean manufacturing, the focus lays on creating a smooth and even flow of work, by using small batch sizes, reducing work in progress (WIP), preventing rework, and optimizing the system towards global goals. These principles are equally applicable to technology work as well as all knowledge work. This application to technology work describes the focus on deployment lead time that the DevOps principles rely on. (Page 9) The lead time describes the time between the moment a change is checked into version control and the moment that this change is running in production. This phase of work includes Testing and Operations and it is akin to Lean Manufacturing in the way that it follows the goal of achieving outputs with minimized variability and high predictability.

The Three Ways

One main concept behind DevOps is that testing and operations work happens simultaneously with design and development. This should allow for fast flow, introducing the first way. Flow enables fast left to right flow of work from development to operations (See graphic: The Three Ways). To do this, work must be visible, batch sizes and intervals of work must be reduced and build quality must be ensured constantly.In addition, work in progress must be limited. Flow can be further amplified through reducing the number of hand-offs. The constraints within the value stream must be continually identified, evaluated and elevated. Specific practices are introduced in the following chapter How to accelerate Flow.

The second way enables the fast flow of feedback from right to left (See graphic: The Three Ways). This helps prevention of problems, enables faster detection of issues as well as faster recovery. Safer systems of work are created that find problems long before they would cause a catastrophic failure. This is critical to achieve quality, reliability and safety standards. Problems must bee seen as they occur and quality must be pushed closer to the source. Specific practices are introduced in the following chapter How to accelerate Feedback.

The third way enables a high trust culture. A dynamic, disciplined and scientific approach to experimentation and risk taking is established to facilitate organizational learning from success and failures. Daily work should be continually improved and local learning must be transformed into global learning. Feedback looks are continually shorted (See graphic: The Three Ways) to further amplify the results of the second way, showing that the third way is interwoven with the first and second way. Specific practices are introduced in the following chapter How to accelerate Continual Learning.

How and where to start

The following chapters will describe an approach to the application of DevOps principles. This chapter will be mostly relevant to IT organizations but it can be easily adapted to other organizations. This will be reflected in the following section. Moreover, the section will discuss an approach for an organizational transformation towards the DevOps principles. For a more detailed view on technical and organizational structures that can facilitate the principles, please see the Annotated bibliography.

To start a DevOps transformation several factors must be considered: (Page 49)

• Which value stream should the transformation start with?
• What work must be done in the value streams?
• How can an organization and architecture be designed?
• How can market oriented outcomes be enabled?
• How can teams be protected and enabled?

Choosing a value stream

Value Streams are evaluated in a number of ways: (Page 54 ff.)

• Greenfield vs Brownfield services:

Greenfield projects are software projects describe a new project or initiative. New applications and infrastructure are built with few constraints. Starting with such a project can be easier since there are no existing code bases, processes, and teams that must be considered.

Brownfield projects are existing products or services that are already in service. They often come with a significant amount of technical debt, complicating the DevOps steps. This does however not mean that such projects are not applicable for DevOps. Existing performance gaps between customer needs and business deliveries can be closed. In addition, existing teams may be open to experimentation with DevOps if they believe traditional methods might not be sufficient to reach their goals.

A significant factor is therefore not the age of the project but rather its architecture and whether it could be re-architected for testability and deployability.

• Systems of engagement and systems of record

Systems of record are systems where the correctness of transactions and data is most important. These ERP-like systems typically show a slower pace of change and are subject to regulatory and compliance requirements. The focus is on "doing in right".

Systems of engagement are customer- or employee-facing systems such as e-commerce systems that typically show a faster pace of change. This enables experimentation to assess customer needs. The focus is in "doing it fast".

This divide between correctness and speed can be resolved when using the DevOps principles. However, because of the interdependence of systems the most difficult system to change securely limits the ability to make changes. Such a system is almost always a brownfield system of record. This concludes that when changing such a system, one should not only focus on improving its safety and reliability, but also improve on the ability to make fast and safe changes to the system. This allows to add new functionality to greenfield systems that rely on the existing system without impacting stability of this system.

• Risk/reward balance

Changing a brownfield system of record typically has a higher risk. Since many systems rely on the existing infrastructure, the impact could be bigger compared to starting the transformation with a less risky greenfield system of engage. However, making the downstream systems safer to change, helps the entire organization to more quickly and safely reach its goals by allowing the organization to quicker change the systems of engage that rely on the existing infrastructure. To assess whether such a system could be used to start the transformation the next factor should be taken into consideration. The team supporting the system must be assessed.

• Resistance from teams

When deciding to transform a IT organization using the DevOps principles it is not a good strategy to use the big bang approach (starting everywhere at the same time). Effort and focus should rather be set in few areas, where success must be ensured.

For this first area of focus a team of Innovators and Early Adopters should be set up or found. These are people wanting to help the transformation because they already believe in the need for change and because they are likely to embrace new ideas. Ideally, this should be a team of respected and influential people.

Afterwards, a critical mass and a silent majority must be built for a stable bass of support for the DevOps practices. One should start working with teams receptive to the idea, even if they are not the most visible. This creates a bandwagon effect while bypassing political battles.

Finally, holdouts must be identified. These are influential people likely to resist the transformation. This group should only be tackled after establishing a silent mass to ensure success of the initiative.

Evaluating work that must be done

After identifying a value stream considering the factors explained above, a understanding about the way value is delivered to the customer must be reached. It is important to understand what work is done by whom and what steps can be taken to improve the flow of work. To do this the following steps can be taken: (Page 61 ff.)

• Identify the teams required to create customer value

No one person knows all the work required in a single value stream. Many teams are involved in the process. Typically the following functions must be considered: Product owner, Development, Quality Assurance (QA), Operations, Infosec, Release managers, Technology executives/Value stream manager.

• Create a value stream map

After identifying all relevant teams supporting the value stream, a concrete understanding of all work that is performed to deliver value to the customer must be reached. This can be documented in a value stream map. Building such a map is time consuming and must liekly would require a multi-day workshop.

Most likely, work begins with the product owner and a customer request or a business hypothesis. Secondly, Development accepts this work, builds features and implements them in code before checking them into a version control repository. Finally, after integration and testing, code is deployed into production. Typically, this process would consist of a high number of individual steps requiring an equally high number of people to perform them.

However, the goal is not to document every single step. It is rather to gain an understanding of the areas hindering a fast flow of work, preventing short lead times and jeopardizing reliable outcomes for the customer. To achieve this understanding, one can focus on the following areas:

• Places where work must wait for long periods of time
• Places where significant rework is generated or received

• Guide teams to better and more quickly create value

To achieve significant change withing ongoing business operations one should set up a dedicated transformation team. This team should be held responsible for reaching a clearly defined and measurable output. The team should consist of member that are only dedicated to the transformation effort. They should show skills over a variety of domains and they should be respected within the organization. Furthermore, a separate physical space should be set up for the team.

In addition, the planning horizons for the team should be kept short to allow flexibility. Furthermore, the team must not only focus on functional requirements but also on non-functional requirements and reducing existing technical debt. Finally, the visibility of work should be increased to allow everyone in the organization to assess the initiative.

Designing an organization and architecture

The DevOps transformation starts with choosing a value stream and building a dedicated transformation team for the effort. Next, one should think about the way one should be organized to achieve the value stream goals. In doing so, one should keen Conway´s Law in mind: The organization of the software and the organization of the software team will be congruent; "commonly stated as: 'if you have four groups working on a compiler, you´ll get a 4-pass compiler'". (Page 224)

Thinking of the organization answers also the next question of enabling market oriented outcomes. This is achieved through market oriented teams. To achieve the DevOps principles, effects of functional orientation ("Optimizing for cost") must be reduced. Goals should rather be market oriented ("Optimizing for speed"). This follows the definition of three primary organizational types as defined by Dr. Roberto Fernandez.

Large, top-down organizations should be replaced by teams in which all functional skills are embedded. These teams are responsible for testing, operations and security in their daily work. However it should be mentioned that a functional orientation is also feasible as long as all stakeholders of the value stream follow a shared goal (the customer and organizational outcomes).

Finally, looking at the organization can also answer the question of how to enable and protect teams. First of all, team members must be enabled to become generalist with the possibility of acquiring all relevant skills instead of developing a team of specialist. In addition, one could argue that products and services should be funded rather than projects. This strengthens the emphasis on reaching organizational goals rather that project goals, like a timely completion. Furthermore, the software architecture of the organization should be as independent as possible, to allow the independent teams to make changes without interfering with the work of other teams. In addition, the size of the individual teams should be limited. (Page 85 ff.)

A small case story

The following chapter will provide a short description of a successful case of applying the DevOps principles. This should help in further understanding of the theory and application. One of the most referenced case-stories is the story of Etsy, an online marketplace, who were among the first to successfully adapt the DevOps principles. Part of the success is due to the fact that in 2009, when the transformation started, the change was initiated from the top negating the need for senior management buy in. This is however not true for most DevOps transformations. However, Jon Cowy (Senior Engineer at Etsy) has some advice for successful transformation processes in all organizations:

• focus on the business benefits it can provide, rather than getting too bogged down in tech talk
• start with a concrete project and show senior management how it has positively affected the business
• the idea should be discussed business terms because IT and development are service organisations that exist to fulfil the priorities of the business

In 2015 the Etsy VP of Technical Operations Michael Rembetsy spoke in an interview with John Dix about the steps of the transformation:

• The need for a change arose when small feature releases required laborious code deploys and deploys of the complete website
• That realization led to the creation of a new team, that build tooling to let people deploy quicker.
• During this phase, the team also realized the need to make the deployment process simpler and to simplify the interdependence between different system.
• The next step for Etsy was the decision to let developers deploy code themselves. If developers felt the responsibility for deploying code to the site they would also, take responsibility for if the site was up or down, take into consideration performance, and gain an understanding of the stress and fear of a deploy.

Rembetsy however describes no intended change with the DevOps principles in mind. It was rather a organic development with the goal of making the processes faster, better and stronger. Most important for this organic transition from his point of view was the high trust culture in the company. Building this trust requires time and faith in the work of peers, showing one of the challenges of any DevOps transformation. Therefore, he mentions these points to consider before adopting the DevOps methods for work:

• Why are you planning the transformation? Good reasons include the following:
  • improve the overall structure of the engineering culture
  • enable people to feel more motivated and to take ownership
  • improve the community in which one is responsible
  • Improve the product one is responsible for
• Remember that the transformation takes time and will not be an overnight process
• It has to be a cultural change in the way people are interacting and disagreements and discussions along the way are ok

Applicability for other sectors and limitations

As the adaptation from principles taken from physical manufacturing shows, the DevOps principles and ideas can hold true for different sectors. However, the concrete realization behind DevOps, the collaboration between development, operations, and QA is primarily applicable to IT projects. What can be said is that every organization, independent of the core business, must conduct technology work in some way, to be successful competitors in nowadays market landscape. This shows the importance and relevance of DevOps, which can assist in creating value from IT projects.

However, the DevOps principles and ideas are not without shortcomings and the transition should be carefully evaluated. Using the DevOps principles when structuring or managing a new project or portfolio does not only require a new set of tools but also a new mindset and new culture. These changes must reach further than the IT operations departments. Every department and every team that relies on the services provided by IT must be ready to provide continuous feedback. Therefore, to achieve a successful transition, first a comprehensive understanding of DevOps must be reached in in the organization. In addition, funding should be secured. This includes both financial as well as time related resources. Because, if you chose to transition to DevOps, you should do it correctly. “A half-baked, insufficiently-funded DevOps “effort” will likely leave you worse off than when you began.” . And DevOps is not the only possible way to reach the desired goal. It is cost intensive, both in labour cost as well as in operational cost due to extended automation. If speed of continuous updates is not business critical, DevOps might not be the right choice.

Therefore, the following points should be considered:

• Organizational challenges/disadvantages

DevOps requires a number of organizational changes. Previously required skills and skillsets might not be sufficient anymore, while new skills must be added to teams. These leaves old personal frustrated and opens the need to change personal through hiring people from the external market, where resources are limited.

• Process related challenges/disadvantages

Continuous feedback is at the core of the DevOps principles. However, if not managed correctly, this may lead to difficulties. At least early on in the process an organization must be fault tolerant. Suggested changes might be implemented without careful consideration, overwhelming the user or destabilizing the system.

• Technology related challenges/disadvantages

One key technology behind the DevOps principles is the automation of work, for example automating are feedback loop back from users to developers or quality assurance testing. This must be done so that the pace of operation Cn keep up with development. However, if set up incorrectly, it might lead to unforeseen consequences.

• Speed and security related challenges/disadvantages

Typically, speed and security are not two factors that work together well. However, in DevOps both must be ensured. Traditional security related work, where a design of a system is analysed for weak points, has to be changed. In DevOps systems are run in production quickly as a MVP, often without a proper design. The security team must then depend upon the advice of the developers and the operations people, showing the need for a good communication between these functions. .

Annotated bibliography

The following list provides resources for further research and study on DevOps principles:

• Kim, G.; Humble, J.; Debois, P.; Willis, J. (2016): The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

- The DevOps Handbook describes how using the DevOps principles can integrate Product Management, Development, QA, IT Operations, and Information Security to help effective management of technology. It provides a theoretical explanation of the principles as well as a practical approach to applying the DevOps principles.

• Davis, J.; Daniels, R. (2016): Effective DevOps : building a culture of collaboration, affinity, and tooling at scale

-Effective DevOps provides actionable strategies to achieve organizational changes. These changes must come from within and this book seeks to be a practical guide to the DevOps principles. It primarily focuses on human factors and job organization in general. It offers a collection of ideas and approaches for improving on key factors of the DevOps principles, like improving collaboration within teams, creating affinity among teams or promoting efficient tool usage.

• Forsgren, N.; Humble, J.; Kim, G. (2018): Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations

-This book provides a summary of the latest research on DevOps and Software Development. It explains, how technology can be applied to drive business value. Findings and the science behind ongoing research on the topic of software delivery performance is presented. The information is presented in a way that makes the application to other organizations possible. For example, explanation on how to measure performance of teams or on what capabilities to invest in to drive higher performance are presented.

• Kim, G.; Behr, K.; Spafford, G, (2013): The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business

- The Phoenix Project is a novel about an IT manager that uses the DevOps principles to lead a technology organization to new success. It introduces the Three Ways and shows a fictive success story of the DevOps principles.

• Kim, Gene (2019): The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data

- The Unicorn Project is the continuation of The Phoenix Project. It takes the perspective of the software development department and introduces the "Five Ideals" to further introduce the theory of DevOps.

• Goldratt, Eliyahu M. (1984): The Goal

- The Goal is the inspiration for The Phoenix Project. It explains the theory of constraints in physical manufacturing. This theory is used in the DevOps principles extensively. It is regarded as an important work in operations research to teach students about the importance of strategic capacity planning and constraint management.

• Puppet Lab: State of DevOps Report

- This yearly report (2011 - 2020) provides new insights into how organizations evolve the DevOps practices. The ninth publication of the report includes over 2,400 participants who work in IT, development, information security and related areas. The newest edition focuses on a platform approach to software delivery and applying DevOps principles to change management.

• Farley, D.; Humble, J. (2010): Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation

- Continuous Delivery describes principles and technical practices that enable rapid, incremental delivery of high quality, valuable new functionality to users and therefore laying the basis for the DevOps principles. Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers and IT operations, teams can release changes quicker and safer.

• Allspaw, J.; Hammond, P (2009): 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr [conference]; available at: https://tech-talks.code-maven.com/ten-plus-deploys-per-day.html

- Video taken from the Velocity conference in 2009. Key concepts for DevOps are introduced (shared goals between development and operations, and continuous integration practices) through an exploration of the technology work performed at flickr (American image hosting and video hosting service).

• Ries, Eric (2011): The Lean Startup

- The Lean Startup describes how to create a successful entrepreneurial business and explains how the principles could be used in other organizations. The principles resemble the DevOps principles. The book argues that the customer needs should be at the center of each organization. Working towards the business objectives can help the creation of shared goals between development and operations, a key concept behind DevOps.