DevOps

Latest revision as of 16:14, 27 February 2021

DevOps, the combination of Development and Operations: How to create agile, reliable, & secure technology organizations.

More than ever, how technology work is managed and performed predicts the success of the entire organization. Typically, in IT organizations software development and IT operations try to achieve different goals (development aims for speed of release and new features; while reliability and security are the focus for operations). DevOps describes an approach to break this "wall" between development and operations in order to create agile, reliable and secure technological organizations. There is no clear academic definition of DevOps yet, besides the approach to call it a cross-functional combination of the terms and concepts for "development" and "operations". However, it is both a cultural and professional movement. DevOps is a way of thinking and acting, and it is also describing the set of tools and procedures one can use. Adam Jacob (CTO of Chef) described it as: "DevOps -A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners". ^[1] The name Devops was coined by Patrick Deboi. ^[2]

The core of the concept is the cooperation of different functions (See figure) in small teams which leads to overall organizational success. "Safe systems of work" are created that are able to quickly and independently develop, test and deploy code and therefore add value to customers safely, securely, and reliably. DevOps can have a major impact on the way technology organizations operate. It can be understood as the application of principles taken from physical manufacturing to the IT value stream. DevOps relies for example on Lean manufacturing concepts, on the Theory of Constraints, and the Toyota Kata movement. However, it may also be viewed as the continuation of agile software development.^[3] (Page 4)

DevOps as the intersection of Development (Software Engineering), Operations and Quality Assurance (QA)^[4]

It is therefore directly relevant for the project management knowledge base as a principle in its own right just like Agile Project Management and Lean Project Management. It allows for a project to achieve general established goals described by PMI standards, for example to meet business objectives. IT work is hereby important for nearly all business objectives. Nowadays, every organization, company or NGO, needs a stable and reliable technology infrastructure to be successful. Therefore, DevOps has a relevance for all industries and the DevOps principles can be considered vital for effective and efficient project management. It can be viewed as a strategic competency to allow organizations to: ^[5] (Page 52)

• Tie technology project results to business goals,
• Compete more effectively in their markets,
• Sustain the organization, and
• Respond to the impact of business environment changes on projects by appropriately adjusting project management plans

Even though the principles can be applicable to every organization, this article will focus on IT organizations and on transforming IT project and portfolio management by using the DevOps principles. In addition, DevOps is not the "one answer suits all" solution for IT organizations. It requires changes in the company culture and the transition must be carefully considered since it may be costly. These shortcomings will be further discussed after an introduction to the theory and the application of DevOps.

A main concept of DevOps is, as introduced prior, to create shared goals between software development and IT operations. In addition, continuous integration practices are used to make deployment part of daily work. These practices were first described at the 2009 Velocity conference by John Allspaw and Paul Hammond. ^[6] This leads to the theoretical foundation behind DevOps, the three ways, which lay the basis for further study and application of the principles. After an organization has made the decision to start their DevOps journey, there are three main areas, correlating to the three ways, which need to be accelerated.

DevOps has been subject to a novel published in January 2013 and written by Gene Kim, Kevin Behr, and George Spafford. “The Phoenix Project” describes the typical issues faced by an IT organization and how DevOps helps to solve them. This homage to “The Goal”, the 1984 novel by Eliyahu M. Goldratt, was accompanied by “The DevOps Handbook” in 2016, which describes how organizations can replicate the success of DevOps.

[edit] The three ways

To understand the big idea behind DevOps, several important movements in management and technology must be explored. Most importantly, value streams, the application of Lean principles to the technology value stream as well as the three ways must be understood. ^[3] (Page 3)

Firstly, What is a value stream? A value stream is defined as "the sequence of activities an organization undertakes to deliver upon a customer request" or "the sequence required to design, produce, and deliver a good or service to a customer, including the dual flows of information and material".^[7] (Page 16) In technology work, the value stream is defined "as the process required to convert a business hypothesis into a technology enabled service".^[3] (Page 8)

Secondly, What is Lean? In lean manufacturing, the focus lays on creating a smooth and even flow of work, by using small batch sizes, reducing work in progress (WIP), preventing rework, and optimizing the system towards global goals. These principles are equally applicable to technology work as well as all knowledge work. This application to technology work describes the focus on deployment lead time that the DevOps principles rely on. ^[3] (Page 9) The lead time describes the time between the moment a change is checked into version control and the moment that this change is running in production. This phase of work includes Testing and Operations and it is akin to Lean Manufacturing in the way that it follows the goal of achieving outputs with minimized variability and high predictability.

The Three Ways ^[8]

Thirdly, What do the Three Ways constitute? They can be summarized as follows: ^[3] (Page 3)

• The principles of Flow (accelerate the delivery of work from Development to Operations and to the customer)
• The principles of Feedback (enabling the creation of safer systems of work)
• The principles of Continual Learning and Experimentation (promote trust and a scientific approach to improvement)

One main concept behind DevOps is that testing and operations work happens simultaneously with design and development. This should allow for fast flow, introducing the first way. Flow enables fast left to right flow of work from development to operations (See graphic: The Three Ways). To do this, work must be visible, batch sizes and intervals of work must be reduced and build quality must be ensured constantly. In addition, work in progress must be limited. Flow can be further amplified through reducing the number of hand-offs. The constraints within the value stream must be continually identified, evaluated and elevated. ^[8]

The second way enables the fast flow of feedback from right to left (See graphic: The Three Ways). This helps prevention of problems, enables faster detection of issues as well as faster recovery. Safer systems of work are created that find problems long before they would cause a catastrophic failure. This is critical to achieve quality, reliability and safety standards. Problems must bee seen as they occur and quality must be pushed closer to the source. ^[8]

The third way enables a high trust culture. A dynamic, disciplined and scientific approach to experimentation and risk taking is established to facilitate organizational learning from success and failures. Daily work should be continually improved and local learning must be transformed into global learning. Feedback loops are continually shorted (See graphic: The Three Ways) to further amplify the results of the second way, showing that the third way is interwoven with the first and second way. ^[8]

Taking the things above in mind, the DevOps principles can be summarized as shown in the picture below (The DevOps Toolchain^[9]) that shows what the value stream can look like in DevOps. Most noteworthy in the figure is the cooperation and co-creation between development and operations that is at the core of the concept. In addition, the figure tries to explain the value stream that is needed to allow for continuous delivery of features, another core concept behind DevOps.

[edit] The Five Ideals

Besides the three ways, another part of the theory behind DevOps are the five ideals. They will be mentioned in the following, however for a more detailed explanation see the Annotated bibliography. The five ideals are: ^[10]

• The First Ideal: Locality and Simplicity
• The Second Ideal: Focus, Flow, and Joy
• The Third Ideal: Improvement of Daily Work
• The Fourth Ideal: Psychological Safety
• The Fifth Ideal: Customer Focus

[edit] How and where to start

The following chapters will describe an approach to the application of DevOps principles. Moreover, the following section will discuss an approach for an organizational transformation towards the DevOps principles. For a more detailed view on technical and organizational structures that can facilitate the principles, please see the Annotated bibliography. This chapter will be mostly relevant to IT organizations but it can be easily adapted to other organizations. This will be reflected in the chapter Applicability for other sectors and limitations.

If an organization chooses to transform their IT work using the DevOps principles to facilitate shared goals between development and operations and to facilitate continuous delivery practices, several factors must be considered that will be further introduced in this chapter:^[3] (Page 49)

• Which value stream should the transformation start with?
• What work must be done in the value stream?
• How can an organization and architecture be designed to facilitate the DevOps principles?
  • E.g. How can market oriented outcomes be enabled?
  • E.g. How can teams be protected and enabled?

[edit] Choosing a value stream

Value Streams are evaluated in a number of ways:^[3] (Page 54 ff.)

• Greenfield vs Brownfield services:

Greenfield projects are software projects that describe a new project or initiative. New applications and infrastructure are built with few constraints. Starting with such a project can be easier since there are no existing code bases, processes, and teams that must be considered.

Brownfield projects are existing products or services that are already in service. They often come with a significant amount of technical debt (implied cost of additional rework caused by choosing an easy/limited solution instead of using a better approach that would take longer), complicating the DevOps transformation. This does however not mean that such projects are not applicable for DevOps. Existing performance gaps between customer needs and business deliveries can be closed. In addition, existing teams may be open to experimentation with DevOps if they believe traditional methods might not be sufficient to reach their goals.

A significant factor is therefore not the age of the project but rather its architecture and whether it could be re-architected for testability and deployability. Automatic deployment is a key enabling factor for the DevOps principles and the system must be able to facilitate this.

• Systems of engagement and systems of record

Systems of record are systems in which the correctness of transactions and data is most important. These ERP-like systems typically show a slower pace of change and are subject to regulatory and compliance requirements. The focus is on "doing it right".

Systems of engagement are customer- or employee-facing systems such as e-commerce systems that typically show a faster pace of change. This enables experimentation to assess customer needs. The focus is on "doing it fast".

This divide between correctness and speed can be resolved when using the DevOps principles. Meaning: systems of record can improve in speed while systems of engagement can improve in stability.

Typically, many systems are interdependent and rely on each other. Therefore, the most difficult system to change securely limits the ability to make changes to any of the other systems. Such a system is almost always a brownfield system of record. Improving such a system could therefore drastically help in improving other systems as well, making it a feasible candidate for the DevOps transformation. One can conclude, when changing such a system, one should not only focus on improving its safety and reliability, but also improve on the ability to make fast and safe changes to the system. This allows to add new functionality to greenfield systems that rely on the existing system without impacting stability.

• Risk/reward balance

Changing a brownfield system of record typically has a higher risk. Since many systems rely on the existing infrastructure, the impact could be bigger compared to starting the transformation with a less risky greenfield system of engage. However, making the downstream systems safer to change helps the entire organization to more quickly and safely reach its goals by allowing the organization to quicker change the systems of engagement that rely on the existing infrastructure. To assess whether a brownfield system could be used to start the transformation, the next factor should be taken into consideration. The team supporting the system must be assessed whether it is suited to be a first area of focus.

• Resistance from teams

When deciding to transform an IT organization using the DevOps principles it is not a good strategy to follow the big bang approach (starting everywhere at the same time). Effort and focus should rather be set in few areas, where success must and can be ensured.

For this first area of focus a team of Innovators and Early Adopters should be set up or found. These are people wanting to help the transformation because they already believe in the need for change and because they are likely to embrace new ideas. Ideally, this should be a team of respected and influential people.

Afterwards, a critical mass and a silent majority must be built for a stable base of support for the DevOps practices. One should start working with teams receptive to the idea, even if they are not the most visible. This creates a bandwagon effect while bypassing political battles.

Finally, holdouts must be identified. These are influential people likely to resist the transformation. This group should only be tackled after establishing a silent mass to ensure success of the initiative.

[edit] Evaluating work that must be done

After identifying a value stream considering the factors explained above, an understanding about the way value is delivered to the customer must be reached. It is important to understand what work is done by whom and what steps can be taken to improve the flow of work. To do this the following steps can be taken:^[3] (Page 61 ff.)

• Identify the teams required to create customer value

No one person knows all the work required in a single value stream. Many teams are involved in the process. Typically the following functions must be considered: Product owner, Development, Quality Assurance (QA), Operations, Infosec, Release managers, Technology executives/Value stream manager.

• Create a value stream map

After identifying all relevant teams supporting the value stream, a concrete understanding of all work that is performed to deliver value to the customer must be reached. This can be documented in a value stream map. Building such a map is time consuming and would likely require a multi-day workshop.

Most probably, work in the value stream begins with the product owner and a customer request or a business hypothesis. Afterwards, Development accepts this work, builds features and implements them in code before checking them into a version control repository. Finally, after integration and testing, code is deployed into production. Typically, this process would consist of a high number of individual steps requiring an equally high number of people to perform them.

However, the goal is not to document every single step. It is rather to gain an understanding of the areas hindering a fast flow of work, preventing short lead times and jeopardizing reliable outcomes for the customer. To achieve this understanding, one can focus on the following areas:

• Places where work must wait for long periods of time
• Places where significant rework is generated or received

• Guide teams to better and more quickly create value

To achieve significant change within ongoing business operations one should set up a dedicated transformation team. This team should be held responsible for reaching a clearly defined and measurable output. The team should consist of members that are only dedicated to the transformation effort. They should show skills over a variety of domains and they should be respected within the organization. Furthermore, a separate physical space should be set up for the team.

In addition, the planning horizons for the team should be kept short to allow flexibility. Furthermore, the team must not only focus on functional requirements but also on non-functional requirements and reducing existing technical debt. Finally, the visibility of work should be increased to allow everyone in the organization to assess the initiative.

[edit] Designing an organization and architecture

The DevOps transformation starts with choosing a value stream and building a dedicated transformation team for the effort. Next, one should think about the way an organization can be set up to achieve the value stream goals. In doing so, one should keep Conway´s Law in mind: The organization of the software and the organization of the software team will be congruent; "commonly stated as: 'if you have four groups working on a compiler, you´ll get a 4-pass compiler'".^[11] (Page 224)

Thinking of the organization answers also the question of enabling market oriented outcomes. This is achieved through market oriented teams. To achieve the DevOps principles, effects of functional orientation ("Optimizing for cost") must be reduced. Goals should rather be market oriented ("Optimizing for speed"). These two orientations are based on the definition of primary organizational types as defined by Dr. Roberto Fernandez.^[3](Page 140)^[12]

Large, top-down organizations should be replaced by teams in which all functional skills are embedded. These teams are responsible for testing, operations and security in their daily work. However it should be mentioned that a functional orientation is also feasible as long as all stakeholders of the value stream follow a shared goal (the customer and organizational outcomes).

Finally, looking at the organization can also answer the question of how to enable and protect teams. First of all, team members must be enabled to become generalist with the possibility of acquiring all relevant skills instead of developing a team of specialists. In addition, one could argue that products and services should be funded rather than projects. This strengthens the emphasis on reaching organizational goals rather that project goals, like a timely completion. Furthermore, the software architecture of the organization should be as independent as possible, to allow the independent teams to make changes without interfering with the work of other teams. Furtheremore, the size of the individual teams should be limited.^[3] (Page 85 ff.)

[edit] A small case story

The following chapter will provide a short description of a successful case of applying the DevOps principles. This should help in further understanding of the theory and application. One of the most referenced case-stories is the story of Etsy, an online marketplace, who were among the first to successfully adapt the DevOps principles. Part of the success is due to the fact that in 2009, when the transformation started, the change was initiated from the top, negating the need for senior management buy in. ^[13] This is however not true for most DevOps transformations. Nonetheless, Jon Cowy (Senior Engineer at Etsy) has some advice for successful transformation processes in all organizations:^[13]

• focus on the business benefits it can provide, rather than getting too bogged down in tech talk
• start with a concrete project and show senior management how it has positively affected the business
• the idea should be discussed in business terms because IT and development are service organisations that exist to fulfill the priorities of the business

In 2015 the Etsy VP of Technical Operations Michael Rembetsy spoke in an interview with John Dix about the steps of the transformation: ^[14]

• The need for a change arose when small feature releases required laborious code deploys and re-deploys of the complete website.
• That realization led to the creation of a new team, that build tooling to let people deploy quicker.
• During this phase, the team also realized the need to make the deployment process simpler and to reduce the interdependence between different system.
• The next step for Etsy was the decision to let developers deploy code themselves, instead of the typical structure where deployment work is done by IT operations. If developers felt the responsibility for deploying code to the site they would also take responsibility for the site being up or down, take into consideration performance, and gain an understanding of the stress and fear of a deploy.

Rembetsy describes no intended change with the DevOps principles in mind. It was rather an organic development with the goal of making the processes faster, better and stronger. ^[14] Most important for this organic transition from his point of view was the high trust culture in the company. Building this trust requires time and faith in the work of peers, showing one of the challenges of any DevOps transformation. Therefore, he mentions these points to consider before adopting the DevOps methods for work:^[14]

• Why are you planning the transformation? Good reasons include the following:
  • improve the overall structure of the engineering culture
  • enable people to feel more motivated and to take ownership
  • improve the community in which one is responsible
  • improve the product one is responsible for
• Remember that the transformation takes time and will not be an overnight process
• It has to be a cultural change in the way people are interacting and disagreements and discussions along the way are ok

[edit] Applicability for other sectors and limitations

As the adaptation of principles taken from physical manufacturing shows, the DevOps principles and ideas can hold true for different sectors. However, the concrete realization behind DevOps, the collaboration between development, operations, and QA is primarily applicable to IT projects. What can be said is that every organization, independent of the core business, must conduct technology work in some way, to be a successful competitor in today´s market landscape. This shows the importance and relevance of DevOps, which can assist in creating value from IT projects.

However, the DevOps principles and ideas are not without shortcomings and the transition should be carefully evaluated. Using the DevOps principles when structuring or managing a new project or portfolio does not only require a new set of tools but also a new mindset and new culture. These changes must reach further than the IT operations departments. Every department and every team that relies on the services provided by IT must be ready to provide continuous feedback. Therefore, to achieve a successful transition, first a comprehensive understanding of DevOps must be reached in in the organization. In addition, funding should be secured. This includes both financial as well as time-related resources. Because, if one chooses to transition to DevOps, you should do it correctly. “A half-baked, insufficiently-funded DevOps “effort” will likely leave you worse off than when you began” ^[15]. And DevOps is not the only possible way to reach the desired goal. It is cost-intensive, both in labor cost as well as in operational cost due to extended automation. If speed of continuous updates is not business critical, DevOps might not be the right choice.

Therefore, the following points should be considered:^[15]

• Organizational challenges/disadvantages

DevOps requires a number of organizational changes. Previously required skills and skillsets might not be sufficient anymore, while new skills must be added to teams. These leaves old personnel frustrated and opens the need to change personnel through hiring people from the external market, where resources are limited.

• Process-related challenges/disadvantages

Continuous feedback is at the core of the DevOps principles. However, if not managed correctly, this may lead to difficulties. At least early on in the process an organization must be fault tolerant. Suggested changes might be implemented without careful consideration, overwhelming the user or destabilizing the system.

• Technology-related challenges/disadvantages

One key technology behind the DevOps principles is the automation of work, for example automating the feedback loop back from users to developers or quality assurance testing. Also automated deployment processes. This must be done so that the pace of operation can keep up with development. However, if set up incorrectly, it might lead to unforeseen consequences.

• Speed and security-related challenges/disadvantages

Typically, speed and security are not two factors that work well together. However, in DevOps both must be ensured. Traditional security-related work, where a design of a system is analysed for weak points, has to be changed. In DevOps, systems are run in production quickly as a minimum viable product, often without a proper design. The security team must then depend upon the advice of the developers and the operations people, showing the need for a good communication between these functions.

[edit] Annotated bibliography

The following list provides resources for further research and study on DevOps principles:

• Kim, G.; Humble, J.; Debois, P.; Willis, J. (2016): The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations

- The DevOps Handbook describes how using the DevOps principles can integrate Product Management, Development, QA, IT Operations, and Information Security to help effective management of technology. It provides a theoretical explanation of the principles as well as a practical approach to applying the DevOps principles. This reference is particularly relevant for technical and organizational structures that can facilitate the principles.

• Davis, J.; Daniels, R. (2016): Effective DevOps : building a culture of collaboration, affinity, and tooling at scale

- Effective DevOps provides actionable strategies to achieve organizational changes. These changes must come from within and this book seeks to be a practical guide to the DevOps principles. It primarily focuses on human factors and job organization in general. It offers a collection of ideas and approaches for improving on key factors of the DevOps principles, like improving collaboration within teams, creating affinity among teams or promoting efficient tool usage.

• Forsgren, N.; Humble, J.; Kim, G. (2018): Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations

- This book provides a summary of the latest research on DevOps and Software Development. It explains, how technology can be applied to drive business value. Findings and the science behind ongoing research on the topic of software delivery performance is presented. The information is presented in a way that makes the application to other organizations possible. For example, explanation on how to measure performance of teams or on what capabilities to invest in to drive higher performance are presented.

• Kim, G.; Behr, K.; Spafford, G, (2013): The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business

- The Phoenix Project is a novel about an IT manager that uses the DevOps principles to lead a technology organization to new success. It introduces the Three Ways and shows a fictive success story of the DevOps principles.

• Kim, Gene (2019): The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data

- The Unicorn Project is the continuation of The Phoenix Project. It takes the perspective of the software development department and introduces the "Five Ideals" to further introduce the theory of DevOps.

• Goldratt, Eliyahu M. (1984): The Goal

- The Goal is the inspiration for The Phoenix Project. It explains the theory of constraints in physical manufacturing. This theory is used in the DevOps principles extensively. It is regarded as an important work in operations research to teach students about the importance of strategic capacity planning and constraint management.

• Puppet Lab: State of DevOps Report

- This yearly report (2011 - 2020) provides new insights into how organizations evolve the DevOps practices. The ninth publication of the report includes over 2,400 participants who work in IT, development, information security and related areas. The newest edition focuses on a platform approach to software delivery and applying DevOps principles to change management.

• Farley, D.; Humble, J. (2010): Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation

- Continuous Delivery describes principles and technical practices that enable rapid, incremental delivery of high quality, valuable new functionality to users and therefore laying the basis for the DevOps principles. Through automation of the build, deployment, and testing process, and improved collaboration between developers, testers and IT operations, teams can release changes quicker and safer.

• Allspaw, J.; Hammond, P (2009): 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr [conference]; available at: https://tech-talks.code-maven.com/ten-plus-deploys-per-day.html

- Video taken from the Velocity conference in 2009. Key concepts for DevOps are introduced (shared goals between development and operations, and continuous integration practices) through an exploration of the technology work performed at flickr (American image hosting and video hosting service).

• Ries, Eric (2011): The Lean Startup

- The Lean Startup describes how to create a successful entrepreneurial business and explains how the principles could be used in other organizations. The principles resemble the DevOps principles. The book argues that the customer needs should be at the center of each organization. Working towards the business objectives can help the creation of shared goals between development and operations, a key concept behind DevOps.

[edit] References

1. ↑ Jacob, Adam (2015): Chef Style DevOps Kung fu [presentation/keynote]; slides available at: http://chef.github.io/devops-kungfu/#/; video recording available at: https://www.youtube.com/watch?v=_DEToXsgrPc&ab_channel=ChefSoftware
2. ↑ Mezak, Steve (2018): The Origins of DevOps: What’s in a Name? [online]; available at: https://devops.com/the-origins-of-devops-whats-in-a-name/
3. ↑ ^{3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9} Kim, G.; Humble, J.; Debois, P.; Willis, J. (2016): The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations
4. ↑ Creative Commons Attribution 3.0 Unported, available at: https://commons.wikimedia.org/wiki/File:Devops.svg#globalusage
5. ↑ Project Management Institute, publisher (2017): A guide to the project management body of knowledge (PMBOK guide)
6. ↑ Allspaw, J.; Hammond, P (2009): 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr [conference]; available at: https://tech-talks.code-maven.com/ten-plus-deploys-per-day.html
7. ↑ Martin, K; Osterling, M (2013): Value Stream Mapping, How to Visualize Work and Align Leadership for organizational Transformation
8. ↑ ^{8.0 8.1 8.2 8.3} Kim, Gene (2012): The Three Ways: The Principles Underpinning DevOps, IT Revolution Press blog, available at: http://itrevolution.com/the-three-ways-principles-underpinning-devops/
9. ↑ Creative Commons Attribution-Share Alike 4.0 International, available at: https://commons.wikimedia.org/wiki/File:Devops-toolchain.svg
10. ↑ Kim, Gene (2020): The Five Ideals Of DevOps [online]; available at: https://itrevolution.com/five-ideals-of-devops/
11. ↑ Raymond, Eric S. (1999): The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary
12. ↑ Fernandez, Roberto (2021): Creating High Velocity Organizations
13. ↑ ^{13.0 13.1} Donnelly, Caroline (2015): Case study: What the enterprise can learn from Etsy's DevOps strategy [online], available at: https://www.computerweekly.com/news/4500247782/Case-study-What-the-enterprise-can-learn-from-Etsys-DevOps-strategy
14. ↑ ^{14.0 14.1 14.2} Rembetsy, Michael (2015): Interview with John Dix; How Etsy makes Devops work [online], available at: https://www.networkworld.com/article/2886672/how-etsy-makes-devops-work.html
15. ↑ ^{15.0 15.1} Tiempo Development (2019): Disadvantages Of Using DevOps [online], available at: https://www.tiempodev.com/blog/devops-disadvantages/