System Readiness Level Index

From apppm
(Difference between revisions)
Jump to: navigation, search
(→‎Perform Iterative Evaluations)
(→‎Limitations)
 
(256 intermediate revisions by one user not shown)
Line 1: Line 1:
 +
The System Readiness Assessment (SRA) is an innovative methodology that provides a system level metric to help reduce integration issues, one of the leading causes of system development failures. The SRA methodology provides decision-makers a snapshot of a system’s holistic state of maturity and quantifies the System Readiness Level Index of component-to-component integration during system development, using a scale from 1 to 9, with 9 signifying the highest level of readiness. Implementation of the SRA methodology aids decision makers in identifying both programmatic and technical risk areas. The process of defining the SRA is done via identifying Technology Readiness levels (TRL) and the Integration Readiness Levels (IRL) between integration points. The SRA methodology is new and is currently being validated through a number of program pilots <ref name="one">[''https://www.researchgate.net/publication/273894745_System_Readiness_Assessment_SRA_an_Illustrative_Example , M. Austin & D. York, 2015.''] ''System Readiness Assessment (SRA) An Illustrative Example. (PDF)'' </ref>.
  
  
−
==Background And Purpose==
+
==Background and Purpose==
 +
The SRA is a follow-up tool that builds upon the TRL and IRL indexes.
 +
In the 1980’s the National Aeronautics and Space Administration (NASA) instituted a 7-level TRL metric as a quantification methodology to assess the maturity of a particular technology and a scale to compare technologies <ref name="two">[''https://www.researchgate.net/publication/228652562_From_TRL_to_SRL_The_concept_of_systems_readiness_levels , B. Sauser et al., 2006.''] ''From TRL to SRL: The Concept of Systems Readiness Levels. (PDF)'' </ref>.
 +
The TRL was initially pioneered by J.C. Mankins at NASA Goddard Space Flight Center as a method to assess the readiness and risk of space technology <ref name="one"/>.
 +
Over time, NASA continued to commonly use readiness levels as part of an overall risk assessment process and by the 1990’s this metric had evolved into the nine levels that exist today <ref name="two"/>.
 +
In 1999, the Department of Defense (DoD) embraced a similar TRL concept in their programs <ref name="two"/>.
 +
 
 +
The TRL scale is a measure of maturity of an individual technology, with a view towards operational use in a system context. However, a more comprehensive set of concerns become relevant when this assessment is abstracted from an individual technology to a system context, which may involve interplay between multiple technologies  <ref name="two"/>. 
 +
 
 +
As systems and technologies become increasingly complex, it is critical to develop a more comprehensive understanding of the readiness of systems to aid in more informed system-level technical and managerial decisions throughout a project’s or program’s life cycle. In order to discover and develop potential system-level metrics, a greater emphasis must be put on the integration between and among individual components. During large-scale developments, it is also critical to measure the system readiness at multiple points along the life cycle to avoid pitfalls that can occur when readiness is only assessed once or twice. The SRA process provides transparency of the entire system’s development life cycle and interfaces to external entities <ref name="one"/>.
 +
 
 +
Unlike the Technology Readiness Assessment (TRA), used to find the TRL metric, the SRA process provides a broader overview of the whole system. This enables the companies to actively trace components throughout the entire system. The SRA process provides a number of additional benefits not provided by the current TRA process. The SRA <ref name="one"/>:
 +
 
 +
* Measures the readiness of all system components. Hereby, critically considering all elements equally.
 +
* Has a focus on the readiness both in terms of internal system integration between components and requires a readiness understanding of other external dependencies.
 +
* Is performed multiple times over the course of the system life cycle and not just at major milestone decisions.
 +
 
 +
 
 +
As the system development progresses, the TRLs and IRLs of the components will mature and the SRA can be calculated several times, providing the decision maker with risk assessment information. SRAs can be performed on any size program and at any time during the system development. The potential technology and integration risks will determine the frequency at which SRAs are performed. A quarterly assesment rate for larger programs are recommended, while for small programs SRAs may be performed every month. Once the system has been defined, the system mapping completed, and the initial SRA done, subsequent or follow-up SRAs can be performed in a reasonably short time <ref name="one"/>.
 +
 
 +
However, SRAs should be performed early in the life cycle to enable the Program Manager to understand and scope the system being built. The initial SRA is best performed at the beginning of the system development life cycle but can be performed at any time during the life cycle. By creating a comprehensive systems view, the SRA enables developers and systems engineers to perform design trade-offs and make sound design decisions. The SRA gives the Program Manager an overall system perspective so that resources can be effectively applied in the most applicable areas. The SRA process also supports portfolio management. When applied to systems across the enterprise, the SRA approach can provide a picture of both developmental and operational systems, with insight not only into the readiness of individual system components and functions but into enterprise capability readiness as well <ref name="one"/>.
  
 
==Preliminary Metrics==
 
==Preliminary Metrics==
 +
This section describes the two metrics, the Technology Readiness Level (TRL) and Integration Readiness Level (IRL).
  
 
===TRL===
 
===TRL===
 +
The TRL is a method for estimating the maturity of technologies during the acquisition phase of a program. The TRLs enables consistent classification of technical maturity across different types of technology. A technology's TRL is determined during a TRA that examines program concepts, technology requirements, and demonstrated technology capabilities. TRLs are based on a scale from 1 to 9, with 9 being the most mature technology <ref name="three">[''https://core.ac.uk/download/pdf/94310086.pdf , Mihaly, Heder, 2017.] ''From NASA to EU: the evolution of the TRL scale in Public Sector Innovation" (PDF)'' </ref>.
 +
 +
{| class="wikitable" style="text-align: center; width:35%"
 +
|+ Technology Readiness Levels
 +
|-
 +
|Level
 +
|TRL Definition
 +
|-
 +
|9
 +
|Actual system proven through succesful mission operations
 +
|-
 +
|8
 +
|Actual system completed and qualified through test and demonstration
 +
|-
 +
|7
 +
|System prototype demonstration in relevant environment
 +
|-
 +
|6
 +
|System/Subsystem model or prototype demonstration in relevant environment
 +
|-
 +
|5
 +
|Component and/or breadboard validation in relevant environment
 +
|-
 +
|4
 +
|Component and/or breadboard validation in relevant environment
 +
|-
 +
|3
 +
|Analytical and experimental critical function and/or characteristic proof-of-concept
 +
|-
 +
|2
 +
|Technology concept and/or application formulated
 +
|-
 +
|1
 +
|Basic principals observed and reported
 +
|-
 +
|}
 +
 +
The TRL is defined as a vector with ''n'' components where ''TRL<sub>i</sub>'' is the TRL of component ''i'':
 +
 +
<math>
 +
[TRL]_{\text{nx1}}=
 +
\begin{pmatrix}
 +
TRL_1 \\
 +
TRL_2 \\
 +
... \\
 +
TRL_n
 +
\end{pmatrix}
 +
</math>
  
 
===IRL===
 
===IRL===
 +
Integration readiness level (IRL) is a systematic measurement of the interfacing of compatible interactions for various technologies and the consistent comparison of the maturity between integration points (TRLs). The IRL can be used to describe the integration maturity of a developing technology with another technology. The IRLs check to what degree a technology is ready on a 7-level integration readiness scale. Moreover, the methodology can help discover a direction for improving integration with other technologies <ref name="one"/>.
  
 +
{| class="wikitable" style="text-align: center; width:45%;"
 +
|+ Integration Readiness Levels
 +
|-
 +
|Level
 +
|IRL Definition
 +
|-
 +
|7
 +
|The integration of technologies has been verified and validated with sufficient detail to be actionable
 +
|-
 +
|6
 +
|The integrating technologies can accept, translate, and structure information for its intended application
 +
|-
 +
|5
 +
|There is sufficient control between technologies neccesary to establish, manage, and terminate the integration
 +
|-
 +
|4
 +
|There is sufficient detail in the quality and assuranceof the integration between technologies
 +
|-
 +
|3
 +
|There is compatibility between technologies to orderly and efficiently integrate and interact
 +
|-
 +
|2
 +
|There is some level of specificity to characterise the interaction between technolgoies through their interface
 +
|-
 +
|1
 +
|An interface between technologies has been identified with sufficient detail to allow characterisation of the relationship
 +
|-
 +
|}
 +
 +
Like the TRA that has been used to assess the risk associated with developing technologies, IRLs are designed to assess the risk of integration <ref name="two"/>.
 +
 +
The IRLs assists the systems engineer in identifying development areas that require additional engineering. IRLs also provide a means to reduce the risk involved in maturing and integrating components into a system. Therefore, IRLs can give a combined systems measure for both new system development and technology insertion <ref name="one"/>. However, The TRL metric does not accurately capture the risk involved in the adopting of a technology <ref name="four">[''https://www.researchgate.net/publication/228752216_An_Approach_to_Technology_Risk_Management , Valerdi, R., and R.J. Kohl., 2004.''] ''An Approach to Technology Risk Management. (PDF)'' </ref>. Furthermore, technologies can have an architectural inequality related to their integration <ref name="five">[''https://apps.dtic.mil/sti/pdfs/ADA443149.pdf , Smith, J.D., 2005.''] ''An Alternative to Technology Readiness Levels for Non-Developmental Item (Ndi) Software. (PDF)'' </ref>. As systems’ complexity increases there must be a reliable method for integration that allows TRLs to collectively combine for develop these complex systems <ref name="two"/>
 +
 +
The IRL matrix represents the integration of different components with each other from a systems perspective. The integration between components ''i'' and ''j'' is represented by ''IRL<sub>ij</sub>'' in the IRL matrix. The theoretical integration of a component ''i'' to itself is denoted by ''IRL<sub>ii</sub>'' and is assumed to be a maximum <ref name="one"/>.
 +
 +
<math>
 +
[IRL]_{\text{nxn}}=
 +
\begin{pmatrix}
 +
IRL_{11} & IRL_{12} & ... & IRL_{1n}  \\
 +
IRL_{21} & IRL_{22} & ... & IRL_{2n}  \\
 +
... & ... & ... & ...  \\
 +
IRL_{n1} & IRL_{n2} & ... & IRL_{nn}  \\
 +
\end{pmatrix}
 +
</math>
  
 
==System Readiness Metrics==
 
==System Readiness Metrics==
 +
There are three system readiness metrics that are computed in the SRA process. These are the Component SRL, Composite SRL, and SRL. System readiness incorporates TRL and IRL metrics. The TRL metric provides component knowledge, while the IRL is a metric that provides a description of architectural knowledge.
  
 
===Component SRL===
 
===Component SRL===
 +
A Component SRL is the System Readiness Level of an individual component of the system and its integration links <ref name="one"/>. Component SRLs are used to identify system component variations, both in terms of being behind and too far ahead in regard to their readiness and thus require Program Management and/or engineering attention <ref name="one"/>.
 +
 +
The ''SRL<sub>i</sub>'' for each component ''i'' is divided by ''m<sub>i</sub>'' to obtain its Component SRL value. The ''m<sub>i</sub>'' term is the number of integrations of component ''i'' with every other component as defined by the system architecture <ref name="one"/>.
 +
 +
<math>
 +
Component SRL_i=\frac{SRL_i}{m_i}
 +
</math>
  
 
===Composite SRL===
 
===Composite SRL===
 +
The Composite SRL measures the SRL of the whole system or all the components of the system integrated together. The SRA approach calculates the Composite SRL by averaging the Component SRLs and rendering the value in a decimal format.
 +
 +
The Composite SRL for the system is the average of the Component SRL values where ''n'' is the number of components <ref name="one"/>:
 +
 +
<math>
 +
Composite SRL=\frac{(\frac{SRL_1}{m_1})+(\frac{SRL_2}{m_2})+...+(\frac{SRL_i}{m_i})}{n}
 +
</math>
 +
 +
Composite SRLs are defined on a scale from 0 to 1 with a value carried out to three decimal places.
  
 
===SRL===
 
===SRL===
 +
The SRL is obtained by converting the Composite SRL to an 1 to 9 integer scale, with 9 being the most ready.
  
 +
{| class="wikitable" style="text-align: center; width:45%"
 +
|+ The System Readiness Levels
 +
|-
 +
|Level
 +
|SRL Definition
 +
|-
 +
|9
 +
|System has achieved initial operational capability and can satisfy mission objectives
 +
|-
 +
|8
 +
|System interoperability should have been demonstrated in an operational environment
 +
|-
 +
|7
 +
|System threshold capability should have been demonstrated at operational performance level
 +
|-
 +
|6
 +
|System component integrability shouldhave been validated
 +
|-
 +
|5
 +
|System high-risk component technology development should have been defined and the vaseline has been allocated
 +
|-
 +
|4
 +
|System performance specifications and constraints should have been defined and the baseline has been allocated
 +
|-
 +
|3
 +
|System high-risk immature technologies have been indetified and prototyped
 +
|-
 +
|2
 +
|System materiel solution should have been considered
 +
|-
 +
|1
 +
|System alternative materiel solutions should have been considered
 +
|-
 +
|}
 +
 +
To calculate a value of the SRL from the TRL and IRL values, an SRL matrix is generated by obtaining the product of the IRL and TRL matrices <ref name="one"/>.
 +
 +
 +
<math>
 +
[SRL]_{\text{nx1}}=[IRL]_{\text{nxn}}*[TRL]_{\text{nx1}}
 +
</math>
 +
 +
 +
The SRL matrix consists of one element for each of the represented components and quantifies the readiness level, of a specific component, with respect to every other component in the system while also accounting for the development state of each.
 +
 +
Mathematically, ''TRL<sub>i</sub>'' represent the individual TRLs and the ''IRL<sub>ij</sub>'' are the individual IRLs between the components. ''SRL<sub>i</sub>'' represents the readiness level of Component ''i'', reflecting the readiness of all of its connections <ref name="one"/>.
 +
 +
 +
<math>
 +
\begin{pmatrix}
 +
SRL_1 \\
 +
SRL_2 \\
 +
... \\
 +
SRL_n
 +
\end{pmatrix}
 +
=
 +
\begin{pmatrix}
 +
IRL_{11} & IRL_{12} & ... & IRL_{1j}  \\
 +
IRL_{21} & IRL_{22} & ... & IRL_{2j}  \\
 +
... & ... & ... & ...  \\
 +
IRL_{i1} & IRL_{i2} & ... & IRL_{ij}  \\
 +
\end{pmatrix}
 +
*
 +
\begin{pmatrix}
 +
TRL_1 \\
 +
TRL_2 \\
 +
... \\
 +
TRL_i
 +
\end{pmatrix}
 +
</math>
  
 
==The 3-Step SRA Process==
 
==The 3-Step SRA Process==
 +
This section describes the System Readiness Assessment process in detail. The approach for conducting an SRA is broken down into the following three core steps <ref name="one"/>:
  
−
===Understand And Bound The System===
+
# Understand and Bound the System
 +
# Decompose and Map System
 +
# Perform Iterative Evaluations
 +
[[File:The_3-Step_SRA_Process.png‎|thumb|upright=4|right|The 3-Step SRA Process.]]
  
−
===Decompose And Map System===
+
 
 +
===Understand and Bound The System===
 +
The team performing the SRA gathers any program information that can help support the understanding of the system. Information such as capabilities statements, test plans, requirements documents, and/or any other documentation that can support the understanding of the system. Product vendor documentation and relevant published reports may also provide additional information to cover knowledge gaps for a complete understanding of the system. For this step, there is close interaction of the team with the program Lead Systems Engineer (LSE) and Subject Matter Experts <ref name="one"/>.
 +
 
 +
===Decompose and Map System===
 +
 
 +
This step consists of mapping the system that provides a relational understanding between the different layers of architecture. At the highest level, this mapping originates with operational requirements and activities. Functions which trace to these operational activities are then generated. System components which perform these functions are identified. The individual components are comprised of technologies. The SRA method is scalable to large systems in case of an increase of new data gathering, assessments and calculations. The system mapping and component interface diagrams together serve as the foundation on which a SRA analysis is performed. Developing system mappings identifies the linkages and traceability between system components and allows systems to be assessed consistently. All hardware and software components that represent the system are identified. Technologies are mapped to specific components when evaluating TRLs. Mappings are based on what is known as-is and are updated as the design, architecture, or other information changes. The same mapping process can be implemented when doing design or system trade-offs, providing significant benefits and insight into analysis of alternatives <ref name="one"/>.
 +
 
 +
 
 +
[[File:System_Mapping.png‎|thumb|upright=4|right|Example of 'Decompose and Map System'.]]
  
 
===Perform Iterative Evaluations===
 
===Perform Iterative Evaluations===
−
{| class="wikitable" style="text-align: center;
+
The third step in the process evaluates the system to determine its readiness. The evaluation is conducted iteratively through the development cycle and can be continued throughout the system life cycle. All component and integration links must be evaluated for technology and integration readiness. TRLs and IRLs are determined using detailed decision criteria and assigned accordingly. Components may be comprised of more than one technology each with its own TRL. The TRL of a component is determined by assigning to it the minimum TRL of the component’s system technologies. Hence, the TRL is assessed at the technology level and the SRL is calculated at the component level. This approach for determining a component’s TRL is a recommended approach, not a required one. Assessment may be performed at a different level as long as consistency is maintained <ref name="one"/>. The following procedure for calculating the SRL Index is given:
−
!colspan="6"|SRL Translation Model
+
 
 +
#'''SRL Matrix'''. In order to calculate a value of the SRL from the TRL and IRL values, an SRL matrix is generated by obtaining the product of the IRL and TRL matrices.
 +
#'''Component SRL'''. The corresponding ''SRL<sub>i</sub>'' for each component ''i'' is then divided by ''m<sub>i</sub>'' to obtain its normalised value.
 +
#'''Composite SRL'''. The Composite SRL for the system is the average of the Component SRL values, where ''n'' is the number of components.
 +
#'''SRL Translation Model'''. Composite SRL values are translated to whole numbers consistent with TRL and IRL scaling for ease of interpretation. To translate the 0 to 1 scale to a 1 to 9 scale, the SRL Translation Model is used to map the decimal values to whole number values. Because the SRA is dependent on the system architecture configuration, a SRL Translation Model is generated for each architecture configuration when performing the SRA.
 +
 
 +
{| class="wikitable" style="text-align:center;width:35%;"
 +
|+ SRL Translation Model
 
|-
 
|-
−
style="text-align: center;
 
 
|TRL
 
|TRL
−
|IRl
+
|IRL
−
|<math>Composite SRL_i</math>
+
|Composite SRL_i
 
|Mean
 
|Mean
−
|<math>Composite SRL_i Range</math>
+
|Composite SRL_i Range
 
|SRL
 
|SRL
 
|-
 
|-
Line 43: Line 265:
 
|0.914 - 1.000
 
|0.914 - 1.000
 
|9
 
|9
 +
|-
 +
|8
 +
|8
 +
|0.828
 +
|0.914
 +
|0.750 - 0.913
 +
|8
 +
|-
 +
|7
 +
|7
 +
|0.672
 +
|0.750
 +
|0.601 - 0.749
 +
|7
 +
|-
 +
|6
 +
|6
 +
|0.530
 +
|0.601
 +
|0.467 - 0.600
 +
|6
 +
|-
 +
|5
 +
|5
 +
|0.404
 +
|0.467
 +
|0.349 - 0.466
 +
|5
 +
|-
 +
|4
 +
|4
 +
|0.293
 +
|0.349
 +
|0.245 - 0.348
 +
|4
 +
|-
 +
|3
 +
|3
 +
|0.197
 +
|0.245
 +
|0.157 - 0.244
 +
|3
 +
|-
 +
|2
 +
|2
 +
|0.116
 +
|0.157
 +
|0.084 - 0.156
 +
|2
 +
|-
 +
|1
 +
|1
 +
|0.051
 +
|0.084
 +
|0.000 - 0.083
 +
|1
 
|}
 
|}
  
 
==Limitations==
 
==Limitations==
 +
The limitations of the SRA are tied to the mathematical biases related to the interpretations of the procedure. Any assumptions based on tacit or biased knowledge may affect the outcome of the TRL, which in turn can change the IRL and the final SRL.
 +
 +
By using a single integer to rate the status of a system’s development and help ensure its success, a company might be one-faceted. System readiness is a multidimensional concept that goes beyond simply assessing the achieved technology and integration status. To fully characterise system readiness requires looking at the multifaceted aspects of system/technology development, including the risk and effort required to reach operational readiness. System readiness is too complex to be characterised by a single number. The way to improve the development of technically advanced system is to develop excellent project teams and implement a comprehensive systems development process that includes a technology development plan based on a sound quantitative assessment of technical, cost, and schedule risks <ref name="six">[''https://www.researchgate.net/publication/260652369_Analysis_and_Critique_of_the_System_Readiness_Level, E. Kujawski, 2013.''] ''Analysis and Critique of the System Readiness Level'' </ref>.
 +
 +
The SRL should not be used uncritically, as a project manager tool, due to the following reasons <ref name="six"/>:
 +
* The SRL is erroneous because it is constructed using nonvalid arithmetic operations on ordinal data.
 +
* The usefulness of the SRL is limited by the data content of the TRL and IRL.
 +
* Being an aggregate index of the TRL and IRL values of the constituent technologies and integration, the SRL filters out important information.
 +
* Quantitative risk management and robustness-based design are proven valid methods available to support the successful development of technically advanced systems.
 +
* Adding Integration Between Technologies Can Increase the SRL.
 +
* The SRL Exhibits Counterfactual Rank Reversal.
 +
 +
 +
Furthermore, the SRA approach calculates the Composite SRL by averaging the Component SRLs and rendering the value in a decimal format. As with any calculation involving an average, the user needs to be aware of the potential risk of masking a Component SRL that may be significantly lagging or leading the average. Furthermore, it could potentially be difficult to understand the difference between system readiness values that are very similar when converting decimals to integer levels via the SRL Translation Model <ref name="one"/>. Furthermore, the conversion from decimal to an integer scale facilitates reporting and interpretation of the results <ref name="one"/>.
 +
 +
Finally, the SRA methodology is currently being validated through a number of program pilots, and the method has therefore not been proven tried-and-true. Further validation of the method will be achieved through broader application across multiple enterprises. Future research areas include mathematically sound weighting techniques and leveraging the principles of the SRA framework to model system availability and for other Risk Management techniques <ref name="one"/>.
 +
 +
==Annotated Bibliography==
 +
The following topics would be of interest to further study in relation to the SRA:
 +
 +
* '''E. Kujawski, 2013. ''Analysis and Critique of the System Readiness Level.'' '''<ref name="six"/> This paper provides a critical angle to the SRA and demonstrates that the SRL is a potentially misleading metric that may distort the assessment of system readiness with harmful consequences. The paper continues to state that the SRL is computed using nonvalid arithmetic operations on ordinal data, the TRL and IRL. The paper investigates the importance of the TRL and IRL and how they only assess the achieved readiness levels. Hence, they do not provide information on the risk and effort required for achieving higher readiness levels. Being a mathematical model, it is limited by this incomplete information. Being an aggregate metric of the constituents TRLs and IRLs, it filters out the microscopic information needed to manage specific risk areas.
  
 +
* '''Project Management Institute, 2019. ''Standard for Risk Management in Portfolios, Programs, and Projects: Project risk resonse strategies.'' '''<ref name="seven">[''https://app.knovel.com/hotlink/toc/id:kpSRMPPP01/standard-risk-management/standard-risk-management Project Management Institute, 2019.''] ''Standard for Risk Management in Portfolios, Programs, and Projects, P. 59.'' </ref> When assessing the SRL, considerations regarding the risk response and the making contigency plans should be taken into account. By having a risk response strategy, companies can reduce the probability of risk due to bias. By looking into the Project Management Institute (PMI), regarding the project risk response strategies, companies can update a project’s baselines or remove activities from these same baselines. Whenever the project is part of a program or is managed as part of a portfolio, escalation of risks to a higher governance level is always one of the responses. Escalation increases the effectiveness or efficiency of dealing with specific risks that impact the program or portfolio or with risks that require funding in excess of the contingency reserves.
  
−
==Annotated References==
+
*'''Project Management Institute, 2019. ''Standard for Risk Management in Portfolios, Programs, and Projects: Qualitative and Quantitative project risk analyses.'' '''<ref name="seven"/> In order to evaluate the risks and whether a contigency plan is actually needed, qualitative and quantitative project risk analyses can be studied to discover project boundaries. By looking into the Project Management Institute (PMI), regarding the Qualitative and Quantitative project risk analyses, an evaluation of risks at the project level can be performed by taking into account the degree of impact on the project objectives and probability of occurrence. The purpose of these analyses is to evaluate whether or not the impact can be contained within the limits of the project budget and the boundary of accountability of the project manager.
  
 +
* '''N. Marlyana et. Al, 2018. ''A Quantitative Analysis of System Readiness Level Plus (SRL+): Development of Readiness Level Measurement.'' '''<ref name="nine">[''https://www.matec-conferences.org/articles/matecconf/pdf/2018/18/matecconf_ijcaet-isampe2018_02067.pdf , N. Marlyana et al., 2018.''] ''A Quantitative Analysis of System Readiness Level Plus (SRL+): Development of Readiness Level Measurement. (PDF)'' </ref> A quantitative combination of levels of readiness can be made to open the potential for expanding the other measures of readiness levels, such as the Manufacturing Readiness Level (MRL). A measurement tool to measure the development of readiness level by involving MRL is called System Readiness Level Plus (SRL+). This research paper focuses on quantitative analysis of SRL+ model. It consists of the mathematical properties method and a readiness reversal method to be conducted to design the SRL+ model.
  
 
== References==
 
== References==
 
<references />
 
<references />

Latest revision as of 23:36, 28 February 2021

The System Readiness Assessment (SRA) is an innovative methodology that provides a system level metric to help reduce integration issues, one of the leading causes of system development failures. The SRA methodology provides decision-makers a snapshot of a system’s holistic state of maturity and quantifies the System Readiness Level Index of component-to-component integration during system development, using a scale from 1 to 9, with 9 signifying the highest level of readiness. Implementation of the SRA methodology aids decision makers in identifying both programmatic and technical risk areas. The process of defining the SRA is done via identifying Technology Readiness levels (TRL) and the Integration Readiness Levels (IRL) between integration points. The SRA methodology is new and is currently being validated through a number of program pilots [1].


Contents

[edit] Background and Purpose

The SRA is a follow-up tool that builds upon the TRL and IRL indexes. In the 1980’s the National Aeronautics and Space Administration (NASA) instituted a 7-level TRL metric as a quantification methodology to assess the maturity of a particular technology and a scale to compare technologies [2]. The TRL was initially pioneered by J.C. Mankins at NASA Goddard Space Flight Center as a method to assess the readiness and risk of space technology [1]. Over time, NASA continued to commonly use readiness levels as part of an overall risk assessment process and by the 1990’s this metric had evolved into the nine levels that exist today [2]. In 1999, the Department of Defense (DoD) embraced a similar TRL concept in their programs [2].

The TRL scale is a measure of maturity of an individual technology, with a view towards operational use in a system context. However, a more comprehensive set of concerns become relevant when this assessment is abstracted from an individual technology to a system context, which may involve interplay between multiple technologies [2].

As systems and technologies become increasingly complex, it is critical to develop a more comprehensive understanding of the readiness of systems to aid in more informed system-level technical and managerial decisions throughout a project’s or program’s life cycle. In order to discover and develop potential system-level metrics, a greater emphasis must be put on the integration between and among individual components. During large-scale developments, it is also critical to measure the system readiness at multiple points along the life cycle to avoid pitfalls that can occur when readiness is only assessed once or twice. The SRA process provides transparency of the entire system’s development life cycle and interfaces to external entities [1].

Unlike the Technology Readiness Assessment (TRA), used to find the TRL metric, the SRA process provides a broader overview of the whole system. This enables the companies to actively trace components throughout the entire system. The SRA process provides a number of additional benefits not provided by the current TRA process. The SRA [1]:

  • Measures the readiness of all system components. Hereby, critically considering all elements equally.
  • Has a focus on the readiness both in terms of internal system integration between components and requires a readiness understanding of other external dependencies.
  • Is performed multiple times over the course of the system life cycle and not just at major milestone decisions.


As the system development progresses, the TRLs and IRLs of the components will mature and the SRA can be calculated several times, providing the decision maker with risk assessment information. SRAs can be performed on any size program and at any time during the system development. The potential technology and integration risks will determine the frequency at which SRAs are performed. A quarterly assesment rate for larger programs are recommended, while for small programs SRAs may be performed every month. Once the system has been defined, the system mapping completed, and the initial SRA done, subsequent or follow-up SRAs can be performed in a reasonably short time [1].

However, SRAs should be performed early in the life cycle to enable the Program Manager to understand and scope the system being built. The initial SRA is best performed at the beginning of the system development life cycle but can be performed at any time during the life cycle. By creating a comprehensive systems view, the SRA enables developers and systems engineers to perform design trade-offs and make sound design decisions. The SRA gives the Program Manager an overall system perspective so that resources can be effectively applied in the most applicable areas. The SRA process also supports portfolio management. When applied to systems across the enterprise, the SRA approach can provide a picture of both developmental and operational systems, with insight not only into the readiness of individual system components and functions but into enterprise capability readiness as well [1].

[edit] Preliminary Metrics

This section describes the two metrics, the Technology Readiness Level (TRL) and Integration Readiness Level (IRL).

[edit] TRL

The TRL is a method for estimating the maturity of technologies during the acquisition phase of a program. The TRLs enables consistent classification of technical maturity across different types of technology. A technology's TRL is determined during a TRA that examines program concepts, technology requirements, and demonstrated technology capabilities. TRLs are based on a scale from 1 to 9, with 9 being the most mature technology [3].

Technology Readiness Levels
Level TRL Definition
9 Actual system proven through succesful mission operations
8 Actual system completed and qualified through test and demonstration
7 System prototype demonstration in relevant environment
6 System/Subsystem model or prototype demonstration in relevant environment
5 Component and/or breadboard validation in relevant environment
4 Component and/or breadboard validation in relevant environment
3 Analytical and experimental critical function and/or characteristic proof-of-concept
2 Technology concept and/or application formulated
1 Basic principals observed and reported

The TRL is defined as a vector with n components where TRLi is the TRL of component i:

[TRL]_{\text{nx1}}= \begin{pmatrix} TRL_1 \\ TRL_2 \\ ... \\ TRL_n \end{pmatrix}

[edit] IRL

Integration readiness level (IRL) is a systematic measurement of the interfacing of compatible interactions for various technologies and the consistent comparison of the maturity between integration points (TRLs). The IRL can be used to describe the integration maturity of a developing technology with another technology. The IRLs check to what degree a technology is ready on a 7-level integration readiness scale. Moreover, the methodology can help discover a direction for improving integration with other technologies [1].

Integration Readiness Levels
Level IRL Definition
7 The integration of technologies has been verified and validated with sufficient detail to be actionable
6 The integrating technologies can accept, translate, and structure information for its intended application
5 There is sufficient control between technologies neccesary to establish, manage, and terminate the integration
4 There is sufficient detail in the quality and assuranceof the integration between technologies
3 There is compatibility between technologies to orderly and efficiently integrate and interact
2 There is some level of specificity to characterise the interaction between technolgoies through their interface
1 An interface between technologies has been identified with sufficient detail to allow characterisation of the relationship

Like the TRA that has been used to assess the risk associated with developing technologies, IRLs are designed to assess the risk of integration [2].

The IRLs assists the systems engineer in identifying development areas that require additional engineering. IRLs also provide a means to reduce the risk involved in maturing and integrating components into a system. Therefore, IRLs can give a combined systems measure for both new system development and technology insertion [1]. However, The TRL metric does not accurately capture the risk involved in the adopting of a technology [4]. Furthermore, technologies can have an architectural inequality related to their integration [5]. As systems’ complexity increases there must be a reliable method for integration that allows TRLs to collectively combine for develop these complex systems [2]

The IRL matrix represents the integration of different components with each other from a systems perspective. The integration between components i and j is represented by IRLij in the IRL matrix. The theoretical integration of a component i to itself is denoted by IRLii and is assumed to be a maximum [1].

[IRL]_{\text{nxn}}= \begin{pmatrix} IRL_{11} & IRL_{12} & ... & IRL_{1n} \\ IRL_{21} & IRL_{22} & ... & IRL_{2n} \\ ... & ... & ... & ... \\ IRL_{n1} & IRL_{n2} & ... & IRL_{nn} \\ \end{pmatrix}

[edit] System Readiness Metrics

There are three system readiness metrics that are computed in the SRA process. These are the Component SRL, Composite SRL, and SRL. System readiness incorporates TRL and IRL metrics. The TRL metric provides component knowledge, while the IRL is a metric that provides a description of architectural knowledge.

[edit] Component SRL

A Component SRL is the System Readiness Level of an individual component of the system and its integration links [1]. Component SRLs are used to identify system component variations, both in terms of being behind and too far ahead in regard to their readiness and thus require Program Management and/or engineering attention [1].

The SRLi for each component i is divided by mi to obtain its Component SRL value. The mi term is the number of integrations of component i with every other component as defined by the system architecture [1].

Component SRL_i=\frac{SRL_i}{m_i}

[edit] Composite SRL

The Composite SRL measures the SRL of the whole system or all the components of the system integrated together. The SRA approach calculates the Composite SRL by averaging the Component SRLs and rendering the value in a decimal format.

The Composite SRL for the system is the average of the Component SRL values where n is the number of components [1]:

Composite SRL=\frac{(\frac{SRL_1}{m_1})+(\frac{SRL_2}{m_2})+...+(\frac{SRL_i}{m_i})}{n}

Composite SRLs are defined on a scale from 0 to 1 with a value carried out to three decimal places.

[edit] SRL

The SRL is obtained by converting the Composite SRL to an 1 to 9 integer scale, with 9 being the most ready.

The System Readiness Levels
Level SRL Definition
9 System has achieved initial operational capability and can satisfy mission objectives
8 System interoperability should have been demonstrated in an operational environment
7 System threshold capability should have been demonstrated at operational performance level
6 System component integrability shouldhave been validated
5 System high-risk component technology development should have been defined and the vaseline has been allocated
4 System performance specifications and constraints should have been defined and the baseline has been allocated
3 System high-risk immature technologies have been indetified and prototyped
2 System materiel solution should have been considered
1 System alternative materiel solutions should have been considered

To calculate a value of the SRL from the TRL and IRL values, an SRL matrix is generated by obtaining the product of the IRL and TRL matrices [1].


[SRL]_{\text{nx1}}=[IRL]_{\text{nxn}}*[TRL]_{\text{nx1}}


The SRL matrix consists of one element for each of the represented components and quantifies the readiness level, of a specific component, with respect to every other component in the system while also accounting for the development state of each.

Mathematically, TRLi represent the individual TRLs and the IRLij are the individual IRLs between the components. SRLi represents the readiness level of Component i, reflecting the readiness of all of its connections [1].


\begin{pmatrix} SRL_1 \\ SRL_2 \\ ... \\ SRL_n \end{pmatrix} = \begin{pmatrix} IRL_{11} & IRL_{12} & ... & IRL_{1j} \\ IRL_{21} & IRL_{22} & ... & IRL_{2j} \\ ... & ... & ... & ... \\ IRL_{i1} & IRL_{i2} & ... & IRL_{ij} \\ \end{pmatrix} * \begin{pmatrix} TRL_1 \\ TRL_2 \\ ... \\ TRL_i \end{pmatrix}

[edit] The 3-Step SRA Process

This section describes the System Readiness Assessment process in detail. The approach for conducting an SRA is broken down into the following three core steps [1]:

  1. Understand and Bound the System
  2. Decompose and Map System
  3. Perform Iterative Evaluations
The 3-Step SRA Process.


[edit] Understand and Bound The System

The team performing the SRA gathers any program information that can help support the understanding of the system. Information such as capabilities statements, test plans, requirements documents, and/or any other documentation that can support the understanding of the system. Product vendor documentation and relevant published reports may also provide additional information to cover knowledge gaps for a complete understanding of the system. For this step, there is close interaction of the team with the program Lead Systems Engineer (LSE) and Subject Matter Experts [1].

[edit] Decompose and Map System

This step consists of mapping the system that provides a relational understanding between the different layers of architecture. At the highest level, this mapping originates with operational requirements and activities. Functions which trace to these operational activities are then generated. System components which perform these functions are identified. The individual components are comprised of technologies. The SRA method is scalable to large systems in case of an increase of new data gathering, assessments and calculations. The system mapping and component interface diagrams together serve as the foundation on which a SRA analysis is performed. Developing system mappings identifies the linkages and traceability between system components and allows systems to be assessed consistently. All hardware and software components that represent the system are identified. Technologies are mapped to specific components when evaluating TRLs. Mappings are based on what is known as-is and are updated as the design, architecture, or other information changes. The same mapping process can be implemented when doing design or system trade-offs, providing significant benefits and insight into analysis of alternatives [1].


Example of 'Decompose and Map System'.

[edit] Perform Iterative Evaluations

The third step in the process evaluates the system to determine its readiness. The evaluation is conducted iteratively through the development cycle and can be continued throughout the system life cycle. All component and integration links must be evaluated for technology and integration readiness. TRLs and IRLs are determined using detailed decision criteria and assigned accordingly. Components may be comprised of more than one technology each with its own TRL. The TRL of a component is determined by assigning to it the minimum TRL of the component’s system technologies. Hence, the TRL is assessed at the technology level and the SRL is calculated at the component level. This approach for determining a component’s TRL is a recommended approach, not a required one. Assessment may be performed at a different level as long as consistency is maintained [1]. The following procedure for calculating the SRL Index is given:

  1. SRL Matrix. In order to calculate a value of the SRL from the TRL and IRL values, an SRL matrix is generated by obtaining the product of the IRL and TRL matrices.
  2. Component SRL. The corresponding SRLi for each component i is then divided by mi to obtain its normalised value.
  3. Composite SRL. The Composite SRL for the system is the average of the Component SRL values, where n is the number of components.
  4. SRL Translation Model. Composite SRL values are translated to whole numbers consistent with TRL and IRL scaling for ease of interpretation. To translate the 0 to 1 scale to a 1 to 9 scale, the SRL Translation Model is used to map the decimal values to whole number values. Because the SRA is dependent on the system architecture configuration, a SRL Translation Model is generated for each architecture configuration when performing the SRA.
SRL Translation Model
TRL IRL Composite SRL_i Mean Composite SRL_i Range SRL
9 9 1.000 0.914 - 1.000 9
8 8 0.828 0.914 0.750 - 0.913 8
7 7 0.672 0.750 0.601 - 0.749 7
6 6 0.530 0.601 0.467 - 0.600 6
5 5 0.404 0.467 0.349 - 0.466 5
4 4 0.293 0.349 0.245 - 0.348 4
3 3 0.197 0.245 0.157 - 0.244 3
2 2 0.116 0.157 0.084 - 0.156 2
1 1 0.051 0.084 0.000 - 0.083 1

[edit] Limitations

The limitations of the SRA are tied to the mathematical biases related to the interpretations of the procedure. Any assumptions based on tacit or biased knowledge may affect the outcome of the TRL, which in turn can change the IRL and the final SRL.

By using a single integer to rate the status of a system’s development and help ensure its success, a company might be one-faceted. System readiness is a multidimensional concept that goes beyond simply assessing the achieved technology and integration status. To fully characterise system readiness requires looking at the multifaceted aspects of system/technology development, including the risk and effort required to reach operational readiness. System readiness is too complex to be characterised by a single number. The way to improve the development of technically advanced system is to develop excellent project teams and implement a comprehensive systems development process that includes a technology development plan based on a sound quantitative assessment of technical, cost, and schedule risks [6].

The SRL should not be used uncritically, as a project manager tool, due to the following reasons [6]:

  • The SRL is erroneous because it is constructed using nonvalid arithmetic operations on ordinal data.
  • The usefulness of the SRL is limited by the data content of the TRL and IRL.
  • Being an aggregate index of the TRL and IRL values of the constituent technologies and integration, the SRL filters out important information.
  • Quantitative risk management and robustness-based design are proven valid methods available to support the successful development of technically advanced systems.
  • Adding Integration Between Technologies Can Increase the SRL.
  • The SRL Exhibits Counterfactual Rank Reversal.


Furthermore, the SRA approach calculates the Composite SRL by averaging the Component SRLs and rendering the value in a decimal format. As with any calculation involving an average, the user needs to be aware of the potential risk of masking a Component SRL that may be significantly lagging or leading the average. Furthermore, it could potentially be difficult to understand the difference between system readiness values that are very similar when converting decimals to integer levels via the SRL Translation Model [1]. Furthermore, the conversion from decimal to an integer scale facilitates reporting and interpretation of the results [1].

Finally, the SRA methodology is currently being validated through a number of program pilots, and the method has therefore not been proven tried-and-true. Further validation of the method will be achieved through broader application across multiple enterprises. Future research areas include mathematically sound weighting techniques and leveraging the principles of the SRA framework to model system availability and for other Risk Management techniques [1].

[edit] Annotated Bibliography

The following topics would be of interest to further study in relation to the SRA:

  • E. Kujawski, 2013. Analysis and Critique of the System Readiness Level. [6] This paper provides a critical angle to the SRA and demonstrates that the SRL is a potentially misleading metric that may distort the assessment of system readiness with harmful consequences. The paper continues to state that the SRL is computed using nonvalid arithmetic operations on ordinal data, the TRL and IRL. The paper investigates the importance of the TRL and IRL and how they only assess the achieved readiness levels. Hence, they do not provide information on the risk and effort required for achieving higher readiness levels. Being a mathematical model, it is limited by this incomplete information. Being an aggregate metric of the constituents TRLs and IRLs, it filters out the microscopic information needed to manage specific risk areas.
  • Project Management Institute, 2019. Standard for Risk Management in Portfolios, Programs, and Projects: Project risk resonse strategies. [7] When assessing the SRL, considerations regarding the risk response and the making contigency plans should be taken into account. By having a risk response strategy, companies can reduce the probability of risk due to bias. By looking into the Project Management Institute (PMI), regarding the project risk response strategies, companies can update a project’s baselines or remove activities from these same baselines. Whenever the project is part of a program or is managed as part of a portfolio, escalation of risks to a higher governance level is always one of the responses. Escalation increases the effectiveness or efficiency of dealing with specific risks that impact the program or portfolio or with risks that require funding in excess of the contingency reserves.
  • Project Management Institute, 2019. Standard for Risk Management in Portfolios, Programs, and Projects: Qualitative and Quantitative project risk analyses. [7] In order to evaluate the risks and whether a contigency plan is actually needed, qualitative and quantitative project risk analyses can be studied to discover project boundaries. By looking into the Project Management Institute (PMI), regarding the Qualitative and Quantitative project risk analyses, an evaluation of risks at the project level can be performed by taking into account the degree of impact on the project objectives and probability of occurrence. The purpose of these analyses is to evaluate whether or not the impact can be contained within the limits of the project budget and the boundary of accountability of the project manager.
  • N. Marlyana et. Al, 2018. A Quantitative Analysis of System Readiness Level Plus (SRL+): Development of Readiness Level Measurement. [8] A quantitative combination of levels of readiness can be made to open the potential for expanding the other measures of readiness levels, such as the Manufacturing Readiness Level (MRL). A measurement tool to measure the development of readiness level by involving MRL is called System Readiness Level Plus (SRL+). This research paper focuses on quantitative analysis of SRL+ model. It consists of the mathematical properties method and a readiness reversal method to be conducted to design the SRL+ model.

[edit] References

  1. ↑ 1.00 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08 1.09 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18 1.19 1.20 1.21 [https://www.researchgate.net/publication/273894745_System_Readiness_Assessment_SRA_an_Illustrative_Example , M. Austin & D. York, 2015.] System Readiness Assessment (SRA) An Illustrative Example. (PDF)
  2. ↑ 2.0 2.1 2.2 2.3 2.4 2.5 [https://www.researchgate.net/publication/228652562_From_TRL_to_SRL_The_concept_of_systems_readiness_levels , B. Sauser et al., 2006.] From TRL to SRL: The Concept of Systems Readiness Levels. (PDF)
  3. ↑ [https://core.ac.uk/download/pdf/94310086.pdf , Mihaly, Heder, 2017.] From NASA to EU: the evolution of the TRL scale in Public Sector Innovation" (PDF)
  4. ↑ [https://www.researchgate.net/publication/228752216_An_Approach_to_Technology_Risk_Management , Valerdi, R., and R.J. Kohl., 2004.] An Approach to Technology Risk Management. (PDF)
  5. ↑ [https://apps.dtic.mil/sti/pdfs/ADA443149.pdf , Smith, J.D., 2005.] An Alternative to Technology Readiness Levels for Non-Developmental Item (Ndi) Software. (PDF)
  6. ↑ 6.0 6.1 6.2 [https://www.researchgate.net/publication/260652369_Analysis_and_Critique_of_the_System_Readiness_Level, E. Kujawski, 2013.] Analysis and Critique of the System Readiness Level
  7. ↑ 7.0 7.1 [https://app.knovel.com/hotlink/toc/id:kpSRMPPP01/standard-risk-management/standard-risk-management Project Management Institute, 2019.] Standard for Risk Management in Portfolios, Programs, and Projects, P. 59.
  8. ↑ [https://www.matec-conferences.org/articles/matecconf/pdf/2018/18/matecconf_ijcaet-isampe2018_02067.pdf , N. Marlyana et al., 2018.] A Quantitative Analysis of System Readiness Level Plus (SRL+): Development of Readiness Level Measurement. (PDF)
Retrieved from "http://wiki.doing-projects.org/index.php?title=System_Readiness_Level_Index&oldid=101606"
Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox