Design validation
(→References) |
(→Validation without verification : Ariane 5 first launch) |
||
| Line 52: | Line 52: | ||
On June 4, 1996, the first flight of the Ariane 5 rocket ended after 37 seconds by the loss of control of the spacecraft, followed by its explosion. The 4 satellites that the rocket was carrying, worth a total of 370 million dollars, were destroyed in the process. | On June 4, 1996, the first flight of the Ariane 5 rocket ended after 37 seconds by the loss of control of the spacecraft, followed by its explosion. The 4 satellites that the rocket was carrying, worth a total of 370 million dollars, were destroyed in the process. | ||
| − | After analysis of the recordings of the flight parameters and the control software, this accident was attributed to a faulty design of the error recovery software in the navigation module. This module was duplicated to face a possible failure. After 37 seconds, an unrecoverable failure occurred in the backup module, which was taken out of service. A fraction of a second later, the same failure occurred in the active module. The simultaneous failure of the two modules was an unforeseen situation, which resulted in an inappropriate response: error diagnostic data was sent to the commands of the rocket. This data, incoherent in this context, triggered a full deflection of the thrusters, which caused the disintegration of the craft and its self-destruction. | + | After analysis of the recordings of the flight parameters and the control software, this accident was attributed to a faulty design of the error recovery software in the navigation module. This module was duplicated to face a possible failure. After 37 seconds, an unrecoverable failure occurred in the backup module, which was taken out of service. A fraction of a second later, the same failure occurred in the active module. The simultaneous failure of the two modules was an unforeseen situation, which resulted in an inappropriate response: error diagnostic data was sent to the commands of the rocket. This data, incoherent in this context, triggered a full deflection of the thrusters, which caused the disintegration of the craft and its self-destruction<ref name="Ariane">. |
Let's now try to understand what, in the design process of the rocket, brought this error to appear. The development of Ariane 5 was, in reality, an enlargement of the Ariane 4 rocket which was functional, in order to be able to transport more satellites, and thus bring more profit. The requirements of the Ariane 5 rocket being the same as those of Ariane 4, these were correctly selected. The rocket created corresponded exactly to the initial need (i.e. to transport more satellites). It was simply a matter of scaling up a design that had already been tested many times, so the validation was fulfilled. However, the designers decided to reuse the Ariane 4 rocket software for their new design, but with an inadequate testing process because the error that occurred had not been detected beforehand. The software was no longer adapted because the size of the rocket implied values that were too large and could no longer be stored in the memory. It is therefore a verification error. However, it could be argued that the idea of scaling up an existing design was also a validation error, as there is no evidence in itself that a larger model is as good a design as its predecessor <ref name="Failures"/>. | Let's now try to understand what, in the design process of the rocket, brought this error to appear. The development of Ariane 5 was, in reality, an enlargement of the Ariane 4 rocket which was functional, in order to be able to transport more satellites, and thus bring more profit. The requirements of the Ariane 5 rocket being the same as those of Ariane 4, these were correctly selected. The rocket created corresponded exactly to the initial need (i.e. to transport more satellites). It was simply a matter of scaling up a design that had already been tested many times, so the validation was fulfilled. However, the designers decided to reuse the Ariane 4 rocket software for their new design, but with an inadequate testing process because the error that occurred had not been detected beforehand. The software was no longer adapted because the size of the rocket implied values that were too large and could no longer be stored in the memory. It is therefore a verification error. However, it could be argued that the idea of scaling up an existing design was also a validation error, as there is no evidence in itself that a larger model is as good a design as its predecessor <ref name="Failures"/>. | ||
Revision as of 02:14, 23 March 2022
Developed by César Delafargue
The Ford company realized in 1957 that making a perfectly functional car was not enough to achieve commercial success. Indeed, nobody wanted to buy it, and the company suffered a loss of 250 million dollars [1]. The goal of validation is to ensure that a design meets the user's needs, and it is just as important as producing a functional design. Indeed, an inadequate validation will result in designing an undesired or unsuitable product, wasting significant amounts of resources, money, and time. The increase in complexity and duration of design projects often causes the initial objectives to get forgotten. Validation closes the production loop and ensures that the functionality intended for the user is fulfilled.
Validation is closely linked to the management of any project that aims to design a product, a system, or a service. It is a very powerful tool to reduce the risks associated with the project and to increase the overall quality of the rendering. This article aims to explain what design validation is, when it should be applied and why it is so important, for example through famous and very costly failures. It then tries to detail how to apply it, through different methods commonly applied by designers. The final point of the article warns the reader about some limitations of the validation such as the difficulty to establish the needs of its users or the long duration of its implementation.
Contents |
What is design validation?
Definition
According to the U.S. Food and Drugs Administration, “Design validation means establishing by objective evidence that device specifications conform with user needs and intended use(s).“[2]. Validation verifies that a product meets the requirements of the users and the market. The question that designers must ask themselves here is: "Are we building the right product?" It can also be accompanied by other more specific questions like "Is the product solving a real problem?" and "Is there a big enough market out there for the product?".
The key to validating a product is to first define exactly what needs the product is trying to meet. Validation will ensure that each of these needs is met, and will evaluate how effective this achievement is. The purpose of validation is to ensure that the product will find a customer base and be a commercial success when released [3]. It is therefore also about ensuring that there are potential users for the designed product.
Opinions on the place of validation in a design process differ. Indeed, in theory, it should be applied at the end of the design process, just before marketing, because this is when the product can be tested as it will be marketed [4]. However, validation is much more effective if it is done iteratively at each phase of the project, testing the product as it goes along and modifying it accordingly. It allows designers to ask fundamental questions about the product early in the design process when modifications are easy and inexpensive. By being carried out throughout the project, it also gives the confidence needed to innovate because each choice is validated. The validation described in this article is a process that must take place during the entire life cycle of a product's development. It starts as soon as the first design ideas are born in the mind of the designers, and can continue even after the final solution is deployed [5].
Difference with verification
Design verification and validation, although often combined, are two different things. They are not necessarily applied in a specific order or a predefined structure. Most of the time, both form a constant process that is applied at the same time throughout the whole project.
Unlike validation, the main question designers try to answer when performing verification is: "Are you designing the product right?". According to the FDA, "Verification means confirmation by examination and provision of objective evidence that specified requirements have been fulfilled" [2]. In other words, the purpose of verification is to ensure that the various components of the system are working as expected. It compares the design inputs (the specifications for the product that the designers chose) with the design outputs (how the product actually works) and verifies their correspondence[4]. It is mostly about ensuring that the product corresponds to the designer's idea. This is usually done through testing, inspection, and analysis of a prototype, or calculations and simulations for a building for example. Verification allows proving that the product that has been built is the product that the designers wanted to build, while validation allows making sure that this product is really useful for the users.
Requirements definition
Design validation evaluates a product for the exact requirements of the end-users and stakeholders. A major issue in order to validate a product is to establish an exhaustive list of all the requirements it must fulfill in order to satisfy those needs. The quality of the validation depends strongly on this list because if some important aspects are forgotten, then the product may have flaws because some of its aspects will not be validated (hence the interest in using pre-established heuristics, see below).
A functional requirement should define how and to what extent the system should satisfy a specific need of users or stakeholders in general. Their establishment and usage are done as follows [1] [7]:
- Determining the needs of the stakeholders
- Transforming these needs into a list of requirements
- Determining which part of the product should fulfill each requirement
- Verifying and validating the requirements by testing that specific part
Examples through famous failures
Verification without validation : The Edsel Ford
In 1957, Ford launched the first vehicle of its new Edsel brand, named after the deceased son of Henry Ford. Promised to thrill the middle classes, the Edsel project wanted to go beyond modernity. It is presented as the first incarnation of the car of the future, full of gadgets and innovations, and determined to "break the codes", including aesthetics. Ford is going all in, and the investments are enormous [8].
However, the long-developed model was immediately condemned by critics and the public. Ford had promised an avant-garde vehicle, but the Edsel was technically not very original and some of its details were mocked. The designers' choices are also criticized, in particular the unusual gaping vertical grille around which the whole front face is structured. The Edsel was a sort of accumulation of all the desires expressed by potential buyers of the time, which made it a real monster almost unsellable [8].
This failure is typically an example of poor validation. The verification was good because the vehicle was fully functional and met the expectations of the designers, who were proud of their product. However, good verification does not necessarily mean good validation. Management produced what management wanted, not what the customers wanted, and they produced the wrong car. They failed to design the right product for their clients, and no one wanted to buy it [1]. In the end, the failed program cost Ford $250 million.
Validation without verification : Ariane 5 first launch
On June 4, 1996, the first flight of the Ariane 5 rocket ended after 37 seconds by the loss of control of the spacecraft, followed by its explosion. The 4 satellites that the rocket was carrying, worth a total of 370 million dollars, were destroyed in the process.
After analysis of the recordings of the flight parameters and the control software, this accident was attributed to a faulty design of the error recovery software in the navigation module. This module was duplicated to face a possible failure. After 37 seconds, an unrecoverable failure occurred in the backup module, which was taken out of service. A fraction of a second later, the same failure occurred in the active module. The simultaneous failure of the two modules was an unforeseen situation, which resulted in an inappropriate response: error diagnostic data was sent to the commands of the rocket. This data, incoherent in this context, triggered a full deflection of the thrusters, which caused the disintegration of the craft and its self-destructionCite error: Closing </ref> missing for <ref> tag
Cite error: Closing </ref> missing for <ref> tag
Cite error:
<ref> tags exist, but no <references/> tag was found
