What is experimental design?

Experimental design is a technique that enables scientists and engineers to efficiently assess the effect of multiple inputs, or factors, on measures of performance, or responses. Compared to one-factor-at-a-time, trial-and-error approaches, a well-designed experiment can provide clear results while dramatically reducing the required amount of testing.

Experiments for most of us are demonstrations of scientific principles. We recall the science class where we put litmus paper into a beaker of lemon juice and watched it turn pink.

In scientific research, many investigators still construct experiments to add support to a current hypothesis or perhaps disprove it. Such experiments may require only one run of the experimental machinery. Any subsequent runs are just to verify the result from the first demonstration.

So, when I say that my research area is experimental design, people ask, “You mean you show people how to be very careful when they set up their experiment?” Certainly careful setup, observation and measurement are important in any kind of experiment. But this is not what experimental design is about.

I want to explain what experimental design is. I also want to motivate the use of designed experiments in engineering problem solving. But first I need to point out that today’s products — and the processes for making these products — are much more complex than they were even a decade ago. The manufacture of the processing units that go into your cell phone may require dozens of steps, each having multiple controls. Even making toilet paper is a high-tech process!

Engineers are responsible for both defining products and building systems or processes that work. This requires attention to detail and ingenuity. Competitive pressure means tight development schedules and careful cost considerations.

The first step on the road from concept to products in stores involves building and testing prototypes. One way to test a prototype product, system or process is to change it in some way and measure the effect of that change on some measure of performance. In experimental design parlance, measures of performance are called responses. Factors are aspects of the prototype that we can change or control. Usually, we are interested in understanding the effects of more than one factor, which requires multiple tests of various prototypes. Here, experimental design is about how to decide which prototypes to test and how to go about testing them.

Initially, the goal is to determine which factors have the most substantial effect on a response. A brainstorming session may identify a dozen or more factors that we think might affect the response, but generally, only a few factors account for the vast majority of the variation in the response. Imagine trying to differentiate the effects of a dozen factors by trial and error. This is a common practice for engineers who have no experience with experimental design. Trial-and-error studies often waste resources and give ambiguous results. An expensive approach is to test all possible combinations of the factors. But with a dozen factors, this approach requires 4,096 trials, which is usually prohibitively time-consuming and expensive. By contrast, a well-designed experiment can provide clear results while dramatically reducing the required amount of testing.

For example, screening experiments separate the vital few factors from the trivial many. Experimental designs constructed for factor screening generally require only a small investment in testing that is between one and two times the number of factors. The ability to distinguish the effects of so many factors with such a small number of test runs is due to the elegant structure of experimental designs for screening.

Until recently, engineers wanting to use experimental designs for factor screening had to rely on catalogues of designs in textbooks or other reference materials. These design tables were restrictive because they were only available for certain combinations of numbers of factors and test runs.

Nowadays, commercial software can construct screening designs on the fly based on the requirements of each unique study. A computer-aided approach to experimental design provides several advantages over the use of reference designs. Computer generation of designs is immediate. This allows for creation and comparison of multiple design alternatives. An additional benefit is the flexibility of creating a design for each unique problem rather than reformulating your problem to match the requirements of a reference design. Finally, the training or knowledge requirements for appropriate use of computer-aided designs are lower than those for traditional designs. This makes the technology potentially accessible to any engineer.

Imagine a world where the use of experimental design was a standard engineering procedure. Products would get to market faster at lower cost and with higher quality and reliability. Such a world would be a better place to live.