Overview of Covering Arrays

You can use covering arrays to test systems where failures occur as a result of interactions among components or subsystems. Covering arrays are often used in areas such as software, circuit, and network design, where the following conditions are likely to be true:

• The cost of testing is usually high.

• Testing focuses on revealing interactions for which failures occur.

• A test run is typically deterministic and results in either success or failure.

• Replicate runs are wasteful because they yield identical results.

• The efficiency of a design is based on how many of the possible conditions are covered without including redundant runs.

Because systems testing is expensive, reducing the amount of testing is critical. Testing all possible interactions is usually prohibitive and often unnecessary. Experience shows that most failures result from the interaction of a small number of components. The size of the largest combination of components likely to drive a failure, called the strength, drives the size of the design.

In the Covering Array platform, you specify the required strength of your design. If appropriate, you define factor level combinations that are not permitted. The Covering Array platform constructs a highly efficient design that meets your requirements. It provides metrics that you can use to assess the quality of the design in terms of its coverage. It also provides a script in the data table for the design that enables you to analyze your results.

Covering arrays are often used in situations where certain combinations of factor level settings are not feasible. The Covering Array platform is able to find very efficient covering arrays even when restrictions are placed on factor level combinations.

For background on the structure of covering arrays and algorithms for computing them, see Colbourn (2004), Colbourn et al. (2011), Hartman and Raskin (2004), and Martirosyan (2003). For more information about covering arrays with restrictions on factor levels, see Cohen et al. (2007) and Morgan (2009).

A covering array of strength t is a design that tests all combinations of t factor level settings. Consider an interaction defined by specific settings for k factors. If failures occur for all tests involving that interaction, then that interaction detects a failure. Using this terminology, a strength t design enables you to detect failures associated with any interaction of up to t factors.

In the literature, covering arrays are also referred to as factor covering designs. For background and more information, see Yilmaz et al. (2014), Cohen et al. (2003), and Dalal and Mallows (1998).

To illustrate the nature of covering arrays, consider a situation involving seven categorical factors each with two levels. You want to test all pairwise combinations of factor levels.

A design that might be used in this situation is the 8-run resolution III main effects design given as follows:

Figure 20.2 A Resolution III Design with Strength 2 

Note that this factorial design is a strength 2 covering array because all pairwise combinations of levels of any two factors appear. For example, for X1 and X2, the following combinations each appear twice:

• L2 and L1

• L2 and L2

• L1 and L1

• L1 and L2

However, a strength 2 covering array needs only 6 runs :

Figure 20.3 Strength 2 Covering Array 

All pairwise combinations of levels of any two factors appear at least once in the six runs. The Covering Array design is more efficient than the Resolution III design because it achieves strength 2 coverage with fewer runs.

The efficiency of a covering array is measured by the number of runs required to achieve the required coverage. The smaller the number of runs, the more efficient the design.