Example Comparing Designs of Different Run Sizes
Use the Compare Design platform to compare three designs with run sizes 16, 20, and 24. The designs are constructed for main effect models. Explore whether the potential benefits of using a larger run size are worth the additional cost in resources.
1. Select Help > Sample Index, click the Sample Index red triangle and select Open Sample > Script. Open Compare Three Run Sizes.jsl.
2. Right-click in the script window and select Run Script.
Three design tables are constructed using Custom Design, with only main effects as entries in the Model section:
– 16-Run Design
– 20-Run Design
– 24-Run Design
You want to compare these three designs. Notice that the 16-Run Design table is active.
3. In the 16-Run Design table, select DOE > Design Diagnostics > Compare Designs.
4. From the Compare ‘16-Run Design’ with list, select 20-Run Design and 24-Run Design.
Panels for each of these designs are added to the launch window. JMP automatically matches the columns in the order in which they appear in the three design tables.
5. Click OK.
6. In the Absolute Correlations report, open the Color Map on Correlations report and the three color map reports under it.
The sections of the Compare Design report are shown and discussed below.
Figure 17.21 Power Analysis Comparison
All three designs have high power for detecting main effects if the coefficients are on the order of the Anticipated RMSE.
Figure 17.22 Prediction Variance Profile Showing Maximum Variance for Three Designs
The maximum relative prediction variance decreases as the run size increases. The plot also shows values of the factors that give this worst-case relative variance. However, keep in mind that many settings can lead to this same maximum relative variance.
Figure 17.23 Fraction of Design Space Comparison
The 24-run design is superior to the other two designs in terms of prediction variance over the entire design space. The relative prediction variance for the 24-run design is uniformly smaller than for the other two designs. The 20-run design has uniformly smaller prediction variance than the 16-run design. The red dotted crosshairs indicate that the relative prediction variance for the 20-run design is less than about 0.23 over about 50% of the design space.
Figure 17.24 Absolute Correlations Comparison
The Absolute Correlations table summarizes the information shown in the Color Maps on Correlations. Recall that the model for all three designs consists of only main effects and the Alias Matrix contains two-factor interactions.
For the 16-run design, the Model x Alias portion of the table indicates that there are 21 confoundings of main effects with two-factor interactions. The Alias x Alias portion indicates that there are 21 confoundings of two-factor interactions.
Figure 17.25 Design Diagnostics Comparison
The Design Diagnostics report compares the efficiency of the 16-run design to both the 20-run and 24-run designs in terms of several efficiency measures. Relative efficiency values that exceed 1 indicate that the reference design is preferable for the given measure. Values less than 1 indicate that the design being compared to the reference design is preferable. The 16-run design has lower efficiency than the other two designs across all metrics, indicating that the larger designs are preferable.
Figure 17.26 Color Map on Correlations Comparison
For the 16-run design, the black cells off the main diagonal indicate that there is confounding of some main effects with some two-factor interactions, and confounding of two-factor interactions. There are 21 instances where model terms (main effects) are confounded with alias terms (two factor interactions), and 21 instances where alias terms are confounded with each other. This is shown in the report under Pairwise Confoundings.
The color maps for the 20- and 24-run designs have no off-diagonal cells that are solid black. It follows that these designs show no instances of confounding between any pair of main or two-way interaction effects. For the 20-run design, the Color Map on Correlations indicates that there are some large correlations between some main effects and some two-factor interactions, and between some two-factor interactions.The 24-run design shows only moderate correlations between main effects and two-factor interactions, and between two-factor interactions.
However, it is interesting to note that the 20- and 24-run designs both have a higher Average Correlation for Model x Alias terms than does the 16-run design. Although the 16-run design shows confounding, the average amount of correlation is less than for the 20- and 24-run designs.