Example of a Custom Design

Use the custom designer to design an experiment to investigate the effect of wine processing factors on wine taste. Your employer grows two varieties of Pinot Noir grapes that can be processed in different ways. Your goal is to determine which factors affect the taste of Pinot Noir wine. Before the grapes are processed, you set up your experimental design. Once processed, the wine samples are aged for 12 months, then filtered and bottled. At this point, the wine samples are rated for quality by expert wine tasters.

Response

Most of your vineyard’s product is sold to five large wine distributors. You arrange for a wine-tasting expert from each distributor to evaluate the wine samples for quality. To maximize the number of factors that you can study, you decide that each expert must rate eight different samples. This means that your design needs to have 40 wine samples, or runs.

The ratings follow a 0 – 20 scale, where 0 is the worst and 20 is the best. Rating, the variable consisting of the experts’ ratings, is the response of interest. You want to identify the wine-related factors that maximize the response.

Blocking Factor

A blocking factor is used to account for variation that is not necessarily of direct interest. A blocking factor is particularly effective when observations taken at one factor level are expected to be more similar than observations at different levels. In your experiment, ratings by one expert are likely to have similar characteristics and to differ from ratings by a different expert. Yet, you are interested in which properties of the wine lead to high ratings by all experts.

Because each rater tastes eight wines, Rater is a blocking factor with eight runs per block. For this experiment, only these five raters are of concern. You are not interested in generalizing to a larger population of raters.

Process Factors

You have identified nine process factors for the study. These include the grape variety, the field on which the grapes were grown, and seven other factors related to processing. You can experiment with any combination of these factors. Also, the factors can be varied at will as part of the experiment. Relative to the experiment, these factors are all “Easy” to change. For information about specifying factor changes, see Changes and Random Blocks.

The factors and their levels appear in Table 4.1. Note that all of these factors are categorical. The factors and their levels are also given in the factor table Wine Factors.jmp in the Design Experiment folder of Sample Data.

To experiment with all possible combinations of these factors would require a staggering 4 x 28 = 1024 runs. However, in this example, you are able to construct a compelling design in only 40 runs.

Now that the experimental goals, factors, and responses have been defined, you can build your custom design.

Add Responses

For this custom design example add your response, the response Goal, and, if appropriate, the Lower Limit, Upper Limit, and Importance. Here, the response is Rating.

1. Select DOE > Custom Design.

2. Double-click Y under Response Name and type Rating.

Note that the default Goal is Maximize. Because you want to maximize the taste rating, do not change the goal.

3. Click under Lower Limit and type 0.

The least desirable rating is 0.

4. Click under Upper Limit and type 20.

The most desirable rating is 20.

5. Leave the area under Importance blank.

Because there is only one response, that response is given Importance 1 by default.

Figure 4.2 shows the completed Responses outline.

Add Factors Manually or Automatically

For this custom design example, enter factors either manually or automatically using a pre-existing table that contains the factors and settings.

• If you are designing a new experiment, you must first enter the factors manually. See Add Factors Manually.

• Once you have saved the factors using the Save Factors option, you can load them automatically using the saved table. See Add Factors Automatically Using Load Factors.

Both methods add these four outlines to the Custom Design window: Define Factor Constraints, Model, Alias Terms, and Design Generation.

Add Factors Manually

1. First, add the blocking factor, Rater. Click Add Factor > Blocking > 8 runs per block.

2. Type Rater over the default Name of X1.

Note that Role is set to Blocking. Note also that only one setting for Values appears. This is because the number of blocks cannot be determined until the desired number of runs is specified. Once you specify the Number of Runs in the Design Generation outline, the number of levels for Rater updates to what is required.

3. Click Add Factor > Categorical > 2 Level.

4. Type Variety over the default Name of X2.

Note that Role is set to Categorical, as requested, and that Changes is set to Easy by default.

5. Click L1 and L2 and change them to Bernard and Dijon.

6. Click Add Factor > Categorical > 4 Level.

7. Type Field over the default Name of X3.

8. Click L1, L2, L3, and L4, and change them to 1, 2, 3, and 4.

9. Click Add Factor > Categorical > 2 Level.

10. Type De-Stem over the default Name of X4.

11. Click L1 and L2 and change them to No and Yes.

12. Type 6 next to Add N Factors, and then click Add Factor > Categorical > 2 Level. This adds six categorical two-level factors to your design.

13. Change the default factor names and values:

– Yeast (Cultured and Wild)

– Temperature (High and Low)

– Press (Hard and Soft)

– Barrel Age (New and Two Years)

– Barrel Seasoning (Air and Kiln)

– Filtering (No and Yes)

Figure 4.2 Completed Responses and Factors Outlines 

14. Click Continue.

The following outlines are added to the Custom Design window:

• Define Factor Constraints (not used in this example)

• Model

• Alias Terms

• Design Generation

Add Factors Automatically Using Load Factors

Enter factors using a table containing factor information:

1. Select Help > Sample Data Library and open Design Experiment/Wine Factors.jmp.

2. Click the Custom Design red triangle and select Load Factors.

After loading the factors, the Custom Design window automatically updates. The following outlines are added to the Custom Design window:

• Define Factor Constraints (not used in this example)

• Model

• Alias Terms

• Design Generation

Define the Model

For this custom design example, the Model outline shows all main effects as Necessary, indicating that the design needs to be capable of estimating all main effects. Your assumed model reflects your interest in main effects only. However, if you wanted to estimate other effects, you could add them to the Model outline. For more information about models, see Model.

Figure 4.3 Model Outline 

Define Alias Terms

The Alias Terms outline specifies the effects to be shown in the Alias Matrix, which appears later. The Alias Matrix shows the aliasing relationships between the Model terms and the effects listed in the Alias Terms outline. Open the Alias Terms outline node to verify that all two-factor interactions are listed. For more information about the alias matrix, see Alias Matrix.

Figure 4.4 Partial View of the Alias Terms Outline 

Set Random Seed to Duplicate Design

The Custom Design algorithm begins with a random starting design. To duplicate a design such as in this example, or in a teaching setting, set the random seed used to define the staring design. If you want to obtain a design with exactly the same runs and run order as the one shown in Figure 4.5, set the random seed and number of starts:

1. Click the Custom Design red triangle and select Set Random Seed.

2. Type 100526291 (the random seed).

3. Click OK.

4. Click the Custom Design red triangle and select Number of Starts.

5. Type 2.

6. Click OK.

Note: Setting the Random Seed and Number of Starts reproduces the exact results shown in this example. In constructing a design on your own, these steps are not necessary.

Generate the Design

In the Design Generation outline, you can enter additional details about the structure and size of your design. In this example, the Default design shows 16 runs. But you have five raters, each of whom can sample eight wines. This means that you want a design with 40 runs. Change the number of design runs:

1. Under Number of Runs, type 40 in the User Specified box.

Because you do not want to replicate runs, leave the Number of Replicate Runs set to 0.

2. Click Make Design.

The Design and Design Evaluation outlines are added to the Custom Design window. The Output Options panel also appears.

Verify the Design

The Design outline shows the runs in the custom design that you have constructed. Later, you are able to randomize the order under Output Options. For now, verify that this design is appropriate for your experiment. For example, check that each of five Raters evaluates eight wines, that all necessary factors are shown, and that none of the settings represent infeasible combinations.

Figure 4.5 Design for Wine Experiment 

Evaluate the Design

The Design Evaluation outline provides different ways to evaluate your custom design. For this example, open the Design Evaluation outline, and examine the Color Map on Correlations, the Alias Matrix, and Design Diagnostics.

Note: For more information about the Design Evaluation outline, see Evaluate Designs.

Color Map on Correlations

The Color Map on Correlations shows the absolute value of the correlation between any two effects that appear in either the Model or the Alias Terms outline.

Figure 4.6 Color Map on Correlations 

The main effects are represented by the 15 terms in the upper left corner of the map. The white corresponding to the correlations of main effects with other main effects indicate correlations of 0. This means that all main effects are orthogonal and can be estimated independently.

The only black in Figure 4.6 is on the main diagonal. Black indicates absolute correlations of one, reflecting that each term is perfectly correlated with itself. It follows that no main effect is completely confounded with any two-way interaction. In fact, the absolute values of the correlations of main effects with two-way interactions are fairly low. This means that estimates of main effects might be only slightly biased by the presence of active two-way interactions.

Tip: Hover over cells in the color map to see the absolute correlations between effects. Right-click below the legend to save the correlations to a data table.

Alias Matrix

In the Alias Matrix, model effects are listed in the column on the left. For a given model effect, a column entry indicates the degree to which the column effect (if active) biases the estimate of the model effect.

Figure 4.7 Partial View of Alias Matrix 

For example, consider the model effect Barrel Seasoning. If Variety*Press is active, then the expected value of the estimate for the Barrel Seasoning effect differs from an unbiased estimate of that effect. The amount by which it differs is equal to 0.4 times the effect of Variety*Press. Therefore, what appears to be a significant Barrel Seasoning estimated effect could in reality be a significant Variety*Press effect.

Design Diagnostics

The Design Diagnostics outline provides information about the efficiency of the design. Efficiency measures compare your design to a theoretically optimal design, which might not exist. The efficiency values are ratios, expressed as percents, of the efficiency of your design to the efficiency of this optimal design. For more information about the efficiency measures, see Estimation Efficiency.

Figure 4.8 Design Diagnostics Outline 

Notice that the D-, G-, and A-efficiency values are all 100%. Because your design is orthogonal for main effects, the design is optimal for the main effects model relative to all three efficiency criteria.

The first line in the Design Diagnostics outline indicates that your design was constructed to optimize the D-efficiency criterion. See the Optimality Criterion description in Custom Design Options. In this case, your design has D Efficiency of 100%.

Specify Design Table Options

The final steps of your custom design generation are to specify options for the design table and to make the table. Specify the order of runs in your data table using the Output Options panel. The default selection, Randomize within Blocks, is appropriate for this example. Simply click Make Table.

A Custom Design table is created and opens, similar to the one in Figure 4.9.

Note: Your table might look different because the algorithm that creates it uses a random starting design. To obtain the precise table shown in Figure 4.9, set the random seed and number of starts as described in Set Random Seed to Duplicate Design.

Figure 4.9 Custom Design Table 

Note the following:

• In the Table panel, the Model, Evaluate Design, and DOE Dialog scripts are added during the design creation process. The Model script opens a Fit Model window containing the effects that you specified as Necessary in the Custom Design dialog window. The DOE Dialog script re-creates the window used to generate the design table.

• In the Columns panel, the asterisks to the right of the factors and response indicate column properties that have been saved to the columns in the data table. These column properties are used in the analysis of the data. For more information about column properties, see Factors and Factor Column Properties.

Analyze the Custom Design

After running an experiment and compiling the data you are ready to analyze the results. Use the Rating column of your Custom Design table to record the Rating data. .

1. Select Help > Sample Data Library and open Design Experiment/Wine Data.jmp.

The Wine Data.jmp table is exactly the same as the Custom Design table shown in Figure 4.9, except that it contains your recorded experimental results.

2. In the Table panel, click the green triangle next to the Model script.

Figure 4.10 Model Specification Window for the Wine Experiment 

Notice that Rater, the blocking factor, is added as a fixed effect, rather than as a random block effect. This is appropriate because the five raters were specifically chosen and are not a random sample from a larger population.

3. Click Run.

Interpret the Full Model Results

The model output from our custom design example is shown below.

Figure 4.11 Partial Model Fit Results 

Note the following:

• The Actual by Predicted Plot shows no obvious evidence of lack of fit.

• The model is significant, as indicated by the Actual by Predicted Plot and by the p-value beneath it.

• The Effect Tests report indicates that seven of the model terms are significant at the 0.05 level. Field, Temperature, and Barrel Age are not significant.

• The Effect Summary report lists these effects in decreasing order of significance. Larger LogWorth values correspond to smaller PValues and greater significance.

Reduce the Model

Reduce the model for the custom design results by removing the effects that you identified as inactive:

1. In the Effect Summary report, press Control and select Temperature, Field, and Barrel Age.

2. Click Remove.

The report updates to show the model fit with these three effects removed.

Interpret the Model Results with the Profiler

The Actual by Predicted Plot for the reduced model shows no lack of fit issues. The Effect Summary and the Effect Test report show that the remaining seven terms are significant at the 0.05 level. Use the prediction profiler to further explore your reduced model.

Figure 4.12 shows the Prediction Profiler. Recall that you specified a response goal of Maximize, with lower and upper limits of 0 and 20. Setting these limits caused a Response Limits column property to be saved to the Rating column in the Custom Design table. The Prediction Profiler uses the Response Limits information to construct a Desirability function, which appears in the right-most plot in the top row in Figure 4.12. The bottom row displays Desirability traces.

The first six plots in the top row show traces of the predicted model. For each factor, the line in the plot shows how Rating varies when all other factors are set at the values defined by the red dashed vertical lines. By default, the profiler appears with categorical factors set at their low settings. By varying the settings for the factors, you can see how the predicted Rating for wines changes. Notice that a confidence interval is given for the mean predicted Rating.

Observe that Rater is not included among the factors shown in the profiler. This is because Rater is a block variable. You included Rater to explain variation, but Rater is not of direct interest in terms of optimizing process factor settings. The predicted Rating for a wine with the given settings is the average of the predicted ratings for that wine by all raters.

Figure 4.12 Profiler for Reduced Model 

Optimize Factor Settings

Use the prediction profiler to identify optimal settings based on your custom design results. You would like to identify settings that maximize Rating across raters.

1. Click the Prediction Profiler red triangle and select Optimization and Desirability > Maximize Desirability.

The red dashed vertical lines in the Prediction Profiler update to show optimal settings for each factor. The optimal settings result in a predicted rating of 19.925. In general, there can be different sets of factor settings that result in the same optimal value.

Figure 4.13 Prediction Profiler with Factor Settings Optimized 

2. To see predicted ratings for all runs, save the Prediction Formula. Click the Response Rating red triangle and select Save Columns > Prediction Formula.

A column called Pred Formula Rating is added to the data table. Note that one of the runs, row 33, was given the maximum rating of 20 by Rater 5. The predicted rating for that run by Rater 5 is 19.550. But the row 33 trial was run at the optimal settings. The predicted value of 19.925 given for these settings in the Prediction Profiler is obtained by averaging the predicted ratings for that run over all five raters.

Lock a Factor Level

You can lock factors in the profiler to explore optimization under specific conditions. When you maximized your design response , you learned that the optimal rating is achieved with the Dijon variety of grapes (Figure 4.13). Your manager points out that it would be cost-prohibitive to replant the fields that are growing Bernard grapes with young Dijon vines. Therefore, you need to find optimal process settings and the predicted rating for Bernard grapes.

1. In the Variety plot of the Prediction Profiler, drag the red dashed vertical line to Bernard.

2. Press Control and click in one of the Variety plots.

The Factor Settings window appears.

3. Select Lock Factor Setting and click OK.

4. Click the Prediction Profiler red triangle and select Optimization and Desirability > Maximize Desirability.

Figure 4.14 Prediction Profiler with Optimal Settings for Bernard Variety 

The optimal settings are unchanged because the model contains no interaction terms. The predicted rating at these settings is 17.975.

Add the Rater to the Profiler

If you want to see the Profiler traces for the levels of Rater, perform the following steps:

1. Click the Prediction Profiler red triangle and select Reset Factor Grid.

A Factor Settings window appears with columns for all of the factors, including Rater. The box under Rater and next to Show is not checked. This indicates that Rater is not shown in the Prediction Profiler.

2. Check the box under Rater in the row corresponding to Show.

3. Deselect the box under Rater in the row corresponding to Lock Factor Setting.

4. Click OK.

The Profiler updates to show a plot for Rater.

5. Click in either plot above Rater.

Figure 4.15 Profiler for Reduced Model Showing Rater 

A dashed vertical red line appears. Drag this line to see the traces for each of the raters. Keep in mind that Variety is still locked at Bernard. To unlock Variety, press Control and click in one of the Variety plots. In the Factor Settings window that appears, deselect Lock Factor Setting.

Summary of Custom Design Example

In your wine tasting experiment, using only 40 runs, you have identified six (out of nine) factors that have an effect on ratings for Pinot Noir grapes. You found that you could achieve a predicted rating of 19.925 (out of a possible 20) at the optimal settings for those factors. You also identified optimal settings for both varieties of grapes.

In this section, you constructed a design using the outlines in the Custom Design window. The next section explains each outline and the design steps in more detail.

Want more information? Have questions? Get answers in the JMP User Community (community.jmp.com).