Summary Across the Folds Report

The Summary Across the Folds report in the Model Screening platform contains a summary of the measures of fit across the folds, as well as across the trials if Repeated K Fold is specified. If the K Fold Crossvalidation option is specified, the measures of fit are summarized across the validation sets. If the Nested Crossvalidation option is specified, the measures of fit are summarized across the test sets. The report contains a table with the following columns:

Method

The name of the method used to fit the model.

N Trials Folds

The total number of models that were fit across all folds and trials, if applicable.

Sum Freq

The average number of observations in the validation or test sets. The test sets are used to estimate model performance.

RSquare

The mean RSquare across all validation or test set folds. This column contains the Entropy RSquare when the response is categorical.

Mean RASE

The mean RASE (Root Average Square Error) across all validation or test set folds.

StdDev RASE

The standard deviation of the RASE across all validation or test sets.

Mean AUC

(Available only for categorical responses.) The mean area under the ROC curve (AUC) across all validation or test sets.

Mean MR

(Available only for categorical responses.) The mean misclassification rate (MR) across all validation or test sets.

The following options are available below the table:

Select Dominant

Selects each model that is better than or equal to all of the other models in terms of a combination of model fitting criteria. This is also referred to as selecting the Pareto Frontier. For continuous responses, RSquare and Sum Freq are considered when determining the dominant model. For categorical responses, Entropy RSquare, Misclassification Rate, AUC, and Sum Freq are considered when determining the dominant model.

Run Selected

Runs the individual model for each selected row. If any type of folded crossvalidation is specified in the launch window, the model run uses the fold, inner fold, and trial combination that corresponds to the final model. The final model is the model that produces the highest weighted average RSquare. The weight average RSquare is the average, weighted by number of observations, of the training RSquare, validation RSquare, and when a test set is used, the test RSquare. A validation column is created in the data table for any fold, inner fold, and trial combination that is required by clicking Run Selected.

Save Script Selected

Saves a model script to the script window for each selected row. If any type of folded crossvalidation is specified in the launch window, the validation set specified in the script uses the fold, inner fold, and trial combination that corresponds to the final model. The final model is the model that produces the highest weighted average RSquare. The weight average RSquare is the average, weighted by number of observations, of the training RSquare, validation RSquare, and when a test set is used, the test RSquare. A validation column is created in the data table for any fold, inner fold, and trial combination that is requested by clicking Save Script Selected.

Want more information? Have questions? Get answers in the JMP User Community (community.jmp.com).