AUC

The AUC tab is shown below.

The (AUC is the area found below the Receiver Operating Characteristics (ROC) Curves, which plot true-positive predictions versus false-positive predictions for a binary-response variable. The greater the AUC, the better the model is at predicting responses. For this example, using a lambda value of 3 resulted in the highest AUC, suggesting it is the best model for predicting future samples. Note the black, horizontal line near the bottom of the graph. This line represents the AUC if there is no any predictive model, and predicts every observation to be in the majority class. Any model whose AUC statistics approach or fall below this reference value is likely to be unreliable.

The plot displays results from JMPs Fit Y by X platform, which in this case performs a one-way analysis of variance along with mean comparisons. Note: the assumptions of independence behind the mean comparisons are not met because each of the models is fitted to the same data set. So you should interpret the results cautiously and use them primarily for relative comparisons. The same holds true for all plots of this type.

The output of this tab is similar to the RMSE tab, and contains the following additional element:

•

Plot and compare ROC curves of selected methods button

Select models of interest in the one-way plot by clicking and dragging a mouse rectangle over them and clicking this button to generate a plot like the one shown below:

This is a comparison of ROC curves for each of the methods. The area under each curve is the AUC statistic plotted in the one-way plot. AUC is a measure of sorting efficiency, with a value of 1 indicating perfect sorting and 0.5 random sorting. ROC curves that approach the upper left corner of the plot indicate better performance. See Receiver Operating Characteristics (ROC) Curves for more information.

The tables below the plots provide confidence intervals for each AUC statistic, an overall difference test, and all pairwise comparisons with confidence limits on differences. Use these to determine which methods are significantly different from each other.