AUC

The plot shows Area Under the Curve ((AUC) statistics for each cross validation model iteration. AUC is the area under the Receiver Operating Characteristics (ROC) curve, which plots sensitivity versus 1 - Specificity for predictions of a binary response variable. The solid black horizontal reference line is the median among cross validation iteration of the AUC values estimated without any model. The dashed horizontal reference lines above and below the solid one are the levels of the whiskers in a box plot for these no-model estimates. The AUC depends on which response value is designated as the event. Usually, the least frequent value is the event. For such cases, the better the model the larger the AUC, and a model whose cross validated AUC is near or below this baseline range is unreliable and should be ignored. The plots use the Mann-Whitney statistic to compute the AUC.

The output of this tab is similar to the RMSE tab, and contains the following additional element:

•

Plot and compare ROC curves of selected models button

Select models of interest in the oneway plot by clicking and dragging a mouse rectangle over them and clicking this button to generate a plot like the one shown below:

This is a comparison of ROC curves for each of the methods. The area under each curve is the AUC statistic plotted in the oneway plot. AUC is a measure of sorting efficiency, with a value of 1 indicating perfect sorting and 0.5 random sorting. ROC curves that approach the upper left corner of the plot indicate better performance. See Receiver Operating Characteristics (ROC) Curves for more information.

The tables below the plots provide confidence intervals for each AUC statistic, an overall difference test, and all pairwise comparisons with confidence limits on differences. Use these to determine which methods are significantly different from each other.