The (
AUC
is the area found below the
Receiver Operating Characteristics (ROC) Curves
, which plot true-positive predictions versus false-positive predictions for a binary-response
variable
. The greater the AUC, the better the
model
is at predicting responses. For this example, using a lambda value of 3 resulted in the highest AUC, suggesting it is the best model for predicting future samples. Note the black, horizontal line near the bottom of the graph. This line represents the AUC if there is no any
predictive model
, and predicts every
observation
to be in the majority class. Any model whose AUC statistics approach or fall below this reference value is likely to be unreliable.
The plot displays results from JMPs
Fit Y by X
platform, which in this case performs a one-way analysis of
variance
along with
mean
comparisons.
Note
: the assumptions of independence behind the mean comparisons are not met because each of the models is fitted to the same data set. So you should interpret the results cautiously and use them primarily for relative comparisons. The same holds true for all plots of this type.
The output of this tab is similar to the
RMSE
tab, and contains the following additional element:
|
•
|
button
|
Select models of interest in the
one-way plot
by clicking and dragging a mouse rectangle over them and clicking this button to generate a plot like the one shown below:
This is a comparison of ROC curves for each of the methods. The area under each curve is the AUC statistic plotted in the one-way plot. AUC is a measure of sorting efficiency, with a value of
1
indicating perfect sorting and
0.5
random sorting. ROC curves that approach the upper left corner of the plot indicate better performance. See
Receiver Operating Characteristics (ROC) Curves
for more information.