Test Set Average Accuracy Range

The Test Set Average Accuracy Range tab is shown below:

The Test Set Average Accuracy Range tab contains the following elements:

•

Plot of Learning Curves

The plot shows the average accuracy learning curve for each model as a solid line, and a shaded region around it indicating the range of individual curves. The width of the shading provides a measure of variability for the average learning curves. Curves with narrow bands are more reliable than those with wide bands.

Interpreting the Learning Curves

Accuracy is a measure of the proportion of the test set samples that are predicted correctly. Learning curves are constructed by using a succession of different sized subsets of the full data and assessing cross validation performance on each. Sample size is plotted on the x-axis while the cross validation performance metric is plotted on the y-axis. The primary goal of this process is to determine whether adding more samples will change performance. This is achieved by inspecting the slope of the curves, especially toward the right-hand side. If the curves have a slope similar to that show in this example, it is likely that adding more samples will improve performance. If the slopes are flat, adding more samples will likely not make much of a difference. Refer to Accuracy for more information about this statistic.

Note: This situation is completely different from classical power and sample size calculations, which are performed in the context of statistical hypothesis testing.