Predictive and Specialized Modeling • Model Comparison • Example of Model Comparison
Previous
 • 
Next
Example of Model Comparison
This section provides an example of using the Model Comparison platform. The example uses demographic data to build a model for median home price. A regression model and a bootstrap forest model are compared.
Begin by selecting Help > Sample Data Library and opening Boston Housing.jmp.
Create a Validation Column
1.
Create a column called validation.
2.
On the Column Info window, select Random from the Initialize Data list.
3.
Select the Random Indicator radio button.
4.
Click OK.
The rows assigned a 0 are the training set. The rows assigned a 1 are the validation set.
Create the Regression Model and Save the Prediction Formula to a Column
1.
Select Analyze > Fit Model.
2.
Select mvalue and click Y.
3.
Select the other columns (except validation) and click Add.
4.
Select Stepwise in the Personality list.
5.
Select validation and click Validation.
6.
Click the Run button.
7.
Select P-value Threshold from the Stopping Rule list.
8.
Click the Go button.
9.
Click the Run Model button.
The Fit Group report appears, a portion of which is shown in Figure 9.2.
10.
Save the prediction formula to a column by selecting Save Columns > Prediction Formula on the Response red triangle menu.
Figure 9.2 Fit Model Report
Create the Bootstrap Forest Model and Save the Prediction Formula to a Column
1.
Select Analyze > Predictive Modeling > Partition.
2.
Select mvalue and click Y, Response.
3.
Select the other columns (except validation) and click X, Factor.
4.
Select validation and click Validation.
5.
Select Bootstrap Forest in the Method list.
6.
Click OK.
7.
Select the Early Stopping check box.
8.
Select the Multiple Fits over number of terms check box.
9.
Click OK.
The Bootstrap Forest report appears, a portion of which is shown in Figure 9.3.
10.
Save the prediction formula to a column by selecting Save Columns > Save Prediction Formula on the Bootstrap Forest red triangle menu.
Figure 9.3 Bootstrap Forest Model
Compare the Models
1.
Select Analyze > Predictive Modeling > Model Comparison.
2.
Select the two prediction formula columns and click Y, Predictors.
3.
Select validation and click Group.
4.
Click OK.
The Model Comparison report appears (Figure 9.4).
Note: Your results differ due to the random assignment of training and validation rows.
Figure 9.4 Model Comparison Report
The rows in the training set were used to build the models, so the RSquare statistics for Validation=0 might be artificially inflated. In this case, the statistics are not representative of the models’ future predictive ability. This is especially true for the bootstrap forest model.
Compare the models using the statistics for Validation=1. In this case, the bootstrap forest model predicts better than the regression model.
5.
Click the Model Comparison red triangle and select Profiler.
Figure 9.5 Prediction Profiler for All Models
The prediction profiler enables you to compare the impact of each factor in the different models. The profiler is especially interesting when comparing different types of models such as here where you have a regression model and a partition model.
Related Information 
•
Model Specification in the Fitting Linear Models book
•
Partition Models topic

Previous
 • 
Next
Help created on 10/11/2018