When you select the Multiple Comparisons option, you can choose the initial comparison to use equivalence testing to test for practical differences. Use this option to conduct one or more equivalence tests. Equivalence tests are useful when you want to detect differences that are of practical interest. You must specify a threshold difference for group means for which smaller differences are considered practically equivalent. In other words, if two group means differ by this amount or less, you are willing to consider them equivalent.
Once you have specified this value, the Equivalence Tests report appears. The bounds that you have specified are given at the top of the report. The report consists of a table giving the equivalence tests and a scatterplot that displays them. The equivalence tests and confidence intervals are based on Student’s t critical values.
The Two One-Sided Tests (TOST) method is used to test for a practical difference between the means (Schuirmann 1987). Two one-sided pooled-variance t tests are constructed for the null hypotheses that the true difference exceeds the threshold values. If both tests reject, the difference in the means does not statistically exceed either threshold value. Therefore, the groups are considered practically equivalent. If only one or neither test rejects, then the groups might not be practically equivalent.
For each comparison, the TOST Tests report contains the following information:
Difference
The estimated difference in the means.
Lower Bound t Ratio, Upper Bound t Ratio
The lower and upper bound t ratios for the two one-sided pooled-variance significance tests.
Lower Bound p-Value, Upper Bound p-Value
The significance probabilities (p-values) that correspond to the lower and upper bound t ratios.
Max p-Value
The maximum of the lower and upper bound p-values.
Lower 90%, Upper 90%
Limits for a 1−2α confidence interval for the difference in the means.
Assessment
An assessment of the hypothesis test for the specified alpha level.
Using colors, this scatterplot indicates which means are practically equivalent and which are not practically equivalent as determined by the equivalence test. This plot is sometimes called a diffogram or a mean-mean scatterplot.
The plot shows a solid reference line on the diagonal as well as a shaded reference band. The width of the band is twice the practical difference. The coordinates of the point on the line segment are the means for the corresponding groups. There is an implied third axis on the diagonal where each line segment corresponds to a 1−2α confidence interval for a pairwise comparison. Hover over one of these points to show a tooltip that indicates the groups being compared and the estimated difference. When a line segment is entirely contained within the diagonal band, it follows that the means are practically equivalent.
The Equivalence Tests Scatterplot has the following option:
Show Reference Lines
Displays reference lines for the points on the scatterplot. This is not recommended if there are many points in the scatterplot. If there are many points, it is better to hover over the points to view the tooltip labels.
In the Forest Plot, the comparison confidence intervals are plotted versus the difference in means or ratio of standard deviations. The intervals are plotted on a difference of means or ratio of standard deviations scale. Shading indicates the equivalent regions.
Tip: Hover over a point to show the groups being compared and the estimated difference or ratio.
This option removes the Equivalence Tests report from the Multiple Comparisons report.