Statistical Tests Used in MarketSight
Tests Comparing Means
When the mean is displayed for a row variable, MarketSight first runs an Analysis of Variance (ANOVA) using an F-test. Doing so tests the hypothesis that the means of multiple normally distributed populations, all having the same variance, are equal.
MarketSight tests whether or not the row variable’s means are equal to one another for all columns in the crosstab. Rejecting the test hypothesis implies that at least one of the column means is significantly different from the others.
Fisher's Least Significant Difference (LSD) test
If the statistics option to ”Correct for Type I errors in all comparisons” is disabled, MarketSight will run Fisher’s LSD test for both Pairwise tests and Contrast tests of means. MarketSight only runs Fisher’s LSD test if the ANOVA F-test first rejects the null hypothesis that all column means are equal to one another.
Fisher’s LSD test is a relatively powerful test because it uses the pooled variance estimate from the F-test, thus taking advantage of the increased sample size of all columns in the crosstab. Pooling the variance is valid because MarketSight explicitly tests for equality of variance among all columns prior to running the associated F-test.
Although the test is more powerful than either the Tukey HSD or Scheffé tests, it is more susceptible to Type I error when running multiple simultaneous tests.
If the statistics option to ”Correct for Type I errors in all comparisons” is enabled, MarketSight will run the Scheffé test for Contrast tests of means. MarketSight only runs the Scheffé test if the ANOVA F-test first rejects the null hypothesis that all column means are equal to one another.
The Scheffé test is a conservative test for running multiple Contrast tests of Means which controls the overall Type I error rate for all possible contrasts based on the selected Confidence Level.
MarketSight will run Tukey-Kramer test for Pairwise tests of means. MarketSight only runs Tukey-Kramer test if the ANOVA F-test first rejects the null hypothesis that all column means are equal to one another.
Tukey-Kramer test is a conservative test for running multiple Pairwise comparisons of Means. It controls the overall Type I error rate across a number of related Pairwise tests based on the selected Confidence Level.
Tests Comparing Column Proportions
When a Row Variable displays the Column % or Count option for individual Values, MarketSight runs a Chi-squared test. This test examines whether there is a relationship between the Column Variable(s) and the Row Variable.
Chi-squared tests involve a comparison of ”actual” cell counts to ”expected” cell counts in a crosstab.
The expected count for each cell is derived from a Row Variable’s actual counts as follows: multiply the cell's row total by its column total, then divide by the sum total of all observations.
If the actual cell counts for one or more cells differ materially from their expected counts, the Chi-squared test may produce a statistically significant result which implies there is a relationship between the Column Variable(s) and the Row Variable.
A modified version of a Chi-Squared test is run for Multiple Response Variables.
For 2x2 crosstabs with small sample sizes, the Chi-squared test may be unreliable. Therefore, MarketSight runs an alternate test, Fisher’s Exact Test, if more than 20% of the cells in a 2x2 crosstab have an expected cell count less than 5, or if any cells in a 2x2 cross-tab have an expected cell count less than 1.
Fisher’s Exact Test calculates the true probability of observing a particular set of actual cell counts in a 2 x 2 crosstab, assuming that row and column totals are held constant.
Fisher's Exact Test is not run for Multiple Response Variables.
MarketSight runs Z-tests for both Contrast and Pairwise tests of Column Proportions. A Z-test is used to test for a difference between two column proportions. The column proportions involved in the test are the cell counts divided by their respective column totals.
A Z-test is only run when the cells being compared have actual counts greater or equal to 5 and the column sample size minus the actual cell counts is greater than or equal to 5. If these data sufficiency conditions are not met, MarketSight runs Fisher’s Exact Test instead.
A modified version of a Z-test is run for Multiple Response Variables.