Significance testing

From mtab wikisupport
Jump to: navigation, search

mTAB™ introduces a powerful new significance testing feature. Using this new functionality, the row question formats t-stat (both equal and unequal variances), percentage significance and z-scores conveniently display significance differences by colorization of cells.

mTAB™ has included row format test statistic options such as t-stat and z-scores for some time now, however, the analyst was required to utilize "rule of thumb" measurements such as +/- 2.00 to determine significance, or alternatively, required to refer to a table of test statistic critical values. The new significance testing feature eliminates this requirement.


To use this new tool, select the row questions containing the test statistics such as t-stat, percentage significance and z-scores as you have in the past when significance testing is required. Now additionally select the toolbar tool with the scale icon as circled in the screenshot shown above. Selecting this tool displays the Significance Indicator dialog as shown below.


Note the Significance Indicator dialog includes selections for the confidence interval associated with the significance test, the selection of two types of reports (comparison to Base Column or comparison Across All Columns), an option for hiding the test statistics (e.g. t-stat values), and the ability to determine the colors to use to identify significantly different values.

How confident do I need to be?

The confidence interval is measure of confidence in the test of significance and is in effect the probability that the values identified as significantly different actually are. mTAB™ allows for three confidence interval values of 90, 95 and 98%, with a default value of 95%. The default value of 95% is normally appropriate for survey data. Selecting a value of 98% will typically identify fewer values as significantly different, as the test includes the additional constraint of a 98% probability of significance vs. a 95% probability.


The screenshot as shown above displays the use of the additional Significance Indicator dialog options Hide Test Values as well as color selection options. In this example, we elected to display the test values (e.g. t-stats) and selected a single cell color combination (green font / khaki background) to identify significance. Note however, that the Significance Indicator dialog allows for different cell colorization for values that are either above or below the comparison column's value, should you wish to draw this distinction.

Compared to What? Changing the Base Column

The Significance Indicator dialog Significance to Base Column dialog implies that each column in the tab is tested for significance relative to a base column. The default base column is the Formatted Sample Total of each individual column question. It is easy to identify the base column if the Hide Test Values option is checked off as in this example; the base column's test statistic values will always be 0.

You can use the existing Base Column toolbar tool to select a new base column, thereby causing every column (important: including across all column questions) to be tested for significance relative to the selected base column. To reset the base column back to each column question's Formatted Sample total, use the menu options Reset Base Column or select any column question's Formatted Sample Total column as the base column.

Test all columns concurrently - the Significance Across All Columns report

The Significance Across All Columns report is a powerful exploratory report that is traditionally performed during a first time review of a survey project. A multiple question column banner is typically defined representing questions of key interest. Examples would include questions that would likely have strong segmentation characteristics such as age categories, life stage or segmentations, and questions pertaining to the focus of the study such as the brand of the item or service purchased or brand of the retail location visited. The remaining key survey questions are then selected as rows against the pre-defined column banner and the columns are compared for significance against each other. This report can quickly identify the survey questions that display little or no significant difference between the key columns, and we may then conclude that there may be little to be gained from involving these questions in the further analysis of the study.

Starting with the tab displayed in the prior example, we can simply select the Significance Indicator dialog Significance Across All Columns option to change the display as shown in the screenshot below.


The letters displayed in place of the test statistic rows (e.g. A,D,E,G..) correspond to the letter suffix added to the question response text labels (e.g. Female (B)). The letters in the test statistic row identify the other columns within that column question that the column is significantly different from. In a single glance, the report captures the equivalent of successively selecting each column as the base column and keeping a running tally of the significant differences between the selected base column and all other columns. Note that if the spreadsheet view columns are resorted or hidden manually or by operations such as zero suppression or threshold, the tab will need to be rerun to redraw both the test statistic letters and corresponding column labels.

Layer, Column View Implications

The Significance Across All Columns report replaces the Significance Across Base Column dialog option Hide Test Values with the checkbox option Calculate Layer Significance across all Columns within each question. As implied, this checkbox option pertains exclusively to tabs formatted to include Layer columns and tests all columns in the spreadsheet against one another when checked. When unchecked, the comparison is limited to the Layer question response columns tiered beneath each individual column question response.

Finally, please note that the colorized spreadsheet views as shown in the screenshots above can be readily exported to Excel or HTML formats using mTAB™'s File Export menu items.