Tests of Statistical Significance

In statistical analysis there are two concepts that are critical for understanding the relationship between two variables. One is the strength of association, which will be covered in the the future. The other concept is statistical significance. Statistical significance is measured by a probability value (p-value), which tells you the chances that the relationship the test identified is due to a Type I error.

When we run statistical procedures, there are two possible errors that can be made (due to sampling or random error). A Type I error is when we reject a null hypothesis (accept the alternative) when in fact the null is correct. This error is a big deal because we are accepting a hypothesis that we really should reject. The p-value of a test tells us the chance of this happening. Most social sciences use .05 (some are more stringent and require .01 or less) as an alpha level; that is for a test to be considered statistically significant, the p-value must be less than .05. This means that we will make a Type I error less than 1 in 20 times. In layman's terms, the p-value tells us how sure we are that the relationship is actually present in the world.

A Type II error is when we accept a null hypothesis, but in fact we should reject it and assume the alternative. This is less of a problem because we aren't assuming anything to be true that really isn't. Our data may not have identified a relationship, but additional testing might, so additional testing is the simple solution to a Type II error.

A good way to think of this is to apply it to our judicial system. A man is on trial, so our (alternative) hypothesis is that he is guilty. Our null hypothesis is that he is innocent (innocent until proven guilty). In order to prove him guilty (accept our hypothesis), we must show that beyond a reasonable doubt (exceed the p-value). Thus, we can make four conclusions, two of which are errors. The correct conclusions we can make are to convict a guilty man or acquit an innocent man. The errors we can make are to acquit a guilty man (Type II error) or convict an innocent man (Type I error). In our justice system the latter is a bigger problem, so we seek to minimize that error. The same is true of scientific testing. Unless we can show something beyond a reasonable doubt (p-value less than .05), we must acquit (assume the null).
-Randy Owen