Samples
and
populations

Probability
theory and statistical inferrence

Inferring a population
mean

Tests of
statistical significance

Tests of
differences between
means

Coefficient
of
correlation

Caution
with
association

Chi square

Other
tests of
significance

Caution in
using
statistical
test results

Aids

Probability and confidence levels

Accepting or rejecting the null hypothesis is a calculated gamble. This decision rests on assessing the probability or odds that favor one choice over the other. Over the years, statisticians and researchers have accepted two probability levels, referred to as "p" levels, as guides in making this decision. One, a p level of .05 indicates that the observed finding could have occurred due to chance variations in sample selection in less than 5 times out of 100. Researchers accept the .05 or 5% level of confidence as the minimum level for rejecting a null hypothesis. We assume, gamble if you like, that the result we found in our one sample is not one of the 5 instances that could have occurred due to chance. We therefore reject the null hypothesis. When a p level is greater than 5%, the null hypothesis is accepted. In this case, we are unwilling to gamble that what was found was not due to random sample variation. We say, in effect, that the odds or probability favor the result as having occurred due to chance and that there is no relationship or difference among variables in the population.

When results fail to reach the 5% level of confidence, they are reported as   nonsignificant   The short hand version of nonsignificant, which you will see in research reports, is "ns."

Some researchers prefer to use the .01 level of significance for accepting or rejecting null hypotheses. This criterion indicates that a finding could have occurred due to chance less then 1 time in 100. The .01 level is a stronger basis for rejecting a null hypothesis than the .05 level. Results at the .05 and .01 levels are referred to as being "significant." As you review literature, you may see reference to even higher levels of significance, such as the .005 or .001 levels of significance. The .001 level indicates that a finding could have occurred by chance only in 1 sample out of 1,000. Probability levels this low are described as being "highly significant."

Steps in conducting a statistical test of significance

Three steps are involved in conducting any statistical test, including the t test. These are:

• Formulate the null hypothesis and, if you did not do it before, do the same for the alternative hypothesis;
• Conduct the statistical test; and
• Interpret the results based on the critical value found.

The purpose of most research is to test for relationships among variables. Research can also focus on identifying differences between groups. Obviously, different tests are used for establishing relationships as compared with discovering differences between groups. Also, regardless of the hypothesis, different tests are used for continuous as opposed to discrete data.

Selecting the right statistical test can be confusing. Fortunately, a lot of easy-to-use help is available through the Internet. Selecting Statistics, is an excellent starting point. This site begins by asking you whether you are analyzing one variable, two variables, or more than two variables. It then asks you to answer additional questions which should lead you to select the right way to analyze your set of data. If this site does not provide the help you want, do a search on Google or some other search engine using search terms like "selecting statistical tests." You will find additional sites that provide help with selecting statistics.  

We will start by explaining how to conduct a test for the statistical significance of a difference between two means following the three steps listed above.

Tests of differences between means

Means are used to summarize distributions based on continuous data (interval or ratio measurement). A statistical measure called the t test is used to test for the significance of the difference between two means. The t test assesses the degree of overlap in the distribution of scores in each of two samples being compared. When the two distributions are highly similar, there will be little difference between the means. The t test, therefore, will produce a high p value, meaning that any difference found can be safely attributed to chance variations in the selection of the samples. When scores in one distribution are distributed differently from the other, there is a greater probability that the difference between the means will be greater. As the difference increases, the p value will be lower, making it less unlikely that any difference found was due to chance.

A t test can be used with large or small samples. However, as the sample size becomes smaller, mean differences have to be larger to become significant. In addition to the requirement of continuous measurement, the t test assumes that the variable being measured is normally distributed in the population from which the sample was selected. Even when distributions for samples are mildly skewed, it may be reasonable to assume a normal distribution for the variable in the population. However, when the distribution for a sample is badly skewed or you doubt that the variable is normally distributed in the population, you should not use a t test. As an alternative you can compare medians or convert continuous data to a set of intervals and conduct a chi square test. We describe how to do perform a chi square calculation later in this chapter.