10. Significance Tests and Statistical Inference

Formulating Hypotheses and Test Direction

Significance testing uses sample data to evaluate a claim about a population. The central question is whether the observed result would be unusual if a specified baseline claim were true. The procedure assesses the strength of evidence against a null model; it does not prove a hypothesis.

A hypothesis must concern a population parameter rather than only a sample statistic. Common parameters include:

• A population proportion, $p$.

• A population mean, $\mu$.

• A difference between two population proportions, $p_1-p_2$.

• A difference between two population means, $\mu_1-\mu_2$.

The null hypothesis contains equality and gives the reference value. The alternative hypothesis states the direction or form of departure that would support the research claim.

For example, to ask whether a population proportion differs from $0.50$, use $H_0:p=0.50$ and $H_a:p\ne0.50$. To ask whether it exceeds $0.50$, use $H_a:p>0.50$. To ask whether a population mean is less than $70$, use $H_0:\mu=70$ and $H_a:\mu<70$.

The alternative must be chosen before examining the results. A two-sided alternative is appropriate when departures in either direction matter; a one-sided alternative is appropriate when only one direction is relevant.

Takeaway: Define the population parameter, include equality in the null hypothesis, and choose the alternative direction in advance.

The Significance-Test Process

A significance test follows a connected sequence of decisions:

1. State the null and alternative hypotheses in terms of the population parameter.

2. Calculate a test statistic that measures the distance between the sample result and the null value in standard-error units.

3. Find and interpret the p-value using the appropriate reference distribution and the direction of the alternative hypothesis.

4. Compare the p-value with the selected significance level $\alpha$.

5. State a conclusion about the population parameter in the context of the problem.

The general structure is:

A large test statistic in magnitude indicates that the sample result is far from the null value relative to the expected sampling variability. The exact meaning of “large” depends on the test and whether the alternative is left-tailed, right-tailed, or two-sided.

The decision rule is:

• If the p-value is less than or equal to $\alpha$, reject $H_0$.

• If the p-value is greater than $\alpha$, fail to reject $H_0$.

“Fail to reject” is more accurate than “accept.” A nonsignificant result may occur because the null is reasonable, but it may also reflect a small sample, high variability, or low-quality measurements.

Takeaway: A decision based on a p-value must be followed by a contextual conclusion, not just the words “significant” or “not significant.”

Testing a Population Proportion

For a one-sample test of a population proportion, let $\hat p$ be the sample proportion, let $p_0$ be the null value, and let $n$ be the sample size. The standard normal statistic is:

The standard error uses $p_0$, not $\hat p$, because the test evaluates the sampling distribution that would result if the null hypothesis were true.

For example, suppose $120$ of $200$ randomly sampled people support a policy. Then $\hat p=120/200=0.60$. To test whether more than half of the population supports it, use $H_0:p=0.50$ and $H_a:p>0.50$. The statistic is:

For a right-tailed test, this gives a p-value of about $0.002$. With $\alpha=0.05$, reject $H_0$. The conclusion is that the sample provides statistically significant evidence that more than $50\%$ of the population supports the policy.

For the usual normal approximation, the sample should be random or representative, and the expected counts under the null should commonly satisfy:

If these conditions fail, an exact or simulation-based method may be more appropriate.

Takeaway: In a one-proportion test, standardize the sample proportion using the null proportion and check the expected-count conditions.

Testing a Population Mean

For a one-sample test of a population mean when the population standard deviation is unknown, let $\bar x$ be the sample mean, $s$ the sample standard deviation, and $\mu_0$ the null value. The statistic is:

with $n-1$ degrees of freedom. The $t$-distribution accounts for the extra uncertainty from estimating the population standard deviation with $s$.

Suppose a random sample of $25$ customer-support calls has mean $72$ seconds and standard deviation $10$ seconds. To test whether the population mean differs from $70$ seconds, use $H_0:\mu=70$ and $H_a:\mu\ne70$. Then:

The test has $24$ degrees of freedom and a two-sided p-value of approximately $0.33$. At $\alpha=0.05$, fail to reject $H_0$. The sample does not provide sufficient evidence that the population mean call time differs from $70$ seconds. This does not prove that the true mean is exactly $70$; it means the observed difference is not unusual enough, given the variability and sample size, to provide strong evidence against the null hypothesis.

Observations should be independent, and the sampling method should be random or reasonably representative. For small samples, the population distribution should be approximately normal and free of extreme outliers. Larger samples generally provide more robustness, although severe skewness or outliers can still cause problems.

Takeaway: A one-mean test uses a $t$-statistic when the population standard deviation is unknown and requires attention to independence, sampling, and distributional conditions.

P-Values, Significance Levels, and Errors

A p-value describes how compatible the observed data are with the null model. A p-value of $0.03$ means that, if the null hypothesis were true and the sampling process were repeated under the stated assumptions, results at least as extreme as the observed result would occur about $3\%$ of the time.

A p-value does not represent:

• The probability that the null hypothesis is true.

• The probability that the alternative hypothesis is true.

• The probability that the result occurred through “random chance alone.”

• The size or practical importance of the effect.

A small p-value provides evidence against the null model, but it does not establish that an effect is large, useful, causal, or scientifically important. Conversely, a large p-value does not prove the null hypothesis; it indicates that the data do not provide sufficiently strong evidence against it under the chosen procedure.

The significance level $\alpha$ is selected before examining the data. It is the maximum tolerated probability of a Type I error, which occurs when a true null hypothesis is rejected. Common levels include $0.10$, $0.05$, and $0.01$, but $0.05$ is a convention rather than a universal requirement.

A Type II error occurs when a false null hypothesis is not rejected. Its probability is denoted by $\beta$. Power is the probability of correctly rejecting a false null hypothesis, so power equals $1-\beta$. Power generally increases with a larger sample, a larger true effect, lower variability, or a less stringent significance level. Reducing $\alpha$ generally makes Type I errors less likely but can make Type II errors more likely, all else equal.

Takeaway: Interpret the p-value under the null model, and consider both kinds of decision error when planning or evaluating a test.

From Statistical Evidence to Practical Meaning

Statistical significance and practical significance answer different questions. Statistical significance asks whether the data provide evidence that an effect differs from the null value relative to sampling variability. Practical significance asks whether the size of the effect matters in the real application.

A very large sample can make a tiny difference statistically significant. For example, a reduction of $0.2$ seconds in processing time among $50{,}000$ employees might produce a very small p-value, while having little effect on cost, safety, or productivity. Conversely, a meaningful effect in a small or highly variable sample may fail to reach statistical significance.

A confidence interval complements a significance test by showing a range of plausible parameter values. It provides information about the direction, possible size, and precision of an effect that a p-value alone does not provide. For a two-sided test at $\alpha=0.05$, a corresponding 95% confidence interval generally excludes the null value when the test rejects $H_0$, and includes the null value when the test fails to reject it.

When assessing practical importance, ask whether the entire confidence interval is above a meaningful benefit threshold, below a meaningful harm threshold, or includes effects that are too small to matter. Report the estimated effect, an uncertainty measure such as a confidence interval, the p-value when appropriate, and the real-world context.

Takeaway: Statistical evidence should be interpreted alongside effect size, uncertainty, study design, sample size, and the consequences of the decision.

Writing a Complete Conclusion

A complete conclusion connects the calculation to the population and the real question. Include these four elements:

1. The decision: reject or fail to reject $H_0$.

2. The evidence: report or characterize the p-value relative to $\alpha$.

3. The population parameter and the direction of the conclusion.

4. The practical meaning and any important limitations.

For the policy example, a complete conclusion is:

Because the p-value of $0.002$ is less than $\alpha=0.05$, we reject $H_0$. The sample provides statistically significant evidence that the population proportion supporting the policy exceeds $0.50$. The estimated proportion is $0.60$, so the effect may also be practically meaningful, although its importance depends on the policy decision and the quality of the sampling process.

Avoid statements such as “the null hypothesis is proven false,” “the null hypothesis is true,” or “there is a $2\%$ chance that the null hypothesis is true.” These statements misinterpret the logic of hypothesis tests and p-values.

Use this final check:

• Did the hypotheses concern a population parameter?

• Was the alternative direction chosen before viewing the results?

• Were the assumptions and conditions considered?

• Was the correct test statistic and reference distribution used?

• Was the p-value interpreted as a probability under $H_0$?

• Were statistical and practical significance distinguished?

• Was the conclusion stated in context?

Takeaway: A strong conclusion reports both what the data support statistically and what the estimated effect means in practice.