6 Hypothesis Testing

Formulating hypotheses

Hypothesis testing uses sample data to evaluate a claim about a population parameter, such as a mean, proportion, or difference between groups. It assesses whether the data are sufficiently inconsistent with a specified null hypothesis; it does not prove a claim true or false.

The null hypothesis $H_0$ represents a benchmark or no-effect claim. The alternative hypothesis, written $H_a$ or $H_1$, represents the effect or difference being investigated. Both hypotheses concern a population parameter, not a sample statistic, and the equality belongs in the null hypothesis.

For example, to ask whether a revised checkout reduces mean service time below $4$ minutes, let $\mu$ be the population mean service time after the revision:

• $H_0: \mu = 4$ minutes

• $H_a: \mu < 4$ minutes

A one-sided alternative tests a specified direction. A two-sided alternative tests for any difference, such as $H_a: \mu \ne 4$. Choose the direction before examining results, based on the question and the consequences of errors.

Conducting a test

A standard test follows an ordered process:

1. Define the business question, the population parameter, and the hypotheses.

2. Choose a significance level $\alpha$ in advance and select an appropriate test for the parameter, data, and design.

3. Check the test's assumptions, including whether observations are independent and whether the method's distributional conditions are reasonable.

4. Use the sample to calculate a test statistic, which measures how far the result is from what the null hypothesis predicts.

5. Find the p-value and compare it with $\alpha$. Reject $H_0$ if $p \leq \alpha$; otherwise, fail to reject $H_0$.

6. State the conclusion in context and consider both the estimated effect and its practical importance.

Choosing the test, its direction, and the significance level before evaluating results helps make the decision process explicit. A failure to reject is not proof that the null hypothesis is true; the data may be inconclusive.

Interpreting the p-value

The p-value is calculated under the assumption that the null hypothesis and the test assumptions are true. It is the probability of observing a test statistic at least as extreme as the one obtained, in the direction or directions specified by the alternative.

A p-value is not the probability that the null hypothesis is true, and it is not the probability that chance alone caused the result. It measures how extreme the observed result is under the null model.

For example, suppose a company tests whether a new checkout process lowers average service time from $4$ minutes. If the test gives $p = 0.032$ and the prespecified significance level is $\alpha = 0.05$, then $0.032 < 0.05$, so the null hypothesis is rejected. The sample provides statistically significant evidence that mean service time is lower.

Significance and practical importance

The significance level $\alpha$ is the test's chosen maximum probability of a Type I error under the null hypothesis. A result is called statistically significant when its p-value is at or below this threshold.

Statistical significance is a decision convention, not a measure of business value. A small effect can be statistically significant, especially with a large sample, while a consequential effect may not reach significance in a small or noisy sample.

In the checkout example, rejecting the null does not establish how large the reduction is or whether it is worth the implementation cost. The estimated reduction and its uncertainty should also be examined.

Errors and power

Because a test uses sample data, its decision can be wrong:

• Type I error: Rejecting a true null hypothesis, also called a false positive. Under the test conditions, its long-run rate is controlled by $\alpha$.

• Type II error: Failing to reject a false null hypothesis, also called a false negative. Its probability is denoted by $\beta$.

The probability of rejecting the null hypothesis when a specified alternative is true is the test's power, $1-\beta$. Power generally increases with larger samples, less variability, and larger true effects.

Lowering $\alpha$ can reduce Type I errors but, all else equal, can also make real effects harder to detect. Test decisions therefore involve risks of both false positives and false negatives.