5 Expectation and Variability

The Center: Expectation

Expectation summarizes the center of a probability distribution. It is a probability-weighted average and represents the long-run average over repeated observations.

For a discrete random variable $X$ with possible values $x$ and probability mass function $p_X(x)$,

For a continuous random variable with density $f_X(x)$,

provided the expectation is well-defined. The expected value does not have to be a value that the variable can actually take. For a fair six-sided die,

No individual roll equals $3.5$, but the average approaches this value over many rolls.

For a function $g(X)$, average the function values with the same probabilities:

in the discrete case, with the corresponding integral for a continuous variable.

Takeaway: Expectation describes the center or long-run average, not necessarily a directly observable outcome.

Adding Expected Values

Linearity of expectation makes totals easier to analyze. For constants $a$ and $b$,

More generally,

This rule holds whether or not the random variables are independent. For example, let $X_i$ indicate whether trial $i$ is a success:

If the success probability on trial $i$ is $p_i$, then $E[X_i]=p_i$. For the total number of successes $S=X_1+\cdots+X_n$,

The trials may be dependent; the expectation formula still applies.

Takeaway: Add expected contributions directly, and do not impose an unnecessary independence assumption.

Spread Around the Mean: Variance

Variance measures spread around the mean. If $\mu=E[X]$, its definition is

A useful computational form is

For a fair die, $E[X]=3.5$ and

Therefore,

Variance is never negative. It is zero exactly when the random variable is constant with probability one. Transformations behave as follows:

Adding a constant changes the center but not the spread. Multiplying by $a$ scales the variance by $a^2$.

Takeaway: Variance quantifies squared spread and must be interpreted in squared units.

Interpreting Standard Deviation

Standard deviation converts variance back to the original measurement scale:

For the fair die,

Unlike variance, standard deviation has the same units as the random variable. It is not necessarily the exact average distance from the mean, but it is a useful measure of typical spread. Interpret its size relative to the scale and units of the variable.

For independent measurements with standard deviations $3$ and $4$, the variances are $9$ and $16$. The standard deviation of their sum is found by adding variances first:

The standard deviations do not generally add.

Takeaway: Use variance for algebraic calculations, then take its square root to obtain an interpretable spread in the original units.

How Variables Move Together

Covariance describes how two variables move together relative to their means:

An equivalent form is

Positive covariance means that large values of $X$ tend to occur with large values of $Y$, while small values tend to occur together. Negative covariance means that large values of one variable tend to occur with small values of the other. Covariance near zero indicates little linear co-movement, but it does not rule out nonlinear dependence.

Variance is the special case

Scaling follows the rule

Adding constants has no effect, whereas multiplying the variables changes covariance according to the product of the scaling factors.

Independence implies zero covariance when the relevant expectations exist. However, zero covariance generally does not imply independence.

Takeaway: Covariance records joint linear movement, including whether co-movement increases or decreases the spread of a total.

Combining Uncertain Quantities

The variance of a sum depends on both individual variability and the relationships among components. For two variables,

Positive covariance increases the variance of the sum. Negative covariance reduces it. For a difference,

For constants $a$ and $b$,

If $X$ and $Y$ are independent, covariance is zero, so their variances add. For example, if $\operatorname{Var}(X)=9$, $\operatorname{Var}(Y)=16$, and $\operatorname{Cov}(X,Y)=2$, then

Takeaway: Never add standard deviations directly; account for covariance and add variances when independence permits it.

Many-Variable Sums

For a total $S=X_1+X_2+\cdots+X_n$, linearity gives

The variance includes every pairwise covariance:

If the variables are mutually independent, all covariance terms vanish:

If they are also identically distributed with mean $\mu$ and variance $\sigma^2$, then

Thus, the expected total grows proportionally to $n$, while the standard deviation of an independent total grows proportionally to $\sqrt{n}$.

Takeaway: Repeated independent contributions accumulate their means linearly but their standard deviations more slowly, at a square-root rate.

From Distributions to Data

Observed data use sample statistics to estimate or summarize the corresponding distributional quantities. For data $x_1,\ldots,x_n$, the sample mean is

The commonly used sample variance and sample standard deviation are

The denominator $n-1$ is used when the sample variance estimates the variance of a larger population. When describing an entire finite population rather than estimating a larger one, a denominator of $n$ is often used.

For paired observations $(x_i,y_i)$, the sample covariance is

These statistics help summarize center and spread, quantify uncertainty, analyze relationships, and evaluate risk. For example, a portfolio's expected return is a weighted sum of expected returns, while its variance depends on both individual asset variances and covariances. Assets with low or negative covariance can reduce overall variability.

Keep population quantities distinct from sample quantities: $\mu$, $\sigma^2$, and $\sigma$ describe a distribution or population, whereas $\bar{x}$, $s^2$, and $s$ are calculated from observed data.

Takeaway: Choose the statistic and denominator according to whether the goal is to describe the observed population or estimate a larger one.

Common Errors and Final Checklist

Several mistakes recur when working with expectation and variability.

1. Adding standard deviations instead of variances: For independent variables, variances add; standard deviations generally do not.

2. Assuming independence is needed for expected values to add: Linearity of expectation holds without independence.

3. Assuming zero covariance means independence: Independence implies zero covariance, but zero covariance does not generally imply independence.

4. Ignoring units: Variance has squared units, while standard deviation has the original units.

5. Confusing population parameters with sample statistics: $\mu$, $\sigma^2$, and $\sigma$ describe a distribution or population; $\bar{x}$, $s^2$, and $s$ are calculated from observed data.

Final takeaway: First identify whether the task concerns a center, individual spread, joint movement, or the variability of a total. Then select the corresponding expectation, variance, standard deviation, or covariance formula and check the assumptions, units, and denominator.