Studying a population

Population distribution (describing possible outcomes and their frequencies) encodes everything we could be interested in.

Sampling variability

Decision making under uncertainty

Population characteristics

Sampling variability

The signal and the noise

Can you spot the differences?

Information accumulates

Histograms of data from uniform (top) and non-uniform (bottom) distributions with increasing sample sizes.

The general recipe of hypothesis testing

Hypothesis tests versus trials

How to assess evidence?

statistic = numerical summary of the data.

requires benchmark / standardization

typically a unitless quantity

need measure of uncertainty of statistic

General construction principles

Wald statistic

\begin{align*} W = \frac{\text{estimated qty} - \text{postulated qty}}{\text{std. error (estimated qty)}} \end{align*}

standard error = measure of variability (same units as obs.)

resulting ratio is unitless!

Impact of encouragement on teaching

From Davison (2008), Example 9.2

In an investigation on the teaching of arithmetic, 45 pupils were divided at random into five groups of nine. Groups A and B were taught in separate classes by the usual method. Groups C, D, and E were taught together for a number of days. On each day C were praised publicly for their work, D were publicly reproved and E were ignored. At the end of the period all pupils took a standard test.

Basic manipulations in R: load data

data(arithmetic, package = "hecedsm")
# categorical variable = factor
# Look up data
str(arithmetic)Copy Code

## 'data.frame':    45 obs. of  2 variables:
##  $ group: Factor w/ 5 levels "control 1","control 2",..: 1 1 1 1 1 1 1 1 1 2 ...
##  $ score: num  17 14 24 20 24 23 16 15 24 21 ...Copy Code

Basic manipulations in R: summary statistics

# compute summary statistics
summary_stat <-
  arithmetic |> 
  group_by(group) |>
  summarize(mean = mean(score),
            sd = sd(score))
knitr::kable(summary_stat, 
             digits = 2)Copy Code

Basic manipulations in R: plot

# Boxplot with jittered data
ggplot(data = arithmetic,
       aes(x = group,
           y = score)) +
  geom_boxplot() +
  geom_jitter(width = 0.3, 
              height = 0) +
  theme_bw()Copy Code

Formulating an hypothesis

Let \(\mu_{C}\) and \(\mu_{D}\) denote the population average (expectation) score for praise and reprove, respectively.

Our null hypothesis is $$\mathscr{H}_0: \mu_{C} = \mu_{D}$$ against the alternative \(\mathscr{H}_a\) that they are different (two-sided test).

Equivalent to \(\delta_{CD} = \mu_C - \mu_D = 0\).

Test statistic

The value of the Wald statistic is $$t=\frac{\widehat{\delta}_{CD} - 0}{\mathsf{se}(\widehat{\delta}_{CD})} = \frac{4}{1.6216}=2.467$$

How 'extreme' is this number?

Assessing evidence

Possible, but not plausible

P-value

Level = probability of condemning an innocent

What is really a p-value?

The American Statistical Association (ASA) published a statement on (mis)interpretation of p-values.

(2) P-values do not measure the probability that the studied hypothesis is true

(3) Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.

(4) P-values and related analyses should not be reported selectively

(5) P-value, or statistical significance, does not measure the size of an effect or the importance of a result

Reporting results of a statistical procedure

Nature's checklist

Pairwise differences and t-tests

The p-values and confidence intervals for pairwise differences between groups \(j\) and \(k\) are based on the t-statistic:

\begin{align*} t = \frac{\text{estimated} - \text{postulated difference}}{\text{uncertainty}}= \frac{(\widehat{\mu}_j - \widehat{\mu}_k) - (\mu_j - \mu_k)}{\mathsf{se}(\widehat{\mu}_j - \widehat{\mu}_k)}. \end{align*}

In large sample, this statistic behaves like a Student-t variable with \(n-K\) degrees of freedom, denoted \(\mathsf{St}(n-K)\) hereafter.

Pairwise differences

Consider the pairwise average difference in scores between the praise (group C) and the reprove (group D) of the arithmetic data.

• Group sample averages are \(\widehat{\mu}_C = 27.4\) and \(\widehat{\mu}_D = 23.4\)
• The estimated average difference between groups \(C\) and \(D\) is \(\widehat{\delta}_{CD} = 4\)
• The estimated pooled standard deviation for the five groups is \(1.15\vphantom{\widehat{\delta}_{CD}}\)
• The standard error for the pairwise difference is \(\mathsf{se}(\widehat{\delta}_{CD}) = 1.6216\)
• There are \(n=45\) observations and \(K=5\) groups

t-tests: null distribution is Student-t

If we postulate \(\delta_{jk} = \mu_j - \mu_k = 0\), the test statistic becomes

\begin{align*} t = \frac{\widehat{\delta}_{jk} - 0}{\mathsf{se}(\widehat{\delta}_{jk})} \end{align*}

The \(p\)-value is \(p = 1- \Pr(-|t| \leq T \leq |t|)\) for \(T \sim \mathsf{St}_{n-K}\).

• probability of statistic being more extreme than \(t\)

Recall: the larger the values of the statistic \(t\) (either positive or negative), the more evidence against the null hypothesis.

Critical values

For a test at level \(\alpha\) (two-sided), we fail to reject null hypothesis for all values of the test statistic \(t\) that are in the interval

$$\mathfrak{t}_{n-K}(\alpha/2) \leq t \leq \mathfrak{t}_{n-K}(1-\alpha/2)$$

Because of the symmetry around zero, \(\mathfrak{t}_{n-K}(1-\alpha/2) = -\mathfrak{t}_{n-K}(\alpha/2)\).

• We call \(\mathfrak{t}_{n-K}(1-\alpha/2)\) a critical value.
• in R, the quantiles of the Student t distribution are obtained from qt(1-alpha/2, df = n - K) where n is the number of observations and K the number of groups.

Null distribution

The blue area defines the set of values for which we fail to reject null \(\mathscr{H}_0\).

All values of \(t\) falling in the red area lead to rejection at level \(5\)%.

Example

• If \(\mathscr{H}_0: \delta_{CD}=0\), the \(t\) statistic is $$t=\frac{\widehat{\delta}_{CD} - 0}{\mathsf{se}(\widehat{\delta}_{CD})} = \frac{4}{1.6216}=2.467$$
• The \(p\)-value is \(p=0.018\).
• We reject the null at level \(\alpha=5\)% since \(0.018 < 0.05\).
• Conclude that there is a significant difference at level \(\alpha=0.05\) between the average scores of subpopulations \(C\) and \(D\).

Confidence interval

Interpretation of confidence intervals

The reported confidence interval is of the form

$$ \text{estimate} \pm \text{critical value} \times \text{standard error}$$

confidence interval = [lower, upper] units

If we replicate the experiment and compute confidence intervals each time

• on average, 95% of those intervals will contain the true value if the assumptions underlying the model are met.

Interpretation in a picture: coin toss analogy

Why confidence intervals?

Test statistics are standardized,

• Good for comparisons with benchmark
• typically meaningless (standardized = unitless quantities)

Two options for reporting:

• \(p\)-value: probability of more extreme outcome if no mean difference
• confidence intervals: set of all values for which we fail to reject the null hypothesis at level \(\alpha\) for the given sample

Example

• Mean difference of \(\widehat{\delta}_{CD}=4\), with \(\mathsf{se}(\widehat{\delta}_{CD})=1.6216\).
• The critical values for a test at level \(\alpha = 5\)% are \(-2.021\) and \(2.021\)
  • qt(0.975, df = 45 - 5)
• Since \(|t| > 2.021\), reject \(\mathscr{H}_0\): the two population are statistically significant at level \(\alpha=5\)%.
• The confidence interval is $$[4-1.6216\times 2.021, 4+1.6216\times 2.021] = [0.723, 7.277]$$

The postulated value \(\delta_{CD}=0\) is not in the interval: reject \(\mathscr{H}_0\).

Pairwise differences in R

library(emmeans) # marginal means and contrasts
model <- aov(score ~ group, data = arithmetic)
margmeans <- emmeans(model, specs = "group")
contrast(margmeans, 
         method = "pairwise",
         adjust = 'none', 
         infer = TRUE) |>
  as_tibble() |>
  filter(contrast == "praise - reprove") |>
  knitr::kable(digits = 3)Copy Code

Recap 1

Recap 2

Recap 3