---
title: "Getting started with lmmresample"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with lmmresample}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---



*This vignette is precomputed: the results shown were obtained with the code
displayed, which takes a few minutes to run.*

`lmmresample` gives permutation tests and bootstrap confidence intervals for
mixed models (`lme4::lmer()`) and linear models (`lm()`) fitted to repeated
measurements organised in blocks: samples within trials, trials within
participants. Resampling always moves whole blocks and refits the model you
already have, so the dependence within blocks is carried into the reference
distribution instead of having to be modelled.

This vignette works through a trial-level analysis: one value per trial, two
conditions, every participant seeing both. The vignette
`vignette("time-courses")` covers full time series, and
`vignette("multiple-comparisons")` covers families of tests and pairwise
comparisons.


```r
library(lmmresample)
library(lme4)
```

## Example data

`sim_blocks()` simulates block-structured data with known effects. Here, 24
participants complete 40 trials each in conditions A and B; condition B
increases the outcome by 0.3 on average, and participants differ in the size
of that effect (`sd_slope`).


```r
d <- sim_blocks("within", level = "trial", n_participants = 24, n_trials = 40,
                effect_condition = 0.3, sd_slope = 0.4, seed = 1)
head(d[c("participant", "trial", "trial_index", "condition", "y")])
#>   participant   trial trial_index condition            y
#> 1         p01 p01_t01           1         A -0.123122022
#> 2         p01 p01_t02           2         B  0.310535946
#> 3         p01 p01_t03           3         B -0.925765442
#> 4         p01 p01_t04           4         B -0.304508300
#> 5         p01 p01_t05           5         B -0.005405251
#> 6         p01 p01_t06           6         B  1.540625229
```

## Why Wald tests can mislead

A common analysis fits a random intercept for each participant and reads the
Wald (t) test of the condition coefficient:


```r
m <- lmer(y ~ condition + (1 | participant), data = d)
summary(m)$coefficients
#>              Estimate Std. Error  t value
#> (Intercept) 0.7194377 0.19870672 3.620601
#> conditionB  0.3982002 0.04152217 9.590063
```

The standard error assumes that the condition effect is the same in every
participant. Because it is not, the 960 trials carry much less independent
information about the mean effect than the model assumes, and the test is
too liberal. Adding a random slope (the maximal model) fixes this here, but in
richer designs the maximal model often fails to converge or is singular, and
in time-series data even the maximal model is not enough (see
`vignette("time-courses")`).

## A permutation test on the same model

`perm_test()` keeps the model as it is and builds the null distribution of the
statistic by resampling. Two arguments describe the design:

* `unit`: the column identifying what is permuted, here the trial;
* `exchange`: which units may be exchanged. `exch_signflip("participant")`
  swaps the condition labels of all trials of a random subset of participants,
  which reverses their individual effects. This tests whether the *mean*
  effect is zero while allowing participants to differ in their effect.


```r
pt <- perm_test(m, "condition", unit = "trial",
                exchange = exch_signflip("participant"), B = 1999, seed = 1)
pt
#> 
#> Permutation test for `condition` (coefficient `conditionB`)
#> 
#> Method:    relabelling of units
#> Exchange:  labels of `trial` swapped (sign-flipped) by `participant` (960 units in 24 blocks)
#> Statistic: t = 9.590
#> p-value:   < 0.001 (two-sided)
#> Resamples: 1999 of 1999 permutations used
```


```r
plot(pt)
```

![plot of chunk perm-plot](figures/start-perm-plot-1.png)

The other exchangeability declarations are `exch_within("participant")`,
which permutes trials within participants and tests the sharp null of no
effect in anyone, and `exch_free()`, which permutes freely and suits
between-group designs (e.g. `unit = "participant"` with group membership as
the tested term). `exch_free(strata = "site")` restricts permutation to
within strata.

The test refits the full model for every permutation, so it can take some
time with large data sets. Computation runs in parallel through the
[future](https://future.futureverse.org/) framework:


```r
future::plan("multisession", workers = 4)
```

Results are reproducible for a given `seed` whatever the number of workers.

## Bootstrap confidence intervals

`boot_ci()` resamples whole participants (case bootstrap) and refits the model
to each resample. The default interval is the *expanded* percentile interval,
which corrects the percentile interval for the small number of participants
typical of experimental studies. Here it is computed for the maximal model:


```r
mf <- lmer(y ~ condition + (1 + condition | participant), data = d)
b <- boot_ci(mf, cluster = "participant", B = 1999, seed = 1)
b
#> 
#> Bootstrap confidence intervals (case bootstrap of `participant`, 24 clusters)
#> 
#> 95% expanded percentile intervals
#>  Coefficient Estimate Lower Upper
#>  conditionB  0.398    0.262 0.538
#> 
#> Replicates: 1999 of 1999 used
```

Other intervals are available through `ci` (`"percentile"`, `"bca"`,
`"basic"`), and `resample = "wild"` gives a wild bootstrap of participants.
With groups of participants (a between-group design), `strata = "group"` keeps
the size of each group fixed in every resample.

## Adjusting for covariates

When the model contains other terms, such as trial order, the tested term is
permuted through the residuals of the reduced model (the Freedman-Lane
procedure), which keeps the covariate effects in place. `method = "auto"`, the
default, chooses this automatically:


```r
d$order <- as.numeric(scale(d$trial_index))
m_order <- lmer(y ~ condition + order + (1 | participant), data = d)
pt_order <- perm_test(m_order, "condition", unit = "trial",
                      exchange = exch_signflip("participant"),
                      B = 1999, seed = 1)
pt_order
#> 
#> Permutation test for `condition` (coefficient `conditionB`)
#> 
#> Method:    relabelling of units
#> Exchange:  labels of `trial` swapped (sign-flipped) by `participant` (960 units in 24 blocks)
#> Statistic: t = 9.633
#> p-value:   < 0.001 (two-sided)
#> Resamples: 1999 of 1999 permutations used
```

## Diagnostics

Two quick checks help interpret a result. `diag_agreement()` compares the
model coefficient with the mean of the participant-level differences; a large
discrepancy suggests that the model weights participants in an unexpected way
or that a confound is at work:


```r
diag_agreement(m, "condition", cluster = "participant")
#> 
#> Agreement for `condition` (B - A)
#> 
#> Model coefficient:              0.398
#> Mean participant-level difference: 0.398 (SE 0.068, 24 participants)
#> Discrepancy:                    0.000 (0%)
#> 
#> The model coefficient agrees with the participant-level differences.
```

`diag_loso()` refits the model leaving out one participant at a time and
reports how much each participant moves the estimate:


```r
lo <- diag_loso(mf, cluster = "participant")
lo
#> 
#> Leave-one-participant-out influence (24 refits)
#> 
#>  Coefficient Estimate Range          Most influential
#>  conditionB  0.398    0.361 to 0.428 p07 (-0.55 SE)
```


```r
plot(lo)
```

![plot of chunk loso-plot](figures/start-loso-plot-1.png)

## Checking the test for your design

`perm_calibrate()` simulates data from the fitted model with the effect
removed and reports how often the permutation and Wald tests reject at the 5%
level. It checks that the chosen exchangeability gives a valid test for a
given design:


```r
cal <- perm_calibrate(m, "condition", unit = "trial",
                      exchange = exch_signflip("participant"),
                      n_sim = 100, B = 99, seed = 1)
cal
#> 
#> Calibration of the permutation test for `condition`
#> 
#> Null:        mean effect zero, participant variation kept
#> Residuals:   independent
#> Exchange:    labels of `trial` swapped (sign-flipped) by `participant`
#> Simulations: 100, with 99 permutations each
#> 
#> Rejection rate at alpha = 0.05:
#>   Permutation  0.050  [0.016, 0.113]  consistent with alpha
#>   Wald         0.030  [0.006, 0.085]  consistent with alpha
#> 
#> Uniformity of permutation p-values (Kolmogorov-Smirnov): p = 0.178
```

Both tests hold the nominal level here. The simulated data reproduce the
structure of the fitted model, which has independent residuals and no
participant variation in the effect, and under that structure the Wald test
is valid too. The calibration reflects only the dependence included in the
simulation: for time series, `ar1` adds the autocorrelation of the residuals,
and `vignette("time-courses")` shows how differently the two tests then
behave.

## Reporting

Every result has a `tidy()` method returning a data frame, convenient for
tables:


```r
tidy(pt)
#>        term       coef statistic df p.value alternative    B B_used
#> 1 condition conditionB  9.590063  1   5e-04   two.sided 1999   1999
#>                  method
#> 1 permutation (relabel)
tidy(b)
#>         term  estimate  conf.low conf.high     type level
#> 1 conditionB 0.3982002 0.2622967 0.5376454 expanded  0.95
```

For reporting, use at least the default number of resamples (`B = 4999`).
