Guide for Rater Agreement under Structural Skew.
Rater-reliability coefficients for binary outcomes are routinely
interpreted using fixed labels from published scales, but those labels
drift with prevalence, rater count, sample size, and latent rater
asymmetry. The same panel can land in different conventional categories
just because prevalence changes. grassr replaces fixed
cutoffs with context-conditioned surface-position
reporting: the rating matrix goes in, and a four-field
Report Card comes out — sample summary, primary
coefficient with calibrated percentile, cross-coefficient asymmetry
diagnostic, and (when the panel disagrees with itself) a per-rater
latent-class fit.
library(grassr)
set.seed(29)
# Five raters, 200 subjects, Se = Sp = 0.85, prevalence 0.30.
truth <- rbinom(200, 1, 0.30)
Y <- sapply(1:5, function(j) {
ifelse(truth == 1, rbinom(200, 1, 0.85), rbinom(200, 1, 0.15))
})
grass_report(ratings = Y)GRASS Report Card
sample = 5 raters, N = 200, pi_hat = 0.35
PABAK = 0.41 -> 62nd percentile | quality 0.78-0.85 <- primary
AC1 = 0.46 -> 62nd percentile | quality 0.77-0.85
Fleiss kappa = 0.36 -> 62nd percentile | quality 0.77-0.85
ICC = 0.46 -> 61st percentile | quality 0.76-0.85 [distribution-sensitive]
read: this panel agreed more tightly than 62% of what panels at this
design can produce; the data are consistent with panel quality 0.78-0.85.
delta = 0.01 pp implied-quality spread (aligned)
matched null = (k=5, N=200, q=0.85): delta_hat at the 19.1 percentile [design snapped]
See `summary(...)` for full panel and CI details.
See `plot(...)` for a surface-position visualization.The percentile is computed against a reference surface calibrated for
the study’s (k, N, pi_hat, tau2_hat) — not against a fixed
1977 cutoff table. The aligned flag
(delta = 1.2 pp) means the panel of unclamped coefficients
agrees about where the study lands, so a single number is safe to cite.
The clamp note in this example is for ICC, whose observed value
(0.4642) sits above the achievable maximum of its reference
surface at this (k, N); ICC is therefore excluded from
delta_hat per the clamping policy (see vignette section 7).
ICC also carries a [distribution-sensitive] marker because
its reference surface depends on the full subject-prevalence
distribution F rather than on (q, pi_+) alone, and is
excluded from delta_hat by construction regardless of clamp
status. At the divergent tier the headline is replaced with the
per-rater sensitivity / specificity table from a latent-class fit (point
estimates at k >= 3, bounded estimates at
k = 2).
The full walkthrough — the easy case, the killer case, layered access
via summary() / as.data.frame() /
plot(), granular building blocks
(position_on_surface(), check_asymmetry(),
latent_class_fit()), and the two-rater branch — is in vignette("grassr").
remotes::install_github("defense031/grassr")
# from the package directory:
devtools::load_all()
# the command above installs the repository default branch (development
# head); the CRAN submission line is the 0.7.x series.grass_report(ratings = Y) — single-call workflow returning
a four-field Report Card object (grass_card) with
print / summary / as.data.frame /
plot methods. Seven plot() view types:
surface (default), panel,
thermometer, intervals,
per_rater, pairwise,
diagnostic.position_on_surface(ratings = Y, metric = ...) places one
observed coefficient on its context-conditioned reference surface and
returns the pooled percentile (the coefficient’s position within the
design’s achievable agreement range) together with a 95% test-inversion
consistency band on panel quality – the quality levels whose sampling
distributions are consistent with the observed value at this design. The
pooled percentile is the categorical score; no four-way labeled band is
interposed between the percentile and the reader. Closed-form references
for PABAK, AC1, Fleiss kappa; bundled fitted-ICC reference for ICC
across
k in {3, 5, 8, 15, 25} x N in {30, 50, 75, 100, 200, 300, 500, 1000}
on a 14-point q-grid.check_asymmetry(ratings = Y) returns the cross-coefficient
implied-quality spread delta_hat (in pp of quality)
computed over the three-coefficient agreement family (PABAK, AC1, Fleiss
kappa) plus an aligned / caution / divergent flag set by
delta_hat’s percentile on the matched (k, N, q_hat) null
distribution (>= 95th caution, >= 99th divergent). ICC is reported
on the panel with a [distribution-sensitive] marker but
does not enter delta_hat by construction, because its
reference surface depends on the full subject-prevalence distribution F
rather than on (q, pi_+) alone.pairwise_agreement() returns a pairwise PABAK matrix on the
k = 2 reference surface plus per-rater pooled-reference
(Se_tilde, Sp_tilde) against the panel-majority of the
other k - 1 raters. Auto-triggered from
grass_report() under the divergent flag.latent_class_fit()
returns per-rater Dawid-Skene (Se_hat, Sp_hat) point
estimates at k >= 3 and Hui-Walter bounds at
k = 2, with bootstrap CIs.plot_surface(metric, pi_hat, observed) renders the
closed-form reference surface as a heatmap with an optional pin marker
at the observed value; useful for “what does my reference surface look
like at this design, before I collect data?”See NEWS.md for the release history and
?grass_roadmap for planned future families (ordinal,
multi-rater nominal, continuous).
v0.7.1 (in development). The pooled-percentile + consistency-band
card redesign and the implied-quality delta_hat (see
NEWS.md); the CRAN submission line is the 0.6.2 series. Binary
inter-rater and intra-rater families fully implemented. For the
intra-rater axis, the inter-rater diagonal calibration is exact for the
idealized intra model (an equivalence proposition); drift and
within-subject dependence bounds are published rather than conditioned
on. See ?grass_roadmap for planned families.