LLMRpanel administers survey and experimental instruments to panels of language model personas. It can be used to pretest questions and pilot conjoint designs. Responses describe a configured model under the supplied personas.
panel_benchmark() compares closed-item response shares
with benchmark shares supplied by the user. It records deviations for
the covered responses and nonresponse rates by item.
The examples use a deterministic response function through
.runner. The live example runs only when
RUN_LIVE is set to TRUE.
Instrument pretesting uses draft items to examine unmatched replies and first-option sensitivity.
Design piloting uses conjoint tasks to examine responses to recorded profile assignments.
Model measurement compares response distributions under different personas or model configurations. Displayed option order can also be varied.
library(LLMRpanel)
# Closed items receive the first displayed option. Open items receive a
# fixed sentence.
first_shown <- function(experiments, ...) {
experiments$response_text <- vapply(seq_len(nrow(experiments)), function(i) {
shown <- experiments$option_order[i]
if (is.na(shown)) "Because it reaches the most people."
else strsplit(shown, "|", fixed = TRUE)[[1]][1]
}, character(1))
experiments$success <- TRUE
experiments
}panel_from_margins() samples attribute values from the
supplied marginal distributions. set.seed() makes this draw
reproducible.
set.seed(110)
panel <- panel_from_margins(
list(
age = c("18 to 34" = .30, "35 to 64" = .45, "65 plus" = .25),
party = c(left = .45, right = .45, independent = .10)
),
n = 12,
persona_template = "A {age} year old voter who leans {party}."
)
panel
#> <silicon_panel | 12 persona(s) | attributes: age, party>
#> e.g. A 18 to 34 year old voter who leans left.
instrument <- panel_instrument(list(
item_likert("wk4", "A four-day work week would benefit society."),
item_choice(
"fund",
"Which should the city fund first?",
c("public transit", "road repair")
),
item_open(
"why",
"What should the city consider when choosing between public transit and road repair?"
)
))
instrument
#> <panel_instrument | 3 item(s) | randomized: option_order>
#> [wk4] (likert) A four-day work week would benefit society.
#> [fund] (choice) Which should the city fund first?
#> [why] (open) What should the city consider when choosing between public transit and road repair?Margins are useful when targets are published as tables. When
microdata is available, panel_from_data() is the joint
distribution counterpart. It draws personas from observed rows and
therefore preserves relationships among attributes rather than sampling
each margin independently. LLMR::report() identifies
whether a panel came from supplied margins, microdata rows, or supplied
personas.
panel_administer() sends each persona-item pair as an
independent request. It returns a panel_responses object.
Response rows are stored in $data; the panel and instrument
are retained in separate fields.
The default instrument settings permute choice options for each
request. A Likert scale retains its original order or is reversed, each
with probability one-half. option_order records the
displayed options. item_position records the item’s fixed
position in the instrument, rather than an order of exposure.
When comparing models, reset the seed before each administration to reproduce the same option assignments.
cfg = LLMR::llm_config("groq", "openai/gpt-oss-20b", temperature = 0.8)
cfg_qwen = LLMR::llm_config("groq", "qwen/qwen3-32b", temperature = 0.8)
set.seed(110)
resp = panel_administer(panel, instrument, cfg, .runner = first_shown)
resp
#> <panel_responses | 12 persona(s) x 3 item(s) | 0 execution failure(s), 0 parse failure(s)>
#> NOT BENCHMARKED: no benchmark comparison has been run; see panel_benchmark().
resp$data
#> # A tibble: 36 × 14
#> persona_id item_id type item_position option_order response_text response_id
#> <int> <chr> <chr> <int> <chr> <chr> <chr>
#> 1 1 wk4 like… 1 strongly ag… strongly agr… <NA>
#> 2 1 fund choi… 2 road repair… road repair <NA>
#> 3 1 why open 3 <NA> Because it r… <NA>
#> 4 2 wk4 like… 1 strongly di… strongly dis… <NA>
#> 5 2 fund choi… 2 public tran… public trans… <NA>
#> 6 2 why open 3 <NA> Because it r… <NA>
#> 7 3 wk4 like… 1 strongly di… strongly dis… <NA>
#> 8 3 fund choi… 2 public tran… public trans… <NA>
#> 9 3 why open 3 <NA> Because it r… <NA>
#> 10 4 wk4 like… 1 strongly di… strongly dis… <NA>
#> # ℹ 26 more rows
#> # ℹ 7 more variables: success <lgl>, error_message <chr>, finish_reason <chr>,
#> # model <chr>, provider <chr>, response <chr>, score <dbl>
panel_bias_audit(resp)
#> # A tibble: 3 × 6
#> item_id n parse_failures execution_failures order_effect_p order_test_note
#> <chr> <int> <int> <int> <dbl> <chr>
#> 1 fund 12 0 0 NA sparse cells; …
#> 2 why 12 0 0 NA open item; no …
#> 3 wk4 12 0 0 NA sparse cells; …
LLMR::diagnostics(resp)
#> # A tibble: 3 × 10
#> item_id n parse_failures execution_failures order_effect_p order_test_note
#> <chr> <int> <int> <int> <dbl> <chr>
#> 1 fund 12 0 0 NA sparse cells; …
#> 2 why 12 0 0 NA open item; no …
#> 3 wk4 12 0 0 NA sparse cells; …
#> # ℹ 4 more variables: benchmark_state <chr>, items_covered <int>,
#> # items_total <int>, mean_abs_dev <dbl>
set.seed(110) # the same assignments for the second model
resp_qwen = panel_administer(panel, instrument, cfg_qwen,
.runner = first_shown)
panel_bias_audit(resp_qwen)
#> # A tibble: 3 × 6
#> item_id n parse_failures execution_failures order_effect_p order_test_note
#> <chr> <int> <int> <int> <dbl> <chr>
#> 1 fund 12 0 0 NA sparse cells; …
#> 2 why 12 0 0 NA open item; no …
#> 3 wk4 12 0 0 NA sparse cells; …panel_benchmark() compares valid model response shares
with benchmark shares for matching item-response pairs. It also records
benchmark coverage and item-level nonresponse in
$benchmark. Before a benchmark is attached, response shares
describe the configured model under the supplied personas, not a human
population. bench_fund supplies shares for one closed
item.
bench_fund = data.frame(
item_id = rep("fund", 2),
response = c("public transit", "road repair"),
share = c(0.41, 0.59)
)
resp_partial = panel_benchmark(
resp,
bench_fund,
benchmark_name = "toy city survey"
)
resp_partial
#> <panel_responses | 12 persona(s) x 3 item(s) | 0 execution failure(s), 0 parse failure(s)>
#> PARTIALLY BENCHMARKED (1/2): vs 'toy city survey', mean abs. deviation 0.007 on covered items.bench_fund covers one of the instrument’s two closed
items. bench_all adds shares for wk4 and
covers both.
bench_all = rbind(
bench_fund,
data.frame(
item_id = rep("wk4", 5),
response = c(
"strongly disagree",
"disagree",
"neutral",
"agree",
"strongly agree"
),
share = c(.05, .20, .25, .35, .15)
)
)
resp = panel_benchmark(
resp,
bench_all,
benchmark_name = "toy city survey"
)
resp
#> <panel_responses | 12 persona(s) x 3 item(s) | 0 execution failure(s), 0 parse failure(s)>
#> BENCHMARKED against 'toy city survey' (2/2 items): mean abs. deviation 0.230 (max 0.450)
LLMR::report(resp)
#> BENCHMARKED (2/2 items). Against 'toy city survey': mean absolute deviation 0.230, max 0.450 (full table in $benchmark$table; nonresponse in $benchmark$nonresponse).
#> PANEL. 12 persona(s) drawn from supplied margins over: age, party.
#> RESPONSES. 36 total; 0 execution failure(s); 0 parse failure(s).
#> FIRST-OPTION SENSITIVITY (chi-squared p by item; small p = which option was listed first moved the answers):
#> fund n = 12 execution failures = 0 parse failures = 0 order p = n/a (sparse cells; chi-square approximation unreliable)
#> why n = 12 execution failures = 0 parse failures = 0 order p = n/a (open item; no options to order)
#> wk4 n = 12 execution failures = 0 parse failures = 0 order p = n/a (sparse cells; chi-square approximation unreliable)
resp$benchmark$nonresponse
#> # A tibble: 2 × 4
#> item_id execution_failures execution_failure_rate nonresponse_rate
#> <chr> <int> <dbl> <dbl>
#> 1 fund 0 0 0
#> 2 wk4 0 0 0resp$benchmark$nonresponse gives the missing response
proportion for each closed item. The comparison shares use nonmissing
responses as their denominator.
conjoint_design() uses R’s random-number generator to
construct a classed design list. Its $profiles field
contains the initial profile table, and its $attributes
field contains the attribute universe. panel_administer()
draws the profiles each respondent sees. Set a seed before
administration to reproduce those respondent-level draws.
set.seed(110)
design = conjoint_design(
list(
price = c("low", "high"),
origin = c("domestic", "imported")
),
n_tasks = 4
)
design
#> <conjoint_design | 4 task(s) x 2 profile(s) | 2 attribute(s)>
design$profiles
#> # A tibble: 8 × 4
#> task profile price origin
#> <int> <int> <chr> <chr>
#> 1 1 1 high imported
#> 2 1 2 high domestic
#> 3 2 1 low imported
#> 4 2 2 low domestic
#> 5 3 1 low domestic
#> 6 3 2 high domestic
#> 7 4 1 low domestic
#> 8 4 2 high imported
design$attributes
#> $price
#> [1] "low" "high"
#>
#> $origin
#> [1] "domestic" "imported"conjoint_design() attempts to use distinct profiles
within each task in $profiles and warns when the attribute
space cannot supply them. conjoint_instrument() creates one
forced-choice item per task. Administration renders a fresh draw for
each respondent and records it with the response.
conjoint_amce() estimates from those recorded profiles
relative to a reference level for each attribute and calculates standard
errors clustered by persona. The reference is the first level, in the
design’s order, that appears in the estimation data.
cj_instr = conjoint_instrument(design, "Which product would you buy?")
set.seed(110)
cj = panel_administer(panel, cj_instr, cfg, .runner = first_shown)
conjoint_amce(cj)
#> <conjoint_amce | 96 profile row(s) | 12 respondent(s), 12 estimation cluster(s) | 0 missing, 0 execution failure(s)>
#> Reference levels: price = low; origin = domestic
#> # A tibble: 4 × 6
#> attribute level estimate std_error ci_lo ci_hi
#> <chr> <chr> <dbl> <dbl> <dbl> <dbl>
#> 1 price low 0 NA NA NA
#> 2 price high 0.0582 0.138 -0.244 0.361
#> 3 origin domestic 0 NA NA NA
#> 4 origin imported -0.135 0.0932 -0.340 0.0703conjoint_amce() returns a classed result with one row
for each observed attribute level. Reference levels have estimate 0 and
missing standard errors. Other rows contain the estimated contrast and
95 percent interval. Run counts remain in separate columns. The printed
header names the reference levels, which are also in
attr(amce, "reference"), and gives the number of personas
whose responses entered the estimate, the clusters for the standard
errors. When every profile showing an attribute’s first level is lost,
for instance because those executions failed, the next level present
becomes the reference and a warning names it.
The identical call against a live provider drops the
.runner argument. This chunk runs only when
RUN_LIVE is set to TRUE in the setup
chunk.
panel_administer() makes one request per persona-item
pair. Option randomization does not add requests. Provider prices and
prompt and response lengths determine cost. A versioned local model can
support later reruns when hosted endpoints change. The
$data field retains response_text,
response_id, success, model, and
provider, including when a reply cannot be matched to a
closed-item option. finish_reason is retained when the
runner supplies it.
LLMR supplies provider configuration and execution. LLMRcontent provides codebook-based text annotation and validation. LLMRagent provides agent experiments. LLMRpanel contains panel constructors, instruments, administration, and response summaries.