Getting started with FactorialSurvey

A factorial survey shows each respondent a few short descriptions – vignettes – whose attributes are varied by design, and asks for a rating or a yes/no answer. Because the attributes are assigned experimentally, their effects can be estimated cleanly, and when one attribute is a cost, they can be expressed as willingness to pay.

This vignette walks through the workflow with take-away lunch boxes as the example: designing the vignettes, deciding how many respondents are needed, and analysing real ratings.

library(FactorialSurvey)

1. Design the vignettes

List the attributes and their levels. Levels may be text or numbers; every level of every attribute will appear in the design.

attrs <- list(
  main  = c("grilled fish", "teriyaki chicken", "tofu hamburger"),
  side  = c("no", "yes"),
  info  = c("no", "yes"),
  price = c(700, 900, 1100, 1300)
)

design <- fs_defficient(attrs, n_vignettes = 16, n_blocks = 2,
                        seed = 1, verbose = FALSE)
design
#> Factorial survey design
#>   Vignettes    : 16 
#>   Blocks       : 2 (optimal) 
#>   Attributes   : 4 
#>   Candidates   : 48 
#>   D-efficiency : 98.3% 
#>   A-efficiency : 96.6% 
#> 
#>                main side info price block
#> 1  teriyaki chicken  yes   no  1300     1
#> 2      grilled fish  yes  yes   900     1
#> 3  teriyaki chicken  yes   no   700     1
#> 4  teriyaki chicken   no   no   900     1
#> 5    tofu hamburger   no   no  1100     1
#> 6      grilled fish  yes  yes  1100     1
#> 7    tofu hamburger   no  yes  1300     1
#> 8      grilled fish   no  yes   700     1
#> 9    tofu hamburger  yes  yes   700     2
#> 10 teriyaki chicken   no  yes  1300     2
#> ... 6 more row(s)

The D-efficiency is reported relative to an orthogonal design of the same size, so 100 percent is the best attainable. fs_efficiency() computes the same measure for any design, including ones built elsewhere; pass the intended levels so that a level that never appears is caught.

fs_efficiency(design)
#> Design efficiency
#>   Vignettes    : 16 
#>   Parameters   : 8 
#>   D-efficiency : 98.3% 
#>   A-efficiency : 96.6%

Respondents are then given blocks and presentation orders that balance carryover, and the rows are turned into the text shown on screen.

design <- fs_vignette_text(
  design,
  "A {{main}} lunch box, side dish: {{side}}, nutrition label: {{info}}, ",
  column = "part1"
)
schedule <- fs_assign(design, n_respondents = 200, seed = 1)
#> Using 8 Williams order(s) of 8 vignettes. Carryover is fully balanced when the number of respondents per block is a multiple of 8.
head(schedule[, c("respondent", "block", "position", "main", "price")])
#>   respondent block position             main price
#> 1          1     1        1 teriyaki chicken  1300
#> 2          1     1        2     grilled fish   700
#> 3          1     1        3     grilled fish   900
#> 4          1     1        4   tofu hamburger  1300
#> 5          1     1        5 teriyaki chicken   700
#> 6          1     1        6     grilled fish  1100

fs_export() writes the design or schedule for a survey platform, for example as a Qualtrics Loop & Merge table.

2. Decide how many respondents you need

Before fielding, simulate the survey under plausible effects – from a pilot or a similar study – and see how often each effect would be detected and how precise the willingness to pay would be.

pw <- fs_power(
  design, n_respondents = c(50, 150),
  coef = c("(Intercept)" = 1, "mainteriyaki chicken" = 0.7,
           "maintofu hamburger" = 0.1, sideyes = 0.3, infoyes = 0.15,
           price = -0.0015),
  formula = ~ main + side + info + price, price = "price",
  n_sim = 50, seed = 1, verbose = FALSE
)
pw[pw$term %in% c("sideyes", "infoyes"), ]
#> Power analysis for a factorial survey
#>   Response      : rating 
#>   Simulations   : 50 per sample size
#>   Significance  : 0.05 (two-sided, clustered SE)
#> 
#>  n_respondents    term coef power mwtp mwtp_halfwidth mwtp_relative
#>             50 sideyes 0.30  1.00  200           97.7          0.49
#>             50 infoyes 0.15  0.54  100           96.2          0.96
#>            150 sideyes 0.30  1.00  200           60.3          0.30
#>            150 infoyes 0.15  0.96  100           58.4          0.58

A modest effect such as the nutrition label is easily missed with 50 respondents. mwtp_relative is the half-width of the confidence interval as a share of the true value: 0.3 means roughly plus or minus 30 percent. Use more simulations (200 or more) for a final decision.

3. Analyse the ratings

The bento data hold real ratings of eight lunch-box vignettes by 48 students, on a 0 to 10 purchase-intention scale.

head(bento)
#>   respondent vignette             main side info price rating
#> 1          1        1     grilled fish    1    1   700      7
#> 2          1        2   tofu hamburger    1    0  1300      1
#> 3          1        3     grilled fish    1    0   900      0
#> 4          1        4   tofu hamburger    1    1   900      2
#> 5          1        5 teriyaki chicken    1    0  1100      1
#> 6          1        6 teriyaki chicken    0    0   700      1

fs_betareg() fits a beta regression, which respects the bounds of the rating scale. Each student rated eight vignettes, so the ratings of one person are correlated; cluster makes the standard errors robust to this.

fit <- fs_betareg(rating ~ main + side + info + price, data = bento,
                  cluster = ~ respondent, compute_me = FALSE)
summary(fit)
#> Factorial survey beta regression
#> Call: fs_betareg(formula = rating ~ main + side + info + price, data = bento,     cluster = ~respondent, compute_me = FALSE)
#> Standard errors: clustered by respondent (48 clusters)
#> 
#> 
#> Call:
#> betareg::betareg(formula = formula, data = work)
#> 
#> Quantile residuals:
#>     Min      1Q  Median      3Q     Max 
#> -2.7799 -0.4580  0.2253  0.6149  3.9741 
#> 
#> Coefficients (mean model with logit link):
#>                        Estimate Std. Error z value Pr(>|z|)    
#> (Intercept)           0.3326180  0.2517675   1.321 0.186457    
#> mainteriyaki chicken  0.7632982  0.1504189   5.074 3.89e-07 ***
#> maintofu hamburger    0.1019424  0.1189826   0.857 0.391564    
#> side                  0.3418223  0.1028982   3.322 0.000894 ***
#> info                  0.1379099  0.0838098   1.646 0.099865 .  
#> price                -0.0016285  0.0002745  -5.931 3.00e-09 ***
#> 
#> Phi coefficients (precision model with identity link):
#>       Estimate Std. Error z value Pr(>|z|)    
#> (phi)   1.7937     0.3227   5.559 2.71e-08 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 
#> 
#> Type of estimator: ML (maximum likelihood)
#> Log-likelihood: 127.9 on 7 Df
#> Pseudo R-squared: 0.1052
#> Number of iterations: 26 (BFGS) + 1 (Fisher scoring) 
#> 
#> Fit statistics
#>   Log-likelihood            : 127.9 
#>   Log-likelihood (intercept): 103.1 
#>   AIC                       : -241.7 
#>   BIC                       : -214.1 
#>   LR test vs intercept      : 49.57 on 5 df, p = 1.7e-09 
#>   Pseudo R2 (Cox-Snell)     : 0.1211 
#>   Pseudo R2 (betareg)       : 0.1052

4. Willingness to pay

The marginal willingness to pay (MWTP) values each attribute in yen: the price change that would offset it.

w <- fs_mwtp(fit, price = "price")
w
#> Marginal willingness to pay
#>   Price coefficient : price 
#>   Interval method   : delta method 
#>   Confidence level  : 95% 
#> 
#>              variable     MWTP      SE    lower    upper
#>  mainteriyaki chicken 468.7263 82.2896 307.4416 630.0109
#>    maintofu hamburger  62.6008 68.8856 -72.4125 197.6141
#>                  side 209.9063 56.8564  98.4699 321.3427
#>                  info  84.6877 49.3522 -12.0409 181.4163
plot(w)

The expected willingness to pay (EWTP) values a whole lunch box – a complete scenario – rather than a single attribute. It is the mean of willingness to pay truncated at zero (Hanemann 1989), applied to complete scenarios as in Tanaka et al. (2027).

boxes <- data.frame(
  main = c("grilled fish", "teriyaki chicken", "teriyaki chicken"),
  side = c(0, 0, 1),
  info = c(0, 0, 1),
  row.names = c("Fish, plain", "Chicken, plain", "Chicken, full")
)
e <- fs_ewtp(fit, boxes, price = "price")
e
#> Expected willingness to pay
#>   Price variable    : price 
#>   Interval method   : delta method 
#>   Confidence level  : 95% 
#> 
#>        scenario   EWTP     SE lower  upper
#>     Fish, plain  536.2  69.58 399.8  672.6
#>  Chicken, plain  850.1  85.69 682.1 1018.0
#>   Chicken, full 1083.0 101.50 884.1 1282.0
plot(e) + ggplot2::labs(x = "Expected willingness to pay (yen)")

5. Yes/no responses

When respondents answer whether they would accept a scenario at the stated cost, fs_logit() takes the place of fs_betareg(), and fs_mwtp() and fs_ewtp() work in the same way. Simulated data illustrate it here.

yesno <- fs_simulate_ratings(
  design, n_respondents = 300,
  coef = c("(Intercept)" = 2, "mainteriyaki chicken" = 0.7,
           "maintofu hamburger" = 0.1, sideyes = 0.3, infoyes = 0.1,
           price = -0.002),
  formula = ~ main + side + info + price, response = "binary",
  name = "buy", seed = 2
)
lfit <- fs_logit(buy ~ main + side + info + price, data = yesno,
                 cluster = ~ respondent, compute_me = FALSE)
fs_mwtp(lfit, price = "price")
#> Marginal willingness to pay
#>   Price coefficient : price 
#>   Interval method   : delta method 
#>   Confidence level  : 95% 
#> 
#>              variable     MWTP      SE    lower    upper
#>  mainteriyaki chicken 294.3987 55.6782 185.2715 403.5259
#>    maintofu hamburger  71.9062 56.4467 -38.7272 182.5397
#>               sideyes  97.6170 45.6579   8.1291 187.1049
#>               infoyes   3.1653 43.9347 -82.9452  89.2758

References

Hanemann, W. M. (1989). Welfare evaluations in contingent valuation experiments with discrete response data: Reply. American Journal of Agricultural Economics, 71(4), 1057–1061.

Kuhfeld, W. F., Tobias, R. D. and Garratt, M. (1994). Efficient experimental design with marketing research applications. Journal of Marketing Research, 31(4), 545–557.

Tanaka, K. (2025). A unified framework for estimating willingness to pay using factorial survey experiments. SSRN Working Paper. https://doi.org/10.2139/ssrn.5238343

Tanaka, K., Dill, J., McNeil, N., Geller, R. and Kondo, N. (2027). What are safer streets worth? Bridging resident preferences and policy design in traffic calming. Transportation Research Part A, 215, 105293. https://doi.org/10.1016/j.tra.2026.105293