---
title: "Get started with safeconduct"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Get started with safeconduct}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---
<!-- Generated from vignettes.orig/safeconduct.Rmd -- do not edit. -->
<!-- Rebuild with: Rscript vignettes.orig/precompute.R -->



Some servers refuse any request that does not come from a web browser. The
file is there, a browser gets it, and `download.file()`, `curl` and anything
built on them get a `403` page instead. This vignette walks through two such
files, one behind each kind of refusal that Cloudflare makes, shows what a
plain request gets, and shows how `safeconduct::fetch()` gets the file.

The chunks below were run when this vignette was precomputed, not when the
package is built: they need a Chrome and a network, and what a server does
at a URL changes over time. The output shown was captured on 2026-11-01.

## Two ways to be refused

Cloudflare, the common case, refuses in two ways:

* **A browser check.** The `403` page carries the header `cf-mitigated:
  challenge` and runs a script. A browser that runs the script is given a
  `cf_clearance` cookie and reloaded, and from then on it is served. Sometimes
  the script shows a "Verify you are human" box for the person to click.
* **A block.** The `403` page says "Sorry, you have been blocked" and sets
  nothing, because the rule is about what the client looks like: its address,
  its user-agent string, how it speaks TLS. There is nothing to pass; a client
  the rule covers is refused every time.

`safeconduct::fetch(url, destfile)` first makes the plain request. If the
file comes, that is the end of it and no browser is involved. If the answer
is a check or a block, it opens a real Chrome with a throwaway profile on the
host, lets any check run, and downloads the file: through `curl` carrying the
cookies the browser was given where the server accepts that, and in the
browser itself where it does not. Nothing is simulated, and nothing is
clicked on anyone's behalf.

## Setup


``` r
# install.packages("pak")
pak::pak("mountainMath/safeconduct")
```

A Chrome or Chromium has to be installed; `chromote::find_chrome()` finds
it, and the `CHROMOTE_CHROME` environment variable points at one somewhere
unusual. `browser_available()` says whether a browser can be opened here,
and why not if not.


``` r
safeconduct::browser_available()
#> [1] TRUE
```

## A browser check: a Statistics Canada correspondence file

On October 2, 2026 StatCan moved a bunch of their data behind a Cloudflare
browser check, breaking reproducible analysis. **safeconduct** provides a way
to navigate around this in interactive sessions, but it cannot help where no
one can click, say in a Docker container, should the check insist on a click.

A simple example is the 2021 Census correspondence file for dissemination
areas, which establishes the correspondence between 2016 and 2021
dissemination areas. It is a 3.6 MB zip on www12.statcan.gc.ca.


``` r
url <- "https://www12.statcan.gc.ca/census-recensement/2021/geo/aip-pia/correspondence-correspondance/files-fichiers/2021_92-156-X_DA_AD.zip"
da_ad_zip <- file.path(tempdir(), "da_ad.zip")
download.file(url, da_ad_zip, mode = "wb")
#> Warning in download.file(url, da_ad_zip, mode = "wb"): downloaded length 0 !=
#> reported length 21961
#> Warning in download.file(url, da_ad_zip, mode = "wb"): cannot open URL
#> 'https://www12.statcan.gc.ca/census-recensement/2021/geo/aip-pia/correspondence-correspondance/files-fichiers/2021_92-156-X_DA_AD.zip':
#> HTTP status was '403 Forbidden'
#> Error in `download.file()`:
#> ! cannot open URL 'https://www12.statcan.gc.ca/census-recensement/2021/geo/aip-pia/correspondence-correspondance/files-fichiers/2021_92-156-X_DA_AD.zip'
```

Nothing was written. The reported length is that of the check page, not of
the zip. `curl` says the status outright:


``` r
curl::curl_download(url, da_ad_zip)
#> Error in `curl::curl_download()`:
#> ! HTTP response code said error [www12.statcan.gc.ca]:
#> The requested URL returned error: 403
```

If we are interested in why the download failed, `probe()` makes the same
plain request and reports on the type of refusal:


``` r
p <- safeconduct::probe(url)
str(p[c("status", "challenged", "blocked", "type")])
#> List of 4
#>  $ status    : int 403
#>  $ challenged: logi TRUE
#>  $ blocked   : logi FALSE
#>  $ type      : chr "text/html; charset=utf-8"
p$headers[["cf-mitigated"]]
#> [1] "challenge"
```

The `fetch()` function is the universal access point for **safeconduct**: it
navigates a wide variety of traffic-blocking environments.


``` r
safeconduct::fetch(url, da_ad_zip)
#> 'www12.statcan.gc.ca' is asking for a browser check; sending a browser through.
#>   Trying headless Chrome first.
#>   Passed, but the clearance does not transfer to curl; downloading under escort, in the browser.
#>   The server is asking for a browser check at the file's URL first; headless Chrome passes it without a click, but it may take two minutes.
#>   If it has not passed by then, a window is opened.
#>   Tip: rather than wait, headed = TRUE opens a window at once, where the check shows its box to click.
#>   Tip: for more from 'www12.statcan.gc.ca', keep one browser open instead of one per call:
#>        s <- browser_session("https://www12.statcan.gc.ca/")
#>        fetch(url, destfile, session = s)
#>        close(s)
```

Each line is a step. Headless Chrome was opened on the host and passed the
check at the site's root within a few seconds. The cookies it was given were
tried on a plain request for the file, which was refused again, so the
download runs in the browser. The browser was then sent to the file's URL,
where this host runs a second check before it serves the file. Headless
Chrome passes that one too, with nothing clicked, but Cloudflare holds it
for about two minutes first; the download takes a second after that, and
the call returns the destination file, invisibly. Had the two minutes
passed without, a window would have been opened and the download made there.

The first tip is about that wait. In a window the same check shows a
"Verify you are human" box, and the file is through a few seconds after the
box is clicked: `fetch(url, da_ad_zip, headed = TRUE)` took ten seconds in
all when this was captured, click included. Which to prefer depends on
whether someone is at the screen. The second tip is taken up below, under
what to expect at the console.


``` r
unzip(da_ad_zip, list = TRUE)
#>                                          Name   Length                Date
#> 1 2021_92-156-X_DA_AD/2021_92-156-X_DA_AD.csv 44914832 2021-08-12 16:18:00
da_ad <- read.csv(unz(da_ad_zip, "2021_92-156-X_DA_AD/2021_92-156-X_DA_AD.csv"))
dim(da_ad)
#> [1] 499052      7
names(da_ad)
#> [1] "DAUID2021_ADIDU2021"     "DAUID2016_ADIDU2016"    
#> [3] "DBUID2021_IDIDU2021"     "DARELFLAG_ADINDREL"     
#> [5] "DADGUID2021_ADIDUGD2021" "DADGUID2016_ADIDUGD2016"
#> [7] "DBDGUID2021_IDIDUGD2021"
```

## A block: Vancouver election results from the open data portal

The City of Vancouver's open data portal lists its election results, but the
files themselves are on another host. The portal's API gives the link:


``` r
api <- "https://opendata.vancouver.ca/api/explore/v2.1/catalog/datasets/municipal-election-results/records?limit=20"
elections <- jsonlite::fromJSON(api)$results
url <- elections$link_to_csv[elections$election == "2026 Municipal Election"]
url
#> [1] "https://webtransfer.vancouver.ca/OpenData/csv/2026MunicipalElectionResults.zip"
```

(`VancouvR::get_cov_data("municipal-election-results")` returns the same
table.) The portal answers a plain request; the file's host does not:


``` r
results_zip <- file.path(tempdir(), "results.zip")
download.file(url, results_zip, mode = "wb")
#> Warning in download.file(url, results_zip, mode = "wb"): cannot open URL
#> 'https://webtransfer.vancouver.ca/OpenData/csv/2026MunicipalElectionResults.zip':
#> HTTP status was '403 Forbidden'
#> Error in `download.file()`:
#> ! cannot open URL 'https://webtransfer.vancouver.ca/OpenData/csv/2026MunicipalElectionResults.zip'
```


``` r
p <- safeconduct::probe(url)
str(p[c("status", "challenged", "blocked", "type")])
#> List of 4
#>  $ status    : int 403
#>  $ challenged: logi FALSE
#>  $ blocked   : logi TRUE
#>  $ type      : chr "text/html; charset=utf-8"
```

This time there is no `cf-mitigated` header and no script. The page is
titled "Attention Required! | Cloudflare" and says "Sorry, you have been
blocked": a rule on this host refuses the client, and `curl` is refused
whatever it sends. A browser is not covered by the rule, and it goes through
at once:


``` r
safeconduct::fetch(url, results_zip)
#> 'webtransfer.vancouver.ca' turns plain requests away; sending a browser through.
#>   Trying headless Chrome first.
#>   Passed, but the clearance does not transfer to curl; downloading under escort, in the browser.
#>   Tip: for more from 'webtransfer.vancouver.ca', keep one browser open instead of one per call:
#>        s <- browser_session("https://webtransfer.vancouver.ca/")
#>        fetch(url, destfile, session = s)
#>        close(s)
```

That took three seconds. There was no check to wait for; the browser was
simply served. The cookies it was given do not make `curl` acceptable to the
rule either, so the download runs in the browser as before.


``` r
unzip(results_zip, list = TRUE)[1:2]
#>                                                       Name Length
#> 1      2026MunicipalElectionsResult_Election2026Totals.csv   6778
#> 2                2026MunicipalElectionsResult_Overview.csv    778
#> 3       2026MunicipalElectionsResult_ResultsCouncillor.csv  50269
#> 4            2026MunicipalElectionsResult_ResultsMayor.csv  14059
#> 5 2026MunicipalElectionsResult_ResultsParkCommissioner.csv  42316
#> 6               2026MunicipalElectionsResult_ResultsQ1.csv  10776
#> 7               2026MunicipalElectionsResult_ResultsQ2.csv  10490
#> 8               2026MunicipalElectionsResult_ResultsQ3.csv  10495
#> 9    2026MunicipalElectionsResult_ResultsSchoolTrustee.csv  33772
```

The same rule covers headless Chrome under its own name, which announces
itself as `HeadlessChrome` in its user-agent string: the hosts in this
example refuse it on that word alone and serve the same headless Chrome once
the word is gone. By default the package drops the word and nothing else, so
the string still names the Chrome version and platform it is. The
`useragent` argument takes `NULL` to leave Chrome's own string, at the price
of a window on hosts like these, or a string to use as given.

## What to expect at the console

**A window, sometimes.** Headless Chrome goes first, and a window is opened
only when the headless run ends on a check or a block it does not get past,
or on a check at the file's URL it has not passed in two minutes, and only
when there is a display to show it on. The window is a plain Chrome; when
the check shows a "Verify you are human" box, the window is brought to the
front and stays there for up to two minutes for the person at the screen to
click it; it wants to be left in front until it is through. `headed = TRUE`
opens a window at once and `headed = FALSE` never does. The argument is on
every function that may open a browser.

**Nothing in the working directory.** A file downloaded in the browser lands
under `tempdir()` and is moved to its destination when complete.

**Fewer browsers over time.** Where the server accepts the browser's cookies
on a plain request, they are kept for the rest of the R session and later
files from the same host need no browser. Neither host above accepts that,
so each `fetch()` there opens a browser. For several files or reads from one
host, open a browser once and pass it along:


``` r
s <- safeconduct::browser_session("https://webtransfer.vancouver.ca/")
#> Sending a browser through to 'webtransfer.vancouver.ca'.
#>   Trying headless Chrome first.
safeconduct::fetch(elections$link_to_csv[elections$election == "2022 Municipal Election"],
                   file.path(tempdir(), "results_2022.zip"), session = s)
safeconduct::fetch(elections$link_to_csv[elections$election == "2018 Municipal Election"],
                   file.path(tempdir(), "results_2018.zip"), session = s)
close(s)
```

A session's browser is reopened if it goes away, say because its window was
closed. `read_page(url)` is `fetch()` for a page or a feed: the response
comes back in memory, in the shape `curl::curl_fetch_memory()` returns, and
it takes a session the same way.

**Tips, once per host.** When a host has been found to need a window or the
browser for the transfer, a message says how to skip the finding out next
time. `quiet = TRUE` silences the progress messages and the tips.

**When a browser is not enough.** A server that blocks the browser too, say
by address, is reported within a second or two as a `safeconduct_not_passed`
error; a check that is not passed within its wait is the same class. Where
no browser can be opened at all, the error is `safeconduct_unavailable` and
its message says what is missing. `options(safeconduct.browser = FALSE)`
never opens a browser, for code that would rather tell its user to download
by hand. The function reference has the rest: `clearance()` for the cookies
and user-agent string to use in requests you make yourself, and `probe()`,
`is_challenge()` and `is_block()` for telling the refusals apart.
