| Type: | Package |
| Title: | Fetch Files Through a Real Browser When a Plain Request is Refused |
| Version: | 0.1.0 |
| Description: | Some servers refuse any request that does not come from a web browser, with a browser check or a block page instead of the file, 'Cloudflare' being the common case. This package opens a real 'Chrome' with a throwaway profile, lets any check run, and fetches the file or page: through 'curl' carrying the clearance the browser was given where the server accepts that, and through the browser itself where it does not. Nothing is simulated; where a check asks a person to click, a window waits for them. Built on 'chromote'. |
| License: | MIT + file LICENSE |
| Encoding: | UTF-8 |
| Language: | en-CA |
| SystemRequirements: | Google Chrome or another Chromium-based browser (<https://www.google.com/chrome/>). |
| Depends: | R (≥ 4.1) |
| Imports: | chromote (≥ 0.3.0), curl (≥ 5.0), jsonlite, processx |
| Suggests: | knitr, rmarkdown, testthat (≥ 3.0.0), withr, xml2 |
| VignetteBuilder: | knitr |
| URL: | https://mountainmath.github.io/safeconduct/, https://github.com/mountainMath/safeconduct |
| BugReports: | https://github.com/mountainMath/safeconduct/issues |
| Config/testthat/edition: | 3 |
| Config/roxygen2/version: | 8.1.0 |
| NeedsCompilation: | no |
| Packaged: | 2026-11-02 04:23:37 UTC; jens |
| Author: | Jens von Bergmann [aut, cre] |
| Maintainer: | Jens von Bergmann <jens@mountainmath.ca> |
| Repository: | CRAN |
| Date/Publication: | 2026-11-20 11:50:02 UTC |
safeconduct: Fetch Files Through a Real Browser When a Plain Request is Refused
Description
Some servers refuse any request that does not come from a web browser, with a browser check or a block page instead of the file, 'Cloudflare' being the common case. This package opens a real 'Chrome' with a throwaway profile, lets any check run, and fetches the file or page: through 'curl' carrying the clearance the browser was given where the server accepts that, and through the browser itself where it does not. Nothing is simulated; where a check asks a person to click, a window waits for them. Built on 'chromote'.
Options
Behaviour switches are function arguments, so that nothing leaks between
packages using safeconduct in one R session; the options are these.
safeconduct.browser = FALSE never opens a browser. safeconduct.window
sets the window's size, c(width, height) in pixels (1000 by 800 by
default) or "maximized". Experimentally, safeconduct.focus, a CSS
selector, pins the element holding the check to the window's top left
corner, enlarged by safeconduct.zoom (1.5), and hides the rest of the
page.
Author(s)
Maintainer: Jens von Bergmann jens@mountainmath.ca
Authors:
Jens von Bergmann jens@mountainmath.ca
See Also
Useful links:
Report bugs at https://github.com/mountainMath/safeconduct/issues
Can this package open a browser here?
Description
Can this package open a browser here?
Usage
browser_available()
Value
TRUE, or FALSE with a reason attribute: the
safeconduct.browser option is FALSE, or no Chrome or Chromium was
found (the CHROMOTE_CHROME environment variable points at one).
Examples
browser_available()
attr(browser_available(), "reason")
Keep a browser open on a host
Description
Opens a Chrome with a throwaway profile on the host's root, lets any check
run, and keeps it open for read_page() and fetch() through their
session argument. Close it with close(); otherwise it lives until the
R session ends. A browser that has gone away, its window closed, is
reopened the same way at the next call, which says so.
Usage
browser_session(
url,
quiet = FALSE,
wait = NULL,
headed = "auto",
useragent = TRUE
)
Arguments
url |
A URL on the host. |
quiet |
Suppress progress messages. |
wait |
Seconds to wait for the check to pass: by default two minutes with a window, so that a person can click, and thirty seconds headless. |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
Value
A safeconduct_session, whose clearance is what clearance()
would return. Errors as fetch()'s.
Examples
## Not run:
s <- browser_session("https://results.vancouver.ca/")
r <- read_page("https://results.vancouver.ca/data/arearesults.json",
session = s)
close(s)
## End(Not run)
Get a clearance for a host from a browser
Description
Opens a Chrome on the host, lets any check run, and returns the browser's
user-agent string and cookies, which clearance_handle() puts on a curl
request. Whether the server then serves curl is the server's call: after
a block page it usually does not, since the rule is about the client, and
fetch() then downloads in the browser instead. Clearances are cached per
host for the R session.
Usage
clearance(
url,
refresh = FALSE,
quiet = FALSE,
wait = NULL,
headed = "auto",
useragent = TRUE
)
Arguments
url |
The file's URL. |
refresh |
Get a new clearance even when one is cached. |
quiet |
Suppress progress messages. |
wait |
Seconds to wait for the check to pass: by default two minutes with a window, so that a person can click, and thirty seconds headless. |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
Value
A safeconduct_clearance: a list with host, useragent,
cookie (one Cookie: header value) and time. Errors as fetch()'s.
Examples
## Not run:
cl <- clearance("https://www12.statcan.gc.ca/")
curl::curl_fetch_memory("https://www12.statcan.gc.ca/", clearance_handle(cl))
## End(Not run)
A curl handle that carries a clearance
Description
The user-agent string and the cookies go on a request together: Cloudflare ties the clearance cookie to the user agent that earned it, and the cookie alone is answered with the check again.
Usage
clearance_handle(clearance = NULL, ...)
Arguments
clearance |
A clearance from |
... |
Further options for |
Value
A curl handle with followlocation set, and the clearance's
useragent and cookie when one was given.
Examples
h <- clearance_handle(list(useragent = "Mozilla/5.0", cookie = "cf_clearance=x"))
Download a file in a browser
Description
Opens a Chrome with a throwaway profile on the host, lets any check run,
and downloads url in it, with no plain request first. fetch() does
this only when it has to.
Usage
download_in_browser(
url,
destfile,
quiet = FALSE,
timeout = 3600,
headed = "auto",
useragent = TRUE
)
Arguments
url |
The file's URL. |
destfile |
Where to write it. A failed transfer leaves nothing there. |
quiet |
Suppress progress messages. |
timeout |
Seconds a transfer in the browser may take. |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
Value
destfile, invisibly. Errors as fetch()'s.
Examples
## Not run:
download_in_browser("https://example.org/large.zip", "large.zip")
## End(Not run)
Download a file, through a browser if the server insists on one
Description
Fetches url to destfile. A plain request is made first, and if the
file comes no browser is involved. If the server answers with a browser
check (is_challenge()) or a block page (is_block()), a Chrome with a
throwaway profile is opened on the host, any check runs, and the file is
downloaded: through curl carrying the browser's clearance where the server
accepts that, and in the browser itself where it does not. A clearance
curl can use is kept for the rest of the R session, so later files from
the host need no browser.
Usage
fetch(
url,
destfile,
quiet = FALSE,
timeout = 3600,
session = NULL,
headed = "auto",
useragent = TRUE
)
Arguments
url |
The file's URL. |
destfile |
Where to write it. A failed transfer leaves nothing there. |
quiet |
Suppress progress messages. |
timeout |
Seconds a transfer in the browser may take. |
session |
A |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
Details
Nothing is simulated. A check that shows a "Verify you are human" box
waits in a window, for up to two minutes, for a person to click it. The
window is brought to the front when a check shows in it, and wants to be
left there until it is through: a check run with the window behind
something else is reissued on every click. A
file downloaded in the browser lands under tempdir() and is moved to
destfile when complete.
Value
destfile, invisibly. Errors are classed under
safeconduct_error: safeconduct_unavailable when no browser can be
opened here, safeconduct_not_passed when the check was not passed in
time or the server blocks the browser too, with a reason field, and
safeconduct_download_error when the transfer failed.
Requirements
Chrome or Chromium, found by chromote::find_chrome(); the
CHROMOTE_CHROME environment variable points at one somewhere unusual,
and browser_available() says whether one can be opened here.
options(safeconduct.browser = FALSE) never opens a browser; the window's
size and the other options are described in safeconduct-package.
Tips
At the console, once a host has been found to need a window or the browser
for the transfer, a tip says how to skip the finding out next time:
headed = TRUE, or a browser_session(). Tips come once per host and R
session, not when quiet, and not for calls made from another package.
They are messages of class safeconduct_hint.
Examples
## Not run:
url <- paste0("https://www12.statcan.gc.ca/census-recensement/2021/",
"geo/sip-pis/rnf-frr/files-fichiers/lrnf000r21a_e.zip")
fetch(url, "lrnf000r21a_e.zip")
## End(Not run)
Is a response a block page that a browser would not be shown?
Description
Cloudflare's firewall can refuse a client that does not look like a
browser outright, with a 403 and its own block page, while a real
browser is served.
Usage
is_block(status, headers, body = NULL)
Arguments
status |
The HTTP status code. |
headers |
The response headers, as the raw header block (one string,
lines separated by |
body |
The response body, as a string. |
Value
TRUE for that page: a 403 from server: cloudflare whose body
says "Sorry, you have been blocked".
Examples
is_block(403, "HTTP/2 403\r\nserver: cloudflare\r\n\r\n",
"<html><title>Attention Required! | Cloudflare</title>...")
is_block(403, "HTTP/2 403\r\nserver: nginx\r\n\r\n", "Forbidden")
Does a response carry a browser check instead of the resource?
Description
Does a response carry a browser check instead of the resource?
Usage
is_challenge(headers)
Arguments
headers |
The response headers, as the raw header block (one string,
lines separated by |
Value
TRUE for a check a real browser would pass: Cloudflare's, by its
cf-mitigated: challenge header.
See Also
is_block() for the other refusal a browser gets past.
Examples
is_challenge("HTTP/2 403\r\ncf-mitigated: challenge\r\n\r\n")
is_challenge("HTTP/2 200\r\ncontent-type: application/zip\r\n\r\n")
Parse a response from read_page()
Description
page_text() decodes the body to a UTF-8 string, page_json() parses it
with jsonlite::fromJSON(), and page_html() with xml2::read_html().
Usage
page_text(resp, encoding = NULL, check = TRUE)
page_json(resp, ..., check = TRUE)
page_html(resp, encoding = NULL, check = TRUE)
Arguments
resp |
A response from |
encoding |
The body's charset, overriding the response's. |
check |
Whether an HTTP status outside 200–299 is an error. |
... |
Passed on to |
Details
The charset comes from the response's content-type; without one,
page_text() and page_json() assume UTF-8 and page_html() leaves it
to the page's own <meta charset>. ISO-8859-1 is read as windows-1252, as
browsers do. A body that is not valid in its charset is an error. A "not
found" page served with status 200 passes the status check; page_json()
refuses an HTML body, and for a page, compare resp$url with the URL
asked for.
Value
page_text(): a UTF-8 string. page_json(): what
jsonlite::fromJSON() returns. page_html(): an xml_document whose
relative links resolve against resp$url. Errors are classed under
safeconduct_error: safeconduct_http_error for a status outside
200–299, carrying status and url, and safeconduct_parse_error for
a body that cannot be decoded, or HTML where JSON was asked for.
Examples
## Not run:
page_json("https://results.vancouver.ca/data/arearesults.json")
page_html("https://www12.statcan.gc.ca/census-recensement/index-eng.cfm")
## End(Not run)
Ask a server what it serves at a URL
Description
Sends a GET for the first byte, so that the answer carries the
resource's headers without the resource; a HEAD is not used because
some servers answer it differently.
Usage
probe(url, clearance = NULL)
Arguments
url |
The URL. |
clearance |
A clearance from |
Value
A list: reachable, status (NA when unreachable),
challenged (is_challenge()), blocked (is_block()), type (the
content-type, lower case), size (bytes, from content-range or
content-length, NA when unknown) and headers (a named list). A
browser is called for when challenged or blocked is TRUE.
Examples
## Not run:
probe("https://www12.statcan.gc.ca/census-recensement/index-eng.cfm")
## End(Not run)
Read a page or a feed into memory, through a browser if the server insists
Description
fetch() for a page or a feed: the response is returned rather than
written to disk, shaped like curl::curl_fetch_memory()'s, by the same
route; where only the browser is served, the page the browser holds on
the host reads it. For a feed read every few minutes, pass a
browser_session() instead of opening a browser per call; a check the
host asks for again meanwhile is passed and the read repeated.
Usage
read_page(
url,
session = NULL,
quiet = FALSE,
timeout = 120,
headed = "auto",
useragent = TRUE
)
Arguments
url |
The file's URL. |
session |
A |
quiet |
Suppress progress messages. |
timeout |
Seconds a read in the browser may take. |
headed |
How to run the browser when one is needed. |
useragent |
The browser's user-agent string. |
Value
A list: url (after redirects), status_code, type (the
content-type, lower case), headers (a named list) and content
(raw). An HTTP error status is returned, not raised. Parse it with
page_text(), page_json() or page_html(). Errors as fetch()'s.
Examples
## Not run:
r <- read_page("https://results.vancouver.ca/data/arearesults.json")
page_json(r)
## End(Not run)