| Type: | Package |
| Title: | Download and Manage Optional Package Data |
| Version: | 0.4.1 |
| Maintainer: | Tim Schäfer <ts+code@rcmd.org> |
| Description: | Manage optional data for your package. The data can be hosted anywhere, and you have to give a Uniform Resource Locator (URL) for each file. File integrity checks are supported. This is useful for package authors who need to ship more than the 5 Megabyte of data currently allowed by the the Comprehensive R Archive Network (CRAN). Download functions are supposed to be called by users in interactive sessions only. |
| License: | MIT + file LICENSE |
| Encoding: | UTF-8 |
| URL: | https://github.com/dfsp-spirit/pkgfilecache |
| BugReports: | https://github.com/dfsp-spirit/pkgfilecache/issues |
| Suggests: | knitr, rmarkdown, testthat (≥ 2.1.0), withr |
| Imports: | rappdirs, curl |
| VignetteBuilder: | knitr |
| Config/roxygen2/version: | 8.0.0 |
| NeedsCompilation: | no |
| Packaged: | 2026-08-31 19:17:05 UTC; ts |
| Author: | Tim Schäfer |
| Repository: | CRAN |
| Date/Publication: | 2026-08-31 22:20:02 UTC |
Add a single file download to a curl multi pool.
Description
Add a single file download to a curl multi pool for parallel downloading. The file is streamed to disk as soon as data arrives, using curl's internal file writer (which opens the file lazily and closes it when the transfer completes). On any failure (connection error or HTTP status >= 300), the destination file is removed, so that a partially downloaded file is never mistaken for a successfully downloaded one. The failure is also reported to the error_collector callback, if given, so that callers can surface a meaningful cause (the HTTP status code or the connection error message) to the user. This function is only used internally by download_files_with_md5_mismatch.
Usage
add_file_download_to_curl_pool(url, destfile, pool, error_collector = NULL)
Arguments
url |
string. The URL to download. |
destfile |
string. The absolute path of the local file to write to. |
pool |
curl pool. The pool to add the download to, see |
error_collector |
function or NULL. An optional callback that is called with two string arguments if the download failed, i.e., if the HTTP response had a status code >= 300 or if a connection-level error occurred: the destination file (first argument) and the error message (second argument). Passing the destination file explicitly lets callers attribute each error to the right file, without closing over loop variables. Defaults to NULL, in which case failures are only handled by removing the destination file. |
Value
NULL, invisibly.
Check whether the given files exist in the package cache.
Description
Check whether the given files exist in the package cache. You can pass MD5 sums, which will be verified and only files with correct MD5 hash will count as existing.
Usage
are_files_available(pkg_info, relative_filenames, md5sums = NULL)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
relative_filenames |
vector of strings. A vector of filenames, relative to the package cache. |
md5sums |
vector of strings or NULL. A list of MD5 checksums, one for each file in param 'relative_filenames', if not NULL. If given, the files will only be reported as existing if the MD5 sums match. |
Value
logical vector. For each file, whether it passed the check.
Examples
pkg_info = get_pkg_info("mypackage")
is_available = are_files_available(pkg_info, c("file1.txt", "file2.txt"))
Derive missing URLs in a manifest from a base URL.
Description
For every file in a manifest without an explicit URL, set the URL to paste0(base_url, path). Used internally by read_manifest.
Usage
derive_manifest_urls(manifest, base_url = NULL)
Arguments
manifest |
data.frame. The validated manifest. |
base_url |
string or NULL. The base URL to derive missing URLs from. |
Value
data.frame. The manifest with the url column filled in.
Download files marked as mismatch to package cache.
Description
Download files marked as mismatched to package cache. You should check afterwards whether this was successful, e.g., via 'files_exist_md5'.
Usage
download_files_with_md5_mismatch(
local_files_absolute,
local_files_md5_ok,
urls,
files_are_binary = NULL,
num_connections = getOption("pkgfilecache.num_connections", 2)
)
Arguments
local_files_absolute |
vector of strings. A vector of filenames, must already include the package cache part. |
local_files_md5_ok |
logical vector. For each file, whether the local copy is OK. Only files for which this lists FALSE will be downloaded. |
urls |
vector of strings. For each file, a remote URL where to download the file. Will be passed to 'curl::curl_download', see that function for URL encoding details. |
files_are_binary |
logical vector. For each file, whether it is binary. Only required on Windows, when files need to be downloaded. See 'curl::curl_download' docs for details. |
num_connections |
integer. The number of parallel connections to use when downloading files. Defaults to 2, or to the value of the R option |
Value
Named character vector with one entry per file in local_files_absolute (the names are the absolute file paths). The entry is NA for files that were not downloaded in this call and for files that downloaded successfully; for files that were downloaded but failed it contains the error message of the download attempt. This message can distinguish remote problems (e.g., an HTTP status code like 404, or a connection failure) from local problems (e.g., the local file could not be written). The authoritative success check is done separately via files_exist_md5 afterwards.
Ensure all given files exist in the file cache, download them if they are not.
Description
Ensure all given files exist in the file cache, download them if they are not.
Usage
ensure_files_available(
pkg_info,
relative_filenames,
urls,
files_are_binary = NULL,
md5sums = NULL,
on_errors = "warn",
download = TRUE,
num_connections = getOption("pkgfilecache.num_connections", 2),
num_retries = 2
)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
relative_filenames |
vector of strings. A vector of filenames, realtive to the package cache. |
urls |
vector of strings. For each file, a remote URL where to download the file. Will be passed to 'curl::curl_download', see that function for URL encoding details. |
files_are_binary |
logical vector. For each file, whether it is binary. Only required on Windows, when files need to be downloaded. See 'curl::curl_download' docs for details. |
md5sums |
vector of strings or NULL. A list of MD5 checksums, one for each file in param 'relative_filenames', if not NULL. If given, the files will only be reported as existing if the MD5 sums match. A file that is present in the cache but whose MD5 sum does not match the expected sum is considered invalid and is deleted from the cache, so that it is never mistaken for a valid file. |
on_errors |
string. What to do if getting the files failed. One of c("warn", "stop", "ignore"). At the end, files are checked using ‘files_available'(including MD5 if given). Depending on the check results, the behaviours triggered are: "warn": Print a warning for each file that failed the check. "stop": Stop the script, i.e., the whole application. "ignore": Do nothing. You can still react using the return value. This applies whether or not downloading was attempted, so missing files are never silently ignored, e.g., when ’download' is FALSE. |
download |
logical. Whether to try downloading missing files. Defaults to TRUE. Existing files (with correct MD5 if available) will never be downloaded. When set to FALSE, no files are downloaded, but missing files are still reported according to 'on_errors'. |
num_connections |
integer. The number of parallel connections to use when downloading files. Defaults to 2. With more connections, several files are downloaded at the same time, which can speed up downloading many small files considerably. Use 1 for strictly sequential downloads. A high number of connections may overload the server or trigger rate limits. The default can be changed globally for all calls that do not specify this argument by setting the R option |
num_retries |
integer. How many times to retry downloading files that failed, in addition to the first attempt. Defaults to 2. Each attempt uses fresh connections, which makes downloads robust against transient connection failures (e.g., a server that closes or throttles connections). Set to 0 to disable retrying. |
Value
Named list. The list has entries: "available": vector of strings. The names of the files that are available in the local file cache. You can access them using get_filepath(). "missing": vector of strings. The names of the files that this function was unable to retrieve. "file_status": Logical array indicating whether the files are available. Order is identical to the one in argument 'relative_filenames'.
Examples
pkg_info = get_pkg_info("mypackage");
local_relative_filenames = c("local_file1.txt", "local_file2.txt");
bu = "https://raw.githubusercontent.com/dfsp-spirit/";
url1 = paste(bu, "pkgfilecache/master/inst/extdata/file1.txt", sep="");
url2 = paste(bu, "pkgfilecache/master/inst/extdata/file2.txt", sep="");
urls = c(url1, url2);
md5sums = c("35261471bcd198583c3805ee2a543b1f", "85ffec2e6efb476f1ee1e3e7fddd86de");
res = ensure_files_available(pkg_info, local_relative_filenames, urls, md5sums=md5sums);
erase_file_cache(pkg_info); # clear full cache
Ensure the files described in a manifest are available in the file cache.
Description
Like ensure_files_available, but the files are described in a single declarative manifest (one row per file) instead of three parallel vectors. A manifest can be a CSV file (e.g., shipped with the package in inst/extdata) or a data.frame, see read_manifest for the expected format.
Usage
ensure_files_available_from_manifest(
pkg_info,
manifest,
base_url = NULL,
files_are_binary = NULL,
on_errors = "warn",
download = TRUE,
num_connections = getOption("pkgfilecache.num_connections", 2),
num_retries = 2
)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
manifest |
string or data.frame. The manifest to use, see |
base_url |
string or NULL. Base URL to derive the download URL for files that have no explicit URL, see |
files_are_binary |
logical vector. For each file, whether it is binary. Only required on Windows, when files need to be downloaded. See |
on_errors |
string. What to do if getting the files failed. One of c("warn", "stop", "ignore"). See |
download |
logical. Whether to try downloading missing files. Defaults to TRUE. Existing files (with correct MD5 if available) will never be downloaded. Set to FALSE to only check which files are available without downloading anything. |
num_connections |
integer. The number of parallel connections to use when downloading files, see |
num_retries |
integer. How many times to retry downloading files that failed, in addition to the first attempt, see |
Value
Named list, like for ensure_files_available: the entries "available" and "missing" contain the manifest paths (as '/'-separated strings) that are available in, or missing from, the local file cache. The entry "file_status" is a logical vector in manifest row order indicating for each file whether it is available.
Examples
pkg_info = get_pkg_info("mypackage")
# A manifest as a data.frame: two files, URLs derived from base_url.
manifest = data.frame(path = c("sub/file1.txt", "file2.txt"),
stringsAsFactors = FALSE)
# Only check availability, do not download anything (download = FALSE).
# The files are missing, so a warning is expected (see 'on_errors').
res = suppressWarnings(ensure_files_available_from_manifest(pkg_info, manifest,
base_url = "https://example.com/data/",
download = FALSE))
Delete the full package cache directory for the given package.
Description
Delete the full package cache directory for the given package.
Usage
erase_file_cache(pkg_info)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
Value
integer. The return value of the unlink() call: 0 for success, 1 for failure. See the unlink() documentation for details.
Check whether files exist, optionally with MD5 check.
Description
Check whether files exist. If MD5 hashes are given, they will be verified.
Usage
files_exist_md5(files_absolute, md5sums = NULL)
Arguments
files_absolute |
vector of strings. A vector of filenames. Files are check as given, so they must already include the package cache part of the path. |
md5sums |
vector of strings or NULL. A list of MD5 checksums, one for each file in param 'files', if not NULL. If given, the files will only be reported as existing if the MD5 sums match. |
Value
logical vector. Whether the files exist. If the md5sums were given, whether the files exist and the MD5 sum matches.
Turn a filepath into a flat string.
Description
Turn a filepath into a flat string.
Usage
flatten_filepath(filepath)
Arguments
filepath |
string or list of strings |
Value
string, the flattened filepath
Join all relative filenames to a datadir.
Description
For each file, create a full path by joining the datadir with the filename.
Usage
get_abs_filenames(datadir, relative_filenames)
Arguments
datadir |
string, the path to the package cache directory. |
relative_filenames |
vector of strings. A vector of filenames, relative to the package cache. Can be a list of vectors, which will be interpreted as files with subdirs. |
Value
vector of strings, the absolute file names.
Construct absolute path for package cache files.
Description
Construct absolute path for package cache files.
Usage
get_absolute_path_for_files(pkg_info, relative_filenames)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
relative_filenames |
vector of strings. A vector of filenames, relative to the package cache. |
Value
vector of strings. The absolute paths.
Examples
rel_files = c("file1.txt", "file2.txt")
pkg_info = get_pkg_info("mypackage")
abs_paths = get_absolute_path_for_files(pkg_info, rel_files)
Get the absolute path of the package cache.
Description
Get the absolute path of the package cache directory for the given package.
By default, the cache is stored in the directory returned by 'tools::R_user_dir(packagename, "data")' (for R version 4.0 or later), which is the location recommended by the CRAN repository policy for user-specific data and cache files. If a cache already exists at the legacy location used by older versions of this package (the directory returned by 'rappdirs::user_data_dir'), that legacy directory is reused instead, so that existing downloads are not lost and do not have to be re-downloaded. On R versions before 4.0, the legacy 'rappdirs::user_data_dir' location is used.
The location can be overridden with R options:
'pkgfilecache.cachedir': If set to a directory path, that directory is used as the root of the package cache, and the package name (and version, if any) are appended to it. Useful to place the cache elsewhere, e.g., on a ramdisk or network drive.
‘pkgfilecache.use_tempdir': If set to 'TRUE', the cache is placed in a subdirectory of the R session’s temporary directory (‘tempdir()'). This is handy for unit tests and continuous integration, and in particular for tests that must not write to the user’s home directory (e.g., on CRAN). The value of 'pkgfilecache.cachedir' takes precedence if both options are set.
Usage
get_cache_dir(pkg_info)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
Value
string. The absolute path of the package cache directory. A subdirectory of the directory returned by 'tools::R_user_dir' (for R version 4.0 or later) or 'rappdirs::user_data_dir' (for older R versions), unless one of the options 'pkgfilecache.cachedir' or 'pkgfilecache.use_tempdir' is set.
Examples
pkg_info = get_pkg_info("mypackage")
opt_data_dir = get_cache_dir(pkg_info)
Retrieve the path to a single file from the package cache.
Description
Retrieve the path to a single file from the package cache.
Usage
get_filepath(pkg_info, relative_filename, mustWork = TRUE)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
relative_filename |
string. A filename, relative to the package cache. |
mustWork |
logical. Whether an error should be created if the file does not exist. |
Value
string. The path to the file. If mustWork=TRUE, the file is guaranteed to exist if the function returns (an error will occur if it does not). If mustWork=FALSE and the file does not exist, the empty string is returned.
Examples
pkg_info = get_pkg_info("mypackage")
full_path_of_file = get_filepath(pkg_info, "file1.txt", mustWork=FALSE)
Construct a pkg_info object to be used with all other functions.
Description
This functions constructs an object that uniquely identifies your package, i.e., the package that want to use the package cache. This is not a secret.
Usage
get_pkg_info(packagename, author = NULL, version = NULL)
Arguments
packagename |
string. The name of the package using the package cache. Must be a valid directory name. Should not contain spaces. Passed as 'appname' to 'rappdirs::user_data_dir'. |
author |
string. The author of the package using the package cache, or NULL. Must be a valid directory name if given, no need for the real author name. Should not contain spaces. Defaults to NULL. Passed as 'appauthor' to 'rappdirs::user_data_dir'. Leave at NULL if in doubt. |
version |
string or NULL. An optional version path element to append to the path. You might want to use this if you want multiple versions of your pacakge to be able to have independent data. If used, this would typically be "<major>.<minor>". Must be a valid directory name. Should not contain spaces or special characters. |
Value
named list. This can be passed to all function which require a 'pkg_info' argument. You should not care for the inner structure and treat it as some identifier.
Examples
pkg_info = get_pkg_info("mypackage")
pkg_info = get_pkg_info("mypackage", author="me")
pkg_info = get_pkg_info("mypackage", author="me", version="0.3")
Given a relative file, determine its subdir in the package cache.
Description
Given a relative file, determine its subdir in the package cache.
Usage
get_relative_file_subdir(pkg_info, relative_file)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
relative_file |
string or vector of strings. If a string, this function does nothing. If a vector of strings, a path is created from the elements using file.path, and the directory of it (determined by dirname()) is created. |
Value
named list. The entries are: "has_subdir": logical, whether the file has a subdir. "relative_filepath": string. The input relative_file, flattened to a string. For files without subdir, this is identical to string in the parameter 'relative_file'. For others, it is the result of applying file.path() to the elements of the vector 'relative_file'. If "has_subdir" is TRUE, the following 2 fields also exist: "relative_subdir": string, subdir path relative to package cache dir. "absolute_subdir": string, absolute subdir path.
List files that are available locally in the package cache.
Description
List files that are available locally in the package cache.
Usage
list_available(pkg_info)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
Value
vector of strings. The file names available, relative to the package cache. The returned names may include a subdirectory part. The subdirectories are not listed separately.
Examples
pkg_info = get_pkg_info("mypackage")
available_files_in_cache = list_available(pkg_info)
Given a relative file, create the subdir in the package cache if needed.
Description
Given a relative file, create the subdir in the package cache if needed.
Usage
make_pgk_cache_subdir_for_all_relative_files(pkg_info, relative_filenames)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
relative_filenames |
vector of strings. A vector of filenames, relative to the package cache. Can be a list of vectors, which will be interpreted as files with subdirs. |
Given a relative file, create the subdir in the package cache if needed.
Description
Given a relative file, create the subdir in the package cache if needed.
Usage
make_pgk_cache_subdir_for_relative_file(pkg_info, relative_file)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
relative_file |
string or vector of strings. If a string, this function does nothing. If a vector of strings, a path is created from the elements using file.path, and the directory of it (determined by dirname()) is created. |
Run the manifest generation command line interface (CLI).
Description
Generate a manifest CSV from a directory of files from the command line. This function is meant to be run with Rscript, not to be called interactively. It reads the command line arguments, parses them, and calls write_manifest_from_dir. It is the engine behind the standalone make_manifest.R script that ships with the package (see manifest_script).
Usage
manifest_cli(args = commandArgs(trailingOnly = TRUE))
Arguments
args |
character vector. The command line arguments. Defaults to the arguments actually passed to the R script ( |
Details
Supported arguments (each flag can be given as --flag value or as --flag=value):
-
--dir <path>: The directory containing the data files (required). -
--out <path>: The path of the CSV file to write the manifest to (required). -
--url-base <url>: Optional base URL used to derive the download URL of every file, aspaste0(url_base, path). -
--help: Print usage information and exit without generating anything.
A leading --args or -- marker (used by Rscript to separate command line arguments from R options) is ignored, so both the one-liner form and the script form shown in the examples work.
Value
The generated manifest as a data.frame, invisibly, or NULL if --help was given.
Examples
## Not run:
# One-liner from the shell (no script file needed):
Rscript -e 'pkgfilecache::manifest_cli()' --args --dir ~/mydata --out files.csv
# Same thing using the standalone script shipped with the package
# (--url-base is optional and derives the download URLs):
Rscript "$(Rscript -e 'cat(pkgfilecache::manifest_script())')" --dir ~/mydata --out files.csv
## End(Not run)
Get the usage message of the manifest generation CLI.
Description
Internal helper that returns the usage text printed by manifest_cli for --help and on errors.
Usage
manifest_cli_usage()
Value
character vector with the usage text, one element per line.
Get the path of the standalone manifest generation script.
Description
Return the full path to the make_manifest.R script that ships with the package (installed into the package's exec subdirectory). This script is a thin wrapper around manifest_cli and can be run with Rscript to generate a manifest from the command line. On Unix-like systems you can also copy it to a directory on your PATH and make it executable with chmod +x, then run it like a normal command.
Usage
manifest_script()
Value
character string. The full path to the script, or "" if the script cannot be found (e.g., because the package is not installed correctly).
Decide between the new and the legacy package cache directory.
Description
After migrating the default cache location to 'tools::R_user_dir', this function decides whether to use the new default directory or the legacy 'rappdirs' directory: the legacy directory is used if it exists and the new one does not, so that users with an existing cache do not have to re-download their files.
Usage
pick_cache_dir(new_dir, legacy_dir)
Arguments
new_dir |
string. The path of the new default cache directory (e.g., a subdirectory of 'tools::R_user_dir'). |
legacy_dir |
string. The path of the legacy cache directory (e.g., as returned by 'rappdirs::user_data_dir'). |
Value
string. Either 'new_dir' or 'legacy_dir', depending on which directories exist.
Append the optional package version to a cache directory path.
Description
Append the optional package version to a cache directory path.
Usage
pkg_cache_dir_with_version(dir, pkg_info)
Arguments
dir |
string. The cache directory path (already including the package name). |
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
Value
string. The directory path, with the version appended if the package info contains one.
Read a file manifest for optional data files.
Description
Read a manifest that describes a set of optional data files, one file per row. A manifest is either a CSV file or a data.frame with the columns described below. Manifests can be written by hand or generated with write_manifest_from_dir.
Usage
read_manifest(manifest, base_url = NULL)
Arguments
manifest |
string or data.frame. The manifest to read: either the path to a CSV file, or a data.frame that is used directly. Comment lines in a CSV manifest that start with |
base_url |
string or NULL. A base URL used to derive the download URL for all files that do not have an explicit URL. For such files, the URL is set to |
Details
The manifest must contain a column named path, giving the file path relative to the package cache directory, using / as separator. Two further columns are optional:
-
url: The remote URL to download the file from. If empty (or missing), it is derived frombase_url(see above). -
md5: The MD5 checksum of the file, used to verify file integrity. If empty (or missing), no integrity check is performed for that file (only its existence is checked).
Additional columns are allowed and ignored.
Value
data.frame with the columns path, url and md5. The url column never contains missing values after this function ran: either it was present in the input, or it was derived from base_url and path. The md5 column may contain NA for files without a checksum.
Delete all the given files from the package cache.
Description
Delete all the given files from the package cache.
Usage
remove_cached_files(pkg_info, relative_filenames)
Arguments
pkg_info |
named list. Package identifier, see get_pkg_info() on how to get one. |
relative_filenames |
vector of strings. A vector of filenames, relative to the package cache. |
Value
logical vector. For each file, whether it was deleted. Note that files which did not exist were not deleted! You should check the results using 'files_available'.
Examples
pkg_info = get_pkg_info("mypackage")
deleted = remove_cached_files(pkg_info, "some_file.txt")
Validate a file manifest.
Description
Check a manifest data.frame for well-formedness, normalize the path and optional columns, and reject paths that would leave the package cache. Used internally by read_manifest.
Usage
validate_manifest(manifest)
Arguments
manifest |
data.frame. The manifest to validate. |
Value
data.frame. The validated and normalized manifest.
Generate a file manifest from a directory of local files.
Description
Create a manifest (CSV file) that describes all files in the given directory. This is meant to remove the tedious manual work of adding many files to a package: put the files into a directory (subdirectories are preserved), run this function once, and a ready-to-use manifest is written. The MD5 checksum of every file is computed automatically.
Usage
write_manifest_from_dir(dir, out, url_base = NULL)
Arguments
dir |
string. The directory containing the local files to describe. Files in subdirectories are included, and their relative paths (using '/' as separator) become the |
out |
string. The path of the CSV file to write the manifest to. Typically something like |
url_base |
string or NULL. If given, the |
Value
The generated manifest as a data.frame, invisibly.
Examples
## Not run:
manifest = write_manifest_from_dir("~/mydata", "inst/extdata/files.csv",
url_base = "https://example.com/data/")
## End(Not run)