Package {data.checker}

data.checker: Data Checker for Validating Data Frames Against Defined Schema

Description

Validates data frames against a defined schema. Produces a report of the checks performed and any issues found, with index and entry value where appropriate. Backend checks are performed using pointblank Richard Iannone et al (2025) doi:10.32614/CRAN.package.pointblank.

Author(s)

Maintainer: Analysis Standards and Pipelines Team (ONS) ASAP@ons.gov.uk

Other contributors:

• Crown Copyright [copyright holder]

See Also

Useful links:

• https://onsdigital.github.io/data.checker/

Pipe operator

Description

See magrittr::%>% for details.

Usage

lhs %>% rhs

Arguments

Value

The result of calling rhs(lhs).

Add a custom check to the validator

Description

This function allows you to add a custom check to the Validatorobject.

Usage

add_check(validator, description, condition)

Arguments

Value

The updated Validator object with the custom check added.

Add a custom check to the validator

Description

This function allows you to add a custom check outcomes to the validator log. The outcomes must be a logical vector.

Usage

add_check_custom(
  validator,
  description,
  outcome,
  type = c("error", "warning", "note")
)

Arguments

Value

The updated Validator object with the custom check added.

Add a QA Entry to the validator's QA Log

Description

This function adds a new entry to the validator's QA log with details such as a description, type of entry, timestamp, pass status, and failing IDs.

Usage

add_qa_entry(
  validator,
  description,
  failing_ids,
  outcome = NA,
  entry_type = c("info", "warning", "error")
)

Arguments

Value

The updated validator object with the new entry appended to its QA log.

Validate a Validator Object

Description

This function runs the full suite of validation checks on a Validator object.

Usage

check(validator, ...)

Arguments

Value

The validated Validator object if all checks pass. If any check fails, an error is thrown.

Examples

# create schema
schema <- list(
  check_duplicates = FALSE,
  check_completeness = FALSE,
  columns = list(
    age = list(type = "double", optional = FALSE),
    sex = list(type = "character", optional = FALSE)
  )
)

# create dataframe
df <- data.frame(
  age = c(10, 11, 13, 15, 22, 34, 80),
  sex = c("M", "F", "M", "F", "M", "F", "M")
)

# create validator object
validator <- new_validator(
  data = df,
  schema = schema
)
# validate the data
validator <- check(validator)

Validate data against a schema and output results

Description

This function validates data against a given schema, performs checks, and exports the validation results to a specified file in a given format.

Usage

check_and_export(
  data,
  schema,
  file,
  format,
  hard_check = FALSE,
  backseries = NULL,
  name = deparse(substitute(data))
)

Arguments

Value

The exported validation results.

Examples

# create schema
schema <- list(
  check_duplicates = FALSE,
  check_completeness = FALSE,
  columns = list(
    age = list(type = "double", optional = FALSE),
    sex = list(type = "character", optional = FALSE)
  )
)

# create dataframe
df <- data.frame(
  age = c(10, 11, 13, 15, 22, 34, 80),
  sex = c("M", "F", "M", "F", "M", "F", "M")
)

# validate and export log
check_and_export(
  data = df,
  schema = schema,
  file = paste0(tempdir(),"\\validation_results_example.html"),
  format = "html",
  hard_check = TRUE
)

Check backseries consistency

Description

Checks if the latest data is consistent with previous data.

Usage

check_backseries(validator)

Arguments

Value

The updated Validator object with outcomes logged.

Check Column Names against schema

Description

This function performs checks on the column names of a Validator object to ensure they follow specific naming conventions and meet schema conditions.

Usage

check_colnames(validator)

Arguments

Details

The function performs the following checks on the column names:

• Ensures column names do not contain spaces.

• Ensures column names do not contain symbols other than underscores.

• Ensures column names do not contain uppercase letters.

For each check, a QA entry is added to the Validator object with details about the check, whether it passed, and the IDs of failing columns (if any). # nolint: line_length_linter.

Value

The updated Validator object with QA entries for each check.

Check Column Contents against schema and checks

Description

This function performs checks on the columns of Validator$data to ensure they meet the specified schema conditions and checks.

Usage

check_column_contents(validator)

Arguments

Value

The updated Validator object with QA entries for each check.

Check dataset for missing columns

Description

Check dataset for missing columns

Usage

check_completeness(validator)

Arguments

Value

The updated validator object with new log entries appended.

Check for duplicate rows. Can use subset of columns to check for duplicates if duplicates_cols is specified in the schema. Otherwise, all columns are used for duplicate check.

Description

Check for duplicate rows. Can use subset of columns to check for duplicates if duplicates_cols is specified in the schema. Otherwise, all columns are used for duplicate check.

Usage

check_duplicates(validator)

Arguments

Value

The updated validator object with new log entries appended.

Check schema contents against the data frame provided

Description

This function checks that the contents of the schema are consistent with the data frame provided. It checks for unused schema entries, incompatible schema entries, and that any columns specified in the schema are present in the data frame.

Usage

check_schema_contents_against_df(validator)

Arguments

Value

The updated Validator object with QA entries added for any issues found in the schema.

Check Column Types and Classes

Description

This function checks the types and classes of the columns in the data against the schema defined in the Validator object.

Usage

check_types(validator)

Arguments

Value

The updated Validator object with quality assurance (QA) entries added for type and class checks. Each QA entry includes a description, pass/fail status, and any failing column IDs.

Generic export function

Description

This function exists For ease of use - see export.Validator() for details.

Usage

export(object, ...)

Arguments

Value

The result of the export operation, specific to the object type.

Export Validator Log

Description

This function exports the log of a Validator object to a file in the specified format.

Usage

## S3 method for class 'Validator'
export(object, file, format = c("yaml", "json", "html", "csv"), ...)

Arguments

Value

Writes the log to the specified file. No value is returned.

Check the status of errors and warnings in the validator log

Description

This function raises errors or warnings if any checks flagged as error or warnings fail.

Usage

hard_checks_status(validator, hard_check)

Arguments

Value

Warning if there are any warnings or errors in the log when hard_check is FALSE. Error if there are any errors and hard_check is TRUE.

Flag outliers based on Interquartile Range (IQR). Outliers are flagged if they are below Q1 - (mulitplier * IQR) or above Q3 + (multiplier * IQR).

Description

Flag outliers based on Interquartile Range (IQR). Outliers are flagged if they are below Q1 - (mulitplier * IQR) or above Q3 + (multiplier * IQR).

Usage

iqr_bounds(x, multiplier = 1.5)

Arguments

Value

A vector the same size as x, with TRUE for values that are outliers and FALSE otherwise

Check column contents valid

Description

This wrapper calls is_valid_column_values for each column in the schema

Usage

is_column_contents_valid(schema)

Arguments

Value

TRUE if all column values are valid, otherwise an error is raised.

Check type of column in schema is valid

Description

Check type of column in schema is valid

Usage

is_type_valid(schema)

Arguments

Value

TRUE if all column types are valid, otherwise an error is raised.

Check that max values are not less than min values in column schema

Description

This function checks that for any column schema, the max values (e.g., max_string_length, max_date) are not less than the corresponding min values (e.g., min_string_length, min_date). If any such inconsistency is found, an error is raised with a descriptive message.

Usage

is_valid_column_values(column_schema, col_name)

Arguments

Value

TRUE if all max values are greater than or equal to their corresponding min values, otherwise an error is raised.

Check if the schema is valid

Description

Check if the schema is valid

Usage

is_valid_schema(schema)

Arguments

Value

TRUE if the schema is a valid named list, otherwise FALSE.

Generate HTML Representation of a Log

Description

Generate HTML Representation of a Log

Usage

log_html(validator)

Arguments

Value

A string containing the HTML representation of the log.

Log pointblank validation outcomes to a validator log

Description

This function extracts validation results from a pointblank agent and appends them to the validator's log.

Usage

log_pointblank_outcomes(validator)

Arguments

Details

Each entry in the log will contain the timestamp, description, outcome, failing row indices, number of failures, and entry type for each validation step.

Value

The updated validator list with new log entries appended.

Convert Validator Log to Table

Description

This function converts a validator log into a formatted data frame (table) for exports.

Usage

log_to_table(log)

Arguments

Value

A data frame containing the formatted log entries.

Validator Constructor

Description

Creates a Validator object to validate data against a given schema.

Usage

new_validator(
  data,
  schema,
  backseries = NULL,
  name = deparse(substitute(data))
)

Arguments

Value

An object of class Validator.

Examples

# create schema
schema <- list(
  check_duplicates = FALSE,
  check_completeness = FALSE,
  columns = list(
    age = list(type = "double", optional = FALSE),
    sex = list(type = "character", optional = FALSE)
  )
)

# create dataframe
df <- data.frame(
  age = c(10, 11, 13, 15, 22, 34, 80),
  sex = c("M", "F", "M", "F", "M", "F", "M")
)

# create validator object
validator <- new_validator(
  data = df,
  schema = schema
)

Print Validator Log

Description

This function prints the log of a Validator object in a markdown table format.

Usage

## S3 method for class 'Validator'
print(x, ...)

Arguments

Value

A markdown-formatted table of the validator log.

Run column checks

Description

To be used by check_column_contents - not intended to be run separately.

Usage

run_checks(validator, i_col)

Arguments

Value

validator object

Convert complex types to the correct types and classes

Description

This function modifies a schema by converting column types to their corresponding R classes.

Usage

types_to_classes(schema)

Arguments

Value

The modified schema with updated type and class fields for each column.

Validate date formats in the schema This function checks that any date formats specified in the schema are valid and can be parsed correctly.

Description

Validate date formats in the schema This function checks that any date formats specified in the schema are valid and can be parsed correctly.

Usage

validate_and_convert_date_formats(schema)

Arguments

Value

The original schema if all date formats are valid. If any date format is invalid, an error is thrown with a message indicating the issue.

Check Z Score of Numeric Columns

Description

This function calculates the maximum z-score for a numeric column.

Usage

z_score(x)

Arguments

Value

A vector of the same length as x, indicating the z-score for each element.