madshapR 2.0.0 (release : 2025-06-15)

Attention: Some changes to functions in the current version of madshapR may require updates of existing code.

Superseded object.

previous version (1.1.0 and older)	version 2.0.0
madshapR_DEMO	madshapR_examples

Superseded parameters.

previous version (1.1.0 and older)	current version (2.0.0)
dataset_evaluate(as_data_dict_mlstr)	dataset_evaluate(is_data_dict_mlstr)
data_dict_evaluate(as_data_dict_mlstr)	data_dict_evaluate(is_data_dict_mlstr)
dossier_evaluate(as_data_dict_mlstr)	dossier_evaluate(is_data_dict_mlstr)

Superseded function behaviors and/or output structures.

In dataset_evaluate(), data_dict_evaluate() and dossier_evaluate(), the columns generated in the outputs have been renamed as follows :

previous version (1.1.0 and older)	current version (2.0.0)
index	Index
name	Variable name
label	Variable label
valueType	Data dictionary valueType
Categories::label	Categories in data dictionary
Categories::missing	Non-valid categories

In dataset_summarize() and dossier_summarize(), the columns generated in the outputs have been renamed as follows :

previous version (1.1.0 and older)	current version (2.0.0)
index in data dict.name	Index
name	Variable name
label	Variable label
Estimated dataset valueType	Suggested valueType
Actual dataset valueType	Dataset valueType
Total number of observations	Number of rows
Nb. distinct values	Number of distinct values
Nb. valid values	Number of valid values
Nb. non-valid values	Number of non-valid values
Nb. NA	Number of empty values
% total Valid values	% Valid values
% Non-valid values	% Non-valid values
% NA	% Empty values
————————————	———————————

Bug fixes and improvements

The package now handles the valueType datetime, which formerly was considered either as a text or date.

https://github.com/maelstrom-research/madshapR/issues/123

https://github.com/maelstrom-research/madshapR/issues/112

https://github.com/maelstrom-research/madshapR/issues/75

The valueType object (present in columns in a data dictionary or as an attribute of a variable) had some errors and bugs that have been corrected.

https://github.com/maelstrom-research/madshapR/issues/87

https://github.com/maelstrom-research/madshapR/issues/82

https://github.com/maelstrom-research/madshapR/issues/81

https://github.com/maelstrom-research/madshapR/issues/76

When a column in a dataset is all NA (empty), the previous version had some issues that have been has been corrected.

https://github.com/maelstrom-research/madshapR/issues/116

https://github.com/maelstrom-research/madshapR/issues/115

https://github.com/maelstrom-research/madshapR/issues/109

A bug in data_dict_pivot_longer() when ‘Source’ or ‘Target’ column was not present has been corrected.

https://github.com/maelstrom-research/madshapR/issues/86

The SPSS format, which haven package uses to produce labelled variables, define integers different form madshapR, which ultimately . That has been taken in account and corrected.
The SPSS format in the haven package used to produce labelled variables defines integers differently from madshapR, which was causing errors. The difference has been taken into account.

https://github.com/maelstrom-research/madshapR/issues/83

The group_by parameter has been redesigned.

dataset_preprocess() now handles grouped dataset, using parameter “group_by”.
Users can now define groups in summaries and visual reports using a variable that is not categorical or has empty values.

https://github.com/maelstrom-research/madshapR/issues/47

Previously, the “group_by” argument had some flaws, resulting in bugs that have been corrected.

https://github.com/maelstrom-research/madshapR/issues/114

https://github.com/maelstrom-research/madshapR/issues/113

https://github.com/maelstrom-research/madshapR/issues/110

https://github.com/maelstrom-research/madshapR/issues/105

Enhancements in the assessment and summary reports!

The assessment and summary reports had some updates, such as renamed columns and bug corrections.

https://github.com/maelstrom-research/madshapR/issues/126

https://github.com/maelstrom-research/madshapR/issues/104

https://github.com/maelstrom-research/madshapR/issues/98

https://github.com/maelstrom-research/madshapR/issues/97

https://github.com/maelstrom-research/madshapR/issues/96

https://github.com/maelstrom-research/madshapR/issues/95

https://github.com/maelstrom-research/madshapR/issues/94

https://github.com/maelstrom-research/madshapR/issues/93

https://github.com/maelstrom-research/madshapR/issues/92

https://github.com/maelstrom-research/madshapR/issues/91

https://github.com/maelstrom-research/madshapR/issues/90

https://github.com/maelstrom-research/madshapR/issues/89

https://github.com/maelstrom-research/madshapR/issues/88

https://github.com/maelstrom-research/madshapR/issues/85

https://github.com/maelstrom-research/madshapR/issues/80

https://github.com/maelstrom-research/madshapR/issues/79

Enhancements in the visual reports!

The visual reports have been improved, including better visual outputs and color palettes, and new features such as total number of rows next to the bar charts.

https://github.com/maelstrom-research/madshapR/issues/108

https://github.com/maelstrom-research/madshapR/issues/107

https://github.com/maelstrom-research/madshapR/issues/106

https://github.com/maelstrom-research/madshapR/issues/100

https://github.com/maelstrom-research/madshapR/issues/84

https://github.com/maelstrom-research/madshapR/issues/64

New functions

typeof_convert_to_valueType() converts typeof (and class if any) into its corresponding valueType.
valueType_convert_to_typeof() converts valueType into its corresponding typeof and class in R representation.
data_dict_update() updates a data dictionary from a dataset.
data_dict_trim_labels() adds shortened labels to data dictionary.
first_label_get() gets the first label from a data dictionary.
has_categories() tests if a dataset has categorical variables.

madshapR 1.1.0

Bug fixes and improvements

for assessment, summary and visualization, the character columns in dataset are put to lower to avoid duplicated informations in outputs.

https://github.com/maelstrom-research/madshapR/issues/63

bug in the function variable_visualize() when the column was empty after removing internally stopwords.

https://github.com/maelstrom-research/Rmonize/issues/53

https://github.com/maelstrom-research/Rmonize/issues/49

Some elements were missing in the reports in dataset_evaluate()

https://github.com/maelstrom-research/madshapR/issues/66

Problem with names containing underscores in variables when visualized fixed.

https://github.com/maelstrom-research/madshapR/issues/62

Functions involving valueType (such as data_dict_apply(),valueType_guess() and valueType_adujst()) have been corrected to be more consistent in the usage of these functions.

https://github.com/maelstrom-research/madshapR/issues/61

The bug affecting tibbles which contain a variable named “test” has been corrected in the package fabR.

https://github.com/maelstrom-research/madshapR/issues/60

functions such as data_dict_summarize() and dataset_evaluate() have cells in tibble generated that can have more than accepted characters in a cell in Excel. the function truncates the cells in tibbles to a maximum of 10000 characters.

https://github.com/maelstrom-research/madshapR/issues/59

Problem with dataType in the function dataset_cat_as_labels() when the values found in the dataset are not in the data dictionary, and the valueType is text, and the dataType is “integer” has been fixed.

https://github.com/maelstrom-research/madshapR/issues/58

Functions involving date formatted variables have been corrected in the package fabR.

https://github.com/maelstrom-research/madshapR/issues/57

The inconsistent error in dataset_evaluate() has been corrected in the package fabR.

https://github.com/maelstrom-research/madshapR/issues/46

deprecated functions

To avoid confusion with help(function), the function madshapR_help() has been renamed madshapR_website().

Dependency changes

set a minimum dplyr dependence to avoid bugs

madshapR 1.0.3

Bug fixes and improvements

Some of the tests were made with another package (Rmonize) which as “madshapR” as a dependence.

Enhance reports

in visual reports, void confusing changes in color scheme in visual reports.
Histograms for date variables display valid ranges.
in reports, change % NA as proportion in reports.
dossier_visualize() report shows variable labels in the same lang.
in visual reports, the bar plot only appears when there are multiple missing value types, otherwise only the pie chart is shown.
in reports, all of the percentages are now included under “Other values (non categorical)”, which gives a single value.
https://github.com/maelstrom-research/madshapR/issues/51

suppress overwrite parameter in dataset_visualize().

https://github.com/maelstrom-research/madshapR/issues/42

in dataset_summary() minor issue (consistency in column names and content).

correction of the function variable_visualize() when valueType_guess = TRUE

Correct Data dictionary functions

https://github.com/maelstrom-research/madshapR/issues/50

enhance the function check_data_dict_valueType(), which was too slow.

https://github.com/maelstrom-research/madshapR/issues/49

valueType_adjust() now works with empty column (all NAs)

allow the format date to be transformed into text in dataset_zap_data_dict() when the format is unclear.

New functions

col_id() function which is a short cut for calling the attribute madshapR::col_id of a dataset.
as_category(),is_category(),drop_category() function which coerces a vector as a categorical object. Typically a column in a dataset that needs to be coerced into a categorical variable (The data dictionary is updated accordingly).

Deprecated functions

Rename and update example rda Object (in data) of DEMO_files into madshapR_DEMO for consistency across our other packages.

madshapR 1.0.2

Creation of NEWS feed !!

Addition of NEWS.md for the development version use “(development version)”.

Bug fixes and improvements

Some improvements in the documentation of the package has been made.
internal call of libraries (using ::) has been replaced by proper import in the declaration function.
get functions in fabR have been changed in its last release. the functions using them as dependencies ( check_xxx()) have been updated accordingly.
DEMO files no longer include harmonization files that are now in the package harmonizR

Dependency changes

New Imports: haven, lifecycle

No longer in Imports: xfun

New functions

These functions are imported from fabR

bookdown_template() replaces the deprecated function bookdown_template().
bookdown_render() which renders a Rmd collection of files into a docs/index.html website.
bookdown_open() Which allows to open a docs/index.html document when the bookdown is rendered

This separation into 3 functions will allow future developments, such as render as a ppt or pdf.

deprecated functions

Due to another package development (see fabR), The function open_visual_report() has been deprecated in favor of bookdown_open() imported from fabR package.

madshapR 1.0.0

This package is a collection of wrapper functions used in data pipelines.

This is still a work in progress, so please let us know if you used a function before and is not working any longer.

Helper functions

madshapR_help() Call the help center for full documentation

functions to generate, shape and format meta data.

These functions allows to create, extract transform and apply meta data to a dataset.

Transform and shape:

data_dict_collapse(),data_dict_expand(),data_dict_filter(), data_dict_group_by(),data_dict_group_split(),data_dict_list_nest(), data_dict_pivot_longer(),data_dict_pivot_wider(),data_dict_ungroup()

extract/apply meta data:

data_dict_match_dataset(),data_dict_apply(), data_dict_extract()

evaluate and apply attributes:

as_data_dict(), as_data_dict_mlstr(),as_data_dict_shape(), is_data_dict(), is_data_dict_mlstr(), is_data_dict_shape() as_taxonomy(), is_taxonomy()

functions to generate, shape and format data.

These functions allows to create, extract transform data/meta data from a dataset. A dossier is a list of datasets.

evaluate and apply attributes:

as_dataset(), as_dossier() is_dataset(), is_dossier()

Extract/transform meta data: data_extract(), dossier_create(), dataset_zap_data_dict(), dataset_cat_as_labels()

Functions to work with data types

These functions allow user to work with, extract or assign data type (valueType) to values and/or dataset.

as_valueType(), is_valueType(), valueType_adjust(), valueType_guess(), valueType_self_adjust(), valueType_of()

Unit tests and QA for datasets and data dictionaries

These helper functions evaluate content of a dataset and/or data dictionary to extract from them irregularities or potential errors. These informations are stored in a tibble that can be use to assess inputs.

check_data_dict_categories(), check_data_dict_missing_categories(), check_data_dict_taxonomy(), check_data_dict_variables(), check_data_dict_valueType(), check_dataset_categories(), check_dataset_valueType(), check_dataset_variables(), check_name_standards()

Summarize information in dataset and data dictionaries

These helper functions evaluate content of a dataset and/or data dictionary to extract from them summary statistics and elements such as missing values, NA, category names, etc. These informations are stored in a tibble that can be use to summary inputs.

dataset_preprocess(), summary_variables(), summary_variables_categorical(),summary_variables_date(), summary_variables_numeric(),summary_variables_text()

Write and read excel and csv

read_csv_any_formats() The csv file is read twice to detect the number of lines to use in attributing the column type (guess_max parameter of read_csv). This avoids common errors when reading csv files.
read_excel_allsheets() The Excel file is read and the values are placed in a list of tibbles, with each sheet in a separate element in the list. If the Excel file has only one sheet, the output is a single tibble.
write_excel_allsheets() Write all Excel sheets using xlsx::write.xlsx() recursively.

Plot and summary functions used in a visual report

plot_bar(), plot_box(), plot_date(), plot_density(), plot_histogram(), plot_main_word(), plot_pie_valid_value(), summary_category(), summary_numerical(),summary_text()

aggregate information and generate reports

assess data

data_dict_evaluate() dataset_evaluate() dossier_evaluate()

summarize data

dataset_summarize() dossier_summarize()

visualize data

dataset_visualize() variable_visualize() open_visual_report()