Introduction to RenyiExtropy

Overview

RenyiExtropy provides a unified interface for computing entropy and extropy measures for discrete probability distributions. All functions accept a probability vector (or matrix for joint measures) and return a single numeric value measured in nats (natural-logarithm base).

The package covers three families of measures:

Probability vectors

Every function that accepts a vector p requires it to satisfy:

p <- c(0.2, 0.5, 0.3)   # valid 3-outcome distribution

Shannon entropy

The Shannon entropy is defined as \[H(\mathbf{p}) = -\sum_{i=1}^n p_i \log p_i\] and equals zero for a degenerate distribution and \(\log n\) for the uniform distribution.

shannon_entropy(p)               # three-outcome distribution
#> [1] 1.029653
shannon_entropy(rep(0.25, 4))    # uniform: H = log(4)
#> [1] 1.386294
shannon_entropy(c(1, 0, 0))      # degenerate: H = 0
#> [1] 0
normalized_entropy(p)            # H(p) / log(n), always in [0, 1]
#> [1] 0.9372306

Renyi entropy

The Renyi entropy of order \(q > 0\) is \[H_q(\mathbf{p}) = \frac{1}{1-q} \log\!\left(\sum_i p_i^q\right)\]

For \(q \to 1\) it reduces to the Shannon entropy; the function automatically returns the Shannon limit when \(|q - 1| < 10^{-8}\).

renyi_entropy(p, q = 2)         # collision entropy
#> [1] 0.967584
renyi_entropy(p, q = 0.5)       # Renyi entropy, q = 0.5
#> [1] 1.063659
renyi_entropy(p, q = 1)         # limit: equals shannon_entropy(p)
#> [1] 1.029653
shannon_entropy(p)               # same value
#> [1] 1.029653

Tsallis entropy

The Tsallis entropy is \[S_q(\mathbf{p}) = \frac{1 - \sum_i p_i^q}{q - 1}\] a non-extensive generalisation of Shannon entropy widely used in statistical physics.

tsallis_entropy(p, q = 2)
#> [1] 0.62
tsallis_entropy(p, q = 1)       # limit: equals shannon_entropy(p)
#> [1] 1.029653

Extropy

Extropy is the dual complement of entropy, measuring information via complementary probabilities: \[J(\mathbf{p}) = -\sum_{i=1}^n (1-p_i)\log(1-p_i)\]

extropy() and shannon_extropy() compute the same quantity.

extropy(p)
#> [1] 0.7747609
shannon_extropy(p)               # identical
#> [1] 0.7747609

Renyi extropy

The Renyi extropy of order \(q\) is \[J_q(\mathbf{p}) = \frac{-(n-1)\log(n-1) + (n-1)\log\!\left(\sum_i(1-p_i)^q\right)}{1-q}\]

For \(n = 2\) it equals the Renyi entropy. For \(q \to 1\) it returns the classical extropy.

renyi_extropy(p, q = 2)
#> [1] 0.7421274
renyi_extropy(p, q = 1)          # limit: equals extropy(p)
#> [1] 0.7747609

# n = 2: Renyi extropy == Renyi entropy
renyi_extropy(c(0.4, 0.6), q = 2)
#> [1] 0.6539265
renyi_entropy(c(0.4, 0.6), q = 2)
#> [1] 0.6539265

# Maximum Renyi extropy over n outcomes (uniform distribution)
max_renyi_extropy(3)
#> [1] 0.8109302
renyi_extropy(rep(1/3, 3), q = 2)
#> [1] 0.8109302

Joint and conditional entropy

joint_entropy() accepts a matrix of joint probabilities; rows correspond to outcomes of \(X\), columns to outcomes of \(Y\).

Pxy <- matrix(c(0.2, 0.3, 0.1, 0.4), nrow = 2, byrow = TRUE)

joint_entropy(Pxy)               # H(X, Y)
#> [1] 1.279854
conditional_entropy(Pxy)         # H(Y | X) = H(X,Y) - H(X)
#> [1] 0.586707

# Independent distributions: H(X,Y) = H(X) + H(Y)
px <- c(0.4, 0.6)
py <- c(0.3, 0.7)
Pxy_indep <- outer(px, py)
joint_entropy(Pxy_indep)
#> [1] 1.283876
shannon_entropy(px) + shannon_entropy(py)
#> [1] 1.283876

Conditional Renyi extropy

conditional_renyi_extropy(Pxy, q = 2)
#> [1] 0.1039623
conditional_renyi_extropy(Pxy, q = 1)   # limit: conditional Shannon extropy
#> [1] 0.13636

Divergences

KL divergence

The Kullback-Leibler divergence \(D_\text{KL}(P \| Q)\) is asymmetric and can be infinite when \(q_i = 0\) but \(p_i > 0\) (a warning is emitted in that case).

q_dist <- c(0.3, 0.4, 0.3)
kl_divergence(p, q_dist)         # KL(P || Q)
#> [1] 0.03047875
kl_divergence(q_dist, p)         # KL(Q || P) -- asymmetric
#> [1] 0.03238211
kl_divergence(p, p)              # 0
#> [1] 0

Jensen-Shannon divergence

The Jensen-Shannon divergence is the symmetrised, always-finite version, bounded in \([0, \log 2]\):

js_divergence(p, q_dist)         # symmetric
#> [1] 0.0078174
js_divergence(q_dist, p)         # same value
#> [1] 0.0078174
js_divergence(p, p)              # 0
#> [1] 0
js_divergence(c(1, 0), c(0, 1)) # maximum: log(2)
#> [1] 0.6931472
log(2)
#> [1] 0.6931472

Cross-entropy

Cross-entropy relates to KL divergence via \(H(P, Q) = H(P) + D_\text{KL}(P \| Q)\):

cross_entropy(p, q_dist)
#> [1] 1.060132
shannon_entropy(p) + kl_divergence(p, q_dist)   # same value
#> [1] 1.060132
cross_entropy(p, p)                              # equals shannon_entropy(p)
#> [1] 1.029653

Input validation

All functions produce informative errors when given invalid inputs:

shannon_entropy(c(0.2, 0.3, 0.1))   # does not sum to 1
#> Error in `shannon_entropy()`:
#> ! Elements of 'p' must sum to 1 (tolerance 1e-8).
renyi_entropy(p, q = NA)            # NA not allowed
#> Error in `renyi_entropy()`:
#> ! 'q' must not be NA or NaN.
max_renyi_extropy(1)                # n must be >= 2
#> Error in `max_renyi_extropy()`:
#> ! 'n' must be at least 2.
kl_divergence(p, c(0.5, 0.5))      # length mismatch
#> Error in `kl_divergence()`:
#> ! 'p' and 'q' must have the same length.