SWORD (Support vector machine Weighted Oblique Random Decision forests) fits random forests of oblique regression trees. Each internal node uses a Weighted-SVM classifier with a linear kernel to define the oblique splitting hyperplane — a sparse linear combination of predictors — rather than a single axis-aligned threshold.
The base learner is TORS (Tree Oblique for Regression with weighted SVM), a single oblique regression tree introduced in Carta & Frigau (2025). SWORD embeds TORS within a bootstrap aggregation framework, adding predictor subspacing and optional randomisation of the number of features per split to increase tree diversity.
The package provides:
TORS() — fit a single TORS oblique regression
tree.SWORD() — fit a SWORD forest of TORS trees with
bootstrap sampling and out-of-bag (OOB) error estimation.predict() — S3 method for both objects, with automatic
encoding of factor and character predictors.VI_SWORD() — Oblique Impurity-Weighted Variable
Importance (OIW-VI): deviance reduction at each node, distributed among
predictors proportionally to their normalised absolute Weighted-SVM
coefficients.We use the Boston housing dataset from the
MASS package (506 observations, 13 numeric predictors,
response medv = median house value). The data are split
80/20 into training and test sets.
if (!requireNamespace("MASS", quietly = TRUE))
stop("Please install the MASS package: install.packages('MASS')")
library(SWORD)
data(Boston, package = "MASS")
set.seed(42)
n <- nrow(Boston)
train <- sample(n, floor(0.8 * n))
test <- setdiff(seq_len(n), train)
X_train <- Boston[train, setdiff(names(Boston), "medv")]
y_train <- Boston$medv[train]
X_test <- Boston[test, setdiff(names(Boston), "medv")]
y_test <- Boston$medv[test]
cat("Training set:", nrow(X_train), "obs |",
"Test set:", nrow(X_test), "obs\n")
#> Training set: 404 obs | Test set: 102 obsTORS() grows one SVM oblique rotation tree. Key tuning
parameters:
| Parameter | Meaning | Default |
|---|---|---|
nmin |
minimum node size | 5 |
cp |
complexity threshold for splitting | 0.01 |
n_top_cor |
top-correlated features fed to the SVM at each node | 2 |
relation |
feature selection criterion: "Pearson" or
"MI" |
"Pearson" |
weight_scheme |
observation weighting: "scale" or
"robust" |
"scale" |
cost_c |
SVM cost C | 1 |
set.seed(1)
tree <- TORS(X_train, y_train, nmin = 15, cp = 0.02)
print(tree)
#> ---------- TORS (Tree Oblique for Regression with weighted SVM)
#>
#> N observations: 404
#> N predictors: 13
#> N nodes: 25 (12 internal, 13 leaves)
#> Max depth: 4
#> Avg obs in leaf: 31.1
#> Min obs in leaf: 7
#> Top-correlated feats: 2
#> Correlation: Pearson
#> Weight scheme: scale
#>
#> -----------------------------------------plot_tors_text() prints the tree as an indented list of
nodes. Each internal node shows the oblique hyperplane equation
(top-k largest coefficients); each leaf shows the predicted
mean.
plot_tors_text(tree, top_k = 3)
#> TORS — text representation
#> ------------------------------------------------------------
#> Node 1 n=404 [ - 0.957·rm + 0.282·lstat - 2.837 = 0 ]
#> L-- Node 2 n=193 [ - 1.636·rm + 0.296·lstat - 8.985 = 0 ]
#> L-- Node 3 n=77 [ - 2.047·rm + 0.226·dis - 13.886 = 0 ]
#> L-- Node 4 n=30 [ 0.262·ptratio + 0.683·lstat + 7.896 = 0 ]
#> L-- Leaf n=18 mean=45.4833
#> R-- Leaf n=12 mean=35.8583
#> R-- Node 7 n=47 [ - 5.578·rm + 1.041·lstat - 33.09 = 0 ]
#> L-- Leaf n=26 mean=34.4731
#> R-- Leaf n=21 mean=27.4952
#> R-- Node 10 n=116 [ 0.321·dis - 0.019·rad + 0.978 = 0 ]
#> L-- Node 11 n=35 [ 1.117·dis + 0.307·lstat + 5.236 = 0 ]
#> L-- Leaf n=7 mean=29.8000
#> R-- Leaf n=28 mean=23.9643
#> R-- Leaf n=81 mean=22.9605
#> R-- Node 15 n=211 [ 0.085·crim + 0.267·lstat + 5.252 = 0 ]
#> L-- Node 16 n=131 [ - 0.009·black + 0.430·lstat + 3.32 = 0 ]
#> L-- Node 17 n=82 [ 0.529·dis + 0.254·ptratio + 6.816 = 0 ]
#> L-- Leaf n=56 mean=20.9036
#> R-- Leaf n=26 mean=19.0615
#> R-- Leaf n=49 mean=17.3367
#> R-- Node 21 n=80 [ 0.013·crim + 0.007·tax + 4.194 = 0 ]
#> L-- Leaf n=20 mean=15.5300
#> R-- Node 23 n=60 [ 5.951·nox + 0.270·lstat + 10.225 = 0 ]
#> L-- Leaf n=33 mean=13.8242
#> R-- Leaf n=27 mean=10.0074plot() on a tors_flat object draws a base-R
layout of the tree (node size proportional to the number of
observations). If the visNetwork package is installed
it renders an interactive zoomable widget instead.
cp on tree sizecp is a complexity threshold: a candidate split is
accepted only if the relative deviance reduction exceeds
cp. Larger values lead to earlier stopping and shallower
trees; cp = 0 lets the tree grow until every node reaches
nmin.
set.seed(1)
t_deep <- TORS(X_train, y_train, nmin = 10, cp = 0.000)
t_moderate <- TORS(X_train, y_train, nmin = 10, cp = 0.010)
t_shallow <- TORS(X_train, y_train, nmin = 10, cp = 0.050)
cp_tbl <- data.frame(
cp = c(0.000, 0.010, 0.050),
n_nodes = c(t_deep$n_nodes, t_moderate$n_nodes, t_shallow$n_nodes),
n_leaves = c(sum(t_deep$is_leaf), sum(t_moderate$is_leaf), sum(t_shallow$is_leaf)),
max_depth = c(max(t_deep$depth), max(t_moderate$depth), max(t_shallow$depth))
)
print(cp_tbl, row.names = FALSE)
#> cp n_nodes n_leaves max_depth
#> 0.00 107 54 8
#> 0.01 37 19 5
#> 0.05 13 7 4With cp = 0 the tree grows until every node reaches
nmin, producing the deepest structure; increasing
cp results in more aggressive pruning and shallower
trees.
n_top_cor on split richnessn_top_cor controls how many top-correlated features are
passed to the Weighted-SVM at each node. Because TORS splits are always
oblique (linear combinations of predictors), the minimum meaningful
value is 2; larger values allow the SVM to combine more predictors into
denser oblique hyperplanes.
set.seed(1)
t_nc2 <- TORS(X_train, y_train, nmin = 15, cp = 0.02, n_top_cor = 2)
t_nc4 <- TORS(X_train, y_train, nmin = 15, cp = 0.02, n_top_cor = 4)
cat("--- n_top_cor = 2 (default, sparse oblique) ---\n")
#> --- n_top_cor = 2 (default, sparse oblique) ---
plot_tors_text(t_nc2, top_k = 2)
#> TORS — text representation
#> ------------------------------------------------------------
#> Node 1 n=404 [ - 0.957·rm + 0.282·lstat - 2.837 = 0 ]
#> L-- Node 2 n=193 [ - 1.636·rm + 0.296·lstat - 8.985 = 0 ]
#> L-- Node 3 n=77 [ - 2.047·rm + 0.226·dis - 13.886 = 0 ]
#> L-- Node 4 n=30 [ 0.262·ptratio + 0.683·lstat + 7.896 = 0 ]
#> L-- Leaf n=18 mean=45.4833
#> R-- Leaf n=12 mean=35.8583
#> R-- Node 7 n=47 [ - 5.578·rm + 1.041·lstat - 33.09 = 0 ]
#> L-- Leaf n=26 mean=34.4731
#> R-- Leaf n=21 mean=27.4952
#> R-- Node 10 n=116 [ 0.321·dis - 0.019·rad + 0.978 = 0 ]
#> L-- Node 11 n=35 [ 1.117·dis + 0.307·lstat + 5.236 = 0 ]
#> L-- Leaf n=7 mean=29.8000
#> R-- Leaf n=28 mean=23.9643
#> R-- Leaf n=81 mean=22.9605
#> R-- Node 15 n=211 [ 0.085·crim + 0.267·lstat + 5.252 = 0 ]
#> L-- Node 16 n=131 [ - 0.009·black + 0.430·lstat + 3.32 = 0 ]
#> L-- Node 17 n=82 [ 0.529·dis + 0.254·ptratio + 6.816 = 0 ]
#> L-- Leaf n=56 mean=20.9036
#> R-- Leaf n=26 mean=19.0615
#> R-- Leaf n=49 mean=17.3367
#> R-- Node 21 n=80 [ 0.013·crim + 0.007·tax + 4.194 = 0 ]
#> L-- Leaf n=20 mean=15.5300
#> R-- Node 23 n=60 [ 5.951·nox + 0.270·lstat + 10.225 = 0 ]
#> L-- Leaf n=33 mean=13.8242
#> R-- Leaf n=27 mean=10.0074
cat("\n--- n_top_cor = 4 (richer oblique splits) ---\n")
#>
#> --- n_top_cor = 4 (richer oblique splits) ---
plot_tors_text(t_nc4, top_k = 4)
#> TORS — text representation
#> ------------------------------------------------------------
#> Node 1 n=404 [ - 0.031·indus - 0.934·rm + 0.172·ptratio + 0.294·lstat + 0.241 = 0 ]
#> L-- Node 2 n=181 [ - 1.278·rm + 0.203·dis + 0.162·ptratio + 0.342·lstat - 2.681 = 0 ]
#> L-- Node 3 n=70 [ - 0.261·crim - 1.894·rm + 0.207·dis + 0.405·lstat - 11.347 = 0 ]
#> L-- Node 4 n=22 [ 0.837·rm - 0.112·rad + 0.511·ptratio - 0.012·black + 9.77 = 0 ]
#> L-- Leaf n=12 mean=48.2417
#> R-- Leaf n=10 mean=40.9100
#> R-- Node 7 n=48 [ - 0.086·indus - 5.187·rm - 0.000·tax - 0.006·black - 39.064 = 0 ]
#> L-- Leaf n=27 mean=34.0519
#> R-- Leaf n=21 mean=30.1143
#> R-- Node 10 n=111 [ 0.151·crim - 0.328·rad + 0.011·tax + 0.107·lstat + 2.842 = 0 ]
#> L-- Node 11 n=66 [ 0.038·crim - 0.032·age + 0.030·rad + 0.041·black + 14.063 = 0 ]
#> L-- Leaf n=13 mean=29.3231
#> R-- Leaf n=53 mean=24.3283
#> R-- Leaf n=45 mean=22.6933
#> R-- Node 15 n=223 [ 0.066·crim + 0.046·age + 0.001·tax + 0.277·lstat + 10.061 = 0 ]
#> L-- Node 16 n=132 [ - 2.085·nox + 0.042·age - 0.007·black + 0.138·lstat + 1.603 = 0 ]
#> L-- Leaf n=80 mean=20.4100
#> R-- Leaf n=52 mean=18.2769
#> R-- Node 19 n=91 [ 0.039·crim - 0.340·dis + 0.009·tax + 0.237·lstat + 10.901 = 0 ]
#> L-- Leaf n=54 mean=15.0241
#> R-- Leaf n=37 mean=10.5054With n_top_cor = 2 each split combines two predictors;
with n_top_cor = 4 the Weighted-SVM can involve up to four
predictors per split, producing denser oblique hyperplanes at the cost
of higher per-node fitting time.
SWORD() fits a random forest of TORS trees with
bootstrap sampling. Setting oob = TRUE (the default)
accumulates out-of-bag predictions at each iteration, which lets you
monitor convergence without a separate validation set.
set.seed(42)
forest <- SWORD(X_train, y_train,
m = 50,
nmin = 10,
cp = 0.01,
oob = TRUE,
verbose = FALSE)
print(forest)
#> ---------- SWORD (Support vector machine Weighted Oblique Random Decision forest)
#>
#> N observations: 404
#> N predictors: 13
#> N trees: 50
#> Predictors per split: 13
#> Top-correlated feats: random [2, 13]
#> Avg leaves per tree: 12.920
#> Min obs in leaf: 2
#> OOB stat type: RMSE / R²
#> OOB stat value: 4.2225 / 0.7764
#> Weight scheme: scale
#> Correlation: Pearson
#>
#> -----------------------------------------plot(forest, which = "oob") plots the OOB MSE as a
function of the number of trees. The dashed line marks the minimum;
check that it has levelled off before using the forest for
inference.
y_hat <- predict(forest, X_test)
rmse <- sqrt(mean((y_hat - y_test)^2))
mae <- mean(abs(y_hat - y_test))
r2 <- cor(y_hat, y_test)^2
cat(sprintf("Test RMSE : %.3f\n", rmse))
#> Test RMSE : 3.511
cat(sprintf("Test MAE : %.3f\n", mae))
#> Test MAE : 2.240
cat(sprintf("Test R^2 : %.3f\n", r2))
#> Test R^2 : 0.891The predict() method also accepts a
data.frame that contains extra columns (e.g. the response),
so you can pass the original data frame directly:
y_hat2 <- predict(forest, Boston[test, ]) # medv column is silently dropped
all.equal(y_hat, y_hat2)
#> [1] TRUEVI_SWORD() computes the OIW-VI (Oblique
Impurity-Weighted Variable Importance): at each internal node, the
deviance reduction is distributed among the predictors involved in the
oblique split proportionally to their normalised absolute Weighted-SVM
coefficients; these scores are then averaged across all trees.
plot_vi_sword() draws a horizontal barplot sorted by
importance.
vi <- VI_SWORD(forest)
print(round(vi, 2))
#> lstat rm ptratio rad tax dis nox crim age indus
#> 9283.94 4907.46 2487.42 1841.03 1742.81 1721.06 1529.35 1503.95 1058.03 1014.01
#> black zn chas
#> 652.43 468.43 455.07pdp_sword() computes the marginal effect of a single
predictor by fixing it to a grid of values and averaging forest
predictions over the full training set (Friedman 2001). A rug of
observed values is added by default.
old_par <- par(mfrow = c(1, 2), mar = c(4, 4, 3, 1))
pdp_sword(forest, X_train, "lstat", main = "PDP: lstat")
pdp_sword(forest, X_train, "rm", main = "PDP: rm")Top-4 predictors in a 2x2 grid:
top4 <- names(vi)[1:4]
old_par <- par(mfrow = c(2, 2), mar = c(4, 4, 3, 1))
for (v in top4)
pdp_sword(forest, X_train, v, main = paste("PDP:", v))Four parameters most likely to affect predictive accuracy are
relation, weight_scheme, rf_var,
and rand_ntopcor. In this section we fit compact forests
(m = 20, nmin = 10, cp = 0.01)
varying each parameter in turn and compare test-set RMSE on the same
Boston hold-out.
relationrelation controls how the top-correlated features are
identified at each node: "Pearson" uses the absolute
Pearson correlation with the node residuals; "MI" uses
mutual information, which can capture non-linear associations between
each predictor and the response.
set.seed(42)
f_pearson <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
relation = "Pearson", verbose = FALSE)
set.seed(42)
f_mi <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
relation = "MI", verbose = FALSE)
cat(sprintf("Pearson — test RMSE: %.3f\n", rmse_f(y_test, predict(f_pearson, X_test))))
#> Pearson — test RMSE: 3.586
cat(sprintf("MI — test RMSE: %.3f\n", rmse_f(y_test, predict(f_mi, X_test))))
#> MI — test RMSE: 3.592Both criteria are competitive on continuous predictors;
"MI" may offer an advantage when predictor–response
associations are strongly non-linear.
weight_schemeweight_scheme determines how observations are weighted
when fitting the Weighted-SVM at each node. Both schemes assign higher
weight to observations whose response is far from the local centre,
guiding the SVM to concentrate on the most informative thresholds. The
two formulas differ in how the centre and spread are estimated:
\[ w_i^{\text{scale}} = \frac{|y_i - \bar{y}|}{s_y}, \qquad w_i^{\text{robust}} = \frac{|y_i - \text{median}(y)|}{\text{MAD}(y)} \]
where \(s_y\) is the standard
deviation and \(\text{MAD}(y) =
\text{median}(|y_i - \text{median}(y)|)\). Because
"robust" centres on the median and scales by the MAD rather
than the mean and standard deviation, it is less influenced by extreme
values in the response.
set.seed(42)
f_scale <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
weight_scheme = "scale", verbose = FALSE)
set.seed(42)
f_robust <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
weight_scheme = "robust", verbose = FALSE)
cat(sprintf("scale — test RMSE: %.3f\n", rmse_f(y_test, predict(f_scale, X_test))))
#> scale — test RMSE: 3.586
cat(sprintf("robust — test RMSE: %.3f\n", rmse_f(y_test, predict(f_robust, X_test))))
#> robust — test RMSE: 3.556"robust" is recommended when the training response
contains outliers or extreme values that would inflate \(s_y\) and distort the "scale"
weights.
rf_varrf_var sets how many predictors are randomly sub-sampled
at each node before the top-n_top_cor correlated ones are
selected for the SVM. Reducing rf_var increases tree
decorrelation — analogously to mtry in standard random
forests. Three common choices with p = 13 predictors:
set.seed(42)
f_full <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
rf_var = p, verbose = FALSE) # all predictors
set.seed(42)
f_half <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
rf_var = floor(p / 2), verbose = FALSE) # half
set.seed(42)
f_sqrt <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
rf_var = floor(sqrt(p)), verbose = FALSE) # sqrt rule
#> Warning in cor(y, x_mat): la deviazione standard è zero
#> Warning in cor(y, x_mat): la deviazione standard è zero
#> Warning in cor(y, x_mat): la deviazione standard è zero
#> Warning in cor(y, x_mat): la deviazione standard è zero
#> Warning in cor(y, x_mat): la deviazione standard è zero
cat(sprintf("rf_var = %2d (all) — test RMSE: %.3f\n",
p, rmse_f(y_test, predict(f_full, X_test))))
#> rf_var = 13 (all) — test RMSE: 3.586
cat(sprintf("rf_var = %2d (floor p/2) — test RMSE: %.3f\n",
floor(p / 2), rmse_f(y_test, predict(f_half, X_test))))
#> rf_var = 6 (floor p/2) — test RMSE: 3.592
cat(sprintf("rf_var = %2d (floor sqrt) — test RMSE: %.3f\n",
floor(sqrt(p)), rmse_f(y_test, predict(f_sqrt, X_test))))
#> rf_var = 3 (floor sqrt) — test RMSE: 4.488Unlike axis-aligned forests, where reducing mtry
decorrelates trees and often lowers test error, in SWORD a
larger candidate pool consistently improves accuracy:
allowing the Weighted-SVM to screen more predictors at each node leads
to better oblique splits without sacrificing tree diversity (which is
already provided by bootstrap sampling and the random- mechanism).
Restricting rf_var limits the quality of the oblique
hyperplane and in practice tends to worsen RMSE. The default
(rf_var = p, all predictors) is therefore recommended.
rand_ntopcorWhen rand_ntopcor = TRUE (the SWORD default) the number
of top-correlated features passed to the SVM at each node is itself
drawn at random from , introducing an extra source of tree diversity
beyond bootstrap sampling. Setting it to FALSE fixes that
count at n_top_cor for every split.
set.seed(42)
f_rnd_t <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
rand_ntopcor = TRUE, verbose = FALSE)
set.seed(42)
f_rnd_f <- SWORD(X_train, y_train, m = 20, nmin = 10, cp = 0.01,
rand_ntopcor = FALSE, verbose = FALSE)
cat(sprintf("rand_ntopcor = TRUE — test RMSE: %.3f\n",
rmse_f(y_test, predict(f_rnd_t, X_test))))
#> rand_ntopcor = TRUE — test RMSE: 3.586
cat(sprintf("rand_ntopcor = FALSE — test RMSE: %.3f\n",
rmse_f(y_test, predict(f_rnd_f, X_test))))
#> rand_ntopcor = FALSE — test RMSE: 3.766Randomisation usually helps when m is large enough; for
very small forests it can occasionally hurt by introducing excess
variance.
#> Parameter Value Test_RMSE
#> relation Pearson (default) 3.586
#> relation MI 3.592
#> weight_scheme scale (default) 3.586
#> weight_scheme robust 3.556
#> rf_var p = 13 — all (default) 3.586
#> rf_var floor(p/2) = 6 3.592
#> rf_var floor(sqrt(p)) = 3 4.488
#> rand_ntopcor TRUE (default) 3.586
#> rand_ntopcor FALSE 3.766
SWORD handles factor and character columns automatically: they are
one-hot-encoded internally, new factor levels in predict()
are mapped to all-zero dummies, and VI_SWORD() sums
dummy-column importances back to the original variable name.
data(diamonds, package = "ggplot2")
set.seed(7)
idx <- sample(nrow(diamonds), 2000)
X_dia <- diamonds[idx, c("carat","cut","color","clarity",
"depth","table","x","y","z")]
y_dia <- diamonds$price[idx]
forest_dia <- SWORD(X_dia, y_dia, m = 50, oob = TRUE, verbose = FALSE)
print(forest_dia)
# VI grouped by original variable (not by dummy column)
VI_SWORD(forest_dia)
# PDP for an ordered factor
pdp_sword(forest_dia, X_dia, "cut")Both TORS() and SWORD() accept an R formula
with a data argument:
df <- cbind(X_train, medv = y_train)
tree2 <- TORS(medv ~ ., data = df, nmin = 15, cp = 0.02)
forest2 <- SWORD(medv ~ ., data = df, m = 50, verbose = FALSE)
predict(forest2, Boston[test, ])| Task | Function |
|---|---|
| Fit one tree | TORS(X, y) |
| Fit a forest | SWORD(X, y, m = 100) |
| Predict | predict(forest, newdata) |
| OOB convergence | plot(forest, which = "oob") |
| OOB scatter | plot_oob_fit(forest, y) |
| OIW-VI (variable importance) | VI_SWORD(forest) /
plot_vi_sword(forest) |
| Partial dependence | pdp_sword(forest, X, "varname") |
| Tree text print | plot_tors_text(tree) |
| Tree graph | plot(tree) |
Carta, A. and Frigau, L. (in press). Oblique random forests for regression via weighted support vector machine. Advances in Data Analysis and Classification.
Carta, A. and Frigau, L. (2025). Tree oblique for regression with weighted support vector machine. Computational Statistics, 40, 5257–5291. https://doi.org/10.1007/s00180-025-01647-w
Yang, X., Song, Q. and Cao, A. (2005). Weighted support vector machine for data classification. In Proceedings of the 2005 IEEE International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/IJCNN.2005.1555965
Xu, T. et al. (2024). WeightSVM: Subject/Instance Weighted Support Vector Machines. R package version 1.7-16. https://doi.org/10.32614/CRAN.package.WeightSVM