User guide#

Practice

The goal of this page is to point to the parts of the documentation that showcase the main features of the package: the scikit-learn and the PyTorch interfaces. We will demonstrate the main functionalities on a simple Linear Regression example.

Linear Regression#

Given some feature vectors \(x_1,\dots,x_n \in \mathbb{R}^d\) and the corresponding target values \(y_1,\dots,y_n \in \mathbb{R}\), the goal is to learn a linear model \(w \in \mathbb{R}^d,\ b \in \mathbb{R}\) that predicts the target value from the feature vector, i.e., \(y_i \approx w^T x_i + b\) for all \(i=1,\dots,n\).

The most common approach to learn the parameters \(w\) and \(b\) is to minimize the empirical risk with the squared loss function:

\[\min_{w, b} \frac{1}{n} \sum_{i=1}^n (y_i - w^T x_i - b)^2.\]

Solving the regression problem with scikit-learn#

The linear regression problem can now be solved with the sklearn.linear_model.LinearRegression estimator from scikit-learn. We assume that we are given X_train of shape (n_train, n_features) and y_train of shape (n_train,) as training data and X_test of shape (n_test, n_features) as test data.

Scikit’s LinearRegression interface#
1from sklearn.linear_model import LinearRegression
2
3# Fit the model
4model = LinearRegression()
5model.fit(X_train, y_train)
6
7# Predict the target values
8y_pred = model.predict(X_test)

Solving the robust regression problem with skwdro#

Robust estimators from skwdro can be used as drop-in replacements for scikit-learn estimators (they actually inherit from scikit-learn estimators and classifier classes.) skwdro provides robust estimators for standard problems such as linear regression or logistic regression. skwdro.linear_models.LinearRegression is a robust version of sklearn.linear_model.LinearRegression and may be used in the same way. The only difference is that now an uncertainty radius rho is required.

SkWDRO’s LinearRegression interface#
 1>>> import numpy as np
 2>>> from sklearn.linear_model import LinearRegression as ERMRegression
 3>>> from skwdro.linear_models import LinearRegression as DRORegression
 4>>>
 5>>> # Some toy linear problem: e.g. additive noise level shift
 6>>> rng = np.random.RandomState(666)
 7>>> X_train = rng.randn(10, 1)
 8>>> X_test = rng.randn(5, 1) + .5
 9>>> y_train = 2. * X_train.flatten() + .01 * rng.randn(10)
10>>> y_test = 2. * X_test.flatten() + .1 * rng.randn(5)
11>>>
12>>> # Uncertainty radius
13>>> rho = 0.1
14>>>
15>>> # Fit the model
16>>> erm_model = ERMRegression()
17>>> robust_model = DRORegression(rho=rho)
18>>> erm_model.fit(X_train, y_train)
19>>> robust_model.fit(X_train, y_train)
20>>>
21>>> # Predict the target values
22>>> y_pred = erm_model.predict(X_test)
23>>> print(np.mean((y_pred - y_test)**2))
240.009900357816198937
25>>> y_pred = robust_model.predict(X_test)
26>>> print(np.mean((y_pred - y_test)**2))
270.009643423384431925

As a consequence, robust estimators can be tried and used without much change to existing pipelines!

By default, the LinearRegression estimator from skwdro uses will solve the robust optimization problem with entropic regularization and by calling a stochastic first-order solver in PyTorch. A dedicated solver from CvxPy can be used by setting the solver parameter in the constructor to 'dedicated'.

robust_model = LinearRegression(rho=rho, solver='dedicated')

Solving the regression problem with the PyTorch interface#

The next section now describe the PyTorch interface of skwdro: it allows more flexibility, custom models and optimizers.

Assume now that the (training) data is given as a dataloader train_loader.

SkWDRO’s PyTorch-type interface#
 1import torch as pt
 2import torch.nn as nn
 3import torch.optim as optim
 4
 5from skwdro.torch import robustify
 6
 7# Toy data
 8n_features = 3
 9X = pt.randn(32, n_features)
10y = X @ pt.rand(n_features, 1) + 1.
11train_loader = pt.utils.data.DataLoader(
12    pt.utils.data.TensorDataset(X, y),
13    batch_size=4
14)
15
16# Uncertainty radius
17rho = pt.tensor(.1)
18
19# Define the model
20model = nn.Linear(n_features, 1)
21
22# Define the loss function
23loss_fn = nn.MSELoss(reduction='none')
24
25# Define a sample batch for initialization
26sample_batch_x, sample_batch_y = X[:16, :], y[:16, :]
27
28# Robust loss
29robust_loss = robustify(loss_fn, model, rho, sample_batch_x, sample_batch_y)
30
31# Define the optimizer
32optimizer = optim.AdamW(model.parameters(), lr=.1)
33
34# Training loop
35for epoch in range(100):
36    avg_loss = 0.
37    robust_loss.get_initial_guess_at_dual(X, y)
38    for batch_x, batch_y in train_loader:
39        optimizer.zero_grad()
40        loss = robust_loss(batch_x, batch_y)
41        loss.backward()
42        optimizer.step()
43        avg_loss += loss.detach().item()
44    print(f"=== Loss (epoch \t{epoch}): {avg_loss/len(train_loader)}")

This is the simplest use of the PyTorch interface: just wrap the usual loss and model with the robustify function and use the resulting loss function in the training loop.

To make the optimization of the robust model more efficient, we also provide an learning-rate free optimizer tailored to this problem, taken from pieces of the literature: [1] and [2].

Fetch the optimizer from the robust loss!#
 # Adaptive optimizer
 optimizer = robust_loss.optimizer

Next#

Scikit part of the library

Tutorial on how to use pre-implemented examples with their scikit-learn interface.

sklearn.html
PyTorch part of the library

Tutorial on how to robustify your model easily with the pytorch wrappers.

pytorch.html
What is WDRO

Gentle introduction to the world of Distributionally Robust Optimization, and motivations for its Wasserstein version.

wdro.html
Sinkhorn-WDRO

More about why and how to regularize the WDRO formulation with the Sinkhorn divergence.

why_skwdro.html
API

More details about the exposed API.

api_deepdive/submodules.html

References#