.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "examples/Custom/polynomial_regression.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_examples_Custom_polynomial_regression.py: Polynomial regression ===================== Polynomial regression is a simple 1D regression. The samples are of the form :math:`\xi = (x,y) \in \mathbb{R}\times\mathbb{R}` and the sought predictor is of the form :math:`f(x) = \sum_{i=0}^d a_i x^i` where :math:`(a_0,..,a_d)` are the :math:`d+1` coefficients to lean. In the following example, we seek to learn a polynomial fitting the function .. math:: f^\star(x) = \frac{10}{e^{x}+e^{-x}} + x from :math:`n=100` samples uniformly drawn from :math:`[-2,2]` and corrupted by a Gaussian noise with zero mean and variance :math:`0.1`. .. GENERATED FROM PYTHON SOURCE LINES 17-28 .. code-block:: Python from typing import Iterable import numpy as np import matplotlib.pyplot as plt import torch as pt import torch.nn as nn from torch.utils.data import TensorDataset, DataLoader from tqdm import tqdm from skwdro.torch import robustify from skwdro.solvers.oracle_torch import DualLoss .. GENERATED FROM PYTHON SOURCE LINES 29-31 Problem setup ~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 31-49 .. code-block:: Python n = 100 # Number of observations var = pt.tensor(0.1) # Variance of the noise def f_star(x): # Generating function return 10/(pt.exp(x)+pt.exp(-x)) + x xi = pt.rand(n)*4.0 - 2.0 # x_i's are uniformly drawn from (-2,2] xi = pt.sort(xi)[0] # we sort them for easier plotting yi = f_star(xi) + pt.sqrt(var)*pt.randn(n) # y_i's are f(x_i) + noise dataset = DataLoader(TensorDataset(xi.unsqueeze(-1), yi.unsqueeze(-1)), batch_size=n//2, shuffle=True) degree = 4 # Degree of the regression device = "cuda" if pt.cuda.is_available() else "cpu" .. GENERATED FROM PYTHON SOURCE LINES 50-52 Polynomial model ~~~~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 52-68 .. code-block:: Python class PolynomialModel(nn.Module): def __init__(self, degree : int) -> None: super().__init__() self._degree = degree self.linear = nn.Linear(self._degree, 1) def forward(self, x): return self.linear(self._polynomial_features(x)) def _polynomial_features(self, x): return pt.cat([x ** i for i in range(1, self._degree + 1)],dim=-1) model = PolynomialModel(degree).to(device) # Our polynomial regression model loss = nn.MSELoss(reduction='none') # Our error will be measure in quadratic loss .. GENERATED FROM PYTHON SOURCE LINES 69-71 Training loop ~~~~~~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 71-97 .. code-block:: Python def train(dual_loss: DualLoss, dataset: Iterable[tuple[pt.Tensor, pt.Tensor]], epochs: int=10): lbfgs = pt.optim.LBFGS(dual_loss.parameters()) # LBFGS is used to optimize thanks to the nature of the problem def closure(): # Closure for the LBFGS solver lbfgs.zero_grad() loss = dual_loss(xi, xi_label, reset_sampler=True).mean() loss.backward() return loss pbar = tqdm(range(epochs)) for _ in pbar: # Every now and then, try to rectify the dual parameter (e.g. once per epoch). dual_loss.get_initial_guess_at_dual(*next(iter(dataset))) # * # Main train loop inpbar = tqdm(dataset, leave=False) for xi, xi_label in inpbar: lbfgs.step(closure) pbar.set_postfix({"lambda": f"{dual_loss.lam.item():.2f}"}) return dual_loss.primal_loss.transform .. GENERATED FROM PYTHON SOURCE LINES 98-100 Training ~~~~~~~~ .. GENERATED FROM PYTHON SOURCE LINES 100-116 .. code-block:: Python radius = pt.tensor(0.001) # Robustness radius dual_loss = robustify( loss, model, radius, xi.unsqueeze(-1), yi.unsqueeze(-1) ) # Replaces the loss of the model by the dual WDRO loss model1 = train(dual_loss, dataset, epochs=5) # type: ignore model1.eval() # type: ignore .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0/5 [00:00=0.0: polyString += "+ {:3.2f}x**{:d} ".format(a,i+1) else: polyString += "- {:3.2f}x**{:d} ".format(abs(a),i+1) print(polyString) .. image-sg:: /examples/Custom/images/sphx_glr_polynomial_regression_001.png :alt: polynomial regression :srcset: /examples/Custom/images/sphx_glr_polynomial_regression_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Polynomial regressor (degree 4, radius 1.00e-03): 4.76 + 1.03x**1 - 1.64x**2 - 0.00x**3 + 0.20x**4 .. GENERATED FROM PYTHON SOURCE LINES 147-148 Different degree and radius, with a more compact formulation. .. GENERATED FROM PYTHON SOURCE LINES 149-161 .. code-block:: Python radius2 = pt.tensor(1e-6) # Robustness radius degree2 = 7 model2 = train(robustify( nn.MSELoss(reduction='none'), PolynomialModel(degree2).to(device), radius2, xi.unsqueeze(-1), yi.unsqueeze(-1) ), dataset, epochs=5) # type: ignore .. rst-class:: sphx-glr-script-out .. code-block:: none 0%| | 0/5 [00:00=0.0: polyString += "+ {:3.2f}x**{:d} ".format(a,i+1) else: polyString += "- {:3.2f}x**{:d} ".format(abs(a),i+1) print(polyString) .. image-sg:: /examples/Custom/images/sphx_glr_polynomial_regression_002.png :alt: polynomial regression :srcset: /examples/Custom/images/sphx_glr_polynomial_regression_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Polynomial regressor (degree 7, radius 1.00e-06): 4.91 + 0.93x**1 - 2.31x**2 + 0.29x**3 + 0.71x**4 - 0.21x**5 - 0.10x**6 + 0.04x**7 .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 2.238 seconds) .. _sphx_glr_download_examples_Custom_polynomial_regression.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: polynomial_regression.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: polynomial_regression.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: polynomial_regression.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_