|
| 1 | +Migrating to MAPIE v1 |
| 2 | +=========================================== |
| 3 | + |
| 4 | +MAPIE v1 introduces several updates, enhancements, and structural changes that simplify the API by breaking down ``MapieRegressor`` functionality into dedicated classes for different conformal prediction methods. This guide outlines the key differences between MAPIE v0.9 and MAPIE v1 and provides instructions for adapting your code to the new structure. |
| 5 | + |
| 6 | +1. Overview of class restructuring |
| 7 | +----------------------------------- |
| 8 | + |
| 9 | +MAPIE v1 organizes the ``MapieRegressor`` functionality into specific regressor classes, each optimized for a particular type of conformal prediction: |
| 10 | + |
| 11 | +- ``SplitConformalRegressor``: Handles split conformal prediction. |
| 12 | +- ``CrossConformalRegressor``: Implements cross-validation-based conformal prediction. |
| 13 | +- ``JackknifeAfterBootstrapRegressor``: Supports jackknife-after-bootstrap conformal prediction. |
| 14 | +- ``ConformalizedQuantileRegressor``: For quantile-based conformal prediction. |
| 15 | + |
| 16 | +This modular approach makes it easier to select and configure a specific conformal regression method. Each class includes parameters relevant to its own methodology, reducing redundancy and improving readability. |
| 17 | + |
| 18 | +Migration summary of ``MapieRegressor`` to new classes |
| 19 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 20 | + |
| 21 | +In MAPIE v0.9, ``MapieRegressor`` managed all conformal regression methods under a single interface, which sometimes led to parameter redundancy and ambiguity. In MAPIE v1, each method-specific class includes only the parameters and methods relevant to its method. |
| 22 | + |
| 23 | ++--------------------+--------------------------------------------------------------------------+ |
| 24 | +| MAPIE v0.9 Class | MAPIE v1 Classes | |
| 25 | ++====================+==========================================================================+ |
| 26 | +| ``MapieRegressor`` | ``SplitConformalRegressor``, ``CrossConformalRegressor``, | |
| 27 | +| | | |
| 28 | +| | ``JackknifeAfterBootstrapRegressor``, ``ConformalizedQuantileRegressor`` | |
| 29 | ++--------------------+--------------------------------------------------------------------------+ |
| 30 | + |
| 31 | + |
| 32 | +2. Key parameter changes |
| 33 | +------------------------ |
| 34 | + |
| 35 | +``conformity_score`` |
| 36 | +~~~~~~~~~~~~~~~~~~~~ |
| 37 | +A parameter used to specify the scoring approach for evaluating model predictions. |
| 38 | + |
| 39 | +- **v0.9**: Only allowed custom objects derived from ``BaseRegressionScore``. |
| 40 | +- **v1**: Now accepts both strings (like ``"absolute"``) for predefined methods and custom ``BaseRegressionScore`` instances, simplifying usage. |
| 41 | + |
| 42 | +``confidence_level`` |
| 43 | +~~~~~~~~~~~~~~~~~~~~ |
| 44 | +Indicates the desired coverage probability of the prediction intervals. |
| 45 | + |
| 46 | +- **v0.9**: Specified as ``alpha`` during prediction, representing error rate. |
| 47 | +- **v1**: Replaced with ``confidence_level`` to denote the coverage rate directly. Set at model initialization, improving consistency and clarity. ``confidence_level`` is equivalent to ``1 - alpha``. |
| 48 | + |
| 49 | +``method`` |
| 50 | +~~~~~~~~~~ |
| 51 | +Specifies the approach for calculating prediction intervals, especially in advanced models like Cross Conformal and Jackknife After Bootstrap regressors. |
| 52 | + |
| 53 | +- **v0.9**: Part of ``MapieRegressor``. Configured for the main prediction process. |
| 54 | +- **v1**: Specific to ``CrossConformalRegressor`` and ``JackknifeAfterBootstrapRegressor``, indicating the interval calculation approach (``"base"``, ``"plus"``, or ``"minmax"``). |
| 55 | + |
| 56 | +``cv`` (includes ``groups``) |
| 57 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 58 | +The ``cv`` parameter manages the cross-validation configuration, accepting either an integer to indicate the number of data splits or a ``BaseCrossValidator`` object for custom data splitting. |
| 59 | + |
| 60 | +- **v0.9**: The ``cv`` parameter was included in ``MapieRegressor``, where it handled cross-validation. The option ``cv="prefit"`` was available for models that were already pre-trained. |
| 61 | +- **v1**: The ``cv`` parameter is now only present in ``CrossConformalRegressor``, with the ``prefit`` option removed. Additionally, the ``groups`` parameter was removed from the ``fit`` method, allowing groups to be directly passed to ``cv`` for processing. |
| 62 | + |
| 63 | +``prefit`` |
| 64 | +~~~~~~~~~~ |
| 65 | +Controls whether the model has been pre-fitted before applying conformal prediction. |
| 66 | + |
| 67 | +- **v0.9**: Indicated through ``cv="prefit"`` in ``MapieRegressor``. |
| 68 | +- **v1**: ``prefit`` is now a separate boolean parameter, allowing explicit control over whether the model has been pre-fitted before applying conformal methods. |
| 69 | + |
| 70 | +``fit_params`` (includes ``sample_weight``) |
| 71 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 72 | +Dictionary of parameters specifically used during training, such as ``sample_weight`` in scikit-learn. |
| 73 | + |
| 74 | +- **v0.9**: Passed additional parameters in a flexible but less explicit manner. |
| 75 | +- **v1**: Now explicitly structured as a dedicated dictionary, ``fit_params``, ensuring parameters used during training are clearly defined and separated from other stages. |
| 76 | + |
| 77 | +``predict_params`` |
| 78 | +~~~~~~~~~~~~~~~~~~ |
| 79 | +Defines additional parameters exclusively for prediction. |
| 80 | + |
| 81 | +- **v0.9**: Passed additional parameters in a flexible but less explicit manner, sometimes mixed within training configurations. |
| 82 | +- **v1**: Now structured as a dedicated dictionary, ``predict_params``, to be used during calibration (``conformalize`` method) and prediction stages, ensuring no overlap with training parameters. |
| 83 | + |
| 84 | +``aggregation_method`` |
| 85 | +~~~~~~~~~~~~~~~~~~~~~~ |
| 86 | +The ``aggregation_method`` parameter defines how predictions from multiple conformal regressors are aggregated when making point predictions. |
| 87 | + |
| 88 | +- **v0.9**: Previously, the ``agg_function`` parameter specified the aggregation method, allowing options such as the mean or median of predictions. This was applicable only when using ensemble methods by setting ``ensemble=True`` in the ``predict`` method. |
| 89 | +- **v1**: The ``agg_function`` parameter has been renamed to ``aggregation_method`` for clarity. It now serves the same purpose in selecting an aggregation technique but is specified at prediction time rather than during class initialization. Additionally, the ``ensemble`` parameter has been removed, as ``aggregation_method`` is relevant only to the ``CrossConformalRegressor`` and ``JackknifeAfterBootstrapRegressor`` classes. |
| 90 | + |
| 91 | +``Other parameters`` |
| 92 | +~~~~~~~~~~~~~~~~~~~~ |
| 93 | +No more parameters with incorrect ``None`` defaults. |
| 94 | + |
| 95 | +- **v0.9**: Eg: ``estimator`` had a ``None`` default value, even though the actual default value is ``LinearRegression()``. This was the case for other parameters as well. |
| 96 | +- **v1**: All parameters now have explicit defaults. |
| 97 | + |
| 98 | +Some parameters' name have been improved for clarity: |
| 99 | + |
| 100 | +- ``optimize_beta`` -> ``minimize_interval_width`` |
| 101 | +- ``symmetry``-> ``symmetric_intervals`` |
| 102 | + |
| 103 | + |
| 104 | +3. Method changes |
| 105 | +----------------- |
| 106 | + |
| 107 | +In MAPIE v1, the conformal prediction workflow is more streamlined and modular, with distinct methods for training, calibration, and prediction. The calibration process in v1 consists of four steps. |
| 108 | + |
| 109 | +Step 1: Data splitting |
| 110 | +~~~~~~~~~~~~~~~~~~~~~~ |
| 111 | +In v0.9, Data splitting is done within two-phase process. First, data ``(X, y)`` was divided into training ``(X_train, y_train)`` and test ``(X_test, y_test)`` sets using ``train_test_split`` from ``sklearn``. In the second phase, the split between training and calibration was either done manually or handled internally by ``MapieRegressor``. |
| 112 | + |
| 113 | +In v1, a ``conf_split`` function has been introduced to split the data ``(X, y)`` into training ``(X_train, y_train)``, calibration ``(X_calib, y_calib)``, and test sets ``(X_test, y_test)``. |
| 114 | + |
| 115 | +This new approach in v1 gives users more control over data splitting, making it easier to manage training, calibration, and testing phases explicitly. The ``CrossConformalRegressor`` is an exception, where train/calibration splitting happens internally because cross-validation requires more granular control over data splits. |
| 116 | + |
| 117 | +Step 2 & 3: Model training and calibration |
| 118 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 119 | +In v0.9, the ``fit`` method handled both model training and calibration. |
| 120 | + |
| 121 | +In v1.0: MAPIE separates between the training and calibration: |
| 122 | + |
| 123 | +- ``.fit()`` method: |
| 124 | + - In v1, ``fit`` only trains the model on training data, without handling calibration. |
| 125 | + - Additional fitting parameters, like ``sample_weight``, should be included in ``fit_params``, keeping this method focused on training alone. |
| 126 | + |
| 127 | +- ``.conformalize()`` method: |
| 128 | + - This new method performs calibration after fitting, using separate calibration data ``(X_calib, y_calib)``. |
| 129 | + - ``predict_params`` can be passed here, allowing independent control over calibration and prediction stages. |
| 130 | + |
| 131 | +Step 4: Making predictions (``predict`` and ``predict_set`` methods) |
| 132 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 133 | +In MAPIE v0.9, both point predictions and prediction intervals were produced through the ``predict`` method. |
| 134 | + |
| 135 | +MAPIE v1 introduces two distinct methods for prediction: |
| 136 | +- ``.predict_set()`` is dedicated to generating prediction intervals (i.e., lower and upper bounds), clearly separating interval predictions from point predictions. |
| 137 | +- ``.predict()`` now focuses solely on producing point predictions. |
| 138 | + |
| 139 | + |
| 140 | +4. Migration example: MAPIE v0.9 to MAPIE v1 |
| 141 | +-------------------------------------------- |
| 142 | + |
| 143 | +Below is a side-by-side example of code in MAPIE v0.9 and its equivalent in MAPIE v1 using the new modular classes and methods. |
| 144 | + |
| 145 | +MAPIE v0.9 code |
| 146 | +~~~~~~~~~~~~~~~ |
| 147 | + |
| 148 | +.. code-block:: python |
| 149 | +
|
| 150 | + from sklearn.linear_model import LinearRegression |
| 151 | + from mapie.estimator import MapieRegressor |
| 152 | + from mapie.conformity_scores import GammaConformityScore |
| 153 | + from sklearn.model_selection import train_test_split |
| 154 | +
|
| 155 | + # Step 1: Split data |
| 156 | + X_train, X_conf_test, y_train, y_conf_test = train_test_split(X, y, test_size=0.4) |
| 157 | + X_conf, X_test, y_conf, y_test = train_test_split(X_conf_test, y_conf_test, test_size=0.5) |
| 158 | +
|
| 159 | + # Step 2: Train the model on the training set |
| 160 | + prefit_model = LinearRegression().fit(X_train, y_train) |
| 161 | +
|
| 162 | + # Step 3: Initialize MapieRegressor with the prefit model and gamma conformity score |
| 163 | + v0 = MapieRegressor( |
| 164 | + estimator=prefit_model, |
| 165 | + cv="prefit", |
| 166 | + conformity_score=GammaConformityScore() |
| 167 | + ) |
| 168 | +
|
| 169 | + # Step 4: Fit MAPIE on the calibration set |
| 170 | + v0.fit(X_conf, y_conf) |
| 171 | +
|
| 172 | + # Step 5: Make predictions with confidence intervals |
| 173 | + prediction_intervals_v0 = v0.predict(X_test, alpha=0.1)[1][:, :, 0] |
| 174 | + prediction_points_v0 = v0.predict(X_test) |
| 175 | +
|
| 176 | +Equivalent MAPIE v1 code |
| 177 | +~~~~~~~~~~~~~~~~~~~~~~~~ |
| 178 | + |
| 179 | +.. code-block:: python |
| 180 | +
|
| 181 | + from sklearn.linear_model import LinearRegression |
| 182 | + from mapie.estimator import SplitConformalRegressor |
| 183 | + from mapie.utils import conf_split |
| 184 | +
|
| 185 | + # Step 1: Split data with conf_split (returns X_train, y_train, X_conf, y_conf, X_test, y_test) |
| 186 | + X_train, y_train, X_conf, y_conf, X_test, y_test = conf_split(X, y) |
| 187 | +
|
| 188 | + # Step 2: Train the model on the training set |
| 189 | + prefit_model = LinearRegression().fit(X_train, y_train) |
| 190 | +
|
| 191 | + # Step 3: Initialize SplitConformalRegressor with the prefit model, gamma conformity score, and prefit option |
| 192 | + v1 = SplitConformalRegressor( |
| 193 | + estimator=prefit_model, |
| 194 | + confidence_level=0.9, # equivalent to alpha=0.1 in v0.9 |
| 195 | + conformity_score="gamma", |
| 196 | + prefit=True |
| 197 | + ) |
| 198 | +
|
| 199 | + # Step 4: Calibrate the model with the conformalize method on the calibration set |
| 200 | + v1.conformalize(X_conf, y_conf) |
| 201 | +
|
| 202 | + # Step 5: Make predictions with confidence intervals |
| 203 | + prediction_intervals_v1 = v1.predict_set(X_test) |
| 204 | + prediction_points_v1 = v1.predict(X_test) |
| 205 | +
|
| 206 | +
|
| 207 | +5. Additional migration examples |
| 208 | +-------------------------------- |
| 209 | + |
| 210 | +We will provide further migration examples : |
| 211 | + |
| 212 | +- **Prefit Models**: Using ``SplitConformalRegressor`` with ``prefit=True`` |
| 213 | +- **Non-Prefit Models**: |
| 214 | + |
| 215 | + - ``SplitConformalRegressor`` without ``prefit`` |
| 216 | + - ``CrossConformalRegressor`` with ``fit_params`` (e.g., ``sample_weight``) and ``predict_params`` |
| 217 | + - ``ConformalizedQuantileRegressor`` with ``symmetric_intervals=False`` |
| 218 | + - ``JackknifeAfterBootstrapRegressor`` with custom configurations |
0 commit comments