Binary risk control - multi risk

CONTRIBUTING.rst

@@ -5,7 +5,11 @@ Contribution guidelines
 What to work on?
 ----------------
 
-You are welcome to propose and contribute new ideas.
+Issues tagged "Good first issue" are perfect for open-source beginners.
+
+For the more experienced, issues tagged "Contributors welcome" are recommended if you want to help.
+
+You are also welcome to propose and contribute to new ideas.
 We encourage you to `open an issue <https://github.com/scikit-learn-contrib/MAPIE/issues>`_ so that we can align on the work to be done.
 It is generally a good idea to have a quick discussion before opening a pull request that is potentially out-of-scope.
 
@@ -43,73 +47,65 @@ Finally, install ``mapie`` in development mode:
 
     $ pip install -e .
 
+Implementing your change
+------------------------------------------
 
-Documenting your change
------------------------
-
-If you're adding a public class or function, then you'll need to add a docstring with a doctest. We follow the `numpy docstring convention <https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html>`_, so please do too.
-Any estimator should follow the `scikit-learn API <https://scikit-learn.org/stable/developers/develop.html>`_, so please follow these guidelines.
-
-In order to build the documentation locally, you first need to create a different virtual environment than the one used for development, and then install some dependencies using ``pip`` with the following commands:
+The linter must pass:
 
 .. code-block:: sh
 
-    $ pip install -r requirements.doc.txt
-    $ pip install -e .
+    $ make lint
 
-Finally, once dependencies are installed, you can build the documentation locally by running:
+The typing must pass.
 
 .. code-block:: sh
 
-    $ make clean-doc
-    $ make doc
+    $ make type-check
 
 
-Updating changelog
-------------------
 
-You can make your contribution visible by:
+Testing your change
+---------------------
 
-1. Adding your name to the Contributors section of `AUTHORS.rst <https://github.com/scikit-learn-contrib/MAPIE/blob/master/AUTHORS.rst>`_
-2. If your change is user-facing (bug fix, feature, ...), adding a line to describe it in `HISTORY.rst <https://github.com/scikit-learn-contrib/MAPIE/blob/master/HISTORY.rst>`_
+See `the tests README.md <https://github.com/scikit-learn-contrib/MAPIE/blob/master/mapie/tests/README.md>`_ for guidance.
+
+The tests absolutely have to pass.
 
-Testing
--------
+.. code-block:: sh
 
-Linting
-^^^^^^^
+    $ make tests
 
-These tests absolutely have to pass.
+The coverage should absolutely be 100%.
 
 .. code-block:: sh
 
-    $ make lint
+    $ make coverage
 
+Documenting your change
+-----------------------
 
-Static typing
-^^^^^^^^^^^^^
+If you're adding a public class or function, then you'll need to add a docstring with a doctest. We follow the `numpy docstring convention <https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html>`_, so please do too.
+Any estimator should follow the `scikit-learn API <https://scikit-learn.org/stable/developers/develop.html>`_, so please follow these guidelines.
 
-These tests absolutely have to pass.
+In order to build the documentation locally, you first need to create a different virtual environment than the one used for development, and then install some dependencies using ``pip`` with the following commands:
 
 .. code-block:: sh
 
-    $ make type-check
-
-
-Unit tests
-^^^^^^^^^^
+    $ pip install -r requirements.doc.txt
+    $ pip install -e .
 
-These tests absolutely have to pass.
+Finally, once dependencies are installed, you can build the documentation locally by running:
 
 .. code-block:: sh
 
-    $ make tests
+    $ make clean-doc
+    $ make doc
 
-Coverage
-^^^^^^^^
 
-The coverage should absolutely be 100%.
+Updating changelog
+------------------
 
-.. code-block:: sh
+You can make your contribution visible by:
 
-    $ make coverage
+1. Adding your name to the Contributors section of `AUTHORS.rst <https://github.com/scikit-learn-contrib/MAPIE/blob/master/AUTHORS.rst>`_
+2. If your change is user-facing (bug fix, feature, ...), adding a line to describe it in `HISTORY.rst <https://github.com/scikit-learn-contrib/MAPIE/blob/master/HISTORY.rst>`_

doc/theoretical_description_risk_control.rst

@@ -138,7 +138,7 @@ Let's first give the settings and the notations of the method:
 - Let :math:`R` be the risk associated to a set-valued predictor:
 
 .. math::
-    R(\mathcal{T}_{\hat{\lambda}}) = \mathbb{E}[L(Y, \mathcal{T}_{\lambda}(X))]
+    R(\mathcal{T}_{\lambda}) = \mathbb{E}[L(Y, \mathcal{T}_{\lambda}(X))]
 
 The goal of the method is to compute an Upper Confidence Bound (UCB) :math:`\hat{R}^+(\lambda)` of :math:`R(\lambda)` and then to find
 :math:`\hat{\lambda}` as follows:

mapie/control_risk/ltt.py

@@ -1,5 +1,5 @@
 import warnings
-from typing import Any, List, Tuple, Union
+from typing import Any, List, Tuple
 
 import numpy as np
 
@@ -12,7 +12,7 @@ def ltt_procedure(
     r_hat: NDArray,
     alpha_np: NDArray,
     delta: float,
-    n_obs: Union[int, NDArray],
+    n_obs: NDArray,
     binary: bool = False,
 ) -> List[List[Any]]:
     """
@@ -24,28 +24,36 @@ def ltt_procedure(
         - Apply a family wise error rate algorithm, here Bonferonni correction
         - Return the index lambdas that give you the control at alpha level
 
+    Note that in the case of multi-risk, the arrays r_hat, alpha_np, and n_obs
+    should have the same length for the first dimension which corresponds
+    to the number of risks. In the case of a single risk, the length should be 1.
+
     Parameters
     ----------
-    r_hat: NDArray of shape (n_lambdas, ).
+    r_hat: NDArray of shape (n_risks, n_lambdas).
         Empirical risk with respect to the lambdas.
         Here lambdas are thresholds that impact decision-making,
         therefore empirical risk.
 
-    alpha_np: NDArray of shape (n_alpha, ).
+    alpha_np: NDArray of shape (n_risks, n_alpha).
         Contains the different alphas control level.
         The empirical risk should be less than alpha with
         probability 1-delta.
+        Note: MAPIE 1.2 does not support multiple risks and multiple alphas
+        simultaneously.
+        For PrecisionRecallController, the shape should be (1, n_alpha).
+        For BinaryClassificationController, the shape should be (n_risks, 1).
 
     delta: float.
         Probability of not controlling empirical risk.
         Correspond to proportion of failure we don't
         want to exceed.
 
-    n_obs: Union[int, NDArray]
+    n_obs: NDArray of shape (n_risks, n_lambdas).
         Correspond to the number of observations used to compute the risk.
         In the case of a conditional loss, n_obs must be the
         number of effective observations used to compute the empirical risk
-        for each lambda, hence of shape (n_lambdas, ).
+        for each lambda.
 
     binary: bool, default=False
         Must be True if the loss associated to the risk is binary.
@@ -62,11 +70,19 @@ def ltt_procedure(
     M. I., & Lei, L. (2021). Learn then test:
     "Calibrating predictive algorithms to achieve risk control".
     """
-    p_values = compute_hoeffding_bentkus_p_value(r_hat, n_obs, alpha_np, binary)
+    if not (r_hat.shape[0] == n_obs.shape[0] == alpha_np.shape[0]):
+        raise ValueError(
+            "r_hat, n_obs, and alpha_np must have the same length."
+        )
+    p_values = np.array([
+        compute_hoeffding_bentkus_p_value(r_hat_i, n_obs_i, alpha_np_i, binary)
+        for r_hat_i, n_obs_i, alpha_np_i in zip(r_hat, n_obs, alpha_np)
+    ])
+    p_values = p_values.max(axis=0)  # take max over risks (no effect if mono risk)
     N = len(p_values)
     valid_index = []
-    for i in range(len(alpha_np)):
-        l_index = np.where(p_values[:, i] <= delta/N)[0].tolist()
+    for i in range(alpha_np.shape[1]):
+        l_index = np.nonzero(p_values[:, i] <= delta/N)[0].tolist()
         valid_index.append(l_index)
     return valid_index
 
@@ -95,7 +111,7 @@ def find_lambda_control_star(
         the empirical risk is less than alpha.
 
     valid_index: List[List[Any]].
-        Contain the valid index that satisfy fwer control
+        Contain the valid index that satisfy FWER control
         for each alpha (length aren't the same for each alpha).
 
     lambdas: NDArray of shape (n_lambda, )
@@ -104,7 +120,7 @@ def find_lambda_control_star(
     Returns
     -------
     l_lambda_star: ArrayLike of shape (n_alpha, ).
-        The lambda that give the highest precision
+        The lambda that gives the minimum precision
         for a given alpha.
 
     r_star: ArrayLike of shape (n_alpha, ).
@@ -113,7 +129,8 @@ def find_lambda_control_star(
     if [] in valid_index:
         warnings.warn(
             """
-            Warning: At least one sequence is empty!
+            Warning: the risk couldn't be controlled for at least one value of alpha.
+            The corresponding lambdas have been set to 1.
             """
         )
     l_lambda_star = []  # type: List[Any]