Skip to content

Implementations

Classes:

Name Description
SelectiveRegression

Selective regression-based decision-making.
SelectiveClassification

Selective classification-based decision-making.
BestClassDecision

Best classification-based decision-making.
MultiLabelDecision

Standard multi-label classification-based decision-making.
BinaryDecision

Binary classification-based decision-making.

SelectiveRegression

Bases: AdvancedRegressionDecision

Selective regression-based decision-making.

  1. Predict the estimated prediction and residual using two separate estimators.
  2. If the residual is less than or equal to the threshold, return the prediction; otherwise, NaN.

When using this class?

  • When the prediction and residual are estimated separately.
  • When the decision is based on the residual.
  • When you want to select predictions with low residuals.
  • When you want to deleguate the decision to a human operator when the confidence is low.

Parameters:

Name Type Description Default
estimator BaseEstimator

The estimator used to predict the output values.

 required
residual BaseEstimator

The estimator used to predict the residuals.

 required
threshold float

The threshold for the residual. If the residual is less than or equal to the threshold, the prediction is returned; otherwise, NaN. The default is 0.0.

 required

Attributes:

Name Type Description
estimator BaseEstimator

The estimator used to predict the output values.
residual BaseEstimator

The estimator used to predict the residuals.
threshold float

The threshold for the residual.

Methods:

Name Description
make_decision

Predict the decision for the input data based on the output values

make_decision 

make_decision(y_output)

Predict the decision for the input data based on the output values and residuals.

Parameters:

Name Type Description Default
y_output Tuple[ndarray, ndarray]

The output values and residuals for the input data. The first element is the output values, and the second element is the residuals

 required

Returns:

Type Description
ndarray

The predicted decision for the input data. The decision is made based on the output values and residuals. If the residual is less than or equal to the threshold, the decision is the output value; otherwise, the decision is NaN.
Source code in risk_control/decision/decision.py 
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
def make_decision(self, y_output: Tuple[np.ndarray, np.ndarray]) -> np.ndarray:
    """
    Predict the decision for the input data based on the output values
    and residuals.

    Parameters
    ----------
    y_output : Tuple[np.ndarray, np.ndarray]
        The output values and residuals for the input data.
        The first element is the output values, and the second element is
        the residuals

    Returns
    -------
    np.ndarray
        The predicted decision for the input data.
        The decision is made based on the output values and residuals.
        If the residual is less than or equal to the threshold, the decision is
        the output value; otherwise, the decision is NaN.
    """
    y_pred, y_res = y_output
    y_post = np.where(
        y_res <= self.threshold, y_pred, np.empty_like(y_pred) * (_abs)
    )
    return y_post

SelectiveClassification

Bases: BaseClassificationDecision

Selective classification-based decision-making.

  1. Predict the estimated probability or score using the estimator.
  2. If the probability or score of the top-1 class is greater than or equal to the threshold, return the top-1 class; otherwise, return _ABS.

When using this class?

  • When you want to select the top-1 class with high confidence.
  • When you want to control the false discovery rate.
  • When you want to deleguate the decision to a human operator when the confidence is low. So, there is abstention when the confidence is low.

Parameters:

Name Type Description Default
estimator BaseEstimator

The estimator object used for making predictions.

 required
threshold float

The threshold value used for decision-making.

 required
predict_mode str

The mode to use for prediction. Can be 'proba' or 'score'.

 required

Attributes:

Name Type Description
estimator BaseEstimator

The estimator object used for making predictions.
threshold float

The threshold value used for decision-making.
predict_mode str

The mode to use for prediction. Can be 'proba' or 'score'.

Methods:

Name Description
make_decision

Predict the decision for the input data based on the output values.

make_decision 

make_decision(y_output)

Predict the decision for the input data based on the output values.

Parameters:

Name Type Description Default
y_output ndarray

The output values predicted by the estimator.

 required

Returns:

Type Description
ndarray

The predicted decisions based on the output values and threshold.
Source code in risk_control/decision/decision.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
def make_decision(self, y_output: np.ndarray) -> np.ndarray:
    """
    Predict the decision for the input data based on the output values.

    Parameters
    ----------
    y_output : np.ndarray
        The output values predicted by the estimator.

    Returns
    -------
    np.ndarray
        The predicted decisions based on the output values and threshold.
    """
    if y_output.ndim == 1:
        y_output = np.vstack([1 - y_output, y_output]).T

    y_idx = np.argmax(y_output, axis=-1)
    y_score = np.max(y_output, axis=-1)
    y_post = np.where(
        y_score >= self.threshold, y_idx, np.empty_like(y_idx) * (_abs)
    )

    return y_post

BestClassDecision

Bases: BaseClassificationDecision

Best classification-based decision-making.

  1. Predict the estimated probability or score using the estimator.
  2. If the probability or score of the top-1 class is greater than or equal to the threshold, return the top-1 class; otherwise, return empty set.

When using this class?

  • When you want to select the top-1 class with high confidence.
  • When you want to control the false discovery rate.
  • When you want to return empty set when the confidence is low. There is no abstension in this case.

Parameters:

Name Type Description Default
estimator BaseEstimator

The estimator object used for making predictions.

 required
threshold float

The threshold value used for decision-making.

 required
predict_mode str

The mode to use for prediction. Can be 'proba' or 'score'.

 required

Attributes:

Name Type Description
estimator BaseEstimator

The estimator object used for making predictions.
threshold float

The threshold value used for decision-making.
predict_mode str

The mode to use for prediction. Can be 'proba' or 'score'.

Methods:

Name Description
make_decision

Predict the decision for the input data based on the output values.

make_decision 

make_decision(y_output)

Predict the decision for the input data based on the output values.

Parameters:

Name Type Description Default
y_output ndarray

The output values predicted by the estimator.

 required

Returns:

Type Description
ndarray

The predicted decisions based on the output values and threshold.
Source code in risk_control/decision/decision.py 
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
def make_decision(self, y_output: np.ndarray) -> np.ndarray:
    """
    Predict the decision for the input data based on the output values.

    Parameters
    ----------
    y_output : np.ndarray
        The output values predicted by the estimator.

    Returns
    -------
    np.ndarray
        The predicted decisions based on the output values and threshold.
    """
    if y_output.ndim == 1:
        y_output = np.vstack([1 - y_output, y_output]).T

    n_samples, n_classes = y_output.shape

    best_score_idx = np.argmax(y_output, axis=1)
    y_post = np.zeros((n_samples, n_classes), dtype=bool)
    y_post[np.arange(n_samples), best_score_idx] = (
        y_output[np.arange(n_samples), best_score_idx] >= self.threshold
    )

    return y_post

MultiLabelDecision 

MultiLabelDecision(
    estimator, *, predict_output="set", **kwargs
)

Bases: BaseClassificationDecision

Standard multi-label classification-based decision-making.

  1. Predict the estimated probability or score using the estimator.
  2. If the probability or score of any class is greater than or equal to the threshold, return True; otherwise, return False.

When using this class?

  • When you want to select best candidates with high confidence.
  • When you want to control the false discovery rate.
  • When you want to return empty set when the confidence is low. There is no abstension in this case.

Parameters:

Name Type Description Default
estimator BaseEstimator

The estimator object used for making predictions.

 required
threshold float

The threshold value used for decision-making.

 required
predict_mode str

The mode to use for prediction. Can be 'proba' or 'score'.

 required
predict_output str

The output type of the prediction. Can be 'class' or 'set'.

 'set'

Attributes:

Name Type Description
estimator BaseEstimator

The estimator object used for making predictions.
threshold float

The threshold value used for decision-making.
predict_mode str

The mode to use for prediction. Can be 'proba' or 'score'.
predict_output str

The output type of the prediction. Can be 'class' or 'set'.

Methods:

Name Description
make_decision

Predict the decision for the input data based on the output values.
Source code in risk_control/decision/decision.py 
230
231
232
233
234
def __init__(
    self, estimator: BaseEstimator, *, predict_output: str = "set", **kwargs: Any
):
    super().__init__(estimator, **kwargs)
    self.predict_output = predict_output

predict_output instance-attribute 

predict_output = predict_output

make_decision 

make_decision(y_output)

Predict the decision for the input data based on the output values.

Parameters:

Name Type Description Default
y_output ndarray

The output values predicted by the estimator.

 required

Returns:

Type Description
ndarray

The predicted decisions based on the output values and threshold.
Source code in risk_control/decision/decision.py 
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
def make_decision(self, y_output: np.ndarray) -> np.ndarray:
    """
    Predict the decision for the input data based on the output values.

    Parameters
    ----------
    y_output : np.ndarray
        The output values predicted by the estimator.

    Returns
    -------
    np.ndarray
        The predicted decisions based on the output values and threshold.
    """
    y_post = y_output >= self.threshold

    if self.predict_output == "class":
        return np.nanargmax(y_post, axis=-1)
    elif self.predict_output == "set":
        return y_post
    else:
        raise ValueError(f"Invalid predict_output: {self.predict_output}")

BinaryDecision 

BinaryDecision(
    estimator, *, predict_output="class", **kwargs
)

Bases: BaseClassificationDecision

Binary classification-based decision-making.

  1. Predict the estimated probability or score using the estimator.
  2. If the probability or score is greater than or equal to the threshold, return True; otherwise, return False.

When using this class?

  • When you want to control the precision-recall trade-off.

Parameters:

Name Type Description Default
estimator BaseEstimator

The estimator object used for making predictions.

 required
threshold float

The threshold value used for decision-making.

 required
predict_mode str

The mode to use for prediction. Can be 'proba' or 'score'.

 required
predict_output str

The output type of the prediction. Can be 'class' or 'set'.

 'class'

Attributes:

Name Type Description
estimator BaseEstimator

The estimator object used for making predictions.
threshold float

The threshold value used for decision-making.
predict_mode str

The mode to use for prediction. Can be 'proba' or 'score'.
predict_output str

The output type of the prediction. Can be 'class' or 'set'.

Methods:

Name Description
make_decision

Predict the decision for the input data based on the output values.
Source code in risk_control/decision/decision.py 
294
295
296
297
298
299
300
301
302
def __init__(
    self,
    estimator: BaseEstimator,
    *,
    predict_output: str = "class",
    **kwargs: Any,
):
    super().__init__(estimator, **kwargs)
    self.predict_output = predict_output

predict_output instance-attribute 

predict_output = predict_output

make_decision 

make_decision(y_output)

Predict the decision for the input data based on the output values.

Parameters:

Name Type Description Default
y_output ndarray

The output values predicted by the estimator.

 required

Returns:

Type Description
ndarray

The predicted decisions based on the output values and threshold.
Source code in risk_control/decision/decision.py 
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
def make_decision(self, y_output: np.ndarray) -> np.ndarray:
    """
    Predict the decision for the input data based on the output values.

    Parameters
    ----------
    y_output : np.ndarray
        The output values predicted by the estimator.

    Returns
    -------
    np.ndarray
        The predicted decisions based on the output values and threshold.
    """
    if y_output.ndim == 2:
        y_output = y_output[..., 1]

    y_post = y_output >= self.threshold

    if y_post.ndim == 1:
        y_post = np.vstack([~y_post, y_post]).T

    if self.predict_output == "class":
        return np.nanargmax(y_post, axis=-1)
    elif self.predict_output == "set":
        return y_post
    else:
        raise ValueError(f"Invalid predict_output: {self.predict_output}")