Quantification Metrics Base
quack.metrics.base.QuantificationMetric
Bases: ABC
Abstract class to all quantification metrics. It uses the strategy design pattern.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Human-readable name of the metric (used in reports/plots). |
required |
lower_is_better
|
bool
|
Whether lower values of this metric indicate better quantification
performance. All metrics currently shipped with |
= True
|
Source code in quack/metrics/base.py
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 | |
__call__(p_true, p_pred)
Call and perform the input validation and metric computation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
p_true
|
ndarray
|
Array-like with all true prevalences. |
required |
p_pred
|
ndarray
|
Array-like with all predicted prevalences. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
result |
float
|
The computed metric value. |
Source code in quack/metrics/base.py
101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | |
compute(p_true, p_pred)
abstractmethod
Each metric implements its own logic and mathematics.
Source code in quack/metrics/base.py
96 97 98 99 | |
Absolute Error (AE)
quack.metrics._ae.AbsoluteError
Bases: QuantificationMetric
Mean Absolute Error (AE) between true and predicted prevalence vectors.
Averages the absolute per-class deviation across all classes, bounding
the metric to [0, 1] regardless of the number of classes (0 = perfect
quantification, 1 = maximally wrong).
AE(p, p_hat) = (1 / n_classes) * sum_c |p(c) - p_hat(c)|
References
George Forman. Quantifying counts and costs via classification. Data Mining and Knowledge Discovery, 17(2):164-206, 2008.
Source code in quack/metrics/_ae.py
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | |
Relative Absolute Error (RAE)
quack.metrics._rae.RelativeAbsoluteError
Bases: QuantificationMetric
Relative Absolute Error (RAE) between true and predicted prevalence vectors.
Averages the per-class absolute deviation relative to the true
prevalence. Both p_true and p_pred are additively smoothed (see
QuantificationMetric._smooth) before the ratio is computed, since
dividing by a true prevalence of exactly 0 would otherwise make the
metric undefined for classes absent from the test bag.
p_s(c) = (p(c) + eps) / (1 + n_classes * eps) RAE(p, p_hat) = (1 / n_classes) * sum_c |p_s(c) - p_hat_s(c)| / p_s(c)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
epsilon
|
float
|
Smoothing factor applied to both |
= 1e-5
|
References
George Forman. Quantifying counts and costs via classification. Data Mining and Knowledge Discovery, 17(2):164-206, 2008.
Source code in quack/metrics/_rae.py
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | |
Kullback Leibler Divergence (KLD)
quack.metrics._kld.KullbackLeiblerDivergence
Bases: QuantificationMetric
Kullback-Leibler Divergence (KLD) between true and predicted prevalence vectors.
Both p_true and p_pred are additively smoothed and renormalized
(see QuantificationMetric._smooth) so they remain valid probability
distributions before computing the divergence, avoiding log(0) /
division issues for classes with zero true or estimated prevalence.
p_s(c) = (p(c) + eps) / (1 + n_classes * eps) KLD(p, p_hat) = sum_c p_s(c) * log( p_s(c) / p_hat_s(c) )
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
epsilon
|
float
|
Smoothing factor applied to both |
= 1e-5
|
References
Esuli, A. & Sebastiani, F. (2015). Optimizing text quantifiers for multivariate loss functions. ACM Transactions on Knowledge Discovery from Data, 9(4), 1-27.
Source code in quack/metrics/_kld.py
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | |
Normalized Kullback Leibler Divergence (NKLD)
quack.metrics._nkld.NormalizedKullbackLeiblerDivergence
Bases: QuantificationMetric
Normalized Kullback-Leibler Divergence (NKLD).
Squashes the unbounded KLD into the [0, 1) range via a logistic-style
transform, making it comparable across experiments/datasets:
NKLD(p, p_hat) = max(0, 2 * exp(KLD(p, p_hat)) / (1 + exp(KLD(p, p_hat))) - 1)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
epsilon
|
float
|
Smoothing factor forwarded to the internal |
= 1e-5
|
References
Esuli, A. & Sebastiani, F. (2015). Optimizing text quantifiers for multivariate loss functions. ACM Transactions on Knowledge Discovery from Data, 9(4), 1-27.
Source code in quack/metrics/_nkld.py
7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | |
Normalized Absolute Error (NAE)
quack.metrics._nae.NormalizedAbsoluteError
Bases: QuantificationMetric
Normalized Absolute Error (NAE) between true and predicted prevalence vectors.
Normalizes the (unaveraged) Absolute Error by its theoretical maximum
given p_true, bounding the metric to [0, 1] regardless of how
skewed the true prevalence is. This makes NAE more comparable across
experiments/datasets with very different training or test prevalences
than the plain AbsoluteError.
NAE(p, p_hat) = sum_c |p(c) - p_hat(c)| / (2 * (1 - min_c p(c)))
References
Esuli, A. & Sebastiani, F. (2015). Optimizing text quantifiers for multivariate loss functions. ACM Transactions on Knowledge Discovery from Data, 9(4), 1-27.
Source code in quack/metrics/_nae.py
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | |