ZMetric
- class ZMetric(base_cls=None, **kwargs)[source]
Bases:
pykeen.metrics.ranking.DerivedRankBasedMetric
A z-score adjusted metrics.
\[\mathbb{M}^* = \frac{\mathbb{M} - \mathbb{E}[\mathbb{M}]}{\sqrt{\mathbb{V}[\mathbb{M}]}}\]In terms of the affine transformation from DerivedRankBasedMetric, we obtain the following coefficients:
\[\begin{split}\alpha &= \frac{1}{\sqrt{\mathbb{V}[\mathbb{M}]}} \\ \beta &= -\alpha \cdot \mathbb{E}[\mathbb{M}]\end{split}\]Note
For non-increasing metrics, i.e., where larger values correspond to better results, we additionally change the sign of the result such that a larger z-value always corresponds to a better result irrespective of the base metric’s direction.
Warning
This requires a closed-form solution to the expected value and the variance
Initialize the derived metric.
- Parameters
base_cls (
Union
[str
,RankBasedMetric
,Type
[RankBasedMetric
],None
]) – the base class, or a hint thereof. If None, use the class-attributekwargs – additional keyword-based parameters used to instantiate the base metric
Attributes Summary
The rank-based metric class that this derived metric extends
whether the metric needs binarized scores
Z-adjusted metrics are formulated to be increasing
Return the key for use in metric result dictionaries.
whether the metric requires the number of candidates for each ranking task
Z-adjusted metrics can only be applied to realistic ranks
synonyms for this metric
the value range
Methods Summary
__call__
(ranks[, num_candidates])Evaluate the metric.
adjust
(base_metric_result, num_candidates)Adjust base metric results based on the number of candidates.
expected_value
(num_candidates[, num_samples])Compute expected metric value.
get_coefficients
(num_candidates)Compute the scaling coefficients.
Get the description.
get_link
()Get the link from the docdata.
Get the math notation for the range of this metric.
get_sampled_values
(num_candidates, num_samples)Calculate the metric on sampled rank arrays.
numeric_expected_value
(num_candidates, ...)Compute expected metric value by summation.
numeric_expected_value_with_ci
(**kwargs)Estimate expected value with confidence intervals.
numeric_variance
(num_candidates, ...)Compute variance by summation.
numeric_variance_with_ci
(**kwargs)Estimate variance with confidence intervals.
std
(num_candidates[, num_samples])Compute the standard deviation.
variance
(num_candidates[, num_samples])Compute variance.
Attributes Documentation
- base_cls: ClassVar[Optional[Type[pykeen.metrics.ranking.RankBasedMetric]]] = None
The rank-based metric class that this derived metric extends
- needs_candidates: ClassVar[bool] = True
whether the metric requires the number of candidates for each ranking task
- supported_rank_types: ClassVar[Collection[typing_extensions.Literal[optimistic, realistic, pessimistic]]] = ('realistic',)
Z-adjusted metrics can only be applied to realistic ranks
- value_range: ClassVar[pykeen.metrics.utils.ValueRange] = ValueRange(lower=None, lower_inclusive=False, upper=None, upper_inclusive=False)
the value range
Methods Documentation
- __call__(ranks, num_candidates=None)
Evaluate the metric.
- adjust(base_metric_result, num_candidates)
Adjust base metric results based on the number of candidates.
- Parameters
- Return type
- Returns
the adjusted metric
Note
since the adjustment only depends on the number of candidates, but not the ranks of the predictions, this method can also be used to adjust published results without access to the trained models.
- expected_value(num_candidates, num_samples=None, **kwargs)[source]
Compute expected metric value.
The expectation is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).
- Parameters
num_candidates (
ndarray
) – the number of candidates for each individual rank computationnum_samples (
Optional
[int
]) – the number of samples to use for simulation, if no closed form expected value is implementedkwargs – additional keyword-based parameters passed to
get_sampled_values()
, if no closed form solution is available
- Return type
- Returns
the expected value of this metric
- Raises
NoClosedFormError – raised if a closed form expectation has not been implemented and no number of samples are given
Note
Prefers analytical solution, if available, but falls back to numeric estimation via summation, cf.
RankBasedMetric.numeric_expected_value()
.
- get_coefficients(num_candidates)[source]
Compute the scaling coefficients.
- Parameters
num_candidates (
ndarray
) – the number of candidates- Return type
- Returns
a tuple (scale, offset)
- get_sampled_values(num_candidates, num_samples, generator=None, memory_intense=True)
Calculate the metric on sampled rank arrays.
- Parameters
- Return type
- Returns
shape: (num_samples,) the metric evaluated on num_samples sampled rank arrays
- numeric_expected_value(num_candidates, num_samples, **kwargs)
Compute expected metric value by summation.
The expectation is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).
- Parameters
num_candidates (
ndarray
) – the number of candidates for each individual rank computationnum_samples (
int
) – the number of samples to use for simulationkwargs – additional keyword-based parameters passed to
get_sampled_values()
- Return type
- Returns
The estimated expected value of this metric
Warning
Depending on the metric, the estimate may not be very accurate and converge slowly, cf. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_discrete.expect.html
- numeric_expected_value_with_ci(**kwargs)
Estimate expected value with confidence intervals.
- Return type
- numeric_variance(num_candidates, num_samples, **kwargs)
Compute variance by summation.
The variance is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).
- Parameters
num_candidates (
ndarray
) – the number of candidates for each individual rank computationnum_samples (
int
) – the number of samples to use for simulationkwargs – additional keyword-based parameters passed to
get_sampled_values()
- Return type
- Returns
The estimated variance of this metric
Warning
Depending on the metric, the estimate may not be very accurate and converge slowly, cf. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_discrete.expect.html
- numeric_variance_with_ci(**kwargs)
Estimate variance with confidence intervals.
- Return type
- std(num_candidates, num_samples=None, **kwargs)
Compute the standard deviation.
- Parameters
num_candidates (
ndarray
) – the number of candidates for each individual rank computationnum_samples (
Optional
[int
]) – the number of samples to use for simulation, if no closed form expected value is implementedkwargs – additional keyword-based parameters passed to
variance()
,
- Return type
- Returns
The standard deviation (i.e. the square root of the variance) of this metric
For a detailed explanation, cf.
RankBasedMetric.variance()
.
- variance(num_candidates, num_samples=None, **kwargs)[source]
Compute variance.
The variance is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).
- Parameters
num_candidates (
ndarray
) – the number of candidates for each individual rank computationnum_samples (
Optional
[int
]) – the number of samples to use for simulation, if no closed form expected value is implementedkwargs – additional keyword-based parameters passed to
get_sampled_values()
, if no closed form solution is available
- Return type
- Returns
The variance of this metric
- Raises
NoClosedFormError – Raised if a closed form variance has not been implemented and no number of samples are given
Note
Prefers analytical solution, if available, but falls back to numeric estimation via summation, cf.
RankBasedMetric.numeric_variance()
.