ExpectationNormalizedMetric

class ExpectationNormalizedMetric(base_cls=None, **kwargs)

Bases: pykeen.metrics.ranking.DerivedRankBasedMetric

An adjustment to create an expectation-normalized metric.

\[M^* = \frac{M}{\mathbb{E}[M]}\]

In terms of the affine transformation from DerivedRankBasedMetric, we obtain the following coefficients:

\[\begin{split}\alpha &= \frac{1}{\mathbb{E}[M]} \\ \beta &= 0\end{split}\]

Warning

This requires a closed-form solution to the expected value

Initialize the derived metric.

Parameters

• base_cls (Union[str, RankBasedMetric, Type[RankBasedMetric], None]) – the base class, or a hint thereof. If None, use the class-attribute

• kwargs – additional keyword-based parameters used to instantiate the base metric

Attributes Summary

base_cls	The rank-based metric class that this derived metric extends
binarize	whether the metric needs binarized scores
key	Return the key for use in metric result dictionaries.
needs_candidates	whether the metric requires the number of candidates for each ranking task
supported_rank_types	the supported rank types.
synonyms	synonyms for this metric

Methods Summary

__call__(ranks[, num_candidates])	Evaluate the metric.
adjust(base_metric_result, num_candidates)	Adjust base metric results based on the number of candidates.
expected_value(num_candidates[, num_samples])	Compute expected metric value.
get_coefficients(num_candidates)	Compute the scaling coefficients.
get_description()	Get the description.
get_link()	Get the link from the docdata.
get_range()	Get the math notation for the range of this metric.
get_sampled_values(num_candidates, num_samples)	Calculate the metric on sampled rank arrays.
numeric_expected_value(num_candidates, ...)	Compute expected metric value by summation.
numeric_expected_value_with_ci(**kwargs)	Estimate expected value with confidence intervals.
numeric_variance(num_candidates, ...)	Compute variance by summation.
numeric_variance_with_ci(**kwargs)	Estimate variance with confidence intervals.
std(num_candidates[, num_samples])	Compute the standard deviation.
variance(num_candidates[, num_samples])	Compute variance.

Attributes Documentation

base_cls: ClassVar[Optional[Type[pykeen.metrics.ranking.RankBasedMetric]]] = None

The rank-based metric class that this derived metric extends

binarize: ClassVar[bool] = False

whether the metric needs binarized scores

key

Return the key for use in metric result dictionaries.

Return type

str

needs_candidates: ClassVar[bool] = True

whether the metric requires the number of candidates for each ranking task

supported_rank_types: ClassVar[Collection[typing_extensions.Literal[optimistic, realistic, pessimistic]]] = ('optimistic', 'realistic', 'pessimistic')

the supported rank types. Most of the time equal to all rank types

synonyms: ClassVar[Collection[str]] = ()

synonyms for this metric

Methods Documentation

__call__(ranks, num_candidates=None)

Evaluate the metric.

Parameters

• ranks (ndarray) – shape: s the individual ranks

• num_candidates (Optional[ndarray]) – shape: s the number of candidates for each individual ranking task

Return type

float

adjust(base_metric_result, num_candidates)

Adjust base metric results based on the number of candidates.

Parameters

• base_metric_result (float) – the result of the base metric

• num_candidates (ndarray) – the number of candidates

Return type

float

Returns

the adjusted metric

Note

since the adjustment only depends on the number of candidates, but not the ranks of the predictions, this method can also be used to adjust published results without access to the trained models.

expected_value(num_candidates, num_samples=None, **kwargs)

Compute expected metric value.

The expectation is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).

Parameters

• num_candidates (ndarray) – the number of candidates for each individual rank computation

• num_samples (Optional[int]) – the number of samples to use for simulation, if no closed form expected value is implemented

• kwargs – additional keyword-based parameters passed to get_sampled_values(), if no closed form solution is available

Return type

float

Returns

the expected value of this metric

Raises

NoClosedFormError – raised if a closed form expectation has not been implemented and no number of samples are given

Note

Prefers analytical solution, if available, but falls back to numeric estimation via summation, cf. RankBasedMetric.numeric_expected_value().

get_coefficients(num_candidates)

Compute the scaling coefficients.

Parameters

num_candidates (ndarray) – the number of candidates

Return type

AffineTransformationParameters

Returns

a tuple (scale, offset)

classmethod get_description()

Get the description.

Return type

str

classmethod get_link()

Get the link from the docdata.

Return type

str

classmethod get_range()

Get the math notation for the range of this metric.

Return type

str

get_sampled_values(num_candidates, num_samples, generator=None, memory_intense=True)

Calculate the metric on sampled rank arrays.

Parameters

• num_candidates (ndarray) – shape: s the number of candidates for each ranking task

• num_samples (int) – the number of samples

• generator (Optional[Generator]) – a random state for reproducibility

• memory_intense (bool) – whether to use a more memory-intense, but more time-efficient variant

Return type

ndarray

Returns

shape: (num_samples,) the metric evaluated on num_samples sampled rank arrays

numeric_expected_value(num_candidates, num_samples, **kwargs)

Compute expected metric value by summation.

The expectation is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).

Parameters

• num_candidates (ndarray) – the number of candidates for each individual rank computation

• num_samples (int) – the number of samples to use for simulation

• kwargs – additional keyword-based parameters passed to get_sampled_values()

Return type

float

Returns

The estimated expected value of this metric

Warning

Depending on the metric, the estimate may not be very accurate and converge slowly, cf. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_discrete.expect.html

numeric_expected_value_with_ci(**kwargs)

Estimate expected value with confidence intervals.

Return type

ndarray

numeric_variance(num_candidates, num_samples, **kwargs)

Compute variance by summation.

The variance is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).

Parameters

• num_candidates (ndarray) – the number of candidates for each individual rank computation

• num_samples (int) – the number of samples to use for simulation

• kwargs – additional keyword-based parameters passed to get_sampled_values()

Return type

float

Returns

The estimated variance of this metric

Warning

Depending on the metric, the estimate may not be very accurate and converge slowly, cf. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.rv_discrete.expect.html

numeric_variance_with_ci(**kwargs)

Estimate variance with confidence intervals.

Return type

ndarray

std(num_candidates, num_samples=None, **kwargs)

Compute the standard deviation.

Parameters

• num_candidates (ndarray) – the number of candidates for each individual rank computation

• num_samples (Optional[int]) – the number of samples to use for simulation, if no closed form expected value is implemented

• kwargs – additional keyword-based parameters passed to variance(),

Return type

float

Returns

The standard deviation (i.e. the square root of the variance) of this metric

For a detailed explanation, cf. RankBasedMetric.variance().

variance(num_candidates, num_samples=None, **kwargs)

Compute variance.

The variance is computed under the assumption that each individual rank follows a discrete uniform distribution \(\mathcal{U}\left(1, N_i\right)\), where \(N_i\) denotes the number of candidates for ranking task \(r_i\).

Parameters

• num_candidates (ndarray) – the number of candidates for each individual rank computation

• num_samples (Optional[int]) – the number of samples to use for simulation, if no closed form expected value is implemented

• kwargs – additional keyword-based parameters passed to get_sampled_values(), if no closed form solution is available

Return type

float

Returns

The variance of this metric

Raises

NoClosedFormError – Raised if a closed form variance has not been implemented and no number of samples are given

Note

Prefers analytical solution, if available, but falls back to numeric estimation via summation, cf. RankBasedMetric.numeric_variance().