Metrics
A module for PyKEEN ranking and classification metrics.
Classes
|
A base class for metrics. |
|
A value range description. |
A base class for rank-based metrics. |
|
A base class for classification metrics. |
Class Inheritance Diagram
Ranking metrics.
This module comprises various rank-based metrics, which get an array of individual ranks as input, as summarize them into a single-figure metric measuring different aspects of ranking performance.
We can generally distinguish:
Base Metrics
These metrics directly operate on the ranks:
The following metrics measures summarize the central tendency of ranks
The Hits at K metric is closely related to information retrieval and measures the fraction of times when the correct result is in the top-\(k\) ranked entries, i.e., the rank is at most \(k\)
The next metrics summarize the dispersion of ranks
pykeen.metrics.ranking.MedianAbsoluteDeviation
and finally there is a simple metric to store the number of ranks which where aggregated
Inverse Metrics
The inverse metrics are reciprocals of the central tendency measures. They offer the advantage of having a fixed value range of \((0, 1]\), with a known optimal value of \(1\):
Adjusted Metrics
Adjusted metrics build upon base metrics, but adjust them for chance, cf. [berrendorf2020] and [hoyt2022]. All
adjusted metrics derive from pykeen.metrics.ranking.DerivedRankBasedMetric
and, for a given evaluation set,
are affine transformations of the base metric with dataset-dependent, but fixed transformation constants. Thus, they
can also be computed when the model predictions are not available anymore, but the evaluation set is known.
Expectation-Normalized Metrics
These metrics divide the metric by its expected value under random ordering. Thus, their expected value is always 1
irrespective of the evaluation set. They derive from pykeen.metrics.ranking.ExpectationNormalizedMetric
, and
there is currently only a single implementation:
Re-indexed Metrics
Re-indexed metrics subtract the expected value, and then normalize the optimal value to be 1. Thus, their expected value
under random ordering is 0, their optimal value is 1, and larger values indicate better results. The classes derive from
pykeen.metrics.ranking.ReindexedMetric
, and the following implementations are available:
z-Adjusted Metrics
The final type of adjusted metrics uses the expected value as well as the variance of the metric under random ordering
to normalize the metrics similar to z-score normalization.
The z-score normalized metrics have an expected value of 0, and a variance of 1, and positive values indicate better
results. While their value range is unbound, it can be interpreted through the lens of the inverse cumulative
density function of the standard Gaussian distribution to retrieve a p-value. The classes derive from
pykeen.metrics.ranking.ZMetric
, and the following implementations are available:
Functions
|
Generate random ranks from a given array of the number of candidates for each ranking task. |
|
Generate random number of candidates, and coherent ranks. |
|
Calculate the generalized harmonic numbers from 1 to n (both inclusive). |
Pre-calculate variances of inverse rank distributions. |
Classes
A base class for rank-based metrics. |
|
|
A derived rank-based metric. |
|
An adjustment to create an expectation-normalized metric. |
|
A mixin to create an expectation normalized metric with max of 1 and expectation of 0. |
|
A z-score adjusted metrics. |
The (arithmetic) mean rank. |
|
|
The adjusted arithmetic mean rank (AMR). |
|
The adjusted arithmetic mean rank index (AMRI). |
|
The z-scored arithmetic mean rank. |
The inverse arithmetic mean rank. |
|
The (weighted) geometric mean rank. |
|
|
The adjusted geometric mean rank index (AGMRI). |
|
The z geometric mean rank (zGMR). |
The inverse geometric mean rank. |
|
The harmonic mean rank. |
|
The inverse harmonic mean rank. |
|
|
The adjusted MRR index. |
|
The z-inverse harmonic mean rank (ZIHMR). |
The median rank. |
|
The inverse median rank. |
|
|
The Hits @ k. |
|
The adjusted Hits at K (\(AH_k\)). |
|
The z-scored hits at k (\(ZAH_k\)). |
The ranks' standard deviation. |
|
|
The ranks' variance. |
|
The ranks' count. |
The metric does not provide a closed-form implementation for the requested operation. |
|
|
The parameters of an affine transformation. |